Chunking
Lesson 5 of 10
5Chunking
The big pictureYou do not embed a fifty-page document as one vector, nor every sentence alone. Chunking decides where documents are split before indexing, and it is one of the most influential and least noticed decisions for retrieval quality. Chunk size, overlap, respecting headings, and all of it measured, not guessed.
What the picture shows: Each chunk here is a whole topic with its heading, so «قبل أسبوعين» is never cut in half as it was with fixed 80-character chunks. With only three chunks, heading-aware chunking hit 5 questions of 6, against 3 for small chunks without overlap.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
You do not hand a researcher a whole book to find one sentence; you cut it into small cards that are easy to search. But cut a sentence in the middle and its meaning is lost, and make a card too big and two topics mix in it. Chunking is the art of choosing card size and where to cut.
Fixed-length chunks with overlap, structure-based by headings and paragraphs, or semantic by topic shifts. Small chunks are precise but lose context; large ones dilute precision and use the window. You keep each chunk's heading and source, and judge the choice with retrieval metrics.
Before You Start
🧲Vector search
You know every chunk becomes one vector, so what goes into a chunk decides what search can find.
📐Evaluation first
You have golden questions with answers, because chunk size is chosen with them, not guessed.
The Core Idea
Chunking decides where documents are split before indexing. In this lesson you start with chunk size and what it cuts, then overlap, then chunking by heading, then measure every method on the same questions.
1Chunk size
A small chunk is precise but may cut a fact. A large one keeps context but blurs the embedding and costs more in the context window.
Fixed chunking every N characters is simple and fast, but it does not know where a word ends. Look at the picture: a cut every 80 characters split «أسبوعين» ("two weeks") across two chunks.
2Overlap
Overlap makes each chunk start before the previous one ends, so a fact on the border is not lost. An overlap of 10% to 20% of the chunk size is a sensible start, and the price is more chunks.
doc = """# الإجازة السنوية
يحق للموظف واحد وعشرون يوم عمل في السنة. تقدم الطلبات قبل أسبوعين. يمكن ترحيل عشرة أيام إلى السنة التالية.
# الإجازة المرضية
تحتاج تقريرا طبيا إذا زادت عن يومين. يوافق عليها المدير المباشر. لا تخصم من رصيد الإجازة السنوية.
# السفر
تحجز التذاكر عبر البوابة الداخلية. تصرف البدلات خلال عشرة أيام من تقديم الفواتير."""
def fixed(text, size, overlap=0):
text = text.replace("\n", " ")
step = size - overlap
return [text[i:i + size] for i in range(0, len(text) - overlap, step)]
def by_heading(text):
parts = []
for block in text.split("# ")[1:]:
title, body = block.split("\n", 1)
parts.append(f"{title}: {body.strip()}")
return parts
for name, chunks in [("fixed 80", fixed(doc, 80)),
("fixed 80, overlap 30", fixed(doc, 80, 30)),
("by heading", by_heading(doc))]:
print(f"{name}: {len(chunks)} chunks")
print(" first:", chunks[0])
print(" second:", chunks[1])
Fixed chunking every 80 characters cut the word «أسبوعين» in half, so neither the first nor the second chunk holds «قبل أسبوعين» whole. A 30-character overlap fixed that: the second chunk starts before the first ends. And heading-aware chunking gave three chunks, each a whole topic with its heading.
3Chunking by heading
Structured documents have natural borders: headings and paragraphs. Cut on them, and attach the heading to every chunk so it knows its topic before embedding.
import re
def chunk_markdown(text, max_chars=1500, overlap=200):
chunks, path = [], []
for block in re.split(r"(?m)^(?=#{1,3} )", text):
if not block.strip():
continue
heading = re.match(r"(#{1,3}) (.+)", block)
if heading:
level = len(heading.group(1))
path = path[:level - 1] + [heading.group(2).strip()]
title = " > ".join(path)
body = block.split("\n", 1)[1] if heading else block
for i in range(0, max(len(body), 1), max_chars - overlap):
chunks.append({"title": title, "text": f"{title}\n{body[i:i + max_chars]}"})
return chunks
Each chunk carries its full heading path ("Policies > Leave > Sick"), and long sections are split with overlap inside the section only, so different sections never mix in one chunk.
4Measure, do not guess
For each method, one question: does the first retrieved chunk contain the answer? Look at the picture: six golden questions and four methods on the same document.
Six golden questions for the same document. For each chunking method we retrieve the nearest chunk and ask: does it contain the answer?
from sklearn.feature_extraction.text import TfidfVectorizer
doc = """# الإجازة السنوية
يحق للموظف واحد وعشرون يوم عمل في السنة. تقدم الطلبات قبل أسبوعين. يمكن ترحيل عشرة أيام إلى السنة التالية.
# الإجازة المرضية
تحتاج تقريرا طبيا إذا زادت عن يومين. يوافق عليها المدير المباشر. لا تخصم من رصيد الإجازة السنوية.
# السفر
تحجز التذاكر عبر البوابة الداخلية. تصرف البدلات خلال عشرة أيام من تقديم الفواتير."""
golden = [("كم يوم إجازة سنوية؟", "واحد وعشرون يوم"),
("متى أقدم طلب الإجازة السنوية؟", "قبل أسبوعين"),
("هل أرحل أيام الإجازة؟", "ترحيل عشرة أيام"),
("متى أحتاج تقريرا طبيا للإجازة المرضية؟", "زادت عن يومين"),
("من يوافق على الإجازة المرضية؟", "المدير المباشر"),
("متى تصرف بدلات السفر؟", "خلال عشرة أيام من تقديم")]
def fixed(text, size, overlap=0):
text = text.replace("\n", " ")
return [text[i:i + size] for i in range(0, len(text) - overlap, size - overlap)]
def by_heading(text):
return [b.replace("\n", ": ", 1).strip() for b in text.split("# ")[1:]]
def hit_rate(chunks):
vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4)).fit(chunks + [q for q, _ in golden])
m = vec.transform(chunks)
hits = 0
for q, answer in golden:
best = (m @ vec.transform([q]).T).toarray().ravel().argmax()
hits += answer in chunks[best]
return hits
for name, chunks in [("fixed 60", fixed(doc, 60)),
("fixed 60, overlap 25", fixed(doc, 60, 25)),
("fixed 200", fixed(doc, 200)),
("by heading", by_heading(doc))]:
print(f"{name:22} chunks={len(chunks):2} top-1 contains the answer: {hit_rate(chunks)}/{len(golden)}")
Small chunks without overlap hit 3 of 6, and overlap raised that to 4. Heading-aware chunking hit 5 with only three chunks. Large chunks (200 characters) also hit 5, but beware: in a real fifty-page document, chunks this size relative to the whole mean whole pages in the context window for every question. Do not send the model more than it needs just because it "contains" the answer.
How It Works, Step by Step: how chunk_markdown cuts a document
It splits the text at every heading from level one to three.
It updates the heading path for each section, such as "Policies > Leave > Sick".
If a section is longer than 1,500 characters it cuts inside it only, with a 200-character overlap, so two sections never mix in one chunk.
It puts the heading path at the start of every chunk, so each chunk knows its topic before embedding.
Worked example
A short document with three headings. We chunk it three ways and look at the first two chunks of each.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| There is one ideal chunk size for everyone, such as 512 tokens. | The best size depends on your documents, your questions and your embedding model. Try three sizes on your golden set and choose by the number. |
| Large chunks are better because they usually contain the answer. | They contain it, but they blur the embedding and fill the context window with text the question does not need, and in a long document they become whole pages per question. |
| Extracting raw text from a PDF is enough. | Columns, tables and headers come out jumbled, so a chunk becomes half a sentence from one column and half from another. Read a sample with your own eyes before indexing. |
Try it yourself
What is the benefit of attaching the heading to every chunk?
A chunk without a heading can be ambiguous about its topic. The heading is a few words that steer the embedding to the right topic.
Overlap raised hits from 3 to 4. What does it cost?
Every overlap duplicates text. That is why you deduplicate when building the context, as in the context engineering lesson.
Quick review cards
Check Your Understanding
Why not make every sentence its own chunk?
Because a sentence alone loses its context (pronouns, topic), its embedding is weaker for being short, and the chunk count balloons so you must retrieve dozens to cover one answer. Single sentences suit reranking or precise citation, not the main retrieval unit.
Why does deduplication matter when building the context if you used overlap?
Because overlap copies text between neighbouring chunks, so both may be retrieved together and the same text enters the context twice, costing twice and taking another chunk's space.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.