Context engineering
Lesson 2 of 10
2Context engineering
The big pictureThe context window is a limited, paid space, and what you put in it shapes the answer more than the model itself. Context engineering is the craft of choosing what goes in, ordering it and dropping the rest, because longer text does not mean a better answer, often a worse and dearer one.
What the picture shows: The budget fills with the best first: three sections of the leave policy, and 400 tokens left empty instead of padded with weak text. Sending everything would have cost 3,850 tokens, three and a half times as much, and given the model more to mix into the answer.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A model remembers nothing beyond what you put in front of it in the request, like a guest who knows only what you tell them now. But tell them everything and they get lost and miss what matters. Context engineering is choosing what to say, what to drop, and where to put the key fact.
The context window is limited in tokens, answer quality drops with padding and conflicting information, and models neglect the middle more than the ends. You select relevant passages, summarise long history, and lay out instructions and sources in a clear, tagged structure.
Before You Start
📞Calling a model
You know every token in a request is paid for, and that the system message is where fixed instructions go.
📍Position
You know models recall the start and end of a context better than the middle, which is the basis of ordering in this lesson.
The Core Idea
The context window is a limited, paid space. In this lesson you choose what goes in with a budget and a floor, then order it, then trim the conversation, then put it all in one template.
1A budget and a floor
The context window is a ceiling, not a target. You set a limit on retrieved tokens and a quality floor nothing below may cross, then fill with the best first.
Look at the picture: six chunks retrieved for a question about leave, with their similarity scores. The 0.5 floor removes the last two before any budget is counted.
chunks = [
("leave policy, section 2", 0.82, 420),
("leave policy, section 5", 0.79, 380),
("holiday calendar 2026", 0.61, 900),
("leave policy, section 1", 0.58, 300),
("HR newsletter, March", 0.41, 1200),
("travel policy", 0.33, 650),
]
budget = 1500 # tokens we allow for retrieved text
picked, used = [], 0
for name, score, tokens in sorted(chunks, key=lambda c: -c[1]):
if score < 0.5:
break # below this, more text adds noise, not answers
if used + tokens <= budget:
picked.append(name)
used += tokens
print("picked:", picked)
print("tokens used:", used, "of", budget)
Three chunks of the leave policy went in. The holiday calendar (0.61) was left out because it no longer fits after the first two, and the newsletter and travel policy were left out for being below the floor. 1,100 tokens instead of 3,850 if we sent everything: three and a half times cheaper and less distracting for the model.
2Sandwich ordering
Models attend to the start and end of the context more than the middle. This research finding is called "lost in the middle": accuracy drops when the needed fact sits in the middle of a long context.
Look at the picture: five chunks with their scores. The strongest sit at the edges, and the weakest in the middle, where the model pays least attention.
Instead of plain descending order, we put the best first and the second best last, and so on towards the middle.
def sandwich(chunks):
"""Best chunk first, second best last, the weakest in the middle."""
ranked = sorted(chunks, key=lambda c: -c[1])
front, back = [], []
for i, c in enumerate(ranked):
(front if i % 2 == 0 else back).append(c)
return front + back[::-1]
chunks = [("A", 0.91), ("B", 0.85), ("C", 0.70), ("D", 0.62), ("E", 0.55)]
print("by score: ", [c[0] for c in sorted(chunks, key=lambda c: -c[1])])
print("sandwiched:", [c[0] for c in sandwich(chunks)])
A at the start and B at the end, with the weakest E in the middle where the model pays least attention. Does it actually improve your results? Do not assume: run the golden question set (next lesson) with both orderings and compare the number. The trick's effect differs between models and shrinks with newer ones.
3Trimming the conversation
A long conversation eats the budget, because every request carries all of it. Drop what the current question does not need, and keep the important facts in one line.
We keep the last four turns, replace what came before with one line of facts, and drop more if we exceed the budget.
history = [("user", "I need leave next week"), ("assistant", "Which days?"),
("user", "Sunday to Tuesday"), ("assistant", "That is 3 days. Annual or sick?"),
("user", "Annual"), ("assistant", "Done, request 1043 submitted."),
("user", "Can I cancel it?")]
def tokens(text):
return len(text.split()) + 3
def fit(history, budget, keep_last=4):
recent = history[-keep_last:]
older = history[:-keep_last]
facts = "Earlier: user asked for annual leave Sunday-Tuesday, request 1043 submitted." if older else ""
msgs = ([("system", facts)] if facts else []) + recent
while sum(tokens(t) for _, t in msgs) > budget and len(msgs) > 2:
msgs.pop(1)
return msgs
for role, text in fit(history, budget=40):
print(f"{role:9} {text}")
The question "can I cancel it?" needs request number 1043, which survives in the facts line and the last reply. The exact days were dropped because they do not matter to this question. In a real system a small, cheap model writes the facts line by summarising old turns, and you test its quality on real conversations.
4A full context template
A good context has the same structure in every request. Look at the picture: the system message with the role, constraints and answer format, then the documents inside a <documents> tag, each with an identifier, then the question last.
SYSTEM = """أنت مساعد يجيب عن سياسات الموارد البشرية في الشركة.
أجب من المستندات المرفقة فقط، واذكر معرّف المستند بين قوسين مربعين بعد كل معلومة.
إن لم تجد الإجابة في المستندات، فقل ذلك صراحة ولا تخمّن."""
def build_messages(question, chunks):
docs = "\n".join(f'<doc id="{c.id}">{c.text}</doc>' for c in chunks)
return [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": f"<documents>\n{docs}\n</documents>\n\nالسؤال: {question}"},
]
The system message is byte-identical across requests, so it benefits from caching. The documents carry tags and identifiers, and the question comes last. This template is a starting point, and every change to it is measured on the golden set.
How It Works, Step by Step: how to build the context for one question
Sort the retrieved chunks by similarity, and drop everything below the floor, such as 0.5.
Fill the token budget with the best first, skipping a chunk that does not fit and moving to the next.
Order what you picked so the strongest sit at the edges and the weakest in the middle, each chunk with its identifier inside a <documents> tag.
Put the question at the end of the user message, and measure every change to this template on the golden set.
Worked example
Imagine your system retrieved six chunks for a question about leave, with different similarity scores and lengths. The temptation is to send them all "just in case". Instead, set a token budget and a quality floor, and fill with the best first.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A bigger context window means you should fill it. | The window is a ceiling, not a target. Every extra token costs and slows the first reply, and weak chunks distract the model. |
| Sandwich ordering always improves results. | Its effect differs between models and shrinks in newer ones. Run the golden set with both orderings and decide by the number. |
| Instructions work wherever you put them in the request. | An instruction like "answer in Arabic only" in the middle of twenty thousand tokens may be ignored. Its place is always the system message. |
Try it yourself
Where do you put "answer only from the documents and cite their identifiers"?
The system message is the place for fixed instructions: the model attends to it more, it stays separate from the content, and it benefits from caching because it does not change.
A chunk with similarity 0.61 is 900 tokens long, and 700 remain in the budget. What does the budget code do?
That is what the example did: it skipped the holiday calendar and fitted a smaller chunk after it. Truncating is possible, but it may cut the needed fact in half.
Quick review cards
Check Your Understanding
You raised the retrieved chunks from 3 to 15 and answer accuracy fell. How can that be?
The extra chunks are weaker matches and contain information that is close but wrong for the question (an old policy, another department), which the model mixes into the answer. And the right fact may now sit in the middle of a longer context. More context raises recall and sometimes lowers accuracy, and only the number decides.
In the conversation-trimming example, why did dropping the exact days (Sunday to Tuesday) do no harm?
Because the last question, "can I cancel it?", needs only request number 1043, which survived in the facts line and the last reply. What you drop is decided by what the current question needs.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs2
Done with this lesson?
Sign in to save your progress.