Context engineering sounds like a new discipline. Most of it is housekeeping: deciding what the model needs to see, in what order, and what it can safely ignore. The model does not know which paragraph is authoritative unless you tell it, and it cannot tell an instruction from a document unless the structure makes that obvious.
Treat the window as a budget
Every token in the window competes for attention and costs money. Start by writing down what must be present: the task, the constraints, the evidence, the output format. Then cut everything else. A short prompt with the right three paragraphs usually beats a long prompt with twenty.
1. system rules (stable, cached)
2. task and constraints
3. retrieved evidence, delimited and labelled
4. one or two examples, only if they change behaviour
5. conversation summary, not the full transcriptOrder and delimit
Put stable instructions first and user-provided data later. Wrap retrieved documents in explicit tags or separators, and say that content inside them is data, not instructions. This does not make prompt injection impossible, but it removes the trivial confusions.
Be consistent. If a document is quoted one way in one prompt and another way in the next, the model has to relearn the structure every time. A small, stable template is easier to evaluate and cache.
Compress before you expand
Long conversations and large tool outputs are the usual bloat. Summarise old turns, keep structured facts instead of raw JSON, and drop tool outputs once their conclusion is captured. If a tool returns a 400-row table and the task needs three fields, extract those fields before the model sees the table.
- Keep the last few turns verbatim; summarise older ones.
- Store extracted facts, not raw documents, when the task allows it.
- Cut examples that do not change the output.
- Put the output schema once, not in three places.
Cache the stable prefix
If your provider supports prompt caching, put the system prompt, tool schemas and fixed examples first, and keep them byte-identical. Any change invalidates the cache. This is one of the cheapest latency and cost wins available, and it forces you to separate stable instructions from dynamic data.
A concrete budget
For a support assistant, the first version of the prompt was 8,400 tokens: six retrieved documents, the full help-centre index, three examples that repeated the same rule, and a long tone guide. The answers were slow and kept missing the specific policy because the relevant paragraph was buried in the middle.
| Part | Before | After | Why |
|---|---|---|---|
| System rules | 600 | 600 | Stable and cached |
| Retrieved docs | 5,200 | 1,800 | Top three reranked chunks, not six |
| Examples | 1,100 | 350 | One example that changed behaviour |
| Conversation | 1,200 | 400 | Summary plus last two turns |
| Output schema | 300 | 300 | Unchanged |
The smaller prompt was not just cheaper. It scored higher on the eval set because the policy paragraph was no longer competing with five near-duplicate documents. The lesson is not 'always use fewer tokens'; it is that every block has to earn its place in the evidence.
Test with ablations
Do not guess which part of the context matters. Remove one section at a time and run your eval set. If removing a block does not change the pass rate, it is probably noise. If removing it improves the pass rate, it was actively confusing the model.
If you cannot say which paragraph justifies each instruction, the model cannot either.
What to remember
- Treat context as a token budget and cut anything that does not change behaviour.
- Put stable instructions first, label retrieved data, and keep the template consistent.
- Compress conversations and tool output into facts before they reach the model.
- Cache the stable prefix and keep it byte-identical.
- Use ablation tests to find context that is noise or actively harmful.
