What Is Context Engineering? Building the Right Working Context for Every Model Decision

The prompt you spent a week tuning is not what the model reads. It reads everything in the window — and everything else is quietly voting.

A team ships a feature. The prompt is clean, the instructions are specific, the few-shot examples are well chosen. It works in testing. Three weeks into production, the same prompt produces worse answers, and nobody touched it. So the team rewrites the instructions. Then rewrites them again. The real variable was never the wording. It was the retrieved documents piling up, the tool output nobody trimmed, the conversation history growing one turn at a time until the signal the model needed was buried under everything the system had ever seen.

That is the failure mode context engineering exists to fix. If you already write prompts and you already know what a context window is, you have the prerequisites. What you may not have is the right unit of design.

The Prompt Was Never the Unit of Work

Prompt engineering optimizes instructions. Context engineering optimizes the entire token set present at inference: instructions, retrieved knowledge, tool output, memory, and system state. The instruction is one input among many, and in a real system it is often not the one deciding the outcome.

Here is the mental model I want you to keep. The context window is the model's working memory — the RAM to the model's CPU, as the analogy often goes. It is a bounded, attention-weighted budget. But the analogy breaks in a way that matters: RAM is addressable and exact. You can read a byte back and get that byte. A context window is lossy. Tokens compete for attention, position changes influence, and adding more text is not adding more information. It is adding more competition.

Treat that working-set model as a design frame, not a mechanical guarantee. How much ordering, position, and long-context degradation actually affect a given call depends on the model and the runtime serving it. The frame tells you what to control; decision-level evals tell you whether the control worked. We come back to that in the measurement section.

More tokens is not more information. Past a point, it is noise with a receipt.

The unit of design is the decision, not the conversation. An agent loop makes many decisions — which tool to call, which document to trust, whether to ask for clarification — and each one has its own ideal working set. A single conversation-level prompt cannot serve all of them well. The observable consequence: quality degrades as the window fills even when the instruction text is byte-for-byte identical.

Context Has Types, and Each Type Has a Different Failure Mode

Before you can engineer context, you need to know what you are engineering. Most systems carry four types, and each fails differently.

The practical move is an audit. Map your pipeline's inputs onto these four buckets and find the one with no owner. In my experience it is almost always state. Instructions get a prompt file. Knowledge gets a retrieval pipeline. Tools get schemas. State just accumulates, because nobody decided it was a design surface.

The Four Operations: Write, Select, Compress, Isolate

Building a working set is not a pipeline you run once. It is a loop of four composable operations, and the loop is where the engineering lives.

Write. Persist context outside the window — scratchpads, memory stores, files — so it survives and can be re-selected later. Writing is how you stop carrying everything forever. You save it, then decide later whether it earns a seat.

Select. Pull the right subset in: retrieval, routing, ranking, and the decision of what not to include. Selection is the operation that most determines output quality, which is why it gets its own section below.

Compress. Summarize, prune, or restructure so retained tokens carry the most decision-relevant signal. Compression trades fidelity for space, and the trade is not uniform — some things must survive verbatim.

Isolate. Split context across sub-agents, tool calls, or separate invocations so one decision is not polluted by another's working set. Isolation is a scaling move disguised as a hygiene move.

These are not sequential stages. A real system loops through them per step: write a result, select what the next decision needs, compress what is too large, isolate what would contaminate. The loop is the system.

One Decision, Traced End to End

Taxonomies are cheap. Let me show the mechanism moving through a single decision, because this is where the per-decision thesis either earns its keep or stays a slogan.

The decision: an agent must decide whether to call a refund(order_id) tool. The decision contract is narrow — does this order qualify for a refund, and is the user authorized to request it?

Candidate context, before selection:

• Instructions: refund policy, tool description, tone rules.
• Knowledge: the order record, the customer's purchase history, the current refund policy document.
• Tools: the refund schema, plus the last tool call's output.
• State: the conversation so far, including the user's stated reason.

Now the operations run. Select pulls the order record, the current policy, and the authorization rule; it drops the purchase history (irrelevant to eligibility) and an obsolete policy version still sitting in the index. Compress summarizes the conversation into "user requests refund for order 1234, reason: damaged item" instead of replaying twelve turns. Isolate keeps the raw tool output out of the reasoning context and passes only the parsed fields the decision needs. Write persists the full order record and conversation to a store so a later decision — say, issuing a replacement — can re-select them.

The assembled working set is small: policy, order fields, authorization rule, compressed request. Everything else was excluded on purpose.

Here is the failure that proves the point. Suppose selection pulls the obsolete policy version alongside the current one. The model now sees two conflicting refund windows. It may pick either. The output is wrong, but the prompt was never touched — the defect lives in selection, and no amount of instruction rewriting will find it.

That trace is the whole discipline in miniature. Now make it a procedure you can debug.

def assemble_context(decision, state):
    # 1. Define the decision contract: what must be true to decide well?
    contract = decision.required_evidence + decision.forbidden_evidence

    # 2. Gather candidates from every source: instructions, knowledge, tools, state.
    candidates = gather(decision, state)

    # 3. Apply hard constraints first. Drop anything forbidden or stale.
    candidates = [c for c in candidates if c not in contract.forbidden]

    # 4. Rank and select against the contract, not against raw similarity.
    selected = rank(candidates, contract)[:decision.budget]

    # 5. Compress only what is eligible. Never paraphrase loss-sensitive fields.
    selected = [compress(c) if c.lossy_ok else c for c in selected]

    # 6. Assemble with provenance so every token traces to a source.
    return assemble(selected, provenance=True)

The point is not the API. The point is the boundaries: candidates, constraints, selection, transformation, assembly. When a decision goes wrong, you now know which boundary to inspect instead of guessing at the prompt.

Selection Is the Hard Part: Retrieval, Routing, and Ranking

Selection deserves the deepest treatment because it is where most quality is won or lost, and where the most common mistake lives.

Start by defining the decision axis. What does this specific model call need to know to produce the desired behavior? Not the conversation, not the task — the call. Write the answer down. If you cannot, you are not ready to retrieve.

Then separate two measurements that teams routinely conflate: retrieval quality and context quality. A retriever can have excellent recall and still produce a bad working set. Recall measures whether the right document was found. Context quality measures whether the right information reached the decision in a usable form. These are different numbers, and optimizing the first does not guarantee the second.

Routing happens before retrieval even runs. It is the choice of which source, tool, or sub-agent supplies context at all. If your agent has five tools and always calls the wrong one, no amount of retrieval tuning fixes it — the routing decision is upstream.

Ranking and ordering matter more than most teams assume. Position within the window changes how much a passage influences output. The same document placed early and placed late are not the same input — though how much that difference costs you is model- and runtime-dependent, so measure it rather than assuming a fixed rule.

The common failure: recall-maximizing retrieval that floods the window with plausible-but-irrelevant text. It dilutes the evidence that mattered. The model does not get confused because it lacks information. It gets confused because it has too much of the wrong kind.

Long-Horizon Context: What Breaks After Turn Fifty

Everything above holds for a single decision. Once an agent runs for many steps, context engineering becomes a different discipline.

History grows monotonically. Without management, the working set stops being a briefing and becomes an archive — a complete record of everything that happened, optimized for completeness rather than for the next decision. The model does not need the archive. It needs the briefing.

Compaction and summarization are the standard response, and they force a decision: what must survive verbatim versus what can be paraphrased. As a safe default, treat IDs, constraints, and commitments as loss-sensitive fields that survive verbatim. A paraphrased order number is a corrupted order number. A paraphrased constraint is a constraint the model may silently drop. Everything else is negotiable.

Externalized state is the stronger move. Write progress and artifacts outside the window, then re-select them just-in-time instead of carrying them forever. This is the write operation earning its keep at scale.

Isolation scales further. Sub-agents with narrow working sets beat one agent with an enormous one, because each sub-agent's decision is not polluted by context it does not need.

The failure mode to watch for is silent context rot. The system keeps running. No error fires. But the model is reasoning over a degraded summary of its own past, and the degradation is invisible until the output is wrong in a way that traces back fifty turns.

When Context Goes Wrong, Read the Operation

A degraded output is a symptom, not a diagnosis. The repair path depends on which operation failed, and the operations fail in distinguishable ways. Use this mapping before you touch the prompt.

The discipline is the same in every row: name the operation, inspect its boundary, fix the cause. Rewriting the instruction is the wrong repair for four of these five symptoms, and it is the repair teams reach for first.

You Cannot Engineer What You Do Not Measure

Context engineering becomes a real practice the moment you can observe it. Until then it is intuition wearing a job title.

Log the assembled context per decision, not just the final prompt. You cannot debug a working set you never recorded. The log should show what was selected, what was compressed, what was excluded, and how many tokens each contributed.

Measure context quality separately from answer quality. Three numbers worth tracking: relevance of selected items, token cost, and redundancy. A working set that is 40% redundant is paying for tokens that add nothing.

Build small decision-level evals. Given this working set, was the right information present? Was the wrong information absent? That second question is the one teams skip, and it is often the one that explains the failure.

Make the eval concrete with a labeled set of decision cases. For each case, record the required evidence, a harmful distractor, the token cost, and the outcome. Then a single pass/fail criterion does real work: the assembled context must contain the current policy and must exclude the obsolete one. You judge that before you judge the model's answer, because a correct answer from a contaminated working set is luck, not engineering.

Track token budget over time as a first-class metric. Context growth is a slow failure that looks exactly like model degradation. If your window usage climbs 15% a week and quality drops, you have your answer before you touch the prompt.

One caveat on evidence: model behavior varies by version. Treat eval results as version-scoped. A working set that performed well on one model version is a hypothesis about the next one, not a guarantee.

When Not to Engineer Context

Not every problem needs a context pipeline, and over-building one is its own failure.

Single-turn, small-input tasks with stable instructions do not need any of this. A good prompt is the whole job. If your input fits comfortably and the instruction is stable, you are done.

If the failure is a reasoning or capability limit, more context will not fix it. Diagnose before you retrieve. A model that cannot do the arithmetic will not do it better with three more documents.

Every layer you add — retrieval, memory, routing — adds latency, cost, and a new failure surface. Each layer must earn its place.

My rule: add a context operation when you can name the specific decision it improves and the metric that will show it. If you cannot name both, you are adding ceremony.

Start With One Decision

Pick one model decision in your current system. Log the exact working set that reached it — every token, every source, every piece of state. Then audit that working set against the four context types and find the one with no owner.

Then apply one operation. Usually selection or compression. Write down the decision contract first: what evidence is required, what is forbidden, what is merely nice to have. Measure whether the decision improved, and measure it against the metric you named before you started.

Context engineering is the discipline of deciding what the model gets to see before it decides. That decision is yours to make on purpose — or to leave to whatever happened to accumulate in the window.