The Anatomy of Agent Context: Instructions, History, Knowledge, Tools, State, Memory, and Schemas

The agent did the wrong thing. You read the prompt. The prompt looks fine.

This is the debugging scene every agent builder eventually hits, and it usually ends the same way: you rewrite a sentence in the system prompt, the symptom moves somewhere else, and you have learned nothing. The failure was never in a sentence. It was in the assembled bundle — and you were reading one source while the model was reading seven.

Context is not a string you build. It is a set of sources you arbitrate.

Why "The Prompt" Is the Wrong Unit of Analysis

The document model of context is: I write instructions, the model reads them, the model acts. That model is correct for a single-turn, single-source task with no tools and no persistence. It breaks the moment any of those four constraints fails, and production agents break all four at once.

What replaces it is an assembly step. Every turn, your code selects material from several independent sources, serializes it into one token sequence, and hands it to the model. The model never sees your sources. It sees the concatenation. When behavior is wrong, the bug lives in the selection, the ordering, or the write path — not in the prose.

Two axes make the rest of this tractable:

• Lifetime — does this source get rebuilt every turn, checkpointed per session, or persisted across sessions?
• Ownership — who is allowed to write it: you, your runtime, or the model?

Keep one distinction straight before going further. The context window is a capacity owned by the model provider. Agent context is a selection owned by your code. A large window gives you room for context; it does not give you context. After instructions, tool schemas, and forty turns of history, the effective room for evidence is far smaller than the number on the pricing page.

Three Layers, Not Seven Siblings

The seven labels are useful as a checklist. They are misleading as an architecture, because they mix three different kinds of thing:

• Semantic content — material selected to inform this specific model call: instructions, conversation history, retrieved knowledge, tool outputs, selected memory.
• Runtime state — the application-owned data your assembler reads to decide what to include: plan, step counter, session checkpoint, memory store, tool registry.
• Serialization contract — the schema that turns the selected material into a valid request: message roles, tool-call format, structured-output fields.

Schemas are not a seventh payload competing with the other six for budget. They are the contract that determines how the payload is expressed. Tools are not one source either: the tool schema is authored contract, and the tool output is untrusted observation. Keeping those layers separate is what lets you answer the only question that matters during a failure: which layer produced the tokens the model actually saw?

The Seven Sources of Agent Context

Source	Layer	Lifetime	Written by	Typical failure
Instructions	Semantic	Stable across turns	Developer	Silently stale after a model or product change
Conversation history	Semantic	Session	Runtime + model	Crowds out everything else
Knowledge	Semantic	Turn	Retrieval system	Freshness tied to index, not conversation
Tool schema	Contract	Stable	Developer	Format drift breaks parsing silently
Tool output	Semantic	Turn	External system	Untrusted output treated as instruction
State	Runtime	Session	Your code	Two writers, last write wins
Memory	Runtime + semantic	Cross-session	Model or developer	Never written, or never selected

Instructions are what you author: system prompt, policies, persona, few-shot examples, tool descriptions. They are stable across turns, which is exactly why they rot. A model upgrade or a product change can invalidate an instruction that still reads perfectly.

Conversation history is user turns, assistant turns, and prior tool results, appended in order. In the naive design it is append-only and unbounded, which makes it the source most likely to consume your budget by accident.

Knowledge is retrieved per turn: documents, snippets, tickets, code. Its freshness depends on the index, not on the conversation. A retrieval result can be perfectly relevant and six months out of date.

Tool schema is authored context you control. Tool output is untrusted context produced by an external system. Conflating them is a security bug, not a style issue.

State is the runtime scratchpad: plan, step counter, intermediate artifacts. Your code writes it. The model sees only what you choose to expose.

Memory is durable, cross-session material — user preferences, past decisions, procedural rules. It has its own write policy and its own retrieval problem, and it is the source most likely to be missing entirely.

Ownership: Who Is Allowed to Write Each Source

Three write classes, and the distinction matters more than the inventory:

• Developer-authored: instructions, schemas. Reviewed by definition.
• System-authored: state, tool results, retrieved knowledge. Written by code you control, but sourced from systems you do not.
• Model-authored: memory writes, plan updates, self-summaries.

The dangerous class is not "model-authored" in isolation. It is any durable write whose inputs include model output, user text, tool output, or retrieved content. A code-mediated write that persists a model-suggested fact is just as exposed as a direct model write — the model still chose the content, and the content still came from somewhere untrusted.

The trust boundary is not the model. It is the write path. Any durable write influenced by model output or external content needs provenance, validation, and a version history before it becomes state.

I would rather ship an agent with no memory writes than one with unreviewed memory writes. The first is forgetful. The second is confidently wrong in a way that compounds.

One more distinction worth holding: a framework's memory object or state graph is an abstraction. The mechanism underneath is always a store, a selection function, and a write policy. When the abstraction misbehaves, you debug the three parts.

Freshness: Turn-Scoped, Session-Scoped, and Durable Context

Lifetime determines what you rebuild, what you checkpoint, and what you persist.

Turn-scoped sources — retrieved knowledge and tool outputs — are rebuilt or re-selected every turn. This is your main lever on token cost, and the place where caching tempts you most.

Session-scoped sources — conversation history and runtime state — get checkpointed so the agent can resume mid-trajectory without replaying the transcript. Exact checkpoint semantics depend on your runtime, so treat this as a mechanism to implement rather than a guarantee to assume.

Cross-session sources — memory and durable artifacts — are where most production agents quietly fail. Not because the store is broken, but because nothing writes to it, or nothing selects from it. An empty memory layer and a missing one produce identical behavior.

The freshness tradeoff is blunt: re-deriving context each turn is expensive but correct; caching it is cheap but silently wrong when the underlying store changes. A cached retrieval result that outlives the document it came from produces an agent that cites a deleted policy with total confidence.

Precedence: Resolving Conflicts Without a Total Order

Most agent misbehavior is a precedence bug wearing a reasoning costume. The common collisions:

• A retrieved document contradicts an instruction.
• A tool error contradicts the plan.
• A stale memory contradicts the current user turn.

There is no universal ranking that resolves these correctly. Authority is a property of your application's trust model, not of the token position. What you can do is make the resolution explicit:

1. Classify each item by trust level in code: policy, developer instruction, user turn, validated observation, unvalidated observation, model-derived.
2. Validate or normalize observations before they enter the bundle. A tool error is evidence, not an instruction.
3. Label the item's role in the serialized message so the model can distinguish "here is a rule" from "here is what a document said."
4. Serialize according to the runtime's message contract.

Recency and position are behavioral hazards to test, not precedence rules. Material near the end of the window often dominates generation, so a low-authority source placed last can outrank a high-authority source placed first. Concatenation order is a precedence decision whether or not you intended it as one.

If you cannot point to the line of code that decides a conflict, you do not have a precedence rule. You have an ordering accident.

When precedence is genuinely ambiguous, the better move is often to surface the conflict as a labeled conflict — "the retrieved policy says X, your instruction says Y" — rather than silently picking a winner. The model is frequently better at adjudicating a stated conflict than at recovering from a hidden one.

Instrumenting the Assembly Step

The mental model is only useful if it is observable. Log the assembled context per turn as structured records tagged by source, never as one flattened string. A flat string cannot attribute a failure.

Minimum useful fields per record:

{
  "turn": 14,
  "role": "user",
  "order": 3,
  "source": "knowledge",
  "source_id": "policy-2026-04",
  "version": "v7",
  "owner": "retrieval",
  "tokens": 1840,
  "written_at": "2026-05-27T09:14:02Z",
  "transformation": "retrieved",
  "inclusion_reason": "matched query: refund policy",
  "truncated": false,
  "dropped": false
}

role and order reconstruct the model-visible sequence. source_id and version let you diff against the store. transformation records whether the item was retrieved, summarized, or truncated. inclusion_reason is the only field that tells you why the assembler chose this item over the one you expected.

Token accounting per source exposes the real problem behind the marketing number. Instructions plus tool schemas plus history can consume most of the window before retrieval gets a turn, and you will not see it until you count.

A trace diff is the debugging surface. Same task, two consecutive turns: which sources changed, which stayed byte-identical. If history grew by 4,000 tokens and knowledge did not change, you know where the budget went.

One trap: compression and summarization are context transformations, not context sources. Record when history was summarized and what was dropped. Otherwise a later failure gets misattributed to the model when the summary ate the evidence.

Failure Modes and When This Decomposition Is Overkill

Amnesia. Durable context is never written, so every session starts from zero and the agent re-asks what it already knew.

Silent overwrite. Two writers touch the same state or memory key, the later write wins, and there is no version history to explain the change.

Context starvation. High-volume tool output or history crowds out the retrieved evidence the decision actually needed.

Cross-agent leakage. When multiple agents share a context layer, one agent's untrusted input becomes another agent's trusted instruction. Shared context creates a new trust boundary, and it is usually undocumented.

Now the honest boundary. The decomposition earns its complexity when sources have independent lifecycles, independent trust levels, or independent write paths — or when you need to diagnose a failure you cannot attribute. A single-turn, single-source, no-tool task does not need it. Neither does a many-source system where every source is immutable and one deterministic assembler governs the order. Source count is not the signal. Independent failure surfaces are.

The Next Move

Instrument one turn of your own agent. Dump the assembled context tagged by source, with role, order, token counts, and timestamps. Then answer three questions:

1. Which source is largest?
2. Which source was written by the model or influenced by untrusted input?
3. Which source would win a conflict — and which line of code decides that?

The answers usually name the next fix. If the largest source is history, you have a compression problem. If a model-influenced source has no review step, you have a trust problem. If you cannot answer the third question, you have a precedence problem — and that one is worth fixing before you touch another sentence of the prompt.