Designing an Adaptive Knowledge Agent: Routing, Retrieval, Memory, and Context Assembly

The demo works on turn one. By turn five, the agent is answering a question you already answered, citing a document it retrieved three turns ago, and quietly ignoring the constraint you stated at the start. Nothing crashed. No component failed its own eval. The system simply assembled a prompt that no longer contained the evidence the answer depended on.

That is the failure mode this article is about. An adaptive knowledge agent is not adaptive because it has routing, retrieval, memory, and compression bolted together. It is adaptive when every token in the final prompt is traceable to a named producer, a selection rule, and a reason it was included. Everything below builds toward that invariant.

I assume you already know the four context operations — write, select, compress, isolate — and the transient-versus-persistent split. What follows is the integration layer: how to wire those subsystems into one pipeline whose context decisions are inspectable, budgeted, and evaluable.

Why Assembled Agents Rot After Turn Three

The symptom is consistent. Turn-one quality is high because the context is clean: one question, one retrieval, one answer. Turn-five quality collapses because four subsystems that were each tuned in isolation are now writing into the same prompt with no shared contract.

The pattern repeats across teams:

• Retrieval returns evidence the router never intended to fetch, because the query was rewritten into an abstraction when the route called for a verbatim passage.
• Memory writes facts the assembler never reads, because the write path and the read trigger were designed by different people at different times.
• Compression silently drops the constraint the answer depended on, and the resulting wrong answer gets blamed on the model.
• History grows monotonically until it crowds out retrieved evidence, and the agent remembers everything while knowing nothing current.

Here is the diagnostic that matters most: distinguish a context bug from a model bug. If the correct evidence never reached the prompt, no model swap fixes it. You can upgrade the model, lower the temperature, rewrite the system prompt — and the agent will still be wrong, because the information was never there. Before you touch the model, reconstruct the prompt and ask whether the answer was even possible.

The invariant: every token in the final prompt must be traceable to a named producer, a selection rule, and a reason it was included. If you cannot name why a block is present, it is noise until proven otherwise.

The Pipeline Contract: Stages, Artifacts, and Ownership

Define the interfaces before you write any of the stages. Each stage has a typed input, a typed output, and exactly one owner. The output is a structured artifact, not a raw string.

Stage	Input artifact	Output artifact	Failure signal	Detecting metric
Intake / normalize	Raw user turn	Normalized turn + session state	Ambiguous referents unresolved	Rewrite resolution rate
Route decision	Normalized turn + state	Route plan (source, granularity, effort, memory trigger)	Always escalates	Escalation rate vs. evidence density
Retrieval fan-out	Query set	Candidate list	Empty or near-duplicate candidates	Candidate recall, dedupe ratio
Evidence reduction	Candidate list	Reduced evidence set	Drops the span the answer needed	Reduction precision, dropped-span audit
Memory read	Read trigger	Memory slice	Injected wholesale every turn	Triggered-read ratio
Memory write	Candidate facts	Persisted records	Appends instead of overwriting	Correction/overwrite rate
Assembly	All stage artifacts	Prompt + manifest	Untraceable blocks	Manifest coverage
Generation	Prompt	Answer + citations	Fluent but ungrounded	Citation grounding score
Post-hoc eval	Trace	Scored trace	Vague "got worse"	Per-stage regression

Two rules make this contract hold.

Ownership rule: producer stages decide eligibility — what may enter the pipeline. Assembly decides admission and ordering — what actually occupies the budget, in what order, at what cost. A producer cannot force a block into the final prompt; assembly cannot invent a block no producer emitted. If the assembler can inject a memory block the memory-read stage never selected, you have two owners and no contract.

Budget as a first-class input: token, latency, and cost budgets are passed down and consumed, not discovered at assembly time. A stage that exceeds its share must report that it did, not quietly borrow from the next stage.

Routing: Deciding What Kind of Question This Is

Routing is a classification-and-policy step, not a hidden model call. The dimensions that matter in practice:

• Source: internal corpus, structured store, memory, web, or no retrieval at all.
• Granularity: atomic fact, event-level summary, or verbatim passage.
• Effort: single-shot or multi-step.
• Memory trigger: whether a memory read fires this turn, and which memory type.

The cheap-first policy is the whole point. Answer from parametric knowledge or a small index when confidence is high; escalate to full retrieval only on defined triggers. The implementation pattern I prefer is an intent vector — a small set of binary or scored flags that map deterministically to a retrieval operator. This keeps routing testable without a model in the loop.

The snippet below covers only the granularity-to-operator subdecision. It is one field of the route plan, not the whole router.

# Granularity subdecision: intent flags -> retrieval operator.
# Deterministic, unit-testable. One field of the full route plan.
def route_granularity(intent):
    if intent.fine_grained:
        return "raw_passage"       # verbatim phrasing required
    if intent.abstract or intent.event_level:
        return "episodic_summary"  # high-level semantic representation
    return "fact_lookup"           # structured, atomic recall

The full route plan is a structured artifact, not a single string. Source, granularity, effort, and memory trigger are separate fields because they are decided by different signals and validated at different points.

# Full route plan. Fields may be produced by a classifier or by
# deterministic features; either way, validate before dispatch.
route_plan = {
    "source": "internal_corpus",     # or structured_store | memory | web | none
    "granularity": "raw_passage",    # from route_granularity()
    "effort": "single_shot",         # or multi_step
    "memory_read": {"fire": True, "type": "episodic"},
    "confidence": 0.82,              # gate for escalation
}

Which fields are deterministic features versus model-produced hypotheses matters for debugging. A classifier that emits granularity can be wrong in ways a feature rule cannot; log the raw signal alongside the decision so a misroute is traceable to its cause.

The failure mode to watch: routing that always escalates. It looks safe. It quietly doubles cost and latency while diluting the context with low-relevance evidence, which then makes the answer worse, which makes you escalate harder. That loop is how a "careful" agent becomes an expensive, mediocre one.

When not to route: single-corpus agents with a uniform query shape should use one retrieval path and spend the complexity budget elsewhere. Routing is a tool for heterogeneity, not a badge of sophistication.

```python
intent.fine_grained = True
intent.abstract = True
intent.event_level = False
```

Retrieval and Reduction: From Candidates to Evidence

Treat retrieval as a funnel with measurable loss at each narrowing step: broad candidate generation, then rerank, then compress, then cap. Every step must report how many candidates it removed and why.

Two placement decisions matter more than the specific algorithms.

Query transformation belongs upstream of retrieval. If the route says "verbatim passage," do not rewrite the query into an abstraction. The rewrite and the route must agree, or you retrieve the wrong granularity and blame the retriever.

Compression is lossy by design. Record what was dropped. When an answer is wrong, you want to trace it to a dropped span rather than to a phantom model failure. A compression step that cannot report its dropped spans is not a component; it is a liability.

Deduplicate before compression, not after. Near-duplicate chunks consume budget without adding evidence, and compressing duplicates wastes the compressor on redundancy instead of on the material that actually needs shrinking.

The metric that matters is evidence density in the assembled prompt — not recall at the candidate stage. Optimizing recall and never measuring final-context precision is how teams ship a retriever that finds everything and a prompt that says nothing.

Memory: Read Triggers, Write Policy, and Forgetting

Memory is a subsystem with explicit read and write contracts, not an accumulating transcript. Separate memory types by the future decision they serve: durable facts, episodic events, procedural preferences, and task state. Each type answers a different question later, so each needs a different retrieval path.

Write policy: what qualifies for persistence, who confirms it, and how a correction overwrites rather than appends. If corrections append, the agent accumulates contradictions and resolves them by recency, which is not the same as correctness.

Read trigger: memory is queried by the router on a condition, not injected wholesale every turn. A memory block that appears in every prompt is not memory; it is a second system prompt with worse provenance.

Forgetting and staleness: every record needs a timestamp, a source, and an expiry or invalidation rule. Without them, the agent will confidently cite a superseded fact — and because it is stored, the agent treats it as more authoritative than fresh retrieval.

The failure mode is monotonic growth. Memory expands until it crowds out retrieval evidence, and the agent remembers everything while knowing nothing current. The fix is not a bigger window. It is a read trigger that fires on condition and a write policy that refuses most candidates.

Assembly: Ordering, Precedence, and Budget Allocation

Assembly is a deterministic function. Given the stage artifacts, it returns the prompt plus a manifest of what was included, from where, and at what token cost. It does not create content; it admits, orders, and budgets what producers already emitted.

Precedence order when sources conflict (an example policy, not a universal rule — state your own and make it explicit):

1. Current-turn instructions
2. Verified structured state
3. Retrieved evidence
4. Memory
5. History

Placement: put hard constraints and the task frame where they are least likely to be diluted. Keep evidence adjacent to the question it answers, so the model does not have to bridge a long distance between a claim and its support.

Budget allocation is a policy, not a leftover. Reserve fixed shares for instructions, evidence, memory, and history. Let unused shares roll over rather than pre-filling them with whatever is available. A budget that always fills itself will always be full of the cheapest material.

Isolation as a tool: when two evidence sets contradict, present them as separate labeled blocks instead of merging them into one ambiguous passage. Merging forces the model to resolve a conflict you have not acknowledged.

def assemble(artifacts, budget):
    blocks = []
    blocks += take(artifacts.instructions, budget.instructions)
    blocks += take(artifacts.structured_state, budget.state)
    blocks += take(artifacts.evidence, budget.evidence)
    blocks += take(artifacts.memory_slice, budget.memory)
    blocks += take(artifacts.history, budget.history)
    manifest = [
        {"block": b.id, "producer": b.stage,
         "rule": b.selection_rule, "tokens": b.tokens}
        for b in blocks
    ]
    return render(blocks), manifest

The manifest is the deliverable. Without it, you have a prompt; with it, you have a debuggable decision.

A Worked Trace: Three Turns, One Localized Failure

Abstract contracts are easy to nod at. Here is a compact trace that exercises routing, memory, retrieval, and assembly across three turns, with one failure you can localize.

Turn 1. User: "Summarize our Q3 refund policy for EU customers."

route_plan: source=internal_corpus, granularity=raw_passage,
            effort=single_shot, memory_read.fire=False
evidence:   [doc_17 §3.2, 412 tok]  [doc_22 §1.1, 208 tok]
manifest:   instructions(180) + evidence(620) + history(90)
            = 890 tok, budget 2000

The answer cites §3.2. The write policy persists one fact: eu_refund_window = 14 days, source doc_17, timestamp T1.

Turn 2. User: "And for the UK?"

route_plan: source=internal_corpus, granularity=raw_passage,
            effort=single_shot, memory_read.fire=True, type=fact
memory_slice: [eu_refund_window=14d, src=doc_17, T1]
evidence:   [doc_31 §2.4, 356 tok]
manifest:   instructions(180) + evidence(356) + memory(40)
            + history(140) = 716 tok

The memory read fires because the turn references a prior constraint. The manifest shows the memory block is 40 tokens — bounded, not wholesale.

Turn 3. User: "Actually the EU window changed to 30 days last month."

route_plan: source=structured_store, granularity=fact_lookup,
            effort=single_shot, memory_read.fire=False
write:      eu_refund_window = 30 days, src=user_correction, T3
            (overwrites T1 record; T1 retained as superseded)
evidence:   [store_row eu_refund_window, 12 tok]
manifest:   instructions(180) + structured_state(12)
            + history(210) = 402 tok

Now the failure. Suppose the write policy appended instead of overwriting. Turn 4 asks "What's the EU refund window?" and the memory read returns both 14d (T1) and 30d (T3). Assembly admits both because both are eligible. The model picks by recency — usually right, occasionally wrong, and never traceable.

The manifest localizes the bug immediately: two memory blocks with the same key and different timestamps. That is a write-policy defect, not a model defect. Fix the overwrite rule; the prompt heals itself.

Evaluation Boundaries: What Each Stage Must Prove

Attach a specific evaluation to each stage so failures localize instead of surfacing as "the agent got worse."

• Route accuracy against labeled intents.
• Retrieval recall on a held-out question set.
• Reduction precision — did the reduced set keep the span the answer needed?
• Memory read/write correctness — did the trigger fire, and did corrections overwrite?
• Assembly manifest diffing — what changed between two runs on the same input?

End-to-end checks must stay honest. Score answer correctness, citation grounding, and constraint adherence separately, so a fluent wrong answer cannot hide behind a good fluency score.

The sharpest test is contribution analysis, not binary removal. Remove or perturb one evidence block, then score the answer against a reference on correctness, grounding, constraint adherence, and uncertainty behavior. An unchanged answer is a signal to investigate redundancy — it is not proof the block was noise. Corroborating evidence can be load-bearing precisely because it is redundant, and a correct answer may legitimately survive the loss of a supporting block. Treat the delta as a measurement, not a verdict.

Regression discipline: freeze a small set of multi-turn traces and re-run them after any routing, memory, or budget change. Multi-turn traces catch the rot that single-turn evals miss.

Uncertainty note: thresholds and acceptable loss rates depend on your corpus, model version, and task risk. Treat any published number as a starting point, not a target. Measure your own.

Failure Paths and Recovery

Stages fail. Plan for it.

• Empty retrieval: fall back to a declared "insufficient evidence" response rather than letting the model improvise from parametric memory.
• Route misclassification: detect it downstream via low evidence relevance and re-route once, with a hard cap to prevent loops.
• Memory conflict: when a retrieved fact contradicts a stored memory, prefer the fresher sourced record and flag the conflict for review.
• Budget exhaustion: assembly degrades by dropping history first, then memory, then low-ranked evidence — never by truncating instructions.

Observability: log the full stage manifest per turn so a bad answer can be replayed without re-running the model. If you cannot replay a turn from its trace, you cannot debug it; you can only guess.

Build Order: The Smallest Agent That Proves the Contract

Ship a working pipeline before adding sophistication.

Step 1: one corpus, one retrieval path, no memory, no routing — but with the stage artifacts and manifest already in place. The manifest is not optional scaffolding; it is the contract.

Step 2: add the route plan as a deterministic classifier over query features. Measure whether it beats always-retrieve on cost and evidence density. If it does not, delete it.

Step 3: add memory with a single type and an explicit write policy. Verify reads are triggered, not injected.

Step 4: add reduction — rerank, compress, dedupe — and confirm the manifest shows higher evidence density at equal token cost.

Step 5: add the contribution-analysis and regression harness before adding a second corpus or a second agent.

Before any of that, run the drill: take one multi-turn trace and hand-write the manifest for each turn. If you cannot name why each block is present, the pipeline is not ready for code.

An adaptive knowledge agent is not adaptive because it has many components. It is adaptive because each component's contribution to the final context is named, budgeted, and measurable. Start there: instrument one existing agent turn with a stage manifest, run contribution analysis on its evidence blocks, and let the result tell you which stage to fix first.

The write trace shows the 30-day value was stored as a new record while the 14-day record remained active.