Memory Engineering for AI Agents: Short-Term, Semantic, Episodic, and Procedural Memory

The agent remembered everything and still got the user wrong. It recalled a preference the user had abandoned six months ago, re-derived a workflow it had already learned twice, and answered a policy question with a story about a Tuesday. Nothing was broken in the retrieval pipeline. The store was full, the embeddings were fresh, and the context window had room. The memory layer was simply undifferentiated — one bucket holding facts, events, and instructions, retrieved by cosine similarity and injected in whatever order the scores happened to fall.

That is the failure mode memory engineering exists to fix. Memory is not a bucket. It is a set of typed stores, and each type earns its place by changing a specific future decision. If you cannot name the decision a memory changes, you are keeping a log, not building memory.

This article assumes you already have working context assembly and retrieval. The write/select/compress/isolate vocabulary and the transient-versus-persistent distinction are the substrate here, not the subject. What follows is the layer above: how to split memory by type, what earns a write, what triggers a read, how wrong memory gets corrected, and what must be allowed to die.

Memory Is Not One Store

The default architecture is a single vector store of conversation chunks. It works until it doesn't, and it fails in a specific way: retrieval returns the wrong kind of thing at the wrong time. A user asks "what's my deployment process?" and gets back an episode — the time they deployed on a Friday and it went badly — instead of the procedural rule that governs deployments. Both are semantically similar to the query. Only one answers the question.

The fix starts with an admission test. Before anything enters memory, name the future decision it must change. If no decision changes, it is a log entry. Logs are valuable for debugging and audit; they are not memory. The distinction matters because logs can grow without bound and memory cannot.

A trace is evidence of what happened. It becomes memory only when a lesson is converted into context a later run can retrieve and act on. Most of your traces should stay traces.

Every memory type in this article is defined by a four-question contract:

1. What decision does it serve? The future action this memory is supposed to change.
2. What writes it? The trigger that admits new content.
3. What reads it? The trigger that pulls it into context.
4. What retires it? The rule that expires, supersedes, or deletes it.

If you cannot answer all four for a store you are maintaining, you have a store with no owner, and it will drift.

The Lifecycle: From Run State to Retired Memory

Before the taxonomy, the state machine. The most common implementation mistake is collapsing three distinct states into one store: live working context, persisted session history, and durable memory eligible for future retrieval. They have different lifetimes, different write rules, and different failure modes.

run state ──(run ends)──> trace ──(admission test)──> candidate
                                                          │
                                          ┌───────────────┴───────────────┐
                                          │                               │
                                    rejected                        admitted
                                    (stays a log)                   (typed entry)
                                          │                               │
                                          │                    ┌──────────┴──────────┐
                                          │                    │                     │
                                          │              retrieved            superseded
                                          │              (used in a           (dated prior,
                                          │               decision)            new value current)
                                          │                    │                     │
                                          │                    │              ┌──────┴──────┐
                                          │                    │              │             │
                                          │                    │         promoted      retired
                                          │                    │         (gated)       (expired)

The invariant that matters: visibility in the current context is not persistence, and persistence is not eligibility for future retrieval. A tool result visible in the current turn is run state. A compressed session summary persisted to disk is session history. Neither is automatically memory. Promotion from trace to durable memory requires passing the admission test, and promotion from one memory type to another requires passing a gate — not just repetition.

Never auto-promote transient control state — retry counters, scratch variables, intermediate reasoning — into durable memory. It is run-scoped by definition, and persisting it pollutes retrieval with artifacts that were never meant to outlive the task.

The Four Memory Types and the Decisions They Serve

The taxonomy is borrowed from cognitive science, and the borrowing is useful precisely because it forces a separation that flat stores collapse. The analogy breaks in one important place: agents have no biological consolidation. Human memory consolidates whether you plan for it or not. Agent memory consolidates only if you build the pipeline step. That is the whole game.

Short-term (working) memory is the live state of the current task: the thread, recent tool results, intermediate artifacts, scratch files. It serves one decision — what do I do next in this run. It is scoped to the run and dies with it.

Semantic memory holds durable facts, preferences, and entity relationships. It serves the decision what is true about this user, project, or domain. "The user prefers TypeScript." "The staging cluster is in eu-west-1." "This project uses Postgres, not MySQL."

Episodic memory holds specific past interactions, actions, and outcomes with time and context attached. It serves the decision has this happened before, and what happened then. "Last time we migrated this schema, the foreign key constraint failed on the orders table."

Procedural memory holds instructions, workflows, tool-use rules, and learned skills. It serves the decision how should I behave on this class of task. "When asked to deploy, run the test suite first, then tag, then push." Procedural memory is where most visible behavior improvements come from, because it is the type that changes what the agent does rather than what it knows.

Type	Decision served	Evidence required to admit	Scope / precondition	Typical read trigger	Retirement rule	Failure when misclassified
Short-term	What next in this run	Any turn output	Current run only	Always present	Dies with run	Leaks into long-term as noise
Semantic	What is true	Explicit instruction or repeated confirmation	Entity or domain	Entity match or query match	Supersede on contradiction; confidence decay	Stale facts outrank current ones
Episodic	Has this happened before	Post-action capture with outcome	Time-bounded event	Similar-task recall or event trigger	Time-based expiry; access reinforcement	Episodes answer policy questions
Procedural	How to behave	Successful action pattern + scope + review	Task class with preconditions	Task-class match; always-on for pinned rules	Supersede on better rule; review cadence	Over-generalized rules misfire in new contexts

The table is the taxonomy. The prose does not need to repeat it. What the table cannot show is the consolidation pathway that makes the system compound — and that pathway needs gates, not assumptions.

Promotion Gates: Why Repetition Is Not Enough

The attractive teaching model says repeated episodic patterns become semantic facts, and repeated semantic patterns become procedural rules. That model is too smooth for an engineering decision guide. A repeated observation may support a preference. A procedural rule requires more: an action, a scope, preconditions, and evidence that the action improved outcomes.

Episode → candidate fact requires repeated evidence or explicit user confirmation. Three sessions where the user corrected the agent's date format is enough to propose a semantic fact about their preference. One session is not.

Fact or episodes → procedure requires a successful action pattern, a defined scope, preconditions, and a review or rollback path. The fact that the project's test command is pytest -x does not automatically become "always run tests before deploying." That procedural rule needs evidence that running tests before deploying actually prevented failures in this project's context.

Here is a promotion that should be rejected. An agent observes three sessions where the user asked for a summary before a detailed answer. The system proposes a procedural rule: "always summarize first." But the user only asked for summaries in the context of long documents, not in code review or debugging sessions. The repetition is real; the scope is wrong. Without a precondition — "when the input is a long document" — the rule will misfire in every other context.

Promotion is a policy decision, not an automatic hierarchy. Mark it as optional. If you cannot state the scope and preconditions of a procedural rule, you do not have a rule — you have a correlation.

Write Policies: What Earns a Slot

Memory grows by accident unless you make writing a decision. The first lever is the trigger, and the triggers have different cost profiles.

End-of-session consolidation runs after the run completes. It is cheap in latency terms because nothing is waiting on it, and it sees the whole trajectory, which makes it better at extracting durable signal than inline extraction. It is worse at capturing things that only make sense mid-run.

Inline extraction runs during the run. It is fast and contextually accurate but expensive per turn and prone to over-eager capture — a passing remark becomes a permanent preference.

Explicit user instruction is the highest-precision trigger. When a user says "remember that I prefer X," that is a write with a clear owner and a clear justification.

Post-failure reflection runs when something goes wrong. It is the primary source of procedural memory, because failures are where the gap between the agent's rules and reality becomes visible.

Whichever trigger you use, the admission criteria are the same. A candidate memory must be durable (will this still be true later), specific (does it name an entity, constraint, or rule), and decision-relevant (does it change a future action). A memory that fails any of the three is a log entry.

Then there is the update semantics question, and it is a real tradeoff, not a preference. Append-only accumulation preserves auditability — you can see every version of a fact and when it was written — but it lets contradictions coexist, and retrieval has to resolve them at read time. Update-in-place keeps a clean current state but destroys the evidence trail, which makes debugging memory failures nearly impossible. Supersede-with-history is the middle path: the new value becomes current, the old value is retained as a dated prior. I default to supersede-with-history for anything a user might ask about later, and update-in-place only for high-churn, low-stakes fields.

Temporal validity is the part most implementations skip. Store when a fact became true and when it stopped being true. Without those two timestamps, retrieval cannot rank the right dated instance for a current-state question, and the agent will confidently report a preference the user abandoned last quarter.

One more write rule that pays off: when the agent confirms an action, that confirmation is a first-class memory candidate, not a side effect. "I created the branch" is a fact about the world that a later run may need.

The failure mode to plan for is memory poisoning through over-eager extraction. A one-off remark — "I'm thinking about switching to Rust" — becomes a permanent preference, and every subsequent response is colored by a decision the user never made.

Retrieval Triggers: When Memory Should Enter the Context

The read side has two failure modes and they are opposites. Starvation: relevant memory exists but never gets pulled in, so the agent re-derives what it already knew. Flooding: too much memory gets injected, and the signal drowns in its own context.

The defense against flooding starts with pinning. A small set of identity and policy facts can be always-present — the user's name, the project's language, the non-negotiable rules. Everything else should be selected per decision. Pinning is expensive because it consumes attention budget on every turn, so the pinned set should be small enough that you can list it from memory.

But here is the boundary that matters more than pinning: immutable or safety-critical constraints belong in application logic or versioned instructions, not in memory. Authorization rules, compliance requirements, and safety guardrails must be enforced by the runtime, not retrieved from a mutable store. Memory may supply user or project-specific preferences and candidate procedures. It cannot be the sole enforcement layer for anything that must not fail.

For everything else, retrieval triggers fall into three classes. Query-driven retrieval fires when the current request has a semantic match in the store. Entity-triggered lookup fires when a named entity appears — a user, a project, a service — and pulls the facts attached to it. Event-triggered recall fires at task boundaries, pulling episodic memory relevant to the task about to start.

The retrieval mechanism matters as much as the trigger. Semantic similarity alone misses exact identifiers and entity relationships; a query for "the orders table" may not match a memory that says "orders_v2" if the embedding space does not place them close. Multi-signal retrieval — fusing vector search with keyword and entity matching — covers query shapes that similarity alone cannot. This is not a novel claim; it is the same lesson reranking taught, applied to memory instead of documents.

Then cap the injection. Every retrieved memory should be measured for whether it changed the output. Unused retrieved memory is pure cost and interference — it occupies attention budget and gives the model more surface area to be distracted by. If a memory is retrieved and never used across many runs, that is an investigation signal that the trigger may be wrong. It is not a diagnosis — some memories are retrieved for latent influence or defensive context — but it is the first thing to check.

The subtle failure is retrieval that returns a plausible but wrong-typed memory. An episode answering a policy question. A stale fact outranking a current one. The output looks fine until you check it against the decision it was supposed to serve.

Correction Paths: Fixing Memory That Is Wrong

Every memory system will store something wrong. The question is whether correction is a designed subsystem or an afterthought. Silent overwrites are dangerous because they destroy the evidence needed to explain a past decision, and because they hide the fact that the system was wrong.

There are three correction sources. Explicit user edit is the highest-authority source and should be treated as such. Agent-detected contradiction happens when a new fact conflicts with a stored one; the system needs a precedence rule — recency, source authority, or explicit user confirmation — and it needs to apply that rule consistently. Offline review of traces catches the errors that neither the user nor the agent noticed, and it is the only source that can find systematic extraction problems.

The correction action itself has two modes. Supersede when the history matters for audit or for explaining a past decision: keep the old value as a dated prior and mark the new value current. Hard-delete when the content is sensitive or the user asked for removal. The distinction should be a policy, not a judgment call made at the moment of correction.

User-visible memory is the highest-leverage correction feature you can ship. Exposing what the agent retained, with read, edit, and delete controls, converts silent memory drift into a debuggable surface. It also gives the user a reason to trust the system, because they can see what it knows and correct it directly.

Sensitive-data policy belongs here too. Some categories should be excluded by default and stored only on explicit opt-in, with notification when they are saved. This is not just a compliance concern; it is a correctness concern, because sensitive facts are the ones most likely to be wrong and most damaging when they are.

The failure mode that bites hardest: correction that updates the store but not the retrieval index. The agent keeps reading the old value because the index was never rebuilt. If your correction path does not include index invalidation, it is not a correction path.

Forgetting: Decay, Consolidation, and Retention Policy

Unbounded memory degrades. An ever-growing store dilutes retrieval precision, increases latency, and preserves facts that are no longer true. Forgetting is not a concession to storage limits; it is a requirement for retrieval quality.

Three decay mechanisms cover most cases. Time-based expiry retires memories after a fixed window — appropriate for episodic memory tied to a specific project phase. Access-frequency reinforcement keeps memories that are repeatedly retrieved and lets the rest fade, which mirrors how relevance actually works. Confidence decay lowers the weight of a memory that is never confirmed, so an unverified fact gradually loses precedence to a verified one.

Consolidation is the pipeline that makes memory compound instead of accumulate — but only when it runs through the promotion gates described earlier. Repeated episodic patterns get promoted into semantic facts: three sessions where the user corrected the agent's date format becomes a semantic fact about their preference. Repeated semantic patterns get promoted into procedural rules: a fact about the project's test command becomes a procedural rule about running tests before deploying — but only after the gate confirms scope, preconditions, and evidence that the action improved outcomes.

Retention tiers make the policy explicit. Session-scoped memory dies with the run. Project-scoped memory lives as long as the project. User-lifetime memory persists across projects. Permanent pinned facts never expire but are reviewed on a cadence. Each tier needs its own expiry rule and its own review schedule, and the review schedule is what catches the memories that should have died but didn't.

The failure mode here is consolidation that over-generalizes a single episode into a permanent rule. One bad deployment becomes "never deploy on Fridays," and the agent is confidently wrong in a context where Friday deploys are fine. Consolidation needs a threshold — repeated evidence, not a single instance — before it promotes anything.

Evaluating a Memory System

Memory design arguments are unwinnable without measurement. The measurement that matters is at the decision level, not the store level.

Build a test set where each case depends on a specific memory type being present, absent, or superseded. For semantic memory: a case where the correct answer depends on a stored preference. For episodic: a case where the correct answer depends on recalling a past outcome. For procedural: a case where the correct behavior depends on a learned rule. Then write the negative cases explicitly — stale fact present, contradictory facts present, sensitive fact present, no relevant memory present. The negative cases are where most memory systems fail, and they are the cases most teams skip.

Measure retrieval precision at the decision: did the injected memory change the output in the intended direction? Recall at the store is a vanity metric. A memory that is retrieved and ignored is worse than a memory that was never retrieved, because it consumed budget and added noise.

Track operational signals over time: write volume per session, retrieval latency, memory count growth, correction frequency. A memory count that grows without bound while correction frequency stays flat is a sign that nothing is being retired.

Long-horizon evaluation is the hardest and the most informative. Run multi-session scenarios where the correct answer depends on something learned several sessions earlier, and check whether consolidation preserved it. This is where you find out whether your promotion pipeline actually works or just looks like it does.

Published benchmark numbers for memory systems are results under specific test conditions and datasets. They do not transfer directly to your workload. Treat them as directional evidence about what is possible, not as guarantees about what you will get.

Choosing a Memory Architecture for Your Agent

Start from the smallest thing that works. A filesystem with clear naming conventions handles a surprising number of agents. A single store with disciplined metadata handles more. Add typed stores only when a specific decision is failing — when you can name the decision and the failure that occurs without the type.

The decision rule is the four-question contract from the first section. Add a memory type when you can answer all four questions for it. If you cannot name the decision it serves, you are adding complexity without a payoff.

There are cases where typed memory is overkill. Short-lived sessions, single-turn tasks, and agents whose state is fully reconstructible from the current request do not need it. Building typed memory for these is premature architecture, and it will cost you maintenance without changing behavior.

Build-versus-buy is a real decision with a real tradeoff. Managed memory layers handle extraction, deduplication, and retrieval plumbing, which is genuinely hard to get right. But they also own your write policy and your correction semantics, and their defaults may not match your retention and sensitivity requirements. Check the defaults before you adopt. If the managed layer's write policy is append-only and you need supersede-with-history, you are fighting the tool.

The migration path I would follow: instrument first. Log every retrieval and whether it was used. Once you can see which retrievals are being ignored or misused, split the store by type. The instrumentation tells you which type to build first, and it gives you the baseline to measure against.

The Next Move

Pick one recurring failure in an agent you already run. Trace it back to a missing or misclassified memory type. Then write the four-line contract for that type — decision served, write trigger, read trigger, retirement rule — and implement only that one type. Measure whether the failure rate drops.

The durable rule is this: memory is justified by the decision it changes, and every memory you keep is a claim about the future that you are obligated to maintain. Keep fewer, typed, dated claims. Retire the ones that stop being true. The agent that remembers less, but remembers the right things, is the one that gets the user right.