Agent State and Persistence: Typed State, Checkpoints, Stores, Threads, and Artifacts

The model is disposable compute. The run is the durable artifact.

Kill a long agent mid-task and you learn which of those two things your system actually believes. If the process dies and the transcript survives but the plan does not, you have built a very confident amnesiac: it remembers the conversation, forgets the work. That gap is the whole subject of agent state and persistence.

I have watched this failure more than once, and it always looks the same. The agent restarts, greets the user warmly, then re-runs a tool it already ran, re-derives a decision it already made, and asks a question it already answered three steps ago. Nothing crashed. Nothing errored. The run simply lost its mind because the only thing that survived was the message list.

This article assumes you already know what a harness is and roughly where state sits inside it. The question here is narrower and harder: how do you decompose state so a run can resume, replay, and be inspected without dragging the entire world into every checkpoint?

One Run, Five Layers

Before the taxonomy, watch a single run move through it. The task is mundane on purpose: fetch a customer record, generate a summary, write the summary to object storage, and update a cross-thread preference.

step 1  load_customer(cust_42)
        working state: {customer: {...}, plan: "summarize"}
        checkpoint A written at step boundary
        side effect: none

step 2  generate_summary(customer)
        working state: {summary_text: "..."}
        checkpoint B written
        side effect: none

step 3  put_object("summary-42.txt", summary_text)
        artifact written to object storage
        working state: {artifact_ref: {id, sha256, media_type, size_bytes}}
        checkpoint C written
        side effect: object storage write (idempotent by key)

step 4  store.put("user:42:prefs", {last_summary_at: now})
        cross-thread store write
        checkpoint D written
        side effect: store write (idempotent by key)

CRASH between step 3's side effect and checkpoint C.

On resume, the runtime loads checkpoint B. Step 3's object write already happened, but the checkpoint that would have recorded the artifact reference never landed. The recovery decision is not "re-run step 3." It is: query object storage for the key, observe the object exists, reconstruct the artifact reference, write checkpoint C, and continue. If the object did not exist, re-execute step 3 with the same idempotency key.

That trace is the spine of everything below. Each layer exists because one part of that recovery path needs a different lifetime, scope, and failure mode.

Layer	Lifetime	Scope	Primary failure mode
Working state	One step	In-process	Lost on crash
Checkpoints	Thread-scoped snapshots	One thread	Lost on schema drift
Stores	Cross-thread durable data	Application-defined namespace	Lost on key collisions
Threads	Identity and continuity	Addressing boundary	Fragmented by thread explosion
Artifacts	Large or binary outputs	Referenced, external	Lost on garbage collection

Working state is what a single node computes and throws away. Checkpoints are snapshots of thread-scoped state at step boundaries. Stores hold application-defined data that outlives any single run. Threads are the addressing scheme that decides which checkpoints belong together. Artifacts are the big payloads you keep out of the state object entirely.

There is a second-order cost to getting this wrong. When the runtime does not persist execution state, agents learn to externalize it into text. A model trained or prompted under a stateless runtime will re-derive what a persistent runtime would have kept, writing intermediate values into the context window over and over. Research on interpreter persistence calls this the "amnesia tax": stateless-trained agents redundantly externalize state into text even when a persistent runtime is available, burning tokens and stability for information the harness could have held for free. The tax is not a prompt problem. It is a runtime contract problem that leaks into model behavior.

Persisting the conversation is not persisting the run. If your restart path can only rebuild the transcript, you have built a chat log with extra steps.

Typed State: Schemas, Reducers, and Migration Debt

Once you accept that state is a first-class object, you have to define it. The state schema is a contract, and the contract has three kinds of fields.

Model-writable fields are things the agent decides: the current plan, a chosen tool, a classification. Runtime-owned fields are things the harness controls: step counters, retry counts, thread metadata. Derived fields are computed from others and should never be written directly, because two writers will eventually disagree.

The subtle part is merge semantics. When two nodes write the same field, what wins? A scalar overwrite is last-write-wins, which is fine until it is not. An append-only message list behaves completely differently: concurrent appends must be ordered, and a naive overwrite silently drops messages. This is why reducers matter. A reducer is the function that decides how a new write combines with existing state.

from typing import Annotated, TypedDict
from operator import add

class AgentState(TypedDict):
    # append-only: concurrent writes accumulate
    messages: Annotated[list, add]
    # scalar: last writer wins, and that is a decision
    current_plan: str
    # runtime-owned: nodes must not write this
    step_count: int
    # derived: recomputed, never persisted as truth
    is_complete: bool

The snippet names contracts. Here is what they do on a real write. Two nodes run in the same step and both append to messages. The reducer concatenates them in arrival order; nothing is lost. Both also try to set current_plan. Last-write-wins picks one, and the other disappears without a log line. If that plan mattered, the schema is wrong: it should be a list with a reducer, or the write should be rejected. step_count is written by the runtime, not by nodes; a node that writes it is a bug, and the harness should reject the write rather than trust the model. is_complete is recomputed from current_plan and messages on every read; persisting it as truth invites a stale True to outlive the work it described.

The migration problem is where this gets expensive. A checkpointer stores serialized blobs. When you rename a field or restructure the state object, old checkpoints still carry the old shape. A user resumes a thread created three weeks ago, and your tools read a field that no longer exists. New conversations work. Old ones break, quietly, at the worst possible moment.

The forum thread that captures this is almost painfully ordinary: a team starts with item_color and item_size, later needs a list of items, and discovers that the checkpointer stores state across tables in blobs. Bulk-rewriting stored blobs is the move everyone considers and almost nobody should make.

The patterns that actually hold up:

• Additive fields with defaults. Add new fields; do not repurpose old ones. A missing field deserializes to its default.
• Version tags on the state object. A schema_version field lets you branch at read time.
• Read-time upcasting. Convert old shapes into the current shape when you load a checkpoint, not by rewriting every stored blob. The migration lives in one function instead of a database operation.
• A compatibility window. Keep old fields readable for a defined period, then drop them deliberately.

I would not type aggressively for a throwaway single-run script. A dict is genuinely sufficient when nothing resumes and nothing is inspected. The moment a run can outlive a process, the schema stops being a convenience and becomes the thing that decides whether resumption works at all.

The schema declares `step_count` runtime-owned and `is_complete` derived.

Checkpoints: Snapshots, Interrupts, and Time Travel

A checkpoint is a snapshot of thread-scoped state at a step boundary, addressed by thread identity plus a sequence position. That addressing scheme is the entire point: it lets you say "resume thread X from position N" and mean something precise.

What a checkpoint captures is bounded, and knowing the boundary prevents a lot of disappointment. It captures graph and node state, pending writes, and the position in control flow. It does not capture the process, open sockets, or in-flight side effects. A checkpoint is a photograph of a decision point, not a save-state of the machine.

That boundary is exactly what makes interrupts work. A human-in-the-loop workflow pauses at a boundary, persists the checkpoint, and waits. When input arrives, the run resumes from the same checkpoint with the new data merged in. The pause is durable because the state is durable.

Time travel is the same mechanism pointed backward. You replay from an earlier checkpoint to reproduce a bad decision. This only works if nondeterministic outputs were recorded. If your replay re-calls the model and the model returns something different, you are not reproducing the bug, you are generating a new one.

The storage tradeoff is real. Checkpoint frequency trades write volume against how much work you lose on a crash. Checkpointing after every token is almost always the wrong default: it multiplies write load to protect a few milliseconds of progress. Checkpoint at meaningful boundaries, where the state is coherent and the cost of redoing the step is non-trivial.

Threads: Identity, Scope, and Continuity Boundaries

A thread is the unit of continuity. It determines which checkpoints belong together. It does not, by itself, determine which store data is visible. Store visibility is an application-level decision: a key, a namespace, a tenant, or an authorization rule your code chooses. Conflating the two produces the most common scope bug in agent persistence, where a developer assumes that resuming a thread also restores the store view that thread saw last time.

The choice of thread identity changes what "resume" means:

• Per conversation: resume means "continue this dialogue."
• Per task: resume means "finish this unit of work," even across sessions.
• Per user session: resume means "pick up where this person left off today."
• Per workflow instance: resume means "continue this specific execution," which is usually what durable execution actually wants.

The failure mode to watch for is thread explosion. Creating a new thread per retry fragments history: each attempt looks like an unrelated run, and inspection becomes impossible because no single thread tells the whole story. A retry is a new attempt within the same thread, not a new thread.

Cross-thread data belongs in a store, not in thread state. If two threads need the same fact, putting it in thread state forces duplication, and duplicated state drifts. One thread updates the fact, the other keeps the stale copy, and now your agent has two opinions about the same user.

Nested runs add a second trap. Subgraphs and child runs may checkpoint into their own namespace, which means a parent graph does not automatically see child updates. The parent's state is coherent from its own perspective and blind to what the child did. If data genuinely needs to cross that boundary, either share it through a store or deliberately configure the child to write into the parent checkpoint. Do not assume the parent sees everything.

Stores: Cross-Thread Durable Data and Its Access Patterns

A store holds application-defined key-value data outside the run state, scoped across threads. User preferences, learned facts, shared knowledge, cached entity records. The access pattern is the tell: checkpoints are addressed by thread config, while stores are read and written explicitly from nodes or application code.

That difference is the design rule. Anything needed to reconstruct this specific run stays in the checkpoint. Durable facts that outlive the run go to the store. A user's preferred language is a store fact. The partial result of step four is a checkpoint fact.

Two operational hazards deserve names.

Staleness. A store read during a long run can go stale. The value was correct when the node started and wrong by the time the node finished. If correctness depends on freshness, read late or version the read.

Concurrent writers. Without conflict rules, last-write-wins becomes a silent bug. Two nodes update the same key, one update vanishes, and nothing in the logs says so. If multiple writers are possible, define the merge rule explicitly.

And the honest counterpoint: a single-thread agent with no cross-session memory should not pay for a store. The operational cost is real, and a store nobody reads is just a database with a nicer name.

Artifacts: Keeping Big Outputs Out of State

Embedding a file's contents, a screenshot, or a large tool output directly into state inflates every checkpoint and slows every resume. The state object is supposed to be small and inspectable. A megabyte of base64 is neither.

The artifact pattern is straightforward: write the payload to external storage, keep a typed reference in state.

class ArtifactRef(TypedDict):
    id: str
    sha256: str
    media_type: str
    size_bytes: int

Content addressing earns its keep here. Hashing the artifact makes replay verifiable and detects tampering or silent overwrite. If the hash in state does not match the bytes in storage, you know the run's inputs changed underneath it, which is exactly the kind of thing that turns a reproducible bug into an unreproducible one.

Artifacts also inherit a different operational burden. They need retention policies, garbage collection, and access control that a state object does not. And they create a specific debugging failure: a run is only fully inspectable if its artifact references resolve. A dangling reference turns debugging into archaeology, where you know the agent saw something and can no longer see what.

Resume, Replay, and the Side-Effect Invariant

Resumption is not a storage problem. It is a determinism problem wearing a storage costume.

There are two distinct goals, and conflating them causes bad architecture. Operational replay exists to resume correctly. Integrity replay exists to prove what happened. The first needs enough state to continue. The second needs an append-only record you can verify.

The mechanism that makes both work is recording nondeterministic outputs. Tool results, external API responses, model completions: store them. On replay, read the stored result instead of re-calling the outside world. You do not need the model to be deterministic. You need the run to be reproducible, and those are different requirements.

The invariant that makes recovery safe is worth stating once and applying everywhere:

Every side effect has an intent record, an idempotency key, and an observed outcome before the workflow advances its durable progress marker.

Read that against the crash windows you are actually defending against:

• Checkpoint written, side effect not executed. The intent record exists, the outcome does not. Recovery re-executes the side effect with the same idempotency key. Safe.
• Side effect executed, checkpoint not written. The intent record exists, the outcome may or may not be recorded. Recovery queries the external system by idempotency key. If the effect is present, record the outcome and advance. If not, re-execute. Safe.
• Partial write during a crash. The intent record is incomplete. Recovery treats the step as not started and re-executes with the same key. The external system's idempotency handling absorbs the duplicate.

The middle case is the dangerous one, because without an intent record the system believes work is done that never happened, or re-does work that already landed. Idempotency keys plus a durable record of intent let you detect and repair it. The invariant is not a framework feature. It is a contract you enforce in your own step wrapper, and it is the difference between a resumable agent and an agent that merely restarts.

Most systems that need fast resume plus complete audit history end up with both snapshots and logs. Snapshots give fast resume. Append-only logs give complete history. Snapshots answer "where do I continue." Logs answer "what actually happened." You will want both answers.

Observability and Inspection of Persisted State

Persistence is not a storage cost. It is a debugging asset, and it is wasted if nobody looks.

State inspection is the primary debugging tool: a readable snapshot of what the agent knew at each step. Log the step id, thread id, checkpoint position, tool calls, and artifact references alongside it. When something goes wrong, you want to see the exact state the agent reasoned from, not reconstruct it from vibes.

Redaction and access control are part of the design, not an afterthought. Persisted state often contains user data, and retention rules apply to checkpoints and artifacts the same way they apply to any other store of personal information.

The evaluation payoff is the part teams underuse. Replaying stored runs against a changed prompt or model lets you compare behavior without re-running the world. Same inputs, same recorded tool results, different model. That is a controlled experiment, and it is only possible because you persisted the run instead of the conversation.

The anti-pattern is logging everything and inspecting nothing. The goal is not maximum data. The goal is a state view that makes the next failure cheaper to isolate.

Choosing Your Persistence Layers

The decision rule is not about run duration. Duration is one signal among several. The test is whether losing progress, duplicating side effects, or waiting on a human costs more than the persistence and recovery complexity you are about to add.

Apply it across the axes that actually matter: reversibility of side effects, cost of a retry, human wait time, audit requirements, and cross-session memory needs.

• Minimal viable setup: an in-memory checkpointer for development. It costs nothing and forces you to define state properly.
• Persist when the cost of loss exceeds the cost of recovery. Irreversible side effects, expensive retries, human pauses, or audit obligations all clear that bar. A long but cheap and fully idempotent task does not.
• Add a store only when cross-thread memory is a real requirement. Speculative memory is a database you maintain for nobody.
• Skip durable execution entirely for short, cheap, idempotent tasks. Re-running from scratch is often simpler and more reliable than resuming.

Both directions of error are expensive. Under-persist and you lose progress, fragment history, and pay the amnesia tax on every restart. Over-persist and you carry a persistence stack that no one inspects, which is just operational weight with a reassuring name.

Here is the next action I would actually take. Pick one existing agent run. Kill the process mid-task. See what survives. Classify every piece of lost information into working state, checkpoint, store, or artifact, and add only the layer the failure proved necessary. Then write the side-effect invariant into your step wrapper and force one crash between a side effect and its checkpoint. If recovery queries the external system and resumes cleanly, you have a resumable agent. If it re-executes blindly, you have a chat log with extra steps.