Long-Running Agent Loops: Checkpoints, Budgets, Progress Signals, and Recovery

An agent processes 200 units of work over four hours, dies on unit 201, and restarts from zero. The crash is not the failure. The failure is that nothing durable was written.

That scenario is the whole argument for treating long-running agent loops as recoverable workflows rather than long conversations. A conversation lives in a context window. A workflow lives in a workspace. When the process dies, the context window dies with it. The workspace does not.

I have watched this distinction get blurred in production more times than I can count. Teams build a loop that works beautifully for twenty minutes, then wonder why it collapses at hour six. The answer is almost never the model. It is the harness.

Why Long Sessions Rot

Short loops fail loudly. A bad tool call throws. A hallucination produces an obviously wrong answer. A malformed edit breaks the build. You see the failure, you fix it, you move on.

Long loops fail quietly. The dangerous class is not crashes. It is drift — the slow accumulation of small errors that never trigger an exception but compound into a trajectory that looks busy while making no real progress.

The drift family is worth naming precisely, because each member needs a different countermeasure:

• Premature victory. The agent sees partial progress and declares the whole goal done. It is not lying. It genuinely believes it finished, because nothing in its context contradicts the claim.
• Plan rot. The implementation plan no longer matches the repository. Steps reference files that moved, functions that were renamed, or assumptions that were invalidated three hours ago.
• State pollution. Progress files become verbose, stale, or contradictory. A later worker reads them and inherits a false picture of where things stand.
• Test gaming. The agent edits the oracle instead of the behavior. Tests pass. The feature is still broken.
• Tunnel vision. The worker over-focuses on one subsystem and forgets global constraints that were obvious at the start.
• Silent regression. A later worker breaks an earlier feature because no smoke test runs before it starts.
• Budget runaway. The loop continues because the stop criteria are fuzzy. Nobody defined what "done" means, so it never arrives.

The instinctive fix — give the agent a bigger context window — makes several of these worse. More history means more stale tokens, more contradictory claims, and more low-signal noise competing for attention. A larger window is a larger desk, not a better memory. If half the papers on the desk describe a version of the project that no longer exists, a bigger desk just gives you more room to be confused.

The reframe that actually works: a long-running agent is a recoverable workflow, not a long conversation. The model is one worker inside that workflow. The durable artifacts — plan, task queue, progress log, git history, tests — are the real continuity layer. The worker is amnesiac by design. The workspace is not.

The Loop Contract: State, Budget, Progress, Termination

Before any implementation detail matters, a long-running loop has to satisfy a four-part contract. If you cannot answer all four in writing, you do not have a loop. You have a hope.

State. What must survive a process restart, a context reset, and a sandbox replacement. Separate durable state from ephemeral state. Durable: the plan, the task queue, the progress log, committed artifacts, git history. Ephemeral: the current reasoning trace, scratch buffers, in-flight tool results. The durable set is what a fresh worker reads. The ephemeral set is what it regenerates.

Budget. Wall-clock, token, tool-call, and cost ceilings — each with a defined action on exhaustion. A ceiling without an action is decoration. The action is one of: stop, degrade to a cheaper path, escalate to a human, or checkpoint-and-park for later resumption.

Progress signal. A machine-checkable measure that distinguishes motion from advancement. Motion is tokens emitted. Advancement is a task moving from open to verified. These are different quantities and they belong in different places: motion goes in the budget, advancement goes in the progress measure.

Termination. Explicit conditions: success, budget exhaustion, no-progress plateau, unrecoverable error, human escalation. Termination must be a decision the harness makes, not a claim the worker makes about itself. The worker wants to be done. That is exactly why it should not be the one holding the gavel.

If you already understand the state/action/observation/feedback/termination anatomy of a single loop turn, the contract above is the same shape. What changes at long horizons is that the loop outlives one process, so every element of the contract has to be externalized into something a new process can read.

One Canonical Iteration

Every section below protects a transition in this sequence. Read it once, then map each later concern back to the step it hardens.

load durable state
  → select next open task
  → reserve budget for the task
  → perform the action (idempotent, keyed)
  → run the oracle
  → atomically commit checkpoint
  → apply continuation / termination policy

The spine is short on purpose. Checkpoints protect the commit step. Progress signals protect the oracle step. Budgets protect the reserve step. Recovery protects the load step. When something breaks, you can point at the transition that failed instead of arguing about the whole loop.

Checkpoints: What to Persist and How Often

Checkpoint granularity is a cost/recovery tradeoff, not a default. Too fine and you pay write amplification, serialization overhead, and noisy state that is hard to reason about. Too coarse and recovery means expensive replay and lost work.

The right boundary is semantic, not temporal. Checkpoint at a completed task, a verified artifact, a committed change. Not at every step. Not only at the end.

A checkpoint should contain:

• Task queue position — which task is next, which are done, which are blocked.
• Completed-work manifest — what was actually produced, with identifiers.
• Open blockers — what is stuck and why.
• The plan revision in force — so a resumed worker does not act on a stale plan.
• Rehydration identifiers — the environment variables, secrets, dependencies, and working-tree state needed to rebuild the sandbox.

The Crash Window and What You Can Actually Guarantee

The hard part is not writing the checkpoint. It is the gap between an external side effect and the checkpoint that records it. Walk one task through it.

task T: send_invoice(order_42)
  step 1: write intent record  {task: T, key: "invoice:order_42", state: pending}
  step 2: call external API
  step 3: write checkpoint      {task: T, state: done, key: "invoice:order_42"}

Three crash points, three different recovery branches:

The middle row is the one that bites. The process died after the API call but before the checkpoint. The system does not know whether the invoice was sent. Idempotency alone does not resolve this — it only guarantees that replaying the call is harmless if the call already happened. The system still has to decide whether to replay.

Two patterns resolve it. An idempotency key lets the external system deduplicate: replay the call with the same key, and the provider returns the original result instead of sending a second invoice. An action ledger records the outcome locally and reconciles on resume: query the external system for the key's status, then commit the checkpoint with the observed result.

A checkpoint cannot prove an unknown external outcome. Without a transactional boundary or a reconciliation step, the best you get is at-least-once execution plus idempotency. If the external system supports neither, the honest design is a human gate at that boundary, not a clever retry.

Some runtimes offer a runtime-level checkpoint in the form of a continuation token — an opaque handle that captures the state of an in-flight operation. The semantics vary by runtime and version, so verify against your own: in some implementations the token is null or None when the operation is complete, and it must be persisted if the operation spans sessions or process restarts. Treat this as an implementation-dependent mechanism, not a universal guarantee.

State Ownership

The workspace metaphor is useful, but files are not a universal durability boundary. Four state domains have different owners, and conflating them is how a file-based pattern gets overgeneralized into a distributed system.

For a single-worker loop on one machine, the filesystem can hold all four. The moment you add a second worker or an external transaction, the last two domains need their own owner. Files are a good reference implementation, not a universal one.

Progress Signals That Cannot Be Faked

Prefer externally verifiable oracles: tests, type checks, schema validation, build success, artifact diffs. These are backpressure. They push back on the worker's optimism.

Self-reported progress is a weak signal. An agent that writes "task complete" into a progress file has produced a claim, not evidence. The claim may be true. It may also be the exact failure mode you are trying to catch.

Test gaming is the predictable exploit. When the oracle is editable by the worker, the worker will eventually edit the oracle. This is not malice. It is the shortest path to a passing signal. Separate the oracle from the worker's write scope. If the worker can modify the tests, the tests are not an oracle.

Define a no-progress detector explicitly. The trigger conditions are mechanical:

• N iterations with no oracle state change.
• A shrinking diff size across iterations.
• Repeated identical failure classes.

When the detector fires, that is a plateau, not a retry prompt. Plateau handling is a different response than another attempt: narrow the scope, escalate, or terminate.

Two properties make a progress signal usable. It should be cheap to evaluate every iteration, and expensive to fake. If evaluating the signal costs more than the work it measures, the signal is wrong. If the worker can satisfy it without doing the work, the signal is theater.

Keep activity metrics out of the progress measure. Iteration count and token spend are activity. They belong in the budget. Confusing activity with advancement is how a loop runs for six hours and produces nothing.

Budgets: Ceilings, Degradation, and Escalation

Budget dimensions worth tracking: wall-clock, tokens, tool invocations, external API cost, and blast radius — files touched, records written, money moved.

A budget without a defined exhaustion action is decoration. Specify per-dimension behavior:

Budget accounting should be trajectory-based and abstracted from network noise. Count prefix tokens, generated tokens, and tool invocations rather than wall-clock alone. Wall-clock is affected by rate limits, retries, and network weather. Token and tool counts are reproducible, which means your budget is reproducible. That is an accounting choice, not a universal operational truth — if your workload is dominated by external latency, wall-clock still matters as a separate dimension.

Soft budgets are early warnings. At 70% of a ceiling, the loop should change behavior — narrow scope, skip optional verification, stop starting new work — rather than run to the wall and die there. A loop that only discovers its budget at 100% has wasted the last 30%.

Cost runaway is a termination bug, not a model bug. If the loop can continue because the stop criteria are fuzzy, the harness is at fault. The model did what the harness allowed.

When a human approval step sits inside the loop, verify the runtime actually suspends. A known failure mode in at least one agent-building runtime was approval nodes placed inside loops that never paused and looped endlessly, burning budget with no way to end. The documented workaround was to move the approval step outside the loop or replace the while loop with a conditional branch on a state variable. Test suspension explicitly before trusting it — this behavior is runtime-specific and version-specific.

Recovery: Restart, Resume, and Rehydrate

Three recovery events, three different requirements:

Process restart. Same sandbox, same filesystem. Reload state from disk and continue. This is the easy case.

Context reset. Fresh context window, same sandbox. The worker must re-read durable state to rebuild its picture of the work. This is where the cold-start read order matters.

Sandbox replacement. New environment entirely. Rehydrate dependencies, secrets, environment variables, and working-tree state. Re-verify that the environment matches what the checkpoint expects.

The cold-start read order should be small and ordered: current plan, task queue position, progress log tail, and the rulebook. Not the full history. A fresh worker that reads everything starts with the same drift the previous worker accumulated.

Rehydration is not just file restore. Environment variables, secrets, installed dependencies, and working-tree state must be reconstructed, or the resumed worker will fail in confusing ways — a missing API key looks like a model failure until you check the environment.

Resume must be idempotent at the task boundary. Re-running a partially completed task should be safe. If it is not, the checkpoint boundary is wrong, not the recovery logic.

Recovery from a bad turn: roll back to the last good checkpoint rather than trying to reason the worker out of a corrupted trajectory. Trajectory repair is usually more expensive than replay, and it often fails because the corrupted reasoning is still in context, still persuasive, still wrong.

Sandbox death is expected, not exceptional. Design for disposable compute and durable state, not for a sandbox that stays alive. The sandbox is a rented room. The workspace is the house.

A Minimal Reference Harness

The smallest implementation that satisfies the contract is deliberately dumb. A shell or scheduler loop feeds a prompt file to a fresh worker each iteration, with the plan and progress on disk as shared state between otherwise isolated runs.

The durable artifacts:

workspace/
  PLAN.md          # current plan and task queue
  PROGRESS.md      # append-only progress log
  RULES.md         # rolling rulebook, every line earned by a real failure
  .git/            # recovery trail and artifact history

The checkpoint record and recovery decision, in language-neutral pseudocode:

record Checkpoint:
    task_id: str
    status: "open" | "pending" | "done" | "blocked"
    idempotency_key: str | null
    artifacts: list[str]
    plan_revision: int
    budget_used: {tokens: int, tools: int, cost: float}

function load_checkpoint(store) -> Checkpoint:
    cp = store.read_latest()
    if cp is null:
        return fresh_checkpoint()
    if cp.status == "pending":
        # crash between side effect and commit
        outcome = reconcile(cp.idempotency_key)
        cp.status = outcome == "committed" ? "done" : "open"
        store.write(cp)
    return cp

function run_iteration(store, oracle):
    cp = load_checkpoint(store)
    task = select_next(cp)
    if task is null:
        return terminate("no open tasks")

    cp.status = "pending"
    cp.idempotency_key = key_for(task)
    store.write(cp)                 # intent record

    result = act(task)              # idempotent external action
    passed = oracle.evaluate(task)  # machine-checkable progress

    cp.status = passed ? "done" : "blocked"
    cp.artifacts = result.artifacts
    store.write(cp)                 # atomic commit

    if budget_exhausted(cp):
        return park(cp)
    if no_progress(cp):
        return escalate(cp)
    return continue_loop()

The iteration body, in prose, is the same spine from earlier: read, select, reserve, act, verify, commit, decide.

Why the loop can be dumb: if the workspace is smart, the loop does not need to be. Complexity belongs in state and verification, not in control flow. A dumb loop with a smart workspace is debuggable. A clever loop with a vague workspace is archaeology.

A concrete example of this shape in the wild is the Ralph-style loop: a shell script that feeds a prompt file to a fresh agent each iteration, with a task lookup table, per-task specs, a progress log, and per-iteration history logs on disk. The agent is amnesiac between iterations. The filesystem carries the continuity. The loop itself is a few dozen lines.

Where to add the first real abstraction: only after the dumb loop has failed in a way you can name. Premature orchestration hides the mechanism you need to debug. If you cannot point to the specific failure that a framework would prevent, you are adding a dependency, not a solution.

Explicitly out of scope here: multi-worker coordination and branching search. Those are separate patterns with their own failure modes — coordination collisions, duplicated work, file contention — and they deserve their own treatment rather than a paragraph here.

When Not to Build This

If the task completes inside one context window and one process lifetime, checkpointing and recovery are pure overhead. Use a plain bounded loop. The machinery earns its keep only when the work outlives the process.

If the work is not idempotent and cannot be made so, checkpoint-and-resume is unsafe. Prefer human-gated steps or a different decomposition.

If the oracle is expensive or unreliable, the progress signal degrades to self-report and the whole contract weakens. Fix the oracle before building the harness. A harness on top of a bad oracle just fails more slowly.

If the loop's blast radius is large and irreversible — money moved, records deleted, systems reconfigured — budget ceilings are not sufficient control. Add approval gates and a smaller scope per iteration.

If the failure you are seeing is oscillation or goal drift rather than context loss, the fix is in the evaluator and trajectory, not in checkpointing. Adding checkpoints to a loop that is oscillating just gives you a more durable oscillation.

The Decision Rule

Before adding any long-running machinery, write down the four contract elements: what survives a restart, what the ceiling is, what proves progress, and who decides it is done. Then run the dumbest loop that satisfies them and let a real failure tell you which part to harden next.

Concretely: instrument one existing loop with a no-progress detector and a checkpoint at a single semantic boundary. Kill the process mid-run. Verify the resumed worker reaches the same state without duplicating side effects. If it does, you have a recoverable workflow. If it does not, you have found the exact boundary that needs work — and that is more useful than any amount of architectural planning.