Fault Tolerance and Durable Execution: Retries, Resumption, Rollback, and Duplicate Prevention

A research agent runs for twenty minutes. It pays for dozens of model calls, executes a dozen tool calls, and commits three external side effects. Then the worker process dies. The naive fix — restart the run — silently re-fires every side effect it already committed.

That is the failure this article is about. Not "how do I retry?" but "which effects are already committed, and what does resuming actually mean for each one?"

If you already have an agent harness with state, checkpoints, and tool contracts, you have the substrate. This article is about the recovery semantics layered on top: how to classify failures, what to persist, when to resume forward, when to roll back, and how to prevent the duplicate that a naive restart will produce.

Why Retry Logic Alone Breaks Agent Runs

Classic retry logic assumes a stateless, idempotent unit of work. An HTTP request either succeeded or it didn't; a retry wrapper re-sends it and the server handles the rest. That model holds for a single request. It does not hold for an agent loop.

An agent loop spans minutes to hours across many model calls, tool calls, and control-flow decisions. A crash anywhere in that loop can erase all prior work and re-spend tokens. Worse, the work is stateful and often non-idempotent at the tool boundary. A payment call, a file write, a webhook, a database insert — each of these can fire once and only once, and a restart does not know which ones already fired.

The core invariant to reason with before writing any recovery code:

Every step must be classified as replayable, retryable, or committed-and-irreversible before you can decide what recovery is legal.

That classification is the whole game. Everything else — checkpoint granularity, retry policy, idempotency keys, rollback — is downstream of it.

Three failure classes matter, and they are not the same problem:

• Transient infrastructure failure. The network dropped, the model API rate-limited, the worker was evicted. The step did not complete, and re-running it is safe if the step is pure or idempotent.
• Deterministic step failure. A validation error, an authorization denial, malformed tool arguments. Re-running produces the same failure and burns budget.
• Partial completion. The side effect fired but the result was never recorded. This is the dangerous class. The system cannot tell whether the work happened, and a naive retry duplicates it.

Most teams write retry logic for the first class and discover the third class in production, usually via a duplicate charge or a double-sent email.

Checkpointing: What Gets Written, When, and Why It Matters

A checkpoint is a durable record of a state transition — a model response, a tool result, or a control-flow decision — written before the next step begins. The point is not to save memory. The point is to make the run's progress survive the death of the process that produced it.

Granularity is the primary design lever, and it trades off directly against cost.

Per-step checkpoints give fine resumption but multiply write volume and storage. Per-superstep checkpoints reduce overhead but widen the replay window: if the worker dies mid-superstep, every step in that batch must be re-executed, which is fine for pure steps and catastrophic for irreversible ones.

Checkpoint content must be sufficient to reconstruct the decision, not just the output. That means the inputs, the tool arguments, and the identity of the step — not only the model's response text. A checkpoint that records "the model said to call send_email" without recording the arguments cannot be safely resumed, because the model may re-synthesize different arguments on the next pass.

Keying and cursoring matter for the same reason. A stable run identifier plus a monotonic step cursor lets a new worker pick up the run without replaying the whole history. The cursor is a position in the run, not a timestamp; timestamps drift, cursors don't.

The failure mode to watch for:

Checkpointing the model's reasoning but not the tool's side effect creates a resume that believes work happened when it did not — or vice versa.

If the checkpoint says the email was sent but the send call never reached the mail server, the resumed run skips a step that never completed. If the checkpoint says the email was not sent but the send call did reach the server, the resumed run sends it twice. Both are checkpoint-content bugs, not retry-policy bugs.

Classifying Failures Before You Retry Them

Retry policy belongs per-step, not per-run. A read-only search can retry aggressively; a payment call cannot retry without a deduplication key. The classification is the policy.

Retryable. Transient network errors, rate limits, timeouts on read-only calls. These benefit from bounded exponential backoff with jitter and a max-attempt budget. The budget matters: an unbounded retry loop against a rate-limited API is a self-inflicted denial of service.

Non-retryable. Validation errors, authorization denials, malformed tool arguments. Retrying these burns budget and can amplify damage — a malformed insert retried ten times is ten malformed inserts, not one corrected one.

Ambiguous. The call may or may not have reached the downstream system. This is the dangerous class and the one that forces idempotency design. A timeout on a write is ambiguous: the write may have committed, or the request may have died in transit. You cannot resolve ambiguity by retrying; you resolve it by making the operation idempotent or by querying the downstream system for the effect.

A concrete rule I use:

If you cannot state whether a step is safe to run twice, you do not yet have a retry policy — you have a hope.

The fix is not a better retry wrapper. The fix is to make the step's effect class explicit at the tool contract level, so the harness knows what recovery is legal before the failure happens.

Idempotency and Duplicate Prevention at the Tool Boundary

Idempotency keys are the standard mechanism: the caller supplies a stable key derived from the logical operation, and the downstream service deduplicates on it. Send the same key twice, get the same result once.

For traditional programs, this works because a retried call is identical to the original. For LLM agents, that assumption fails.

After a restore, an LLM may re-synthesize a subtly different request for the same logical action. The model does not remember that it already decided to refund order 4471; it re-derives the decision from context and produces a request that is semantically the same but textually different. A naive content hash of the request produces a new idempotency key, and the duplicate slips through.

The mitigation is to derive the key from the logical intent and step identity, not from the model's regenerated arguments:

def idempotency_key(run_id: str, step_name: str, logical_target: str) -> str:
    # logical_target is a stable identifier the harness controls,
    # not the model's regenerated argument text.
    return hashlib.sha256(
        f"{run_id}:{step_name}:{logical_target}".encode()
    ).hexdigest()

The harness, not the model, owns the key. The model proposes an action; the harness maps that action to a stable logical target and derives the key from run identity plus step identity. If the model re-synthesizes the same intent after a restore, the key is identical and the downstream service deduplicates.

Two related hazards deserve the same treatment:

Consumed credentials and one-time tokens must stay consumed across a restore. Replaying a one-time token is a correctness and security failure, not just a duplicate. If a step consumed an OAuth code or a single-use API key, the checkpoint must record that consumption, and the resumed run must not attempt to reuse it.

When idempotency is impossible downstream, the only safe options are to record the effect before firing it or to require human confirmation on resume. Recording-before-firing means the checkpoint says "about to send" and the resume path treats an unconfirmed "about to send" as ambiguous, not as done. Human confirmation on resume is slower but honest: the system admits it does not know, and a person decides.

Resumption vs Rollback: Choosing the Recovery Semantics

Two recovery paths exist, and they are not interchangeable.

Resume-forward. Replay from the last committed checkpoint, skipping completed steps. This is the cheapest option when prior side effects are durable and correct. The run continues from where it stopped, and the only cost is the work between the last checkpoint and the crash.

Rollback. Revert to an earlier state and re-execute. This is necessary when a later step invalidated an earlier decision — the model chose a plan, executed half of it, then discovered the plan was wrong. Rollback cannot undo external effects that already left the system.

That last sentence is the output-commit problem, and it is the boundary that decides everything:

Once an irreversible external effect is committed, no local rollback can retract it. Design must account for this before the effect fires.

You can roll back a database transaction. You cannot roll back a sent email, a charged card, or a webhook that a downstream service already processed. If your recovery design assumes rollback will clean up external effects, it will fail exactly when it matters.

For concurrent input — a user sends a second message while the first run is in flight — the interrupt semantics matter too. The common options are enqueue, reject, interrupt, or rollback, and each has different implications for in-flight tool calls and partial state. Interrupt is the snappiest for chat, but it requires the graph to handle partial tool calls cleanly: a tool call initiated but not completed when the interrupt hits may need cleanup on resume. Rollback is appropriate when the second message replaces the first; interrupt is appropriate when it builds on the first.

The decision rule:

Roll back only when the effects are compensable; otherwise resume forward and reconcile.

Compensable means you have a defined inverse operation — a refund, a delete, a reversal. If you don't, resume forward and reconcile the state by querying the downstream system for what actually happened.

Designing the Recovery Path: A Minimal Implementation

Here is the smallest useful durable-execution skeleton. It is framework-agnostic on purpose, so the mechanism stays visible instead of hiding behind an orchestration library's abstractions.

Structure the run as an ordered sequence of named steps. Each step declares a retry policy and an effect class:

from dataclasses import dataclass
from enum import Enum

class EffectClass(Enum):
    PURE = "pure"              # safe to replay, no external effect
    IDEMPOTENT = "idempotent"  # safe to replay with a dedup key
    IRREVERSIBLE = "irreversible"  # must not replay

@dataclass
class Step:
    name: str
    effect: EffectClass
    max_attempts: int = 3

Persist step results keyed by run identifier and step name. On resume, load committed results and skip those steps:

def resume(run_id: str, steps: list[Step], store):
    cursor = store.load_cursor(run_id)
    for step in steps:
        if step.name in cursor.completed:
            continue  # already committed, skip
        result = execute_with_policy(run_id, step, store)
        store.commit(run_id, step.name, result)

Wrap each side-effecting call with an idempotency key derived from run identity plus logical step identity:

def execute_with_policy(run_id: str, step: Step, store):
    if step.effect is EffectClass.IRREVERSIBLE:
        # Record intent before firing. On resume, an unconfirmed
        # intent is ambiguous, not done.
        store.record_intent(run_id, step.name)
    key = idempotency_key(run_id, step.name, step.name)
    for attempt in range(step.max_attempts):
        try:
            return call_tool(step, idempotency_key=key)
        except TransientError:
            backoff(attempt)
    raise StepFailed(step.name)

The resume path is the important part. A new worker loads the cursor, replays only uncommitted steps, and reconciles any step whose effect status is unknown. For irreversible steps, "unknown" means the harness must query the downstream system or escalate to a human — it must not assume the effect did not fire.

The skeleton is deliberately small. Real systems add leases, heartbeats, and distributed coordination, but those are operational concerns layered on this core. Get the effect classification and the idempotency key derivation right first; the coordination machinery is replaceable.

Observability for Recovery: Making Failures Debuggable

Execution history is a first-class debugging asset. Which step ran, what it received, what it returned, and whether it was replayed or freshly executed — all of that must be recorded, not inferred.

Record the recovery decision itself. Retried, resumed, rolled back, or escalated. Post-incident analysis needs to distinguish a bug from a designed recovery. A run that resumed forward and skipped three steps is behaving correctly; a run that re-executed an irreversible step is not. Without the decision recorded, the two look identical in the logs.

Track duplicate-suppression events. A deduplicated call is a signal that recovery fired. A spike in them indicates a systemic problem — a worker that keeps crashing, a lease that keeps expiring, a step that keeps timing out. The dedup counter is a health metric, not just a correctness mechanism.

Cost observability matters for the same reason. Replayed model calls that should have been cached show up as unexpected token spend. If your token bill spikes after a deploy, the first question is whether the checkpoint store is being read on resume or whether the run is silently restarting from scratch.

The evaluation question to carry forward:

Does your recovery path produce the same observable outcome as an uninterrupted run, or merely a plausible one?

A resumed run that produces a different final state than an uninterrupted run is a correctness bug, even if the output looks reasonable. The only way to know is to compare the two, which means you need the execution history to make the comparison possible.

When Durable Execution Is Overkill

Durable execution adds real overhead: storage, serialization constraints, determinism requirements on orchestration code, and operational surface area. It is not free, and it is not always worth it.

Short runs with cheap, idempotent steps and no external side effects do not need checkpointing. A bounded retry wrapper is sufficient. If every step is pure and the whole run costs seconds, restart-from-scratch is simpler and more debuggable than a durable engine. The restart is cheap, the state is trivial, and the durable machinery is pure ceremony.

The trigger to adopt durable execution is concrete:

• Runs long enough to lose meaningful work on a crash.
• Side effects that cannot be safely repeated.
• Human-in-the-loop pauses measured in hours or days, where the process must hand off its slot and sleep.

Watch for the misuse pattern of using durability to paper over a non-idempotent tool that should have been fixed at the contract level. If a tool cannot be made idempotent, that is a tool-design problem, and a durable runtime will not solve it — it will only make the duplicate harder to reproduce.

The Decision Rule

Classify every step by effect type before writing any retry logic. Pure steps can be replayed freely. Idempotent steps can be retried with a key the harness controls. Irreversible steps must be recorded before they fire and reconciled, not assumed, on resume.

Then choose resume-forward or rollback based on whether the effects are compensable. If you have a defined inverse operation, roll back. If you don't, resume forward and reconcile the state by querying the downstream system.

The next action is an audit, not a rewrite. Take one existing agent run and label each step as pure, idempotent, or irreversible. Then find the first step that would duplicate on a naive restart. That step is your recovery design's real constraint, and everything else in the harness should be arranged around it.