Prompt Chaining: Designing Multi-Stage Cognitive Workflows

A chain is only as reliable as its weakest handoff.

The demo works. You paste a gnarly task into one prompt—extract the entities, reconcile them against the schema, flag the contradictions, write the summary—and the model nails it. You ship it. Two weeks later the output is plausible but wrong: the summary reads fine, the entity list is missing one field, and nothing in the response tells you which instruction the model quietly dropped. You are not debugging a prompt anymore. You are debugging a system whose internal state you never designed.

That is the moment engineers reach for prompt chaining. And most of them reach for the wrong mental model first: "chaining is just calling the model several times." That framing is technically true and practically useless. Calling the model several times is what you already do when you retry. A chain is something more specific: a typed dataflow where each stage declares what it consumes, what it produces, and how it fails. The prompts are the easy part. The joints between them decide whether the pipeline holds.

Why One Prompt Stops Scaling

A single prompt fails in a recognizable way. It does not usually return garbage. It returns something structurally inconsistent—sometimes a clean JSON object, sometimes a JSON object wrapped in an apology, sometimes a paragraph that answers three of your four sub-tasks and silently forgets the fourth. The failure signature is plausible output with unattributable error. You cannot point at the broken instruction because all the instructions share one context window and one generation.

Chaining changes the economics of that failure. Each call solves a smaller problem, so per-stage accuracy tends to improve. The cost is that you now own the seams: latency multiplies by the number of stages, orchestration logic becomes real code, and every handoff is a new place for a defect to enter. You are trading latency and orchestration complexity for per-stage accuracy and inspectability. That trade is worth it under specific conditions, not as a default.

The decision boundary is concrete. Chain when the task has genuinely separable sub-problems, when intermediate validation adds real value, or when a single prompt produces unstable output across runs. Do not chain when the task is one judgment call—splitting a single decision into three calls adds handoffs without adding information. Do not chain when a well-specified single prompt with a clear output contract already produces stable results; the extra calls buy latency, not accuracy.

It helps to place chaining next to the techniques it gets confused with. Decomposition and planning prompts (least-to-most, plan-and-solve) restructure reasoning inside one call. Reasoning scaffolds give the model a representation to think in. Self-critique and verification prompts add a checking pass. Chaining is different in kind: it is separate calls with separate state, where the output of one stage becomes the input to the next. The model does not see the whole task at once. That is the point—and the risk.

I want to be honest about the evidence here, because the marketing around chaining outruns the research. Studies comparing chained refinement against a single stepwise prompt that specifies the same stages inside one call show the advantage is task- and model-dependent, not automatic. In some summarization and information-extraction settings, chaining wins; in others, a single well-structured prompt matches or beats it. The mechanism explains why: chaining helps when isolating a sub-task reduces the model's effective problem, and hurts when the sub-tasks are not actually separable or when each stage needs the full original context. Treat "chain it" as a hypothesis to test, not a law.

A Chain Is a Typed Dataflow, Not a Sequence of Prompts

Here is the mental model I want you to install. A stage is not a prompt. A stage is a record with four fields: a declared input type, a declared output type, a prompt contract, and a failure mode. The prompt is one field of that record. If you cannot fill in the other three, you do not have a stage—you have a string you are hoping will behave.

The chain carries named state. At minimum: the original task, the accumulated artifacts produced so far, control flags (retry counts, abort signals), and provenance (which stage produced which artifact, with which prompt version). Decide explicitly what each stage may read and what it may write. Most chain bugs are state-visibility bugs—a stage reads an artifact it should not see, or writes over one it should have preserved.

The handoff discipline is the part that separates a chain from a pile of calls. Pass structured payloads—JSON, delimited blocks, typed fields—rather than free prose. When stage two receives a paragraph, it has to guess where the entity list ends and the commentary begins. When it receives a JSON object with named fields, it parses. Parsing is deterministic; guessing is not.

The invariant to protect: every stage's output must be independently inspectable and independently testable before it enters the next stage. If you cannot look at a stage's output in isolation and say whether it is correct, the stage is not a stage.

Start with the smallest useful implementation. A table of stages, then a loop that runs them in order and records each result. No framework.

def run_chain(task, stages, state):
    run_id = new_run_id()
    for stage in stages:
        rendered = stage.render(task, state)
        raw = call_model(rendered)
        parsed, ok = stage.parse(raw)
        log(run_id, stage.name, rendered, raw, parsed, ok)
        if not ok:
            return handle_failure(stage, state, run_id)
        state[stage.name] = parsed
    return state

That loop is the whole idea. Everything else—retries, checkpoints, tracing—is elaboration on it. Read it once and you understand more about chaining than a framework's abstraction layer will ever show you.

Designing the Handoff Contract Between Stages

The handoff is where chains die. Specify each stage's output contract the way you would specify an API response: required fields, allowed values, and an explicit representation for "unknown" or "not found." If a field can be absent, say so and give it a null. If a value must come from a fixed set, enumerate the set. Ambiguity at the contract level becomes a parse error or a silent wrong answer downstream.

Validate at the boundary, not downstream. A cheap schema check or parser at the edge catches the failure while the context that produced it is still in hand. If you wait until stage five to notice that stage two emitted a string where you expected a list, you have lost the ability to attribute the defect.

Then decide the failure policy per edge. You have four options: retry the stage, repair the payload, escalate to a human, or abort the chain. Silent coercion—quietly coercing a malformed field into something parseable—is the expensive default, because it converts a loud failure into a quiet one that surfaces three stages later.

Two handoff bugs account for most of the pain I have seen. The first is lossy compression: a stage summarizes its input and drops the exact field the next stage needs. The fix is to make the contract explicit about what must survive. The second is context leakage: a stage inherits stale artifacts from an earlier run or an unrelated branch and reasons over them as if they were current. The fix is to pass state by explicit reference, not by dumping the whole accumulated context into every prompt.

One more piece of judgment: keep the chain's prompts close to the metal. A thin runner you can read beats a framework that hides which prompt actually ran. When a chain breaks at 2 a.m., you want to open one file and see the exact string that hit the API—not trace through three layers of abstraction to reconstruct it.

Observability: Making Each Stage Accountable

A broken chain should produce a diagnosis, not archaeology. That means logging per stage: the rendered prompt (after template substitution, because templates drift), the raw output, the parsed payload, latency, token counts, and the validation result. The rendered prompt matters more than the template. The template is what you wrote; the rendered prompt is what the model saw.

Trace the chain as a single unit with a shared run identifier. When the final artifact is wrong, you want to reconstruct the full path from input to output in one query, not stitch together five disconnected log lines.

Define the metrics that actually change decisions:

• Per-stage failure rate—which stage is the weak link.
• Retry rate—how often the chain is paying for a second attempt.
• Downstream rejection rate—how often a stage's output passes its own check but gets rejected by a later stage. This is the metric that exposes lossy compression and quiet coercion.

Use intermediate artifacts as debugging evidence. When the final answer is wrong, the question is never "why is the model bad." It is "which stage first produced a wrong-but-plausible payload." That question has an answer, and the answer is in your logs.

Finally, distinguish a model failure from a plumbing failure. A truncated response, a parse error, and a genuinely wrong answer need different fixes. Truncation is a token-budget problem. A parse error is a contract problem. A wrong answer is a prompt or task-decomposition problem. Conflating them sends you tuning the wrong thing.

Recovery Points and Error Propagation

Errors compound across stages. A wrong artifact at stage N is consumed as fact by every later stage, so the cost of a defect grows with chain depth. This is the single most important operational fact about chains: the first stage is not always the culprit, but it is always the cheapest place to fix a defect.

Place recovery points where the chain has a natural checkpoint—after extraction, after validation, after a critique pass—and make each checkpoint resumable from stored state. A checkpoint is just the serialized chain state at a known-good boundary. When something breaks, you resume from the last checkpoint instead of re-running from zero.

Choose between local repair and global restart based on whether the defect is isolated or structural. If one stage produced a malformed payload but its inputs were fine, re-run that stage with a corrective instruction. If the defect traces back to a bad extraction that poisoned everything downstream, restart from the last good checkpoint. Local repair is cheaper; global restart is safer. The deciding factor is whether you can trust the artifacts between the defect and the failure.

Guard against the chain that cannot fail loudly. A stage that always returns something plausible removes your ability to detect a bad run. If every stage has a fallback that produces output no matter what, you have built a pipeline that lies to you at scale.

Cap the chain. Maximum retries per stage, maximum total calls, and a defined terminal state so a failing chain cannot loop indefinitely. A chain with no ceiling is a chain that will eventually burn your budget on a task it cannot complete.

Evaluating a Chain Stage by Stage

Evaluation is what tells you whether a stage earns its cost—and it is what lets you change one prompt without re-litigating the whole chain.

Evaluate at two levels. End-to-end quality on a held-out set tells you whether the chain works. Per-stage correctness on the intermediate artifacts tells you where it works and where it does not. Build a small labeled set of stage inputs with expected outputs. This is the asset that makes the chain maintainable: when you change the extraction prompt, you run the extraction eval, not the whole pipeline.

Measure the marginal value of each stage by ablating it. Remove the critique stage. Remove the extraction stage. See whether the final artifact degrades enough to justify the extra call. Some stages earn their latency; some are ceremony. You will not know which without the ablation.

Compare against the honest baselines: the same task in a single prompt, and the same task with a stepwise prompt that specifies the stages inside one call. If the single prompt matches your chain, the chain is not buying accuracy—it is buying structure, and you should decide whether that structure is worth the latency. Track cost and latency per stage alongside quality, because a stage that improves accuracy by a small margin at a large latency cost may not survive contact with production.

When Not to Chain

The most useful skill here is knowing when to stop. Do not chain a task that is one judgment call; splitting it adds handoffs without adding information. Do not chain when a single prompt with a clear output contract and a reasoning scaffold already produces stable results. Do not chain when the sub-tasks are not actually separable, or when each stage needs the full original context to be correct—those are the cases where chaining actively hurts.

And do not chain as a substitute for evaluation. A chain with no per-stage checks is a longer single prompt with more places to hide a bug. The structure is not the value; the inspectability is.

When chaining is the wrong tool, the adjacent techniques usually fit better. Verification prompts add a checking pass without restructuring the task. Candidate selection (self-consistency, best-of-N) buys reliability through sampling rather than decomposition. A single contract-driven prompt handles tasks that are cohesive but need explicit output discipline. The criterion that flips the decision is separability: if the sub-problems can be solved independently and validated independently, chain them. If they cannot, keep them in one call and invest in the contract instead.

The Next Experiment

Take one task you currently handle with a single prompt. Write down its stages as a table: stage name, input, output, failure mode. If you cannot fill in the failure mode column, you have found your first gap. Then implement the thinnest possible runner—the loop above, plus per-stage logging—and run it against a handful of real inputs. Not synthetic ones. The messy ones that broke the single prompt.

The chain will expose which edge breaks first. That edge is your real problem, and it was invisible inside the single prompt. Fix the handoff, not the prompt, and run it again. The invariant to carry forward: a chain is only as reliable as its weakest handoff, and the handoff is the part you designed—or forgot to.