Self-Critique and Verification Prompts: Critique, Revision, Rubrics, and Chain-of-Verification

You ask the model to critique its own draft. It returns: "The response is clear and well-structured, but could be more specific." You ask it to revise. The revision is the same answer wearing different adjectives.

That failure is not a prompt-wording problem. It is a structural one, and it repeats across every task where people bolt a feedback stage onto model-generated work. The critique sounded like a review. It carried no information. Once you see why, the fix becomes mechanical: a critique is only a signal when it is anchored to something outside the draft, and a revision is only real when the diff proves it moved.

Why Self-Critique Usually Produces Nothing

The model that wrote the draft and the model that critiques it share the same context, the same priors, and the same blind spots. When you ask it to review its own output with no external anchor, you are not running a check. You are re-sampling from the same distribution and calling the second sample a verdict. The draft and the critique are correlated because they were produced by the same process looking at the same evidence.

Fluency hides this. A critique can be well-written, specific-sounding, and still carry zero information about whether the output is correct. The tell is simple: if the critique text would apply to almost any draft of the same task, it is decoration. "Could be more specific" is a sentence that fits every document ever written.

Three things get conflated here, and separating them is the first useful move:

• Self-critique is an opinion about quality. It is cheap and often uninformative.
• Verification is checking a claim against a source, a constraint, or a checkable condition. It can be wrong, but it is falsifiable.
• Revision is acting on the result. It either changes the artifact or it does not.

The reason verification can work at all is an asymmetry: for many tasks it is easier to judge whether a claim satisfies a stated condition than to produce the claim in the first place. Checking that a function handles the empty-list case is easier than writing the function. That asymmetry is the entire economic basis for a feedback stage. When it is not real for your task, no amount of prompt engineering will manufacture it.

Verification quality is bounded by the independence and checkability of the thing being verified against. If the anchor is the draft itself, the bound is zero.

What Makes a Critique Signal Useful

Before writing a critique prompt, decide what would make its output worth reading. A usable signal has four properties:

The fastest way to get all four is to anchor the critique to an artifact: a rubric line, a decomposed constraint, a source passage, or a verification question. Unanchored critique drifts toward style commentary because style is the only thing left to talk about when there is no criterion.

Structure matters as much as content. Separate the critique from the revision in the prompt. When you ask for both in one pass, the model tends to produce a critique that justifies the revision it already wanted to write — the critique becomes post-hoc rationalization rather than a check. Two passes cost more tokens but produce a critique you can inspect independently of the output it is supposed to gate.

There is a real result underneath this. In the DeCRIM pipeline, decomposing an instruction into individual constraints and using a critic to decide where refinement is needed improved an open-source model's constraint-following substantially, and the improvement held even with weak feedback. Stronger feedback let the smaller model close the gap with a much larger one. The lesson is not that self-critique is magic. It is that a structured pipeline — decompose, critique, refine — extracts value even from mediocre critique, and strong critique is what lets a cheap model punch above its weight.

Here is the check I run before trusting any critique prompt: take a draft, plant a known defect in it, and run the critique. If the critique does not flag the planted defect, the prompt is decoration. Do this before you wire it into anything.

Rubric Prompting: Turning Judgment Into a Checklist

A rubric is a decomposition of "good" into independently checkable criteria, each with a pass condition the model can state evidence for. The operative word is independently. If two criteria can only be judged together, they are one criterion wearing two labels.

The mistake is deriving criteria from generic quality words. "Clarity" and "accuracy" are not criteria; they are categories. Derive criteria from a failure taxonomy for the specific task. Collect the ways this task actually fails, then write one criterion per failure mode. A taxonomy of observed failures produces criteria that catch real defects. A list of quality adjectives produces a rubric that grades everything as fine.

The anti-hallucination mechanism is the quote. Require a per-criterion verdict plus the quoted span that justifies it:

For each criterion, output:
- criterion: <name>
- verdict: pass | fail
- evidence: "<exact quote from the draft>" or "no supporting span found"
- required_change: <one concrete edit, or "none">

The quote forces the critique to point at text that exists. A verdict with no quotable span is a verdict the model invented. This is the cheapest structural defense against a critique that sounds rigorous and checks nothing.

Rubric granularity is a real tradeoff, not a knob to maximize. Too few criteria and the critique stays generic. Too many and the model starts failing criteria it cannot actually evaluate, producing noise that looks like rigor. I would rather have five criteria the model can apply with evidence than fifteen it applies by guessing.

Rubric-based verification has an empirical edge over plain LLM-as-judge scoring in meta-evaluation, and rubric feedback can be fed back iteratively at inference time without retraining. The gain, though, depends entirely on the rubric matching the actual failure distribution. A rubric built from a failure taxonomy that was automatically constructed from real agent failures is doing work; a rubric built from a brainstorm is not.

When not to reach for a rubric: open-ended generative tasks where "good" is genuinely contested, or tasks where you have not yet collected failure examples. If you cannot name the failure mode, you cannot write the criterion, and a rubric without criteria is just a longer way to say "make it better."

Chain-of-Verification: Making the Model Check Claims It Cannot See

Chain-of-Verification (CoVe) is a structural fix for the shared-context problem. The mechanism has four stages:

1. Generate the draft.
2. Generate verification questions about the draft's individual claims.
3. Answer those questions in a context that does not contain the draft.
4. Revise using the answers.

Stage three is where the mechanism lives, but it is not sufficient on its own. Removing the draft from the answering context reduces anchoring — the model is less likely to confirm a claim just because it already wrote it. That is one lever. It is not the same as independence. A verifier can still reproduce the same unsupported claim if it has no source, no tool result, and no independently generated evidence to check against. Using the same model does not become independent merely because one context item was removed.

Three separable mechanisms are doing the work here, and they protect against different failures:

Use draft-context isolation when the failure mode is confirmation bias. Add evidence grounding when the claim is checkable against a source. Reach for a separate model or sampling path only when you have reason to believe the errors are correlated across paths — and measure whether the diversity actually buys anything before paying for it.

It helps to see CoVe against its neighbors, because they are easy to confuse:

CoVe interrogates claims one at a time. That granularity is what makes it work and what makes it expensive. The cost profile is roughly three extra generation passes — questions, answers, revision — so latency on the verification path roughly triples. That belongs on high-stakes claims, not on every output.

The failure path to watch is question quality. Verification questions that are too broad — "is this correct?" — produce answers that are just the draft restated. Questions must target a single checkable claim. "Does the cited figure match the source?" is a verification question. "Is the analysis sound?" is a request for a vibe.

A Worked Trace: From Draft to Diff

The pipeline is easier to trust once you watch a signal travel through it. Take a constrained technical claim: a draft sentence asserting a specific behavior of a function.

Draft span (the claim under test):

The cache evicts entries once the TTL expires, so stale reads cannot occur.

Criterion (from the failure taxonomy — "unsupported causal claim"):

criterion: causal claim is supported by the implementation
verdict: fail
evidence: "so stale reads cannot occur"
required_change: state the eviction condition and note the read path that can still observe stale data

Verification question (draft-free):

Does the eviction path guarantee that no read observes an expired entry?
Answer using only the implementation, not the draft.

Verifier answer (no draft in context, grounded in the implementation):

Eviction runs on a background sweep. A read that arrives between expiry and the next sweep can still return the expired entry. The guarantee is eventual, not immediate.

Verdict routed into revision:

criterion: causal claim is supported by the implementation
verdict: fail
evidence: "so stale reads cannot occur"
required_change: replace the absolute claim with the actual guarantee

Revised span:

The cache evicts entries on a background sweep after the TTL expires, so a read that arrives before the next sweep can still observe an expired entry.

Diff check:

- The cache evicts entries once the TTL expires, so stale reads cannot occur.
+ The cache evicts entries on a background sweep after the TTL expires, so a read
+ that arrives before the next sweep can still observe an expired entry.

The flagged span changed. The surrounding sentences — which the critique did not flag — are untouched. That is the whole point of routing structured feedback instead of asking for a rewrite: the revision moved exactly where the evidence pointed, and the diff proves it.

Routing Feedback Into Revision

A good critique that produces no change is a failed pipeline. The revision step is where most implementations quietly break, because prose critique gives the model too much room to rewrite everything.

Revision prompts should consume structured critique, not prose. Pass the criterion, the verdict, the quoted span, and the required change as separate fields. Then constrain the revision to the flagged spans. An unconstrained rewrite discards correct content and reintroduces defects the critique already cleared — you paid for verification and then threw away its output.

The iteration budget is smaller than people expect, but "one cycle" is a starting budget, not a law. One critique-revision pass captures most of the gain on tasks where the criteria are independent and the revision is constrained. It is not enough when constraints interact, when the revision introduces new claims that themselves need checking, or when an external check fails after the revision. Cap the loop and route the second pass explicitly:

• Continue only if unresolved failed criteria remain, the revision introduced new claims, or an external check failed.
• Stop on pass, on budget exhaustion, or on repeated non-progress — the same criteria failing across cycles with no diff movement.

Log the exit reason. Without it you cannot tell convergence from a fight between two versions, each "fixing" what the other broke.

The acceptance test is a diff. Compare the draft and the revision. If the flagged spans are unchanged, the revision step failed regardless of how good the critique looked. This is the check that catches the most common silent failure: a critique that correctly identified a problem, followed by a revision that politely ignored it.

There is a compounding angle here. Critique-revision pairs generated this way are usable as synthetic training data — generate the pairs with a strong model, distill the verification behavior into a small one. The distilled self-critique work frames the loop as a Gibbs sampling chain and found that a single step of each stage was sufficient for generating the synthetic data, which keeps the distillation cheap. Treat this as a downstream optimization, not part of the core workflow. It changes what you do after the pipeline works, not whether you should build it.

Evaluation: Does the Verification Stage Earn Its Cost

Do not assume the stage helps. Measure it, and measure the two halves separately, because they fail independently.

Build a small labeled set of drafts with known defects. Then track:

• Detection rate — did the critique flag the planted defects?
• Removal rate — did the revision remove them?
• False-positive rate — how often did the critique flag drafts that were already correct?

The false-positive rate is the one people skip, and it is the one that degrades good outputs. A critique stage that flags everything forces the revision stage to churn, and churn on a correct draft makes it worse. Track misses and false alarms together or you will tune the stage into a machine that never stops editing.

Compare against the baseline you actually have, not against nothing. Single-pass generation, best-of-N sampling, and an external judge are all alternatives, and verification is not automatically the cheapest. The cost accounting is tokens, latency, and the human review time the stage is supposed to remove. If the stage does not reduce downstream review, it is overhead with a nice dashboard.

Report uncertainty honestly. Gains depend on model version, task, and rubric quality, so a result on one task does not transfer without re-measurement. The numbers in the papers are real for the tasks and models they measured; they are not a promise about yours.

When Not to Add a Verification Stage

The default should be no. A verification stage is a cost you add when you can name what it buys.

Skip it when the task has a cheap external check. A compiler, a unit test, a schema validator, or a retrieval lookup is strictly better than a model critiquing itself — but only when the checker actually covers the target property. A schema validator that checks structure does not check whether the content is true. Match the checker to the failure mode you are trying to catch; a deterministic check that misses the real defect is worse than no check, because it produces false confidence.

Skip it when the output is low-stakes and reversible, or when a human reviews everything anyway and the critique just adds a layer they have to read through. Skip it when you cannot state the failure mode you are trying to catch. Verification without a named target is extra tokens and a longer trace.

Prefer it when the failure is subtle, the task is high-stakes, the criteria can be written down, and no deterministic checker covers the target property. That is the narrow band where a model checking a model earns its cost.

One operational note worth taking seriously: as reasoning moves into less legible internal computation, the visible critique trace becomes a more important — and more fragile — observability surface. Chain-of-thought records have been a real tool for diagnosing why an agent behaved the way it did, and there is active concern that opaque reasoning techniques erode that monitorability. Treat your critique logs as evidence you may lose. If a verification stage is load-bearing for your safety or quality story, do not assume the trace will always be there to audit.

The Decision Rule

Add a verification stage only when you can name the failure mode, write the criterion, and diff the revision against the draft. If any of those three is missing, you are adding latency and calling it rigor.

Here is the next action, and it takes an afternoon. Take a task you already run in production. Plant a known defect in a draft. Run your critique prompt against it and check two things: did the critique flag the planted defect, and did the revision remove it? If the critique missed it, the problem is the anchor — you have no criterion that catches this failure. If the critique caught it and the revision ignored it, the problem is the routing — your revision prompt is not constrained to the flagged spans. Fix the anchor or fix the routing. The model is rarely the thing that is broken.