Reflection and Self-Refine: Building Generator-Critic Revision Loops

You add a critique-and-revise step. The output gets longer, more confident, more padded with caveats. Your eval score does not move.

That is the default outcome, and it is worth understanding why before writing a single prompt. The loop you built was a second generation pass wearing a critic's costume. It asked the model how the draft could be better, the model produced fluent suggestions, the reviser produced fluent changes, and nothing in the system ever defined what "better" meant. Fluent revision is not improvement. It is motion.

The lineage matters because it explains the mechanism. Reflexion framed verbal self-feedback as a substitute for gradient updates: the agent fails, receives a reward signal, and writes itself a natural-language lesson stored in memory for the next trial. Self-Refine runs a similar cycle inside a single generation task — evaluate against task constraints, refine, repeat. Both work because the feedback carries information the generator did not have. When your critic has no fixed target, it carries nothing, and you have paid for two inference passes to learn that.

One boundary before we go further, because it changes which architecture you pick. Reflexion operates at the trajectory level: it stores verbal experience across trials so a later action sequence starts smarter. Self-Refine operates inside one generation task. The loop in this article is the narrower, stateless version — it revises a single artifact against explicit criteria and carries no memory between tasks. That is deliberate. Artifact revision is the smallest unit that produces measurable quality gains, and it is the one you can evaluate cleanly. Trajectory-level reflection is a different mechanism with different state, and I am leaving it out.

So treat reflection as a feedback channel, not a second draft. The channel has to be specific, criterion-anchored, and bounded. Everything below is about building that channel.

Why Naive Self-Critique Fails

Two failure modes dominate, and they look different in production.

The first is the agreeable critic. Ask "how can this be better?" and you get back "this is a strong response; consider adding more detail." That is not a finding. It is a compliment with a suggestion-shaped hat. The reviser cannot act on it, so it either does nothing or invents a change to justify the pass.

The second is the vague critic. "The argument could be tighter" is a real complaint with no addressable location. The reviser now has to guess which paragraph, which claim, which transition. It usually guesses wrong, and the revision degrades the parts that were fine.

Both failures share a root cause: the critic's output is a function of the model's general sense of quality rather than a function of explicit criteria. A general sense of quality is not stable across calls, not inspectable, and not comparable across iterations. You cannot measure it, so you cannot tell whether the loop helped.

Reflection pays off only when the critic produces information the generator did not already have. If the critique restates the prompt, the loop is a cost multiplier with no signal.

There is also a budget dimension that beginners miss. Every iteration is a full inference pass over the artifact plus the findings plus the criteria. An unbounded loop is not just slow; it is a budget leak with a quality ceiling, because after the first useful revision the critic starts finding things that do not matter and the reviser starts fixing them.

The Generator-Critic-Reviser Contract

The fix is to stop thinking of this as one model talking to itself and start thinking of it as three interfaces with explicit inputs and outputs.

The critical design decision is the middle row. The critic should emit structured findings, not prose. Each finding carries a criterion identifier, a severity, an evidence span pointing at the specific location in the artifact, and a suggested direction. Structure buys you three things: the reviser can act without re-reading the whole artifact, you can log and diff findings across iterations, and you can measure per-criterion improvement instead of staring at an aggregate score.

Separating the critic from the reviser also means you can swap either one. You can fine-tune the critic on your criteria, replace the reviser with a stronger model, or run the critic as a cheaper model while the reviser stays expensive. None of that requires rewriting the loop.

The obvious question is whether to use one model role-playing all three parts or separate models. My default is a single model with distinct prompts until the loop demonstrably helps, then separate models when independence of judgment matters more than cost. A critic that shares the generator's weights shares its blind spots. A separate critic model — or a critic with a different prompt and temperature — breaks some of that correlation, at the price of another model to operate.

This is a specialization of the general agent loop, not a new control structure. State is the current artifact plus accumulated findings. Action is generate or revise. Observation is the critic's findings. Feedback is the score delta. Termination is explicit. If you have built an agent loop before, you already have the skeleton.

Writing Criteria the Critic Can Enforce

This is the highest-leverage change in the entire system, and it is the part most teams skip.

A criterion must be decidable from the artifact alone, or from the artifact plus a supplied reference. If the critic cannot decide pass or fail by looking at what it has, it is guessing, and a guessing critic produces noise that the reviser faithfully acts on.

Decompose "good" into orthogonal axes. For a technical explanation: factual grounding, constraint compliance, completeness against the request, format, and tone. Orthogonal matters — if two criteria overlap, the critic double-counts the same failure and the reviser over-corrects one dimension.

Each criterion needs a pass condition and a failure example. The failure example is what stops the critic from rubber-stamping. A pass condition alone tells the critic what success looks like; a failure example tells it what to look for. Without the example, the critic drifts toward approval because approval is the path of least resistance.

criteria:
  - id: grounding
    pass: "Every factual claim is traceable to the supplied source or marked as inference."
    fail_example: "States a specific number that does not appear in the source."
    weight: high
  - id: constraint_compliance
    pass: "Response stays within the requested length and format."
    fail_example: "Adds a section the request did not ask for."
    weight: medium
  - id: completeness
    pass: "Addresses every sub-question in the request."
    fail_example: "Answers two of three sub-questions and omits the third."
    weight: high

Weighting matters because the reviser needs to know which failure to fix first. If grounding and format both fail and the reviser fixes format, you have spent an iteration on the cheaper problem.

Anti-pattern: criteria that restate the prompt. If the criterion is already in the task, it adds no new signal to the loop. The critic's job is to check the artifact against a standard, not to re-read the instructions.

Minimal Implementation

Build the smallest loop that runs before adding any framework. The scoring contract is the part most tutorials hand-wave, so I am making it explicit: each criterion is scored 0 or 1 by the critic, weighted, and the aggregate is the weighted fraction of criteria that pass. A finding is a failed criterion. The score is sum(weight of passed criteria) / sum(all weights). That definition makes the threshold meaningful and makes the plateau check comparable across iterations.

WEIGHTS = {"high": 3, "medium": 2, "low": 1}

def score_findings(findings, criteria):
    failed = {f["criterion_id"] for f in findings}
    total = sum(WEIGHTS[c["weight"]] for c in criteria)
    passed = sum(
        WEIGHTS[c["weight"]] for c in criteria if c["id"] not in failed
    )
    return passed / total

def revision_loop(task, criteria, max_iterations=3, score_threshold=0.9):
    artifact = generate(task)
    history = []
    seen_findings = set()

    for i in range(max_iterations):
        findings = critique(artifact, criteria)
        score = score_findings(findings, criteria)
        history.append({"iteration": i, "artifact": artifact, "score": score})

        if score >= score_threshold:
            break

        # Repeated-finding check: critic is recycling, not discovering.
        fingerprint = frozenset(
            (f["criterion_id"], f["evidence_span"]) for f in findings
        )
        if fingerprint and fingerprint == seen_findings:
            break
        seen_findings = fingerprint

        # Plateau check: score stopped moving.
        if i > 0 and score - history[-2]["score"] < 0.05:
            break

        artifact = revise(artifact, findings)

    return max(history, key=lambda h: h["score"])["artifact"]

Two prompt shapes carry the loop. The critic receives the artifact and the criteria and returns structured findings, with a required no-findings path so it can pass an artifact cleanly:

Given the artifact and the criteria below, return findings as JSON.
Each finding: {criterion_id, severity, evidence_span, direction}.
If the artifact satisfies a criterion, omit it. If all criteria pass,
return an empty findings list. Do not invent findings to appear useful.

The reviser receives the artifact and the findings only. Do not re-inject the full criteria list. If you do, the reviser re-litigates the entire task instead of addressing the specific findings, and you get the padding problem back.

Revise the artifact to address each finding below. Change only what the
findings require. Do not add sections, caveats, or hedging that the
findings do not call for.

State to carry across iterations: the current artifact, the accumulated findings, and the score history. The score history is what lets you detect a plateau. The accumulated findings let you check whether the critic is repeating itself, which is the clearest signal that the loop has stopped producing new information.

Here is one iteration traced end to end, so the state transition is visible rather than implied. Task: summarize a source document. Criteria: grounding (high), completeness (high), format (medium). Generator produces a summary that cites one number not in the source and omits the second of two requested sections.

Iteration 0
  findings:
    - criterion_id: grounding
      severity: high
      evidence_span: "revenue grew 40% in Q3"
      direction: "Number not present in source; remove or replace with sourced figure."
    - criterion_id: completeness
      severity: high
      evidence_span: "<end of artifact>"
      direction: "Second requested section (risks) is missing."
  score: (0 + 0 + 2) / (3 + 3 + 2) = 0.25
  action: revise

Iteration 1
  artifact: revised summary, number removed, risks section added
  findings: []   # all criteria pass
  score: 1.0
  action: break (score >= 0.9)
  return: iteration 1 artifact

The trace exposes the contract. The score is not a vibe; it is a weighted pass fraction. The plateau and repeated-finding checks operate on the same state the loop already carries. And the returned artifact is the best-scoring one, not the last one — if iteration 1 had regressed format while fixing grounding, iteration 0 would win.

Deliberate drill. Run one iteration on a fixed input. Print the findings. Read each one and ask: could a reviser act on this without guessing? If any finding fails that test, fix the criterion, not the prompt. Do not add a second iteration until every finding from the first is actionable. Most loops that fail were never worth iterating because the first critique was noise.

Handling Malformed and Contradictory Findings

Structured findings are an interface, and interfaces fail. The critic will occasionally emit invalid JSON, cite an evidence span that does not appear in the artifact, or return two findings that contradict each other. If you do not handle these, the reviser acts on garbage and the loop degrades silently.

Three checks, applied before the reviser sees anything:

• Schema validation. Parse the critic output against the finding schema. On failure, retry once with the parse error appended to the prompt. On second failure, treat the iteration as a no-op and fall through to the best-scoring artifact.
• Evidence-span verification. Confirm each evidence_span is a substring of the current artifact. A span that does not match means the critic hallucinated a location; drop the finding rather than passing it to the reviser.
• Post-revision re-critique. After every revision, re-run the full criteria set, not just the criteria that failed. A revision that fixes grounding can break format, and you will not see the regression unless you re-score everything.

The invariant: every finding the reviser sees must reference a real location in the current artifact and pass schema validation. A finding that violates either is not feedback; it is noise with a structured shape.

Termination and Budgets

Four termination conditions, and you want all four wired in:

1. Criteria satisfied — the score crosses the threshold.
2. Iteration cap — a hard ceiling, typically two or three.
3. Score plateau — the score delta falls below a threshold across two iterations.
4. Repeated findings — the critic emits the same criterion-and-span fingerprint twice, meaning it has stopped discovering new defects.

The plateau and repeated-finding checks are the ones people forget. A critic that keeps finding new things is not necessarily improving the artifact; it may be drifting into lower-severity nitpicks. When findings stop changing between iterations, the loop has extracted what it can and further passes are pure cost.

"The model says it is done" is not a termination condition. Self-reported completion is the least reliable signal in the loop, because the model that produced the artifact is the model judging whether the artifact is finished.

Track tokens, wall-clock, and cost per iteration per request. A runaway loop should be visible in your metrics before it is visible in your bill. And return the best-scoring artifact, not the last one — revision can regress, and the final pass is not privileged.

Measuring Whether the Loop Actually Helps

Build a held-out set with a scoring rubric before you tune anything. Without it you are measuring vibes, and vibes always favor the loop you just built.

Compare three arms:

The middle arm is the diagnostic. If one revision captures all the gain, your loop is a one-shot critic with extra steps, and you should ship the cheaper version. If the full loop beats one revision, the critic is producing genuinely new findings on later passes.

Track per-criterion improvement, not just aggregate score. A loop that fixes format while degrading grounding is a net loss that an aggregate score will hide. Log findings per criterion per iteration and watch which criteria the loop actually moves.

Watch for over-reflection: revisions that add hedging, padding, or caveats without addressing the finding. This is the most common way a working loop degrades. The reviser, told to address a finding, produces text that gestures at the finding without fixing it. The fix is a stricter reviser prompt and a re-critique that checks whether the original finding is actually resolved.

Finally, report cost per quality point. A loop that costs three times as much for a two-point gain may still lose to a single pass from a stronger model. That comparison is the one that decides whether the loop ships.

When Not to Build a Revision Loop

Skip the loop when a deterministic validator exists. Schema checks, unit tests, and compilers are cheaper, faster, and more reliable critics than a model. If your output is JSON, validate it with a parser. If it is code, run the tests. Model critics are for the cases where no deterministic check exists.

Skip it when the task has no verifiable criteria. Without a checkable target, the loop optimizes for fluency, and fluency is not quality.

Skip it when latency budget is tight and a single stronger model pass beats two weaker passes. This is often true, and it is worth measuring rather than assuming.

Prefer external feedback over self-critique when the environment can supply a real signal. Test results, retrieval hits, and tool errors are ground truth. A model critiquing its own output is a fallback for when no external signal exists, not a first choice.

The general rule: reflection is worth its cost only when the critic can produce information the generator did not have. If it cannot, you are paying for a second opinion from the same brain.

The Decision Rule

Build the loop only when you have checkable criteria and a way to measure per-criterion improvement. Start with one iteration and a hard cap. Let the score delta decide whether the loop earns a second pass.

Concretely: take one existing generation task, write a two-criterion critic with a pass condition and a failure example for each, and run the three-arm comparison on twenty held-out inputs. If the middle arm beats the baseline and the full loop does not beat the middle arm, ship the single revision. If the full loop wins, you have evidence that the critic is producing new signal, and you can justify the cost of a second iteration.

The loop is not the goal. The loop is a hypothesis about where quality comes from. Run the experiment before you believe it.