Corrective Retrieval Loops: CRAG, Self-RAG, Query Rewriting, and Evidence Sufficiency

Top-k retrieval always succeeds. That is the problem.

A vector search returns its nearest neighbors whether or not those neighbors answer the question. The pipeline reports no error, the generator receives context, and the answer comes back fluent and confident. Irrelevant context can pull generation below the no-retrieval baseline, so "retrieval happened" tells you nothing about whether the evidence was good enough to answer. Corrective retrieval augmented generation is the discipline of putting a verdict between retrieval and generation: score the evidence, route the request, and terminate on a budget when the evidence never arrives.

This is a control problem, not a prompt problem. The loop has four routing outcomes — use, rewrite, fallback, abstain — and one invariant that keeps it honest: every iteration must either raise evidence sufficiency or terminate. If an iteration cannot do either, it is burning latency for nothing.

If you have built an agent loop before, the anatomy transfers directly: state, action, observation, termination. Here the observation is an evidence-sufficiency verdict, and the action space is retrieval correction rather than tool use.

Why Retrieval Needs a Verdict, Not Just a Score

Similarity answers one question: how close is this document to the query in embedding space? It does not answer the question that actually determines answer quality: is the retrieved set collectively sufficient to answer?

Those are different properties. A document can be topically close and factually useless. A set of five documents can each be individually relevant and still miss the one fact the query requires. And a set can contain the right fact while three noisy neighbors drown it out. The retriever ranks; it does not judge.

The corrective loop adds a decision layer on top of retrieval:

1. Retrieve. Get top-k from the primary index.
2. Assess. Score the retrieved set for relevance and sufficiency.
3. Route. Choose one of four actions based on the assessment.
4. Generate or terminate. Produce an answer from accepted evidence, or return an explicit insufficiency result.

The four routing outcomes and the signal that selects each:

The rest of this article deepens one mechanism at a time: the evaluator that produces the verdict, the trained-reflection alternative, rewriting as a corrective action, sufficiency as a distinct check, and the budget that bounds the whole thing.

The Retrieval Evaluator: What It Scores and How It Routes

CRAG's core mechanism is a lightweight retrieval evaluator: a model that takes a query and the retrieved documents and returns a confidence degree over the quality of that retrieval. The confidence is quantized into three states — Correct, Incorrect, Ambiguous — and each state triggers a different knowledge-retrieval action.

• Correct → refine the retrieved documents and use them.
• Incorrect → discard them and fall back to large-scale web search.
• Ambiguous → combine refined local evidence with fallback retrieval.

The refinement step is the part people skip. CRAG's decompose-then-recompose algorithm strips retrieved documents into smaller knowledge strips, scores each strip, and keeps only the relevant ones before generation. This matters because relevance is not uniform within a document. A long page may contain one useful paragraph and nine paragraphs of noise; strip-level selection keeps the paragraph and drops the rest.

Why a small dedicated evaluator instead of asking the generator to judge its own context? Three reasons:

• Cost. The evaluator runs on every request. A small classifier is cheaper than a frontier-model self-critique.
• Latency. Assessment sits on the critical path. A lightweight model keeps the added round-trip tolerable.
• Separation of concerns. The generator's job is to write. The evaluator's job is to judge. Fusing them makes both harder to debug and harder to swap.

The evaluator is a classifier over (query, document) pairs. Its calibration matters more than its raw accuracy. A miscalibrated threshold silently routes every request to one branch — usually "Correct" — and the corrective loop becomes decoration.

That failure is quiet. If your evaluator's confidence scores cluster in a narrow band and your threshold sits outside that band, you will never see the Incorrect branch fire. You will believe you shipped a corrective system. You shipped a reranker with extra steps.

Self-RAG: Reflection Tokens Instead of an External Judge

CRAG puts the critique in a separate evaluator. Self-RAG puts it inside the generator. The model is trained end-to-end to emit special reflection tokens that decide when to retrieve and how to critique retrieved passages. The critique lives in the weights, not in a sidecar model.

The architectural tradeoff is the whole story:

The decision rule I use: choose the external evaluator when you cannot retrain and need to swap retrievers, corpora, or generators independently. Choose trained reflection when you control the training pipeline and want retrieval decisions fused with generation, so the model learns when retrieval actually helps rather than when it merely returns something.

There is a real cost to the reference implementation that is worth naming. Open-source reproduction work on CRAG reports friction from the original system's dependencies: a paid commercial search API, proprietary fine-tuned weights, and deprecated API calls. That is a property of the reference implementation, not a flaw in the idea. If you build a corrective loop, budget for the fallback retrieval source as a first-class dependency — an API key, a rate limit, and a cost line — not an afterthought.

Query Rewriting as a Loop, Not a Preprocessing Step

Rewriting changes the query, not the corpus. That makes it the only corrective action available when the right document exists in your index but the query never matched it. No amount of reranking fixes a query that was phrased in the wrong vocabulary.

Rewrite intents are not interchangeable:

• Disambiguation. The query is ambiguous; resolve it against conversation context or the dominant reading.
• Decomposition. The query contains multiple sub-questions; split into sub-queries and retrieve each.
• Vocabulary alignment. The query uses terms the index does not contain; rewrite toward indexed terminology.
• Broadening. The query was too narrow and returned nothing useful; loosen it.

The loop hazard is query drift. Rewrite without a sufficiency check and each iteration moves further from the original intent while the retriever keeps returning confident, irrelevant results. The retriever is not lying — it is answering the query you gave it, which is no longer the query you meant.

Three guardrails keep rewriting honest:

1. Anchor the original query. Keep it in the rewrite prompt and in the assessment. The rewrite is a variation, not a replacement.
2. Cap rewrite depth. Two or three rewrites is usually the ceiling. Beyond that, you are searching for a phrasing that does not exist.
3. Require improvement. The new retrieval must beat the previous assessment score. If it does not, stop rewriting and route to fallback or abstain.

Ablation evidence from CRAG shows rewriting contributes measurable accuracy on its own — removing it degrades performance in the reported experiments. That is the argument for giving rewriting its own budget line rather than folding it into a generic "retry" step. It is a distinct corrective action with a distinct cost and a distinct failure mode.

Evidence Sufficiency: The Signal That Decides Abstention

Sufficiency is a set-level property. It is not the max similarity score, and it is not the average relevance. It is a judgment about whether the retrieved set covers what the query requires.

Three practical checks:

• Required-slot coverage. Decompose the query into the facts it needs, then check whether each fact appears in the retrieved set. "What did the company report for Q3 revenue and headcount?" needs two slots. One filled slot is insufficient.
• Claim-level support. For each claim the answer would make, verify a retrieved passage supports it. This catches the case where the set is topically relevant but does not actually contain the answer.
• Conflict detection. Check whether retrieved passages contradict each other. Contradiction is not the same as absence.

That last distinction drives different responses:

• No evidence → fallback retrieval, or abstain if fallback is exhausted.
• Conflicting evidence → surface the conflict. Do not silently pick a side.

Abstention is a valid terminal state, not a failure. Returning "insufficient evidence" with what was found is cheaper and safer than generating from weak context. A wrong answer costs more than an honest non-answer.

The sufficiency threshold needs calibration against a labeled set of answerable and unanswerable queries. An uncalibrated gate has two failure modes, and both are common: it abstains constantly (threshold too high, users lose trust) or it never abstains (threshold too low, the gate is decorative). Build the labeled set before you tune the threshold. Fifty queries with a mix of answerable and unanswerable is enough to see which way your gate is broken.

Bounding the Loop: Budgets, Termination, and Failure Paths

An unbounded corrective loop is a latency generator with a retrieval habit. Termination conditions must be explicit:

• Sufficiency reached. The assessment clears the threshold. Generate.
• Budget exhausted. Iterations, tokens, latency, or cost. Terminate and return the best evidence found, or abstain.
• No progress detected. The assessment score did not improve over the previous iteration. Terminate.

Progress tracking is the loop-engineering analogue of a convergence check. Store the assessment score per iteration and require monotonic improvement. If iteration two scores the same as iteration one, iteration three will too — stop.

Failure modes to instrument:

• Oscillation. The loop alternates between rewrite and fallback without improving. Usually caused by a rewrite that keeps producing the same query in different words.
• Evaluator miscalibration. Everything routes to one branch. Detect by logging the action distribution; if one action is above 90%, suspect the threshold.
• Fallback returning the same corpus. If your fallback source is a web search that indexes the same documents as your local index, fallback adds latency and nothing else. Verify the fallback source is actually independent.

Observability is not optional here. Log the routing decision, the assessment score, the rewrite text, and the terminal reason for every request. Without that trace, a bad answer is archaeology. With it, you can point at the branch that failed.

The cost reality: each corrective iteration adds a retrieval call and an evaluation call. The loop only pays off when the baseline failure rate is high enough to justify the extra latency. If vanilla RAG answers correctly 95% of the time on your workload, the corrective layer is spending latency to fix a 5% problem. Measure the baseline first.

A Minimal Corrective Retrieval Loop

Here is the smallest useful implementation. One callable evaluator, explicit thresholds, a bounded rewrite, and an abstain path.

from dataclasses import dataclass, field

@dataclass
class Assessment:
    score: float          # 0.0 to 1.0, evidence sufficiency
    verdict: str          # "correct" | "ambiguous" | "incorrect"
    reason: str

@dataclass
class LoopTrace:
    iterations: list = field(default_factory=list)
    terminal_reason: str = ""

def assess(query: str, docs: list[str]) -> Assessment:
    """Single callable. Swap or stub during testing."""
    # In production: a small classifier over (query, doc) pairs,
    # plus a set-level sufficiency check.
    ...

def rewrite(query: str, docs: list[str], attempt: int) -> str:
    """Rewrite the query. Keep the original as an anchor."""
    ...

def fallback_retrieve(query: str) -> list[str]:
    """External source. Must be independent of the local index."""
    ...

def corrective_retrieve(
    query: str,
    retrieve,
    generate,
    max_iterations: int = 3,
    use_threshold: float = 0.7,
    abstain_threshold: float = 0.3,
) -> dict:
    trace = LoopTrace()
    original_query = query
    docs = retrieve(query)
    prev_score = -1.0

    for i in range(max_iterations):
        a = assess(query, docs)
        trace.iterations.append({
            "iteration": i,
            "score": a.score,
            "verdict": a.verdict,
            "query": query,
        })

        if a.score >= use_threshold:
            trace.terminal_reason = "sufficient"
            return {"answer": generate(original_query, docs), "trace": trace}

        if a.score <= abstain_threshold and i == max_iterations - 1:
            trace.terminal_reason = "insufficient_evidence"
            return {"answer": None, "trace": trace}

        if a.score <= prev_score:
            trace.terminal_reason = "no_progress"
            break

        prev_score = a.score

        if a.verdict == "incorrect":
            docs = fallback_retrieve(query)
        else:
            query = rewrite(original_query, docs, i)
            docs = retrieve(query)

    trace.terminal_reason = trace.terminal_reason or "budget_exhausted"
    return {"answer": None, "trace": trace}

Three things to notice in this sketch:

The evaluator is one callable with a clear contract. Input: query and documents. Output: a score and a verdict. That contract lets you stub it during testing and swap it during tuning without touching the loop.

Thresholds are named constants. use_threshold and abstain_threshold are not buried magic numbers. They are the two knobs you tune against your labeled set.

The trace is built first. Every run prints the score, the decision, and the terminal reason. Debugging a corrective loop without this is guesswork.

Now break it deliberately. Force assess to always return verdict="incorrect" and a score of 0.5. Run it. Confirm the loop terminates on budget rather than looping forever. If it does not terminate, your no-progress check is wrong, and you found that out in a test instead of in production.

Evaluating the Loop, Not Just the Answer

Final answer accuracy is the wrong primary metric for a corrective system. It tells you the loop worked; it does not tell you which branch worked or whether the branches are pulling their weight.

Measure the routing decisions:

• Action distribution. How often does each branch fire? A branch that never fires is either unnecessary or broken.
• Per-branch accuracy. What is the accuracy when the Correct branch fires versus the Ambiguous branch? Reported reproduction data on CRAG shows the Correct branch far outperforming the Ambiguous branch without fallback — which is the evidence that fallback retrieval is load-bearing, not decorative.
• Counterfactual delta. How often would vanilla RAG have answered correctly on the same query, and how often did the loop change the outcome? This is the number that justifies the latency.

Attribution matters here. Reported figures depend on the generator, the corpus, and the evaluator's calibration. Do not treat any single benchmark number as universal. Run the ablation on your own workload.

When not to use this:

• Latency-sensitive paths. Every iteration adds a retrieval and an evaluation round-trip. If the user is waiting on a sub-second budget, the loop does not fit.
• Corpora where the answer is almost always in the top-k. If your baseline failure rate is low, the corrective layer is overhead.
• Cases where a reranker alone closes the gap. A cross-encoder reranker is cheaper than a full corrective loop. Try it first.

The decision rule: add the corrective loop when retrieval failure is a measured, recurring cause of wrong answers — not because the architecture sounds more advanced.

Where to Start

Do not build the loop first. Instrument the pipeline you already have.

Log every request where the answer was wrong and ask one question: was the evidence absent, present but missed, or present and ignored? Absent evidence points to fallback retrieval. Present but missed points to query rewriting. Present and ignored points to a generation or context-assembly problem that no corrective loop will fix.

Then label a small set — fifty queries, mixed answerable and unanswerable — and measure your baseline. Add the assessment-and-routing layer only where the failure data justifies it. Start with the evaluator and the trace. Add rewriting when you see missed evidence. Add fallback when you see absent evidence. Add abstention when you see confident answers built on nothing.

The invariant holds the whole thing together: every iteration must raise sufficiency or terminate. When a single corrective path is not enough — when the query needs branching exploration rather than one rewrite — the natural extension is a search loop with backtracking, where the state is a partial evidence set and the action space includes revisiting earlier retrieval branches. That is the next concept to study once this one is running and measured.