Context Evaluation: Precision, Recall, Faithfulness, Citation Coverage, and Ablation

A RAG feature ships. The answers read well. Someone wires up a faithfulness evaluator, the number lands at 0.87, and within a week that single scalar is the only thing anyone looks at. Then it drops to 0.81 after a release that touched the chunker, the reranker, and the system prompt, and nobody can say which change caused it — or whether the context is doing anything at all.

That is not a measurement problem. It is an attribution problem. One number cannot describe a pipeline with four distinct stages, and a metric that cannot be traced to a stage cannot tell you what to fix.

Why One Score Cannot Describe Context Quality

The failure mode is predictable. A team treats "context quality" as a scalar, watches it move, and then changes three things at once. The score moves again. The team learns nothing, because the metric was never attributable to a single stage in the first place.

Four different questions get collapsed into that one number:

1. Did retrieval surface the right material? This is retrieval quality — a property of the search step, measured against relevance labels.
2. Did the assembled context contain what the answer needed? This is context usefulness — a property of the prompt after reranking, compression, deduplication, and memory injection have run.
3. Did the model stay inside the supplied context? This is faithfulness — a property of the generated answer relative to its inputs.
4. Did the context change the outcome at all? This is ablation — a counterfactual, and the only question that proves the retrieval layer earned its tokens.

These are not four views of one thing. They are four measurements on four different objects, and a change in any one of them implicates a different part of the system.

The organizing invariant: a metric is only useful if a change in it points at exactly one stage of the pipeline. If you cannot name the stage a regression implicates, the metric is decoration.

Relevance-based ranking metrics are a good example of the gap. A context can rank well and still miss information the answer requires — especially in long-form or multi-document synthesis, where the answer needs coverage across sources rather than a well-ordered list of individually relevant passages. The CRUX work on controlled retrieval-augmented context evaluation makes this concrete: in the long-form setting studied, context coverage and ranked coverage correlated more strongly with final answer coverage than common ranking metrics such as Recall, MAP, and nDCG. Ranking position is a proxy. Coverage is closer to the thing you actually care about.

If you have already internalized the write/select/compress/isolate vocabulary and the transient-versus-persistent distinction, this article is the measurement layer underneath them. Evaluation is how you find out which of those operations is failing.

Define the Unit You Are Scoring

Most evaluation confusion is unit-of-analysis confusion wearing a metrics costume. Before you compute anything, decide what object each metric attaches to.

There are four candidate units:

Unit	What it is	Label source	A regression implicates
Retrieved passage set	Raw output of the search step, before assembly	Query relevance labels (qrels)	Index, embedding model, query rewriting, retriever config
Assembled context	The actual prompt context after rerank, compress, dedupe, memory injection	Answerability or coverage judgments	Reranker, compression, dedup, context assembly order
Generated answer	The model's output	Ground truth or reference answers	Prompt, model, decoding params, context quality
End-to-end outcome	Task success as the user experiences it	Task-specific success criteria	Anything, including the judge

The assembled context is not the same object as the retrieved set. The moment a reranker reorders, a compressor drops sentences, a deduplicator merges near-identical chunks, or a memory layer injects a prior turn, you have a new object with new failure modes. Scoring the retrieved set and calling it "context quality" is how teams end up debugging the wrong layer for a week.

Practical rule: if you cannot name the unit and the label source for a metric, you cannot interpret its movement. Write both down next to the metric before you trust it.

Retrieval Quality: Precision, Recall, and Their Blind Spots

Retrieval metrics are the best-understood layer, which is exactly why teams over-invest here and stop.

Precision is the share of retrieved chunks that are relevant. Recall is the share of relevant material that was retrieved. Both are needed. Neither is sufficient. A retriever that returns one perfect chunk has precision 1.0 and recall near zero; a retriever that returns the entire corpus has recall 1.0 and precision near zero.

Ranked metrics — nDCG, MAP — add position sensitivity. Position matters when the generation step has a fixed attention budget and truncates, or when the model demonstrably attends more to early context. Position matters less when you pass a small, deduplicated set and the model reads all of it. Choose based on whether ranking actually changes what the model sees.

The blind spot is the one that bites: high recall on passages does not guarantee the context covers the information the answer needs. Ten relevant passages can still omit the one fact the answer requires. This is why retrieval metrics alone cannot predict downstream answer quality in long-form settings, and why the context-level layer below exists.

Skip retrieval metrics entirely when you have no relevance labels, when the corpus is small enough that recall is trivially high, or when retrieval is not the suspected bottleneck. Measuring a stage that is not failing is a way to feel productive while the real problem compounds.

Context Usefulness: Coverage, Density, and Answerability

This is the layer most teams skip, and it is the diagnostic one. Evaluate the context before the model writes anything: does the supplied material contain the information required to answer?

Two complementary signals:

• Coverage asks whether required information is present.
• Density asks how much of the context is doing work. A context with high coverage and low density is bloated — it contains the answer plus a lot of noise competing for attention.

The mechanism that makes this measurable is question-based assessment. Instead of judging passage relevance in the abstract, decompose the expected answer into sub-questions and score how many the context can actually answer. The CRUX framework does exactly this: it uses human-written summaries to control the information scope, then applies question-based evaluation with an answerability threshold to score the retrieval context independently of generation.

That control matters. Without a known target — a reference answer, a human summary, a curated sub-question list — "relevance" is a vibe. With one, you can score the context against a defined information scope and get a number that means something.

Why this layer earns its place: it separates "retrieval failed" from "retrieval succeeded but the context was assembled badly." Those two failures have different fixes. If coverage is low, the retriever missed material. If coverage is high but the answer is still wrong, the assembly step — reranking, compression, ordering — destroyed or buried what retrieval found.

Faithfulness and Citation Coverage: Scoring the Answer Against Its Context

Faithfulness is consistency between the answer and the supplied context. It detects fabrication relative to context. It does not detect truth in the world.

That distinction is the whole game. A faithful answer to a bad context is confidently wrong. The model stayed inside its inputs, cited them correctly, and produced a well-grounded answer to a question the context could not actually answer. This is why faithfulness must be read alongside context usefulness and never alone. A faithfulness score of 1.0 on a context with poor coverage is not a win. It is a precise measurement of a confident mistake.

Citation coverage is a stricter, more inspectable variant. Two questions, not one:

1. What fraction of claims carry a citation?
2. Do the cited spans actually support the claims attributed to them?

The first is cheap to compute and easy to game — a model can cite everything and support nothing. The second is the one that matters, and it requires claim-level decomposition: split the answer into atomic claims, then check each against the context. That decomposition is the mechanism that makes both faithfulness and citation coverage auditable rather than vibe-based.

At the answer layer, groundedness and completeness form a precision/recall split. Groundedness is the precision side — the answer contains nothing outside the grounding context. Completeness is the recall side — the answer does not miss critical information from the expected answer. A model can be perfectly grounded and uselessly incomplete, or complete and ungrounded. Track both.

A caveat that deserves its own paragraph: LLM judges introduce their own error. Judge model choice, prompt sensitivity, and score drift across model versions all move the number. Treat judge scores as measurements with error bars, not ground truth. Pin the judge model version, and re-baseline when it changes — a judge upgrade can masquerade as a pipeline regression, and you will spend a day debugging a retriever that never moved.

Ablation: Proving the Context Earned Its Tokens

Every metric above scores the pipeline as it ran. Ablation answers the question the others dodge: did the context change the outcome?

Four runs, cheapest first:

• No-context baseline. Run the query with retrieval disabled. If the model answers correctly from parametric memory, the retrieval layer is being paid for nothing — tokens, latency, and index maintenance for zero marginal value.
• Shuffled-context baseline. Supply context from unrelated queries. If the answer quality holds, the model is pattern-matching from memory and ignoring the context entirely. This is the run that exposes "retrieval theater."
• Truncated-context baseline. Cut the context to half, then a quarter. Measure where quality falls off. This connects directly to attention-budget tradeoffs: the point where truncation hurts is the point where your context was actually load-bearing.
• Reordered-context baseline. Move the most relevant material to the end. If quality collapses, position sensitivity is real and your assembly order is a first-class parameter.

The no-context baseline is the highest-value run per token spent. It is one config flag and it answers the most expensive question: is this layer doing anything?

Cost accounting is not optional. Every ablation run costs tokens and latency, so run them against a small fixed query set — the same 30 to 100 queries you use for the main loop — not the full suite. You are looking for a signal, not a publication-grade result.

Decision rule: if removing context does not degrade the metric tied to your system contract, the context is decoration. That metric is not always answer quality. Context can earn its tokens by improving safe tool selection, forcing abstention when evidence is thin, satisfying freshness or authorization constraints, or cutting tool calls — even when a simple correctness score is flat. Ablate against the outcome the context was introduced to improve, not against faithfulness alone.

Wiring the Loop: Dataset, Harness, and Failure Triage

The loop is small. Start with a fixed evaluation set of 30 to 100 queries with reference answers and, where possible, relevance labels. Small and stable beats large and drifting — a set you can re-run in ten minutes gets re-run; a set that takes an afternoon rots.

The harness captures one row per query. The critical design choice is that every mode returns the same fields, so you can diff them:

def evaluate_query(query, pipeline, judge, mode="full"):
    # mode in {"full", "none", "shuffled", "truncated", "reordered"}
    retrieved = pipeline.retrieve(query)              # passage set + IDs
    context   = pipeline.assemble(retrieved, mode=mode)
    answer    = pipeline.generate(query, context)

    return {
        "query_id":            query.id,
        "mode":                mode,
        "retrieved_ids":       [p.id for p in retrieved],
        "context":             context,
        "answer":              answer,
        "retrieval_precision": score_precision(retrieved, query.qrels),
        "retrieval_recall":    score_recall(retrieved, query.qrels),
        "context_coverage":    score_coverage(context, query.subquestions),
        "context_density":     score_density(context, query.subquestions),
        "task_correctness":    score_correctness(answer, query.reference),
        "faithfulness":        judge.faithfulness(answer, context),
        "citation_support":    judge.citation_support(answer, context),
    }

def run_ablation(query, pipeline, judge):
    modes = ["full", "none", "shuffled", "truncated", "reordered"]
    return [evaluate_query(query, pipeline, judge, m) for m in modes]

task_correctness is the primary outcome — scored against a reference answer or the sub-question set, held constant across modes. Faithfulness and citation support are secondary diagnostics. Without a primary outcome, you cannot tell whether context improved the answer, merely changed grounding, or made the judge's job easier.

Run the main loop over the fixed set. Run the ablation variants over the same query IDs. Store everything in one table so a regression is a diff, not a hunt.

Reading a Paired Result

Here is a synthetic trace for one query, showing the deltas against the full-context control:

Mode	Retrieval recall	Context coverage	Task correctness	Faithfulness	Citation support
full	0.80	0.90	0.85	0.92	0.88
none	—	—	0.40	0.95	—
shuffled	0.80	0.10	0.42	0.90	0.30
truncated	0.80	0.55	0.60	0.91	0.70
reordered	0.80	0.90	0.83	0.92	0.86

Read the deltas, not the absolutes. Retrieval recall is identical across full, shuffled, truncated, and reordered — the retriever did the same work every time. The differences live entirely in assembly and generation.

The no-context run drops task correctness from 0.85 to 0.40 while faithfulness rises to 0.95. That is the signature of a model answering from parametric memory: it produces a fluent, internally consistent answer that happens to be wrong. Faithfulness went up because there was no context to contradict. This is exactly why faithfulness alone cannot certify a system.

The shuffled run confirms it. Task correctness stays near the no-context floor (0.42), and citation support collapses to 0.30 — the model is citing passages that do not support its claims. The context is present but not load-bearing.

The truncated run is the interesting one. Coverage falls from 0.90 to 0.55, and task correctness falls proportionally to 0.60. This is the point where context is demonstrably doing work: cutting it hurts. The reordered run barely moves, which tells you position sensitivity is not a factor for this query — you can deprioritize assembly-order tuning here.

Now a second trace, where the pattern implicates a different stage:

Mode	Retrieval recall	Context coverage	Task correctness	Faithfulness	Citation support
full	0.80	0.45	0.50	0.90	0.85
none	—	—	0.48	0.93	—
shuffled	0.80	0.10	0.45	0.88	0.35

Here retrieval recall is healthy at 0.80, but assembled coverage is only 0.45 — and task correctness barely beats the no-context baseline. The retriever found the material; the assembly step lost it. The suspect is the reranker, compressor, or deduplicator, not the index. The confirming experiment is to compare raw retrieved coverage against assembled coverage on the same query IDs: if raw coverage is high and assembled coverage is low, assembly is the culprit.

Full mode: recall 0.80, coverage 0.45, correctness 0.50. None mode: correctness 0.48.

Triage Order

When a score drops, triage in this order:

1. Retrieval first. Did recall or precision move? If yes, the problem is upstream of generation.
2. Context assembly second. Did coverage or density move while retrieval held? The reranker, compressor, or dedup step is the first suspect. Confirm by diffing raw retrieved coverage against assembled coverage.
3. Generation third. Did faithfulness or citation support move while context held? The prompt or model changed.
4. Judge stability last. Did the judge model version change? Re-baseline before you believe any of the above.

That order exists because a judge change can masquerade as a pipeline regression. Check the cheapest explanation before the expensive one.

Version everything that can move a score: index snapshot, chunking config, reranker, prompt, judge model, and evaluation set. A score without a version is an anecdote.

This loop is overkill for single-source lookups, deterministic tool calls, or systems where a downstream check validates the output anyway. If the answer is checked by something that cannot be fooled, you do not need a faithfulness judge.

Reading the Numbers Without Fooling Yourself

Four interpretation traps, in rough order of how often they bite:

Correlated metrics. Precision, recall, coverage, and faithfulness move together often enough that teams mistake one for the cause of another. When four numbers move at once, you changed something upstream of all of them. Slice before you conclude.

Judge drift. A judge model upgrade silently re-baselines every score. Pin the version. When it changes, re-run the old set and record the delta before you attribute anything to the pipeline.

Aggregate scores hiding segment failures. A flat overall score can conceal a collapse on one query type, one document source, or one answer length. Slice by segment before declaring a win. The average is a summary, not a diagnosis.

The upstream blind spot. Every metric here scores the pipeline, not whether the indexed source material is accurate, current, or authoritative. A perfectly faithful answer to a stale document is still wrong. Context evaluation measures the machinery; it does not audit the corpus.

Symptom	First suspect	Confirming experiment
Answers miss facts present in the corpus	Retrieval	Check recall against qrels
Answers miss facts retrieval found	Context assembly	Diff raw retrieved coverage vs. assembled coverage
Answers cite context but are wrong	Context usefulness	Check coverage against sub-questions
Answers invent details	Generation	Check faithfulness and citation support
Everything moved at once	Judge or upstream change	Re-baseline judge, diff configs
Context seems to do nothing	Ablation	Run the no-context baseline

The Next Move

Pick one query from your current pipeline. Run it five ways: full context, no context, shuffled context, truncated context, and reordered context. Record retrieval precision and recall, context coverage, task correctness, faithfulness, and citation support side by side for all five.

If the no-context run scores as well as the full pipeline on task correctness, you have found a retrieval layer that is not earning its tokens — and you found it in one afternoon instead of one quarter. If the shuffled run holds up, your model is ignoring its context, and every retrieval improvement you ship from here is wasted effort until you fix that. If coverage drops while retrieval recall holds, your assembly step is the problem, not your index.

Never trust a single context-quality number. Never ship a retrieval change without an ablation that shows the context changed the answer. The loop is small, the fixed set is cheap, and the alternative is a dashboard that measures the wrong thing with great precision.