Reranking and Contextual Compression: Improving Evidence Before Generation

The retriever found the right document. The answer is still wrong. Somewhere between the vector store and the prompt, the evidence got buried under five near-misses that each contain one relevant sentence and a lot of fluent noise.

That symptom is rarely a retrieval problem. It is a precision problem wearing a retrieval costume. The first-stage retriever is built to maximize recall over a large candidate set, and it does that job well. What happens after retrieval is a separate engineering stage with a separate objective: raise the density of relevant evidence in the tokens the generator actually reads. Reranking and contextual compression are the two operations that stage performs, and they fail in different ways, which is why they need separate instrumentation.

The Canonical Pipeline Contract

Before the mechanisms, fix the ordering, because the ordering determines what you can measure. The baseline this article builds is:

retrieve N  ->  rerank N  ->  select K  ->  compress K  ->  generate

Each arrow is a boundary where evidence can be lost, and each boundary gets its own metric. This is the contract the code, the metrics, and the final trace all obey.

Two alternative orderings exist and are sometimes correct:

Ordering	When it is justified	What it costs you
Retrieve N → rerank N → select K → compress K (baseline)	Default. Ranking is cheap and reversible; compression is destructive and audited last.	Compression cannot influence which candidates survive.
Retrieve N → compress N → rerank compressed candidates	Compression is very cheap relative to reranking, or the reranker consumes compressed representations directly.	A compression error can remove a candidate before ranking ever sees it; recall loss is now upstream of the reranker.
Joint rerank + compress in one forward pass	You already pay for a cross-encoder pass per candidate and the model emits both a score and a compressed representation.	Couples the two stages; a regression is harder to attribute to one of them.

I default to the baseline because it keeps the destructive operation last and the reversible operation first. Reordering is cheap to reason about. Deleting text is not. When you adopt one of the alternatives, you are trading attribution clarity for latency or cost — make that trade explicitly, not by accident.

Why the Retriever Is Not the Problem

A bi-encoder retriever embeds the query and each document independently, then compares the vectors. That independence is the source of its speed and the source of its ceiling. The model never sees the query and the document together, so it cannot reason about term interaction, negation, or whether a chunk actually contains an answer to the specific question asked. It can only report that the two vectors point in similar directions.

That is fine, because the retriever's job is not to be right. Its job is to make sure the right chunk is somewhere in the candidate set. Recall over a large corpus is the objective, and a cheap independent score is the right tool for it.

The generator, by contrast, pays for every token it receives. Irrelevant tokens do not sit quietly at the edge of the context; they compete with relevant ones for attention and they dilute the signal the model conditions on. The quantity that matters here is evidence density: the ratio of query-relevant content to total context tokens. Volume is not the metric. Density is.

Two operations raise density, and they are not the same operation:

• Reranking changes the order and selection of candidates. It reorders what the retriever returned and decides which subset survives.
• Compression changes the text itself. It removes or rewrites content inside the surviving chunks.

They compose naturally, and they fail differently. A reranker that promotes the wrong chunk produces a confident wrong answer. A compressor that deletes the answer-bearing sentence produces a clean, fluent, wrong answer. You need to measure both.

If you have already internalized the broader vocabulary of context operations — select, compress, isolate — this stage is the concrete, query-time implementation of select followed by compress, applied specifically to retrieved evidence. The rest of this article is about the mechanism and the measurement, not the vocabulary.

Cross-Encoder Reranking: What Actually Changes

A cross-encoder scores the query and document jointly. Instead of two independent embeddings compared by cosine similarity, the model receives the query and the candidate as a single input and produces a relevance score. That joint pass is what lets it model the things a bi-encoder cannot: whether the chunk contains an answer rather than merely related vocabulary, whether a negation flips the meaning, whether the passage covers the specific constraint in the question.

The cost model follows directly from the mechanism. N candidates means N scoring operations through the cross-encoder — typically batched, but still proportional to N. You cannot precompute anything, because the score depends on the query. This is why reranking only works on a bounded candidate set — commonly tens to low hundreds of chunks — and why it sits after retrieval rather than replacing it.

The design has two independent knobs, and conflating them is the most common tuning mistake:

Knob	What it controls	Failure if set wrong
Candidate count (N)	Recall ceiling — the best chunk must be in the pool to survive	Gold chunk never enters the reranker; no downstream fix
Keep count (K)	Precision and token budget — how much reaches the generator	Too low drops evidence; too high re-introduces noise

Tune them as separate experiments. If you change both at once and the answer quality moves, you have learned nothing about which knob did the work.

The empirical picture is consistent across published work: reranking a larger candidate pool tends to help, and passing more chunks to the model tends to help, but both add latency. Anthropic's contextual retrieval work, for example, describes reranking the top ~150 candidates down to the top ~20 before generation, and reports that passing the top-20 chunks outperformed top-10 or top-5 — while noting that reranking adds a runtime step and that the balance between reranking more chunks and reranking fewer is a per-use-case tradeoff. Treat those numbers as a starting point for your own sweep, not as universal constants. The right N and K depend on your corpus, your chunk sizes, and your latency budget.

There is also a choice of what does the reranking. A dedicated cross-encoder reranker model is fast and cheap per candidate. An LLM-as-reranker can be more capable on subtle relevance judgments but costs a generation-scale call per candidate or per batch, which usually makes it viable only at small N or when you are already paying for a generation call in the same request. I default to a dedicated reranker unless I have evidence that its relevance judgments are the bottleneck.

When not to rerank: small corpora where the retriever already returns the answer in the top few chunks, tasks that tolerate a broad recall-oriented context, or latency budgets where a serial model call dominates end-to-end time. Reranking is a precision instrument. If precision is not your problem, it is overhead.

Contextual Compression: Extractive, Abstractive, and Soft

Compression is the operation that changes the text. Three families, separated by what they preserve and what they risk destroying.

Extractive compression prunes sentences or spans and keeps the original wording. It is cheap, auditable, and safe for citation because every surviving token traces back to the source. Its ceiling is set by how the source is segmented: if a relevant fact is split across two sentences and only one survives, you have lost it. Extractive methods are the right default when citations, auditability, or regulated content matter.

Abstractive compression rewrites or summarizes with a model. It reaches higher compression ratios because it can generate new phrasing, but it can also introduce claims that were not in the source, and it adds a generation call to the pipeline. The failure mode is subtle: a fluent summary that reads well and asserts something the source never said.

Soft or embedding-level compression replaces text with learned representations — memory tokens or compressed embeddings that the generator consumes directly. It achieves the highest ratios, but it couples the compressor to the generator and complicates debugging, because there is no text to inspect when something goes wrong. Research on reranking with compressed document representations shows this approach can keep reranking effective while holding input length constant regardless of document length, which is attractive for long documents — but it is a heavier architectural commitment than most pipelines need.

The architectural insight worth internalizing: compression and reranking can share a single forward pass. If a cross-encoder is already scoring each query-document pair, a model trained to emit both a relevance score and a compressed representation gets compression nearly free. This is the design behind systems like OSCAR, which performs query-dependent compression and reranking in one pass and reports inference speedups with minimal accuracy loss. The practical consequence: if you are already paying for a reranker, the marginal cost of adding compression is small — which changes the cost-benefit calculation in favor of doing both.

One property that trips people up: compression is query-dependent. The same chunk compresses differently for different questions, because what counts as relevant depends on what is being asked. That means compressed artifacts generally cannot be cached across queries. You can cache the source chunks; you cannot cache their compressed forms. Offline soft representations that are not query-conditioned are a different case — those can be precomputed and reused, which is exactly why they trade away query-specificity.

The compression failure mode to plan for: the compressor removes the sentence containing the answer while keeping the fluent surrounding text. The output looks clean. It is wrong. This is the regression that no answer-quality metric will catch quickly, because the generator will produce a plausible answer from the remaining context.

A Minimal Implementation: Retrieve, Rerank, Compress, Generate

Build the smallest pipeline that exposes the mechanism before any framework abstraction hides it. Four stages, and the instrumentation is the deliverable.

def answer(query, retriever, reranker, compressor, generator, N=50, K=8):
    # Stage 1: retrieve top-N by vector similarity
    candidates = retriever.search(query, top_n=N)
    log_candidates(query, candidates, stage="retrieve")

    # Stage 2: score each candidate jointly with the query, keep top-K
    scored = [(doc, reranker.score(query, doc.text)) for doc in candidates]
    scored.sort(key=lambda pair: pair[1], reverse=True)
    kept = scored[:K]
    log_rank_movement(query, candidates, kept)  # pre-rank vs post-rank

    # Stage 3: compress surviving chunks, record token counts
    compressed = []
    for doc, score in kept:
        text = compressor.compress(query, doc.text)
        compressed.append({"text": text, "source_id": doc.id, "score": score})
    log_token_delta(query, kept, compressed)

    # Stage 4: assemble prompt, generate, keep text traceable to source
    prompt = build_prompt(query, compressed)
    return generator.generate(prompt), compressed

The stages map cleanly onto framework wrappers you may already have seen. A "contextual compression retriever" that wraps a base retriever with a compressor is just stages 1–3 fused behind one interface; the reranker and the compressor are the base_compressor. Recognizing that mapping matters, because when the wrapper misbehaves you need to know which of the four stages to instrument.

What to log, at minimum:

• The candidate list with scores, before reranking.
• The pre-rerank and post-rerank rank of every document. Rank movement is the reranker's actual output.
• Token counts before and after compression, per chunk and in total.
• Per-stage latency, so you can attribute the added time.

Without the rank-movement log, you cannot tell whether the reranker helped or merely reshuffled. Without the token-delta log, you cannot tell whether compression is buying you budget or just deleting content.

One Worked Trace

Five candidates, one query, one answer-bearing span. This is what the logs should look like when the pipeline is working — and what they look like when it is not.

query: "What GPU was used to train the model, and how long did it take?"

stage=retrieve  N=5
  id=doc_12  sim=0.81  "The model was trained on a cluster of A100 GPUs..."
  id=doc_07  sim=0.79  "Training infrastructure is a major cost center..."
  id=doc_31  sim=0.77  "We used 64 A100s for 21 days..."
  id=doc_44  sim=0.74  "Inference was served on H100s..."
  id=doc_02  sim=0.71  "The dataset contains 1.2T tokens..."

stage=rerank  (cross-encoder scores, joint with query)
  doc_31  score=0.94   rank 3 -> 1   (contains both GPU and duration)
  doc_12  score=0.88   rank 1 -> 2   (GPU only, no duration)
  doc_44  score=0.41   rank 4 -> 3   (wrong phase: inference)
  doc_07  score=0.33   rank 2 -> 4   (topical, no answer)
  doc_02  score=0.29   rank 5 -> 5   (irrelevant)

stage=select  K=3
  kept: doc_31, doc_12, doc_44

stage=compress  (extractive, sentence-level)
  doc_31  tokens 180 -> 42   kept span: "We used 64 A100s for 21 days."
  doc_12  tokens 210 -> 55   kept span: "trained on a cluster of A100 GPUs"
  doc_44  tokens 160 -> 0    all sentences pruned (no query-relevant span)

stage=generate
  context tokens: 550 -> 97
  answer: "64 A100 GPUs for 21 days."

Read the trace as three separate diagnoses. The reranker did real work: it moved the only chunk containing both the GPU and the duration from rank 3 to rank 1. That is the rank-movement log earning its keep — without it, you would only see that the answer was correct, not that the reranker was responsible. The compressor then cut 550 tokens to 97 while preserving the answer span in doc_31. And doc_44 compressed to zero tokens, which is correct here: it was about inference, not training, and the reranker had already demoted it.

Now the failure version of the same trace:

stage=compress  (extractive, sentence-level)
  doc_31  tokens 180 -> 38   kept span: "We used 64 A100s."   <-- duration dropped
  doc_12  tokens 210 -> 55   kept span: "trained on a cluster of A100 GPUs"
  doc_44  tokens 160 -> 0

stage=generate
  answer: "The model was trained on 64 A100 GPUs."

The answer is fluent, plausible, and incomplete. The compressor pruned the sentence containing "for 21 days" because it scored lower against the query than the GPU sentence. Compression recall — the fraction of answer-bearing spans that survive — is the metric that catches this. Answer quality alone will not, because the generator answered the half of the question it could still see.

Structural Evidence Breaks Flat Compression

The trace above treats every chunk as a flat string. That assumption fails on tables, code, and lists, where meaning lives in relationships between spans rather than in any single span.

Consider a table row:

| Model | Params | Training GPU | Duration |
|-------|--------|--------------|----------|
| VILA  | 13B    | A100 x64     | 21 days  |

A sentence-level extractor that keeps "A100 x64" and drops the header row has destroyed the meaning: the reader no longer knows that column is Training GPU rather than Inference GPU. The preservation unit is not the sentence. It is the row plus its header. For code, it is the block plus the surrounding conditions. For multi-chunk evidence, it is the linked group, not the individual chunk.

Make the preservation unit explicit in your compressor configuration. If your corpus is structural, sentence-level pruning is the wrong granularity, and no amount of reranking upstream will repair the damage.

Measuring What You Lost, Not Just What You Gained

Answer quality is a lagging, noisy signal. By the time it moves, you have already shipped a regression. Measure the stage directly.

Post-rerank recall-at-K. Against a labeled gold set, compute the fraction of queries where the gold chunk survives into the top-K after reranking. Compare it to pre-rerank recall-at-K. A reranker that improves ordering but pushes the gold chunk out of top-K is a net loss, and this metric catches it. If your candidate count N is too small, the gold chunk never reaches the reranker and no amount of reranking recovers it — recall-at-K will show the ceiling.

Compression recall. The fraction of answer-bearing spans that survive compression. This is the metric most teams skip, and it is the one that produces silent regressions. The verification rule depends on the compressor family:

Compressor family	Verification rule	What it misses
Extractive	Substring or span-overlap check against the labeled answer span	Paraphrase is impossible, so this is exact — but it assumes the span was labeled correctly
Abstractive	Claim-and-qualification check: does the output preserve the claim, its units, negation, and qualifiers? Requires human or labeled-entailment review	A summary can contain the answer while dropping a qualifier ("up to", "in most cases") that changes its meaning
Soft / embedding	No text to inspect; verify end-to-end answer quality and treat the compressor as a black box	You cannot attribute a failure to compression versus generation

Token reduction is never a preservation test. A compressor that cuts 90% of tokens and drops the answer looks identical to one that cuts 90% and keeps it — until you check the span.

Latency accounting. Reranking and compression add a serial stage. Measure p50 and p95 end-to-end, not just the reranker's own runtime. A reranker that adds 80ms at p50 can add far more at p95 under load, and that tail is what your users feel.

Cost accounting. Fewer tokens to the generator offsets part of the added stage cost. Compare total cost per query — retrieval plus reranking plus compression plus generation — rather than the reranker's price in isolation. The compression pass often pays for itself in generation tokens.

Build a small adversarial set of queries that specifically target both failure modes: queries where the answer sits in a low-ranked chunk (tests the reranker's recall ceiling), queries where the answer is a single sentence inside a long chunk (tests the compressor's span preservation), and queries whose answer depends on a table header or a code block's surrounding context (tests the preservation unit). These are the cases that expose the boundaries. A benchmark of easy queries will tell you the pipeline works; the adversarial set will tell you how it breaks.

Decision Rules and Failure Modes

Start with reranking before compression. Reordering is reversible and cheap to reason about; compression destroys information and is harder to audit. Get the ranking right first, then decide whether you still need to cut tokens.

Tune candidate count and keep count as separate experiments, one variable at a time. Otherwise you cannot attribute the result.

Watch for the reranker that is confidently wrong. A cross-encoder can promote a chunk that is topically similar but factually irrelevant, and the generator will trust it more because it ranked first. Reranking does not guarantee correctness; it guarantees that whatever is wrong is now at the top.

Watch for compression that flattens structure. Tables, code blocks, and lists lose meaning when sentences are pruned independently. If your corpus is structural, extractive sentence pruning will quietly corrupt it.

Skip both stages when the corpus is small enough that the retriever already returns the answer, or when the latency budget cannot absorb a serial model call. Prefer extractive compression when citations, auditability, or regulated content matter. Reach for abstractive or soft compression only when token budget is the binding constraint and you can afford the debugging cost.

The invariant to hold: reranking and compression are a precision stage. Evaluate them for what they drop, not only for what they keep.

The First Row of Your Evaluation Set

Pick one query where your current pipeline fails. Log the pre-rerank and post-rerank rank of every candidate, the token count before and after compression, and whether the answer-bearing span survived. That single trace tells you whether your problem is ranking, compression, or neither — and it is the first row of the evaluation set you should keep building.

If the gold chunk never entered the candidate set, your problem is upstream: raise N or fix retrieval. If it entered but did not survive reranking, your reranker or your K is wrong. If it survived reranking but the answer span vanished in compression, your compressor is the culprit. If the answer span survived but the answer is still wrong, check the preservation unit — a table row without its header, or a code block without its conditions, is a compression failure that span-level checks will miss.

One trace, four diagnoses. Run it before you tune anything.