Query Engineering for Retrieval: Analysis, Decomposition, Rewriting, and Filters

Your retriever is not broken. It is answering a question nobody asked.

Watch a real session. A user types: "Compare the latency and cost of the two reranking options we discussed, for our EU deployment." Your hybrid retriever embeds that whole sentence, scores it against chunks, and returns three documents about reranking in general, one about EU data residency, and nothing about either specific option. The generator produces a plausible, fluent, wrong answer. The retriever did exactly what you told it to do. You told it to search for a conversation.

That is the whole problem. The user's request is an intent. The index is a document surface. Between them sits a translation layer most RAG systems never build: query engineering for retrieval. This article is about building that layer — analysis, decomposition, rewriting, expansion, and filters — and about knowing when each transformation earns its latency.

I assume you already have chunking, hybrid search, and a reranker. This sits in front of the retriever, not inside it.

Why the User's Question Is a Bad Retrieval Query

A retriever scores text against text. A user request is not text in that sense. It is a compressed bundle of intent, prior context, and unstated constraints. Four failure classes recur, and each one has a different fix.

Multi-part questions collapsed into one embedding. "Compare the latency and cost of the two reranking options" contains at least four information needs: latency of option A, cost of option A, latency of option B, cost of option B. A single embedding of the compound sentence lands near the average of those needs — a region of vector space that may be close to nothing in particular. This is the geometric version of asking four questions and getting an answer to none.

Unresolved references. "The two reranking options we discussed" and "that one" and "the second option" point at earlier turns. The retriever has no conversation history. It sees a pronoun with no antecedent and retrieves whatever the pronoun statistically resembles. In the worked example, the conversation history must supply the actual entity names — the retriever cannot resolve them from the query text alone.

Implicit constraints never stated as filters. "Our EU deployment" is a jurisdiction constraint. "The current policy" is a time constraint. Neither appears in the query as a predicate, so neither reaches the retriever as one.

Vocabulary mismatch. The user says "reranking options." The document says "cross-encoder rescoring candidates." Same concept, different tokens, weak lexical overlap, and — depending on your embedding model — a weaker vector match than the concept deserves.

The observable consequence to measure is recall on the subquestion, not on the whole question. If you only score end-to-end answers, you cannot tell which transformation failed. You will blame the generator for a retrieval miss.

The Query Plan: Analysis Before Transformation

The central artifact of this layer is an explicit, inspectable query plan: structured output that describes what the request actually needs before any retrieval happens.

A plan is not prose. It is a record with fields:

Field	Meaning	Extraction reliability
intent	What kind of answer is wanted (compare, explain, locate, summarize)	Inference-heavy
entities	Named things the request is about	Cheap, usually reliable
constraints	Explicit limits (time, jurisdiction, version, scope)	Cheap when stated
subquestions	Independently answerable information needs	Inference-heavy
filters	Constraints converted to structured predicates	Depends on constraint type
retrieval_mode	Per-subquery choice: lexical, vector, hybrid	Deterministic from intent

Take the worked example. "Compare the latency and cost of the two reranking options we discussed, for our EU deployment." A useful plan looks like this:

{
  "intent": "compare",
  "entities": ["reranking option A", "reranking option B"],
  "constraints": ["EU deployment"],
  "subquestions": [
    "latency of reranking option A",
    "cost of reranking option A",
    "latency of reranking option B",
    "cost of reranking option B"
  ],
  "filters": {"region": "EU"},
  "retrieval_mode": "hybrid"
}

Why structured output instead of a free-text rewrite? Because you can log it, diff it across model versions, test it against a labeled set, and route on it. A rewritten string is a black box with a new failure mode: you cannot tell whether a bad result came from a bad rewrite or a bad retriever.

The design rule that keeps this honest: every field in the plan must change a downstream decision, or it is decoration. If intent never alters retrieval mode or reranking strategy, delete it. It costs tokens and adds a failure surface for nothing.

Note the reliability split. Entities and explicit constraints are near-deterministic extractions. Intent classification and implicit constraints are inference-heavy and will be wrong sometimes. Treat the two tiers differently — validate the cheap fields, hedge on the expensive ones.

Choosing the Smallest Transformation That Fits

Before you enable all four transformations by default, route by symptom. Each failure class maps to one transformation, one piece of evidence to collect, and one cost to accept.

Observable symptom	Transformation	Evidence to collect	Default cost
Missing or wrong entities	Analysis	Entity extraction accuracy vs. reference	One LLM call
Compound question, multiple needs	Decomposition	Per-subquestion recall	N retrieval calls
Vocabulary mismatch, sparse index	Rewriting / expansion	Recall lift on the mismatched term	One LLM call + wider pool
Constraints leaking into results	Filters	Filter precision and relaxation events	Predicate evaluation

The worked example needs decomposition (four subquestions) and filters (EU region). It does not need expansion — the corpus vocabulary is already aligned. A single-intent query like "What is the default timeout?" needs none of them. Start with the smallest transformation that matches the observed failure, measure, then add the next one only if the metric demands it.

Decomposition: Splitting One Request into Answerable Subquestions

Decomposition is the highest-leverage transformation for compound questions. It is also the easiest to over-apply.

Recognition clue. The request contains "and," "compare," "why does X do Y," or multiple named entities with different information needs. That is your signal to split.

Rules that keep subqueries useful:

1. Each subquery must be self-contained — no pronouns inherited from the parent.
2. Each must be independently answerable — a single document could satisfy it.
3. Subqueries must be non-overlapping in what they retrieve, or you pay for near-duplicates.

Execution and merging. Run subqueries in parallel. Then deduplicate by document before reranking. The choice that matters: rerank per-subquery, or rerank across the merged pool? Per-subquery reranking preserves the best evidence for each information need but can flood the final context with four documents that all answer the same subquestion. Merged-pool reranking balances across needs but can let one dominant subquery crowd out the others. I default to merged-pool reranking with provenance tracking, because the generator needs coverage more than it needs depth on any single subquestion.

Failure mode: over-decomposition. Five subqueries for a question one document answers costs five retrievals and dilutes the final context with near-duplicates. The reranker then has to choose among documents that are all correct, which is a waste of its budget.

Failure mode: under-specified subqueries. A subquery that inherits "that one" from the parent retrieves the wrong entity with full confidence. Self-containment is not optional.

Cost accounting. Decomposition multiplies retrieval calls and reranker tokens linearly. State the budget before you enable it by default. If your reranker bills per candidate and you fan out to five subqueries, you just multiplied that line item by five.

Rewriting and Expansion: Fixing Vocabulary Mismatch

Rewriting and expansion are different jobs and should be separate stages.

Rewriting makes the query retrievable. Its jobs: strip conversational noise, resolve references from conversation history, restore omitted entities, and translate user vocabulary into index vocabulary. The rewrite of "the two reranking options we discussed" is "cross-encoder reranking vs. LLM-based reranking" — the actual entities, spelled the way the corpus spells them.

Expansion widens the candidate set. Its jobs: synonyms, acronyms, domain terms, alternate phrasings. Expansion helps when the index is sparse or the corpus uses inconsistent terminology. It hurts when your reranker is weak, because it raises recall and lowers precision, and a weak reranker cannot recover the precision you gave away.

Rewriting must be grounded in the actual corpus. An LLM rewriting from world knowledge invents terms your index does not contain. If your corpus says "rescoring" and the model rewrites to "re-ranking," you may have made the query worse. Ground the rewrite in observed vocabulary — sample the index, or constrain the model to terms that appear in retrieved documents.

Failure mode: rewriting that silently changes the question. The model drops a constraint, swaps an entity, or — worse — answers instead of searching. A rewrite that returns "Option A has lower latency" is not a query. It is a hallucination wearing a query's clothes.

Verification pattern. Keep the original query in the candidate pool alongside rewrites. A bad rewrite then cannot eliminate the correct document; it can only fail to add to it. This is cheap insurance and I would not ship a rewriting layer without it.

Filters: Turning Constraints into Structured Predicates

The most reliable query engineering is often not linguistic at all. It is extracting constraints and pushing them into the retriever as filters.

Constraint types worth extracting: time ranges, entity type, source or document class, version, jurisdiction, access scope, and status.

Why filters beat semantic similarity for hard constraints: a date range is a predicate, not a similarity score. "Documents from 2024" is a boolean. No embedding will reliably enforce it, and no reranker will reliably fix a violation. Push it into the retriever.

Extraction reliability. Explicit constraints ("from 2024," "in the EU") are near-deterministic. Implicit ones ("our EU deployment," "the current policy") require inference and should be treated as low-confidence. When confidence is low, apply the filter softly — as a boost rather than a hard cut.

Failure mode: over-filtering to zero results. A misparsed date or an over-narrow scope returns nothing, and the pipeline either errors or silently falls back to unfiltered retrieval. Plan a fallback ladder: relax the filter, widen the range, then fall back to unfiltered retrieval with a warning attached to the result.

Failure mode: silently wrong filters. A filter that is wrong but non-empty is the dangerous one. It returns confidently irrelevant documents with no visible error. This is why the logging requirement matters: record which filters were applied per query. Filter bugs look exactly like relevance bugs, and without the log you will debug the wrong layer.

A Minimal Implementation: Analyze, Plan, Retrieve, Merge

Here is the smallest pipeline that exercises all four transformations, with the plan as the inspectable artifact between stages. The code is conceptual pseudocode — it shows the control flow and the invariants, not production error handling.

Stage 1 — Analyze. One LLM call producing the structured plan. The plan carries a filter mode and a confidence source so downstream code knows what is safe to enforce.

def analyze(request, history):
    prompt = ANALYZE_TEMPLATE.format(request=request, history=history)
    plan = llm(prompt, schema=QueryPlan)  # structured output
    return plan

# QueryPlan fields:
#   subquestions: list[str]
#   filters: list[{field, value, mode: "hard"|"soft", source: "app_state"|"explicit"|"inferred"}]
#   query_budget: int  # max total retrieval calls for this request

Stage 2 — Transform. Deterministic code that turns the plan into concrete queries. The original query stays in the pool as a control. The budget caps fan-out before retrieval starts.

def transform(plan, original):
    queries = [{"text": original, "filters": [], "source": "control",
                "subquestion_id": None}]
    for i, sq in enumerate(plan.subquestions):
        queries.append({"text": sq, "filters": plan.filters,
                        "source": "decomposition", "subquestion_id": i})
    for rw in rewrite(plan, original):
        queries.append({"text": rw, "filters": plan.filters,
                        "source": "rewrite", "subquestion_id": None})
    # Enforce budget: keep control + highest-priority subquestions first.
    return queries[:plan.query_budget]

Stage 3 — Retrieve. Parallel execution, hybrid search per subquery. Hard filters become predicates; soft filters become boosts. A hard filter that returns zero hits triggers a logged relaxation, never a silent fallback.

def retrieve(queries, index):
    results = parallel_map(lambda q: index.hybrid_search(
        q["text"],
        hard=[f for f in q["filters"] if f["mode"] == "hard"],
        soft=[f for f in q["filters"] if f["mode"] == "soft"],
    ), queries)
    return results  # list of (query, hits) pairs

Stage 4 — Merge. Deduplicate, then rerank the merged pool. Keep provenance rich enough to evaluate plan fields and filter behavior later.

def merge(results, reranker):
    seen, pool = set(), []
    for query, hits in results:
        for hit in hits:
            if hit.doc_id not in seen:
                seen.add(hit.doc_id)
                pool.append({
                    **hit,
                    "query_id": query["source"],
                    "subquestion_id": query["subquestion_id"],
                    "filters_applied": query["filters"],
                    "relaxed": query.get("relaxed", False),
                })
    return reranker.rerank(pool)

The deliverable is not the code. It is the activity log: per-subquery text, filters applied, relaxation events, hit counts, and timings. Without it, you are debugging by vibes. With it, "why did this document surface?" has an answer.

A hard filter must never be silently relaxed. If the pipeline drops a predicate to avoid an empty result, that event belongs in the log with the original filter and the reason. Silent relaxation turns a filter bug into a relevance mystery.

Evaluating the Query Layer Separately from the Retriever

You cannot improve what you cannot isolate. Build a small labeled set of complex requests with known supporting documents per subquestion — not just a final answer key.

Measure at three levels:

1. Plan correctness. Did the analyzer extract the right entities and constraints? Score the plan against a hand-written reference.
2. Per-subquery recall. Did the right document appear for each subquestion? This is the metric that tells you whether decomposition worked.
3. Merged-pool quality. Did reranking keep the right evidence after deduplication?

The activity log must carry the fields these metrics need: query_id, subquestion_id, transformation type, filters before and after relaxation, hit count, and latency. If your log only records a free-form provenance string, you cannot reproduce the evaluation.

Ablation is the primary tool. Run original-query-only. Add rewriting. Add decomposition. Add filters. Keep a transformation only if it moves the metric it was supposed to move. This is the discipline that prevents a pipeline from accumulating transformations nobody can justify.

Watch for metric masking. End-to-end answer quality can stay flat while per-subquery recall improves, because the generator compensates by reasoning around missing evidence. If you only watch the final answer, you will conclude the transformation did nothing.

Regression risk. Transformations that help compound questions often hurt simple ones. A single-intent query does not need decomposition, and forcing it through a five-subquery fan-out adds latency and noise. Segment your evaluation set by question complexity, and route simple queries around the planning layer entirely.

When Not to Engineer the Query

The decision boundary matters as much as the technique.

Skip it when queries are already short, single-intent, and vocabulary-aligned with the corpus. The planning layer adds latency and a new failure mode for no recall gain.

Skip it when the bottleneck is elsewhere. Bad chunking, weak embeddings, or a missing reranker will not be fixed by rewriting queries. Fix the retriever first.

Skip it when latency budget is tight and request volume is high. Each transformation is an LLM call on the critical path. At high volume, that is a real cost and a real tail-latency risk.

Prefer deterministic extraction over LLM planning when constraints are structured and already available from the application layer. If your app already knows the user's region and plan tier, do not ask a model to infer them from prose.

The decision rule I use: add a transformation only after you can name the failure class it fixes and the metric it should move. If you cannot name both, you are adding complexity, not capability.

The Next Move

Take five real failing requests from your own logs. For each one, hand-write the query plan it should have produced — intent, entities, constraints, subquestions, filters. Then run your current pipeline and compare what it retrieved against what the hand-written plan implies it should have retrieved.

The gap tells you which transformation to build first. If entities are missing, you need analysis. If compound questions collapse, you need decomposition. If vocabulary mismatches, you need rewriting. If constraints leak, you need filters.

And keep those five hand-written plans. They are the seed of your evaluation set — the first rows of the labeled data that will tell you, six months from now, whether the query layer you built actually earned its latency.