Source Routing and Adaptive RAG: Choosing Where and How to Retrieve

Always-on retrieval is a fixed policy applied to a variable question distribution. It over-retrieves on questions the model already knew, and it under-serves the questions that needed a specific store. Source routing is the fix, but only if you treat it as a decision under uncertainty: a signal, a threshold, and a fallback.

I have watched a single always-on retriever do two opposite things in the same afternoon. It answered a long-tail factual question cleanly, then quietly corrupted a question the model already knew — because the top-k chunks were plausible, adjacent, and wrong. The retrieval call succeeded. The answer did not. That gap is the entire reason source routing exists.

The Always-Retrieve Default and Where It Breaks

A fixed retrieve-then-read pipeline is optimized for one thing: coverage. It assumes every question needs external evidence, so it pays the retrieval cost every time. That assumption holds when your corpus is the only source of truth and your questions are uniformly knowledge-intensive. It fails the moment your question distribution is mixed.

Three failure modes show up in practice, and they are distinct:

• Wrong-source retrieval. The router (or the absence of one) pulls correct-looking context from the wrong system. A SQL aggregation question answered with semantically similar prose chunks is not a retrieval failure — it is a source-selection failure.
• Unnecessary retrieval. The model could answer from parametric knowledge, but you inject chunks anyway. You pay latency and tokens, and you add interference: irrelevant context can pull a correct answer off course.
• Missing escalation. A weak first pass returns mediocre documents, and nothing in the pipeline decides to try a second source. The answer that existed in the graph or the warehouse never gets reached.

Two of these are not about retrieval quality at all. They are about the decision before retrieval. Keep two decisions separate in your head: source routing (which store) and retrieval-mode routing (whether to retrieve at all, how many hops, which retriever pattern). They consume different signals and they fail differently.

I assume you already have chunking, query rewriting, and advanced retrieval patterns in place. Routing is the stage that feeds them, not a replacement for them.

Routing is a decision under uncertainty. If it has no signal, no threshold, and no fallback, it is not a router — it is a coin flip with extra latency.

Define the Decision Before You Define the Router

Most teams reach for a classifier before they can state what the classifier decides. That is backwards. Write the routing contract first.

Enumerate your candidate sources with their real properties. A source is not interchangeable with another just because both return text.

Then name the decision axes explicitly. Does the question need private or proprietary data? Post-training-cutoff information? Structured aggregation? Multi-hop traversal? Or is it answerable from parametric knowledge? Each axis maps to a source property, not to a vibe.

Separate the routing decision from the answer decision. Routing picks a source and a budget. Generation consumes what routing returned. When you conflate them — letting the generator "decide" to retrieve mid-stream with no logged decision — you make both untestable. You cannot tell whether the router chose wrong or the generator ignored good context.

Define the cost model up front: retrieval calls, tokens injected, added latency, and the cost of a wrong route. A confidently wrong answer is more expensive than a slow one. That asymmetry should shape your thresholds, not your intuition.

Signals You Can Actually Route On

At decision time you have four families of signal. Each one can tell you something, and each one has a blind spot.

Lexical and structural signals. Numerals, dates, entity density, aggregation verbs ("total", "average", "trend"), comparison phrasing, and explicit references to internal systems. These are cheap, interpretable, and cacheable. They are also brittle on paraphrase: "how are we doing this quarter" carries no numeral.

Semantic signals. Embedding similarity to labeled routing examples, nearest-neighbor lookup over a policy datastore of past queries and their winning source, and query-type classification. This handles paraphrase and drift better than rules, but it needs labeled data and a refresh path.

Model-internal signals. Token likelihood for a source marker, self-reported uncertainty, hidden-state representations. Research on self-routing RAG shows these can drive source selection inside a single generation pass, with special tokens marking the end of the query and prompting a source choice. The catch: these signals are sensitive to fine-tuning and model version. A threshold tuned on one checkpoint does not transfer after a swap.

Retrieval-derived signals. The shape of the score distribution across a candidate set — concentration versus dispersion — is a useful proxy for whether a question is easy or diffuse. Concentrated scores suggest a small model or a single source can handle it; diffuse scores suggest escalation. But this signal requires a retrieval call, so it cannot gate the first retrieval. It gates the second.

No single signal is reliable alone. The practical design is a cheap gate that resolves the common case, with a stronger signal invoked only when the gate is ambiguous.

Three Routing Architectures and Their Tradeoffs

Compare architectures on a shared axis before you pick one: interpretability, data requirement, latency, adaptability to drift, and blast radius when the router is wrong.

Rule-based routing. Hand-written or LLM-refined rules over extracted features. Interpretable, low latency, easy to debug, easy to cache. It degrades as the question distribution drifts and the rule set multiplies. Do not build a learned router for two sources with a clean lexical split — rules win there.

Classifier or policy-datastore routing. Collect labeled query-to-source outcomes, then classify or retrieve nearest neighbors at inference. This handles paraphrase and drift better and gives you a distribution over sources rather than a single guess. It needs labeled data, a refresh path, and monitoring for label decay. Do not use pure rules when your questions are open-ended and paraphrased.

Model-integrated routing. The model emits a source decision inside the generation pass, optionally augmented with a learned policy signal. Fewer moving parts, lower latency. Harder to inspect, tied to a specific model version, and threshold behavior can shift after fine-tuning. The research direction here is real, but the operational cost is version coupling.

My default for a first build is rule-based with a policy-datastore fallback. Rules get you a working, debuggable loop today. The datastore earns its place once you have logged enough decisions to know where the rules break.

The Evidence Contract: What Makes an Answer Allowed

Before you write escalation logic, write the invariant that governs it. This is the part most teams skip, and it is the part that determines whether your router is safe.

Relevance is not sufficiency. A document can score 0.91 against the query and still fail to contain the requested field, support every subquestion, or come from an authoritative source. If your only condition is max(grade) < threshold, you have built a relevance filter, not an evidence policy.

State the contract explicitly:

Answer only when every required claim is covered by eligible evidence with acceptable authority and freshness. Escalate when coverage is incomplete, source authority is insufficient, or retrieval quality is low. Refuse when the escalation budget is exhausted.

That sentence decomposes into four checks you can implement and log:

A relevant-but-unauthorized document is the dangerous case. It reads well, it grades high, and it should not be allowed to answer. Coverage is the second trap: three relevant chunks that each answer a different subquestion still leave the fourth subquestion unsupported.

Escalation as Action Selection, Not a Fixed Ladder

The instinct is to order your sources and climb: parametric, then vector, then SQL, then web, then human. That ladder is one policy, not the architecture. It breaks the moment a failed internal query should not go to the web — because the missing evidence is a private metric, and no external source can supply it.

Decompose the decision instead. At each step, the system holds a set of missing conditions and selects the next admissible action that can satisfy them.

state = {
  required_claims: [...],       # subquestions or fields the answer must cover
  covered: {...},               # claim -> evidence item that satisfies it
  missing: [...],               # claims with no eligible evidence yet
  attempts: 0,
  budget: 3,
  route_reason: "...",
}

The transition rule is: pick the cheapest admissible action whose source can plausibly satisfy a missing claim. If no admissible action remains and claims are still missing, refuse. That is the whole policy.

Two transitions matter more than the rest, and they are different:

• Poor relevance — the retrieved set does not address the query. Remedy: rewrite the query or change the retrieval mode against the same source.
• Incomplete coverage — the retrieved set addresses the query but misses a claim. Remedy: retrieve from a different source that holds that claim.

Collapsing these into one threshold is the most common design error I see. They demand different next actions, and a single max(score) < 0.5 check cannot tell them apart.

Order actions cheap-first, and bound the loop. Unbounded escalation is a latency and cost bug wearing the costume of thoroughness.

A Minimal Implementation You Can Run

Build the narrow version first: two sources, one cheap gate, two distinct escalation conditions, and a logged decision record. Resist a third source until the first two are measured.

from dataclasses import dataclass, field

@dataclass
class RouteState:
    query: str
    required_claims: list[str]
    covered: dict = field(default_factory=dict)
    attempts: int = 0
    budget: int = 3
    route_reason: str = ""
    terminal: str = ""          # "answer" | "refuse"

def gate(query):
    signals = extract_signals(query)          # numerals, entities, aggregation verbs
    score = gate_score(signals)               # cheap lexical/structural gate
    if score >= 0.6:
        source = "sql" if signals.has_aggregation else "vector"
    else:
        source = "vector"                     # highest-coverage default
    return source, score, signals

def coverage(state, evidence):
    # evidence: list of {claim, grade, authority_ok, fresh_ok}
    for item in evidence:
        if item["grade"] >= 0.5 and item["authority_ok"] and item["fresh_ok"]:
            state.covered[item["claim"]] = item
    state.required_claims = [c for c in state.required_claims
                             if c not in state.covered]

def next_action(state, evidence):
    if not evidence or max(e["grade"] for e in evidence) < 0.5:
        state.route_reason = "poor_relevance"
        return "rewrite_or_switch_mode"       # same source, better query
    coverage(state, evidence)
    if state.required_claims:
        state.route_reason = "incomplete_coverage"
        return "retrieve_other_source"        # different source for missing claim
    state.terminal = "answer"
    return "generate"

def run(query, required_claims, sources):
    state = RouteState(query=query, required_claims=list(required_claims))
    source, score, signals = gate(query)
    while state.attempts < state.budget and not state.terminal:
        state.attempts += 1
        evidence = sources[source].retrieve(query, state.required_claims)
        action = next_action(state, evidence)
        if action == "generate":
            break
        if action == "retrieve_other_source":
            source = pick_source_for(state.required_claims, exclude=source)
        # rewrite_or_switch_mode keeps the source and re-queries
    if not state.terminal:
        state.terminal = "refuse"             # budget exhausted
    return state

The decision record is the core artifact. Log the query, chosen source, signal values, gate score, threshold, retrieved doc ids, per-claim coverage, authority and freshness flags, route_reason, attempt count, and terminal outcome. That record is what makes routing debuggable instead of mystical.

Two Traces That Expose the Mechanism

A mixed-source query that needs decomposition, not a source swap:

query="how did Q3 revenue compare to our forecast, and what did leadership say about it?"
required_claims=["q3_revenue", "forecast_value", "leadership_commentary"]
attempt=1 source="sql" -> evidence=[{claim:"q3_revenue", grade:0.94, authority_ok:True, fresh_ok:True},
                                    {claim:"forecast_value", grade:0.88, authority_ok:True, fresh_ok:True}]
coverage -> covered={q3_revenue, forecast_value}; missing=[leadership_commentary]
route_reason="incomplete_coverage" -> action="retrieve_other_source"
attempt=2 source="vector" -> evidence=[{claim:"leadership_commentary", grade:0.79, authority_ok:True, fresh_ok:True}]
coverage -> missing=[] -> terminal="answer"

Note what happened: the first pass was good. High grades, correct source. It still escalated, because coverage was incomplete. A relevance-only router would have answered here and dropped the commentary.

A false-positive trace where relevance lies:

query="what is our current enterprise pricing?"
required_claims=["current_enterprise_price"]
attempt=1 source="vector" -> evidence=[{claim:"current_enterprise_price", grade:0.91,
                                        authority_ok:False, fresh_ok:False}]
route_reason="incomplete_coverage"   # high grade, but ineligible evidence
action="retrieve_other_source" -> source="sql" (pricing table is authoritative)
attempt=2 source="sql" -> evidence=[{claim:"current_enterprise_price", grade:0.97,
                                     authority_ok:True, fresh_ok:True}]
terminal="answer"

The 0.91 document was a marketing page from two years ago. Relevant, high-scoring, and disqualified by authority and freshness. This is the case your threshold-only router ships to production.

Orchestration libraries give you the graph and the state plumbing. The routing policy, the evidence contract, and the thresholds are yours to define and own.

Evaluating the Router, Not Just the Answer

End-to-end accuracy hides routing quality. Measure them separately: did the router select a source that contained eligible evidence for every required claim, independent of whether generation used it well?

Build a labeled routing set keyed on evidence items, not a single oracle source. For each query, record which claims must be covered and which sources are eligible to cover each one. A single oracle source is too coarse for multi-source and decomposed questions.

Track these metrics:

• Routing precision — fraction of required claims covered by eligible evidence on the chosen path.
• Unnecessary-retrieval rate — how often you retrieved when parametric knowledge sufficed.
• Missed-escalation rate — how often incomplete coverage never triggered a second action.
• Unsupported-claim rate — how often the final answer asserts a claim with no eligible evidence behind it.
• Budget-exhaustion rate — how often the loop hit its cap and refused.

Track cost and latency per route, not just in aggregate. A router that improves accuracy by always escalating to the expensive path has not solved the problem — it has renamed it.

Watch for the silent failure: a router that is right on the benchmark and wrong on the long tail, because the labeled set over-represents the easy distribution. And define the regression check you run after any model swap, prompt change, or index rebuild. Routing signals are coupled to all three.

Failure Modes and When Routing Is Overkill

Design for these failure paths before they surprise you:

• Router confidence collapse. Everything routes to the fallback. Usually a threshold drift or a signal that stopped discriminating.
• Threshold drift after a model upgrade. The old threshold no longer means what it meant.
• Stale policy datastore. Nearest-neighbor routing returns decisions from a distribution you no longer serve.
• Source outage with no degraded path. The primary store is down and the pipeline has no second rung.
• Feedback loops. The router learns from its own unverified outputs and reinforces its mistakes.

Recovery patterns: default to the highest-coverage source on ambiguity, cap escalation depth, cache routing decisions for semantically similar queries, and keep a manual override path.

And be honest about when routing is overkill. One source, a narrow and stable question distribution, or a corpus small enough to fit in context — in those cases a fixed retrieve-then-read pipeline is the correct engineering choice, not a compromise. Routing is mandatory when you have heterogeneous sources with different freshness and authority, mixed structured and unstructured questions, or retrieval cost and latency as first-class constraints.

Route only when the expected accuracy or cost gain exceeds the added failure surface. Measure both before and after.

The Next Move

Pick two sources. Write down the required claims, the evidence contract, and the two escalation conditions — poor relevance versus incomplete coverage. Log every routing decision with its signal values, coverage result, and route_reason. Run a labeled routing set keyed on evidence items before you add a third source or reach for a learned router.

The rule I keep coming back to: routing earns its complexity only when measured accuracy or cost gains exceed the added failure surface. Start with the narrow loop, watch the decision records, and let the evidence tell you when the router has earned the right to grow.