Embeddings, Vector Indexes, and Hybrid Search: Choosing a Retrieval Substrate

A retrieval system fails in a specific, repeatable way. You ship a demo that answers paraphrased questions beautifully, then a user pastes an error code, a part number, or a rare surname, and the system returns confident nonsense.

The tempting diagnosis is "the embedding model is bad." Usually it is not. The substrate is wrong — or the substrate is fine and something upstream of it is broken. Distinguishing those cases is the whole job, and it is the reason this article treats retrieval as a layered decision rather than a single choice between "vector" and "keyword."

Three Layers, Not One Choice

Most retrieval debates collapse three distinct decisions into one. Separate them and the arguments get shorter.

query
  │
  ├─ 1. classification / rewriting        (what is being asked?)
  │
  ├─ 2. hard filters                      (what is even eligible?)
  │
  ├─ 3. candidate generation              (lexical | dense | hybrid)   ← substrate choice
  │
  ├─ 4. fusion                            (RRF or score blending)
  │
  └─ 5. reranking / compression           (reorder the survivors)

The substrate comparison — the subject of this article — owns layer 3. Filters own layer 2. Reranking owns layer 5. Chunking, query rewriting, and multi-query expansion are neighboring topics; they change what gets retrieved, not what the scorer can see. When a retrieval failure shows up, you have to know which layer produced it before you change anything.

A dense miss on an identifier and a lexical miss on a paraphrase look identical in an aggregate metric. They have opposite fixes. The layer decomposition is what makes the miss diagnosable.

Two Scoring Models, One Query

Dense and lexical retrieval disagree because they compute different things, not because one is newer.

Dense retrieval uses a bi-encoder: a model maps the query and each document into the same vector space independently. Scoring is a distance or dot product between two vectors computed in isolation. The consequence is architectural: document vectors are precomputable at index time, so retrieval collapses into a nearest-neighbor lookup. That is what makes vector search fast, and it is also what makes it lossy.

Lexical retrieval analyzes text into terms, builds an inverted index mapping each term to its postings list, and scores overlap with BM25. BM25 adds term-frequency saturation and document-length normalization, so a term appearing fifty times does not score fifty times a term appearing once. The consequence is the mirror image: exact tokens are first-class evidence, and there is no notion of paraphrase at all.

Why they disagree is worth stating precisely. A dense vector compresses a passage into a fixed-width representation. Rare identifiers, negation, and numeric constraints get smoothed into the same neighborhood as everything semantically adjacent. Ask for "error 0x80070005" and the model may return passages about Windows update failures generally, because that is what the training distribution associates with the phrase. The literal token is not privileged. In BM25, it is the only thing that matters.

The bi-encoder constraint is the root cause: query and document never see each other during scoring. That independence is what makes precomputation possible, and it is exactly what a cross-encoder gives up. A cross-encoder scores the pair jointly and can capture term-level interaction, but it cannot precompute document representations, so it cannot serve as a first-stage retriever over a large corpus.

The Decision Axis: Evidence Type, Not Topic

The common mistake is choosing a substrate by corpus topic. "We're a legal tech company, so we need semantic search." That framing hides the actual variable. The substrate must match the query's evidence type, not the corpus's subject matter.

Four axes determine the choice.

Evidence type. Does the answer live in a concept, where paraphrase tolerance helps, or in a literal token — an identifier, error code, name, quoted phrase, or numeric threshold? Concept queries reward dense retrieval. Literal-token queries reward lexical retrieval. Most real systems have both.

Query distribution. What fraction of production traffic is short, keyword-like, or contains out-of-vocabulary strings the embedding model never saw during training? A model trained on web text has weak representations for your internal SKU format. That is not a tuning problem; it is a coverage gap.

Corpus shape. Homogeneous prose behaves differently from a mix of prose, tables, logs, and structured records. When exact fields matter — a version string, a status enum, a date range — lexical and filter-based retrieval carry evidence that dense vectors blur.

Operational budget. Index build time, memory footprint, re-embedding cost when the model changes, and the added query latency of a second retrieval path. These are not afterthoughts. They are the reason hybrid sometimes loses even when it retrieves better.

A Worked Trace

The axes are abstract until you run a query through them. Take a fictional internal knowledge base: API docs, incident postmortems, and a support ticket archive. Three queries arrive.

Query A — "how do I rotate a service account key?" Paraphrase query. The user does not know the exact doc title. Dense retrieval carries the decisive evidence: the relevant passage may say "credential renewal" and never use the word "rotate." Lexical retrieval returns near-misses that share tokens but not meaning. Substrate: dense.

Query B — "ERR_TLS_CERT_ALTNAME_INVALID" Identifier query. The token is rare, exact, and out-of-distribution for a general embedding model. Dense retrieval returns generic TLS troubleshooting; the literal string is smoothed away. Lexical retrieval returns the exact postmortem. Substrate: lexical.

Query C — "postmortems for the payments service in Q3" Constraint query. The decisive evidence is not in the text at all — it is in metadata: service name and date range. Neither scorer should carry this. Layer 2 does: filter to service = payments AND date ∈ Q3, then let either substrate rank the survivors. Treating this as a "hybrid" problem is a category error; the constraint belongs in a filter, not in a similarity score.

That third case is the one the decision table below has to make explicit, because it is where most teams reach for the wrong tool.

The Decision Frame

Query class	Decisive layer	Defensible substrate
Paraphrase over prose	Candidate generation	Dense-only
Identifier / exact token	Candidate generation	Lexical-only
Hard constraint (field, date, enum)	Hard filter (layer 2)	Filter + either substrate
Mixed paraphrase + identifier traffic	Candidate generation	Hybrid
Latency-critical, single path	Candidate generation	One substrate + query analysis

Dense-only is defensible when queries are conversational and the corpus is prose. Lexical-only is defensible when the domain is identifier-heavy and the vocabulary is stable. Hybrid earns its cost when both paraphrase and identifier queries appear in the same traffic — and only then. Constraint queries are not a third substrate; they are a filtering problem that sits in front of whichever substrate you pick.

Index Structures and What They Cost You

Picking a vector database usually means picking an index structure you did not explicitly choose. That structure determines recall, latency, and how painful your next model migration will be.

Exact versus approximate. Brute-force k-NN is the correctness baseline: compare the query against every vector, return the true top-k. It is slow but exact, and it is the only honest way to measure what your approximate index loses. Graph-based indexes such as HNSW build a navigable hierarchy over the vectors and trade exactness for speed. Inverted-file and quantization approaches trade memory for recall by compressing or partitioning the vector space.

The recall dial is a parameter, not a property. HNSW exposes efSearch; IVF-style indexes expose nprobe. These knobs move you along a recall/latency curve. A default value is not a guarantee. Measure recall against brute force on a labeled query set, then pick the operating point your latency budget allows.

Filtered search is the trap that bites production systems. Pre-filtering restricts the candidate set before graph traversal, which can starve the traversal and return poor neighbors. Post-filtering retrieves top-k first and discards non-matching results, which can return fewer than k. Both behaviors depend on filter selectivity. Test the selectivity your product actually produces — a filter that matches 80% of the corpus behaves nothing like one that matches 2%. This is the layer-2 decision from the trace above, and it has its own failure modes independent of the substrate.

Lexical index cost is different in kind. Inverted indexes are cheap to build and update. They do not shrink with better models, and their maintenance surface is vocabulary drift and stemming choices. That is a different operational profile from a vector index, not a worse one.

Embedding model changes are index migrations. A new model means re-embedding the entire corpus. Mixed-model indexes degrade silently, because vectors from different embedding spaces are not comparable — cosine similarity between them is meaningless. Treat the model as part of the index schema.

Fusing Two Ranked Lists Without Fooling Yourself

Hybrid retrieval is not a checkbox. It is a ranking-combination problem with its own failure modes, and the naive implementation is fragile.

Score fusion is the fragile path. Cosine similarity and BM25 live on different, non-comparable scales. Weighted score addition requires normalization that is easy to get wrong and hard to keep stable across queries — a normalization constant tuned on one query distribution drifts on another.

Rank fusion is the more portable default. Reciprocal Rank Fusion combines lists using rank position rather than raw scores:

def rrf(rank_lists, k=60):
    scores = {}
    for ranks in rank_lists:
        for position, doc_id in enumerate(ranks, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + position)
    return sorted(scores, key=scores.get, reverse=True)

RRF sidesteps scale mismatch entirely and needs only a small constant. It is a heuristic, not a learned optimum — but a heuristic that survives query-distribution drift is worth more than a tuned score blend that does not.

The semantic ratio is a product decision. A weight like semanticRatio controls how much lexical evidence can override semantic similarity. The right value depends on your query mix, not on a benchmark. If 30% of your traffic is identifier queries, that number should move the ratio, not a leaderboard.

Reranking is a separate stage with a separate budget. A cross-encoder over the fused top-k can fix ordering that fusion got wrong, at a latency cost proportional to the candidate set. Keep it distinct from fusion in your architecture and your accounting.

The failure mode to watch: hybrid can improve your average while making a specific high-value query class worse. Aggregate metrics hide exactly the regression the substrate choice was supposed to prevent. Slice evaluation by query type before declaring victory.

Evaluating the Substrate Before You Commit

The experiment that produces a decision is small and specific. Build it before you argue about architecture.

Construct a stratified labeled set from real traffic. A few hundred queries with known relevant documents, deliberately split into paraphrase queries, identifier queries, and constraint queries. The stratification is the point — it is what makes the per-stratum result meaningful.

Measure recall@k per stratum for lexical-only, dense-only, and hybrid. Aggregate numbers hide the failure. A hybrid system that scores 0.82 overall while scoring 0.61 on identifier queries has told you something the aggregate buried.

Measure the operational side on the same run: index build time, index size, p95 query latency, and the cost of a full re-embed. Retrieval quality without these numbers is half a decision.

Inspect the misses, not just the metrics. This is where the layer decomposition earns its keep. A dense miss on an identifier points to candidate generation. A dense miss on a paraphrase that should have worked points to representation — chunking, embedding model coverage, or query formulation. A miss on a constraint query points to filtering, not to the substrate at all. The metric tells you that it failed; the miss tells you which layer failed.

Define the kill criterion in advance. If hybrid does not improve the stratum you care about by a margin that justifies its latency and complexity, ship the single substrate and revisit when the query mix changes. Writing this down before the experiment prevents the result from being rationalized after it.

When Not to Reach for Hybrid

The most expensive mistake is defaulting to the most complex option. Hybrid has a real overkill boundary.

Small corpora. If the whole corpus fits in the context budget, or brute-force search returns in acceptable time, index sophistication is premature. Brute force buys decisive simplicity when the corpus is small enough.

Stable, narrow vocabulary. A domain with controlled terminology and identifier-heavy queries may be better served by lexical search plus good field filters than by an embedding pipeline. Adding embeddings to a system that does not need paraphrase tolerance adds cost without adding recall.

Pure semantic workloads. Conversational search over prose with no literal-token queries rarely justifies a second retrieval path. The second path exists to catch what the first misses; if nothing is missed, it is overhead.

Latency-critical paths. Two retrieval calls plus fusion plus reranking is a real budget. If the product cannot absorb it, pick one substrate and invest in query analysis instead.

The adjacent move. Before adding a substrate, check whether query analysis, filters, or chunking changes are the actual bottleneck. Those are cheaper and often decisive. A query that gets rewritten to expose its entities and constraints may not need a second retriever at all.

The Decision Rule

Classify your real queries into paraphrase, identifier, and constraint types. Route constraints to filters, not to a similarity scorer. Run the stratified recall comparison against a brute-force baseline. Let the per-stratum result — not the aggregate score — choose the substrate.

The next concrete action is to build the labeled query set first. Without it, every substrate argument is taste. With it, the choice becomes an observation: this stratum needs lexical evidence, that one needs semantic, the constraint stratum needs a filter, and the overlap tells you whether hybrid has earned its latency.