Prompt Ensembling and Candidate Selection: Self-Consistency, Best-of-N, and Consensus

A single greedy decode is right most of the time and wrong with total confidence the rest. The instinct is to sample more and let the answers vote. But sampling only builds a candidate pool. The selector is what turns that pool into quality, and a selector that cannot tell a good candidate from a bad one makes N samples worse than one.

I want to frame this as a cost-per-correct-answer problem, not an accuracy problem. Anyone can raise accuracy by spending more inference. The engineering question is whether the extra spend buys a better answer than the same budget spent on a stronger prompt, a bigger model, or a deterministic checker.

Sampling Buys Diversity, Selection Buys Quality

A multi-sample pipeline is two independent systems bolted together:

• A generator that produces a candidate pool.
• A selector that collapses the pool to one answer.

Most failures I see in production get misdiagnosed as model failures. They are selector failures. The model produced the right answer somewhere in the pool, and the selector threw it away.

Diversity is a property of the generator, not the ensemble. Temperature, prompt variation, and choice shuffling create the spread. If every sample is a near-copy of the others, you paid N times for one answer.

Selection is a decision problem with its own error rate. Measure it separately from generation quality, or you will never know which half of the pipeline to fix.

The cost model is blunt:

total_cost = N × generation_cost + selection_cost

Quality gain must be expressed per unit of that total. A selector that costs as much as generation doubles your bill for a marginal win.

Three selector families cover almost everything:

Ranking and verification are not the same family, and the distinction is load-bearing. Ranking scores candidates against a proxy for quality — a judge's opinion, a reward model's preference, a logprob. Verification tests a property of the task itself: does the code compile, does the arithmetic recompute, does the output satisfy the schema. A proxy can rank a persuasive wrong answer first. A verifier can reject every candidate. That difference decides your architecture, so keep the two separate before comparing them.

The Decision Axis: What Kind of Answer Are You Selecting?

The right selector is determined by the answer's structure and whether a cheap correctness signal exists. Not by which method sounds strongest.

Discrete, canonicalizable answers — multiple choice, classification, a numeric result, a short extractive span — support voting because agreement is computable. You can define an equivalence class: two answers are the same if they normalize to the same token.

Open-ended answers — prose, code, plans, long reasoning — have no natural equivalence class. Voting degrades into fuzzy similarity, and fuzzy similarity is a coin flip dressed as a metric.

The decisive question: is there a signal that correlates with correctness but costs less than generating the answer? If yes, rank or verify. If no, vote — or accept that you cannot select at all.

Verification strength is bounded by the verifier's own accuracy. A weak verifier caps the whole pipeline, no matter how many samples you feed it.

Read the table as a decision procedure. Find your answer type, find the signal you actually have, and the selector follows. If the signal column is empty, stop — you are about to build a pipeline that cannot select.

Self-Consistency: Voting Over Reasoning Paths

Self-consistency prompting is the cheapest selector and the one most sensitive to answer canonicalization. The mechanism is simple: sample K reasoning paths, extract the final answer from each, take the majority.

The extraction step is where it breaks. If extraction is lossy or inconsistent — one path says "42", another says "forty-two", a third buries the number in prose — voting operates on noise. Canonicalize before you count.

Majority vote assumes the correct answer is the modal one. That holds when errors are diverse and the model is more often right than wrong. It fails when errors are correlated.

Correlated errors are the killer. A systematic bias in the prompt or the model produces the same wrong answer K times. Voting does not catch it. Voting amplifies confidence in it. You now have a wrong answer with a K-sample endorsement.

For multiple choice, choice shuffling is the standard diversity trick. Permute the option order per sample so position bias does not become fake consensus. Then select the answer least sensitive to the shuffle:

import random
from collections import Counter

def self_consistency(question, options, model, k=10):
    votes = Counter()
    for _ in range(k):
        shuffled = options[:]
        random.shuffle(shuffled)
        path = model.generate(question, shuffled)
        answer = extract_choice(path, shuffled)  # map back to original label
        votes[answer] += 1
    return votes.most_common(1)[0][0]

The extract_choice function is the load-bearing part. If it maps a shuffled label back to the wrong original, every vote is corrupted. Test it in isolation before you trust the ensemble.

Cost: K× generation, near-zero selection cost. This is the cheapest selector you can run, and it is the one most likely to fail silently on correlated errors.

Ranking: Best-of-N With a Proxy Scorer

Best-of-N replaces agreement with a scoring function. You need a scalar signal per candidate: a reward model, a learned verifier, a logprob-based score, or a rubric-graded judge. All of these are proxies — they estimate quality without proving it.

The selection ceiling is set by the scorer's ranking accuracy, not by N. Adding samples past the point where the scorer can discriminate yields diminishing or zero returns. If the scorer cannot tell candidate 3 from candidate 7, samples 4 through 7 are wasted money.

Logprob and self-reported confidence are cheap but poorly calibrated. Treat them as weak signals and validate before trusting them. A model that says "I'm 95% confident" is reporting a token distribution, not a probability of correctness.

Weighted aggregation can beat picking a single best candidate when no single scorer is reliable. Combine scores across prompt variants or models, then rank:

def best_of_n(prompt_variants, model, scorer, n_per_variant=4):
    candidates = []
    for prompt in prompt_variants:
        for _ in range(n_per_variant):
            candidates.append(model.generate(prompt))
    scored = [(scorer(c), c) for c in candidates]
    return max(scored, key=lambda x: x[0])[1]

The scorer is often an LLM, which means the selection pass is itself a generation. That cost frequently dominates. Budget for it explicitly, or you will discover it in the invoice.

Consensus and Verification: When Agreement Is Not Enough

Consensus across diverse prompts or models reduces correlated error because the failure modes differ. This is the mechanism behind prompt-blender and multi-judge aggregation: run the same task through distinct prompts or distinct models, then aggregate. Diversity of failure is the whole point. If two prompts fail the same way, you have one prompt with extra steps.

Verification is different from voting and from ranking. A checker re-derives or tests the answer rather than counting votes or scoring proxies, so it can catch a confident unanimous error. That is the property voting can never have.

Verification only works when the checker is cheaper or more reliable than the generator on the same task. The strong cases are code execution, unit tests, schema validation, and arithmetic checks. The weak cases are open-ended quality judgments, where the checker is just another model with its own biases.

Aggregation strategy matters. Averaging scores, majority vote over judgments, and max/min selection produce measurably different results on the same candidate pool. There is no universal winner. Pick the aggregation that matches your signal's calibration, and measure it.

I will flag an open question honestly: optimal panel composition and prompt weighting remain under-explored. Do not treat any single recipe as settled. The research on ensembling judgments shows no single best judge across all settings, and panel selection is explicitly named as future work.

A Worked Trace: Same Pool, Three Selectors

The taxonomy only earns its keep if the selectors actually diverge on the same candidates. Here is a small arithmetic task — "What is 17 × 24?" — with five sampled candidates and their judge scores:

Voting canonicalizes to 408 and takes the majority: 4 of 5. Selected answer: 408.

Ranking trusts the judge and picks the highest score: candidate 4, 412. The judge preferred the confident wrong answer. This is the proxy failure mode in miniature.

Verification recomputes 17 × 24 = 408 and rejects 412 outright. Selected answer: 408.

Now compute the two diagnostics on this pool. The correct answer is 408, so oracle accuracy is 1.0 — the pool contains it. Selected accuracy is 1.0 for voting and verification, 0.0 for ranking. The oracle-vs-selected gap is zero for two selectors and total for the third. That gap is the whole point: the generator did its job, and only the ranking selector threw the answer away.

The trace also exposes the fixed-pool invariant. All three selectors ran on the same five candidates. If ranking had received a different sample pool, the 0.0 would mix generator variance with selector quality and tell you nothing. Generate once, persist the pool, replay every selector over it.

Measuring Whether It Earns Its Cost

Build a labeled set and measure three numbers:

1. Single-sample accuracy — your baseline.
2. Oracle accuracy — the best possible candidate in the pool, conditional on that pool.
3. Selected accuracy — what your selector actually picks.

Oracle accuracy is an upper bound for the pool you generated, not a claim about the model's attainable accuracy. A different pool could have a different oracle. That is why you hold the pool fixed when comparing selectors.

The gap between oracle and selected is your selector's headroom. If it is large, fix the selector. If it is small, the selector is fine and the generator is the bottleneck.

If oracle accuracy barely exceeds single-sample accuracy, the generator is not producing diverse candidates. Fix diversity before touching the selector. More samples of the same answer do not help.

Sweep N and plot accuracy against total cost. The curve usually flattens well before the N you assumed you needed. Report cost per correct answer, not raw accuracy, so the comparison against a single larger model or a better prompt is honest.

Ablate the selector: swap voting for ranking on the same pool and measure the delta. This isolates selection quality from generation quality. If ranking beats voting by ten points on the same candidates, you learned something about your signal, not your model.

The oracle-vs-selected gap is the single most useful diagnostic in this pipeline. It tells you whether to spend on generation or selection. Measure it before you add a single sample.

When Not to Ensemble

Ensembling is not a default. Draw the boundary against nearby alternatives.

If a single prompt change or a better reasoning scaffold closes most of the gap, that is cheaper than N× inference. Fix the prompt first. This is where most teams should start, and most skip it.

If the task has a deterministic checker — compiler, test suite, schema — run the checker instead of voting. For tasks with a valid independent checker, verification is strictly more informative than agreement. Scope that claim to the condition: without a real checker, you are back to proxies.

If errors are correlated, more samples do not help. You need a different model, a different prompt, or an external tool. Voting on correlated errors is a confidence machine, not a correctness machine.

If latency budget is tight and the task is interactive, N× generation may be unacceptable regardless of quality gain. A correct answer that arrives after the user left is a wrong answer.

The Decision Rule

The final rule branches by selector, not by a single condition:

• Canonicalizable answers with diverse errors → vote. Canonicalize first, then count.
• Open-ended answers with a calibrated scorer → rank, and validate the scorer's ranking accuracy before trusting it.
• Independently testable answers → verify, and treat the verifier's own accuracy as the ceiling.
• No reliable signal, or conflicting signals → abstain, escalate, or fall back to a human. Do not force a selection you cannot justify.

Canonicalizability is necessary for voting, not for every useful ensemble. Ranking and verification stay on the table for open-ended and testable outputs respectively.

The next concrete action is not to add samples. It is to instrument one existing pipeline with the oracle-vs-selected measurement. Run your current task, collect a pool of candidates, persist them, and compute the best possible answer in that pool. If the oracle is not meaningfully better than your single decode, you have a generation problem, not a selection problem — and no amount of voting will fix it.

Once selection works as a one-shot choice, the adjacent problem is making it a live control loop: orchestrating retries, routing between models, and feeding verification results back into the next generation. That is where selection stops being a batch decision and becomes runtime orchestration.