Prompt Evaluation: Test Cases, Graders, Variance, and Regression Testing

A prompt is not a sentence you tune. It is a versioned artifact whose behavior is a distribution, and you cannot manage a distribution by reading three outputs.

Here is the failure that motivates everything below. An engineer notices the model mishandles a refund question. They add a clarifying line to the system prompt, paste the refund question into a playground, and the answer comes back clean. They ship. Two weeks later, support tickets show that a different slice of traffic — order-status questions with a missing order ID — now returns confident, wrong answers. The prompt change did not break the refund case. It broke a case nobody re-ran.

That is not a testing discipline problem. It is a model-of-the-world problem. The engineer believed a prompt is validated by reading outputs. The stronger model is that a prompt defines a distribution over inputs and samples, and shipping a change means moving that distribution. The smallest harness that makes the distribution visible is the subject of this article.

I assume you already have a prompt contract: objectives, inputs, constraints, outputs, and acceptance criteria. This turns those acceptance criteria into executable checks.

Why Reading Outputs Is Not Evaluation

Three traps account for most prompt regressions I have seen.

Test once. You run one input, the output looks right, you ship. This fails the moment a user supplies an input shape you never imagined. The single sample told you the prompt can work, not that it does work.

Test a few times. You run a handful of inputs, patch a corner case or two, and ship. Better, but you are still sampling from a distribution by hand and calling the sample the distribution.

Trust the demo. The prompt worked in the notebook with curated inputs. Production traffic is not curated.

All three traps share a root cause: they conflate three distinct quantities that deserve separate names.

• Task success — did the system actually do the job on this input? This is the thing you care about.
• Grader agreement — does your automated judge agree with a human about whether the job was done? This is a property of your measurement instrument, not your prompt.
• Sampling variance — does the same input, run again, produce a different score? This is a property of the model and decoding, not your prompt's correctness.

A pass rate of 92% means nothing until you know how reliable the grader is and how wide the per-case spread is. A judge that passes everything will report 92% on a system that is 60% correct. A case that scores 1.0, 0.0, 1.0, 0.0, 1.0 has a mean of 0.6 and is not a 60%-correct case — it is a coin flip, and it will surprise you in production.

The operating loop that ties these together is the evaluation flywheel: analyze failures qualitatively, measure them with graders, improve the prompt, then re-measure. The flywheel only spins if each phase produces something the next can consume. Qualitative analysis without measurement stays anecdotal. Measurement without analysis grades the wrong thing.

Invariant: every claim about a prompt must be reproducible from a stored test set, a stored prompt version, and a stored grader version. If you cannot reconstruct the run, you cannot attribute the regression.

Anatomy of a Test Case

A test case is the unit your harness runs and diffs. Keep it small and precise:

• Input — the prompt variables plus any grounding data the model needs.
• Expected behavior — a statement of what success looks like, specific to one scenario or intent.
• Assertions — one or more checks that operationalize the expected behavior.

Three properties make a case worth keeping: it is independent (runs without other tests), repeatable (produces a consistent pass or fail), and specific (tests one scenario or intent).

Grounding data is what makes assertions checkable. Consider a support agent:

Prompt: "What's my PTO balance?"
Assertion: "The response contains the correct balance"

That assertion is not a test. It is a wish, because nothing in the case defines what "correct" is. Add grounding data — the employee's actual balance — and the assertion becomes verifiable:

Prompt: "What's my PTO balance?"
Grounding: { "employee_id": "E-4471", "pto_balance_days": 12 }
Assertion: response contains "12"

Now the grader has something to check against. Without grounding data, "contains the correct value" is unfalsifiable, and unfalsifiable assertions pass by accident.

Coverage grows in tiers. Start with a small core set of the scenarios that matter most. Add variations and edge cases before production. Broaden once you are live. The exact counts depend on your system; the shape does not — core first, edges second, breadth third.

Two categories earn their place early because they regress silently:

• Negative cases — the model must decline, route, or refuse. "What's my colleague's salary?" should decline and leak nothing. "I need FMLA leave" should route to HR rather than attempt an eligibility answer.
• Boundary cases — inputs at the edge of the contract, where a clarifying line in the prompt is most likely to push behavior past the boundary.

The practical rule that keeps a suite honest: every production incident becomes a permanent test case. Your suite should grow from real failures, not from imagination. Imagined cases test what you already thought of; incident cases test what actually broke.

Choosing Graders: Deterministic First, Model-Judged Second

A grader decides whether an assertion passes. Choose the cheapest, most deterministic grader that can express the requirement. Work up this ladder only when the rung below cannot do the job.

Each rung down trades determinism for coverage. A judge that is easy to write is also easy to fool. When a deterministic grader can express the requirement, prefer it: it is cheaper, faster, and carries no variance of its own. Reserve the model judge for the requirements that genuinely resist a rule — tone, helpfulness, whether an open-ended answer is actually correct.

Write assertions against observable output structure, not against the model's internal reasoning. You cannot grade a chain of thought reliably, and you should not try. Grade the artifact the system produces.

The boundary to respect: a grader measures the assertion you wrote, not the requirement you meant. Ambiguous acceptance criteria produce confident, useless scores. If you cannot state the requirement as an assertion, the problem is the contract, not the grader.

Aligning an LLM Judge Before You Trust It

A model judge is a measurement instrument, and an uncalibrated instrument produces numbers that look like data. Calibrate it before you let it gate anything.

Split your labeled data three ways:

• Train (~20%) — a handful of clear pass/fail cases embedded as few-shot examples in the judge prompt.
• Validation (~40%) — where you iteratively tune the judge's instructions.
• Test (~40%) — a held-out set you run once, to confirm you did not overfit the judge to the validation set.

The held-out set is the report card. If you tune against it, it stops being a report card and becomes another validation set.

Track true positive rate and true negative rate against human labels, not raw agreement. Raw agreement is a trap: a judge that passes everything looks accurate on a mostly-passing set. TPR and TNR separate the two ways a judge can be wrong — failing good outputs and passing bad ones — and they have different costs depending on your application.

One subtlety: TPR and TNR require a binary decision, but a judge may emit a scalar or rubric score. You have to define the threshold that converts the score into pass or fail before you can compute either rate. That threshold is a product decision, not a formatting detail — it encodes how much you fear false passes versus false failures.

Rubric-based scoring with criterion-specific rationales makes judge errors diagnosable. A scalar score tells you a case failed; a rationale tells you why, which is what you need to fix either the judge or the prompt. A judge that emits a score and a one-line reason per criterion is worth more than one that emits a bare number.

Report judge reliability next to task results. A 92% pass rate judged by an instrument that is 80% reliable is not a 92%-correct system. State both numbers or the pass rate is misleading.

Known failure modes to watch for:

• Verbosity bias — longer answers score higher regardless of quality.
• Position bias — in pairwise comparisons, the first or second option wins more often than it should.
• Version drift — when the judge model version changes, your scores move even though your prompt did not. Pin the judge model version and treat a judge upgrade as a change that must be re-validated.

Measuring Variance Instead of Averaging It Away

A mean hides a distribution. Run each case multiple times and report the per-case score spread, not just the average. A case that scores 1.0, 0.0, 1.0, 0.0, 1.0 averages 0.6 and is bimodal — it will pass or fail unpredictably in production, and the average tells you nothing about that.

Three kinds of variation deserve separate names because they come from different experimental factors and have different fixes:

• Input variance — different test cases score differently. This is signal, not noise. It tells you which inputs the prompt handles well.
• Sampling variance — the same case, run repeatedly under a fixed prompt, scores differently. The controlled factor is the model's stochastic output. Fix with temperature, seed control, or by accepting the spread and budgeting for it.
• Prompt-variant sensitivity — meaning-preserving rewrites of the prompt produce different scores. The controlled factor is the prompt artifact itself, not the sampling. Fix by tightening the contract.

That last distinction matters because it changes what you do next. Sampling variance is a property of the model and decoding; you reduce it by changing how you sample. Prompt-variant sensitivity is a property of the prompt; you reduce it by making the instruction less dependent on one exact phrasing. If you lump them together under one "variance" number, you cannot tell whether to adjust temperature or rewrite the prompt.

Prompt-variant sensitivity is the one teams skip and later regret. Re-run semantically equivalent prompt variants and measure two things: the average variance of scores across the paraphrase set, and a robustness rate — the share of cases whose scores stay within a bounded spread across all paraphrases. A prompt that only works with one exact phrasing is fragile, and paraphrase testing is how you find out before a teammate "cleans up the wording" and ships a regression.

Set a variance budget per case type. Some cases need to be stable — a structured extraction that feeds a downstream system must not flip between runs. Others can tolerate spread — a brainstorming assistant can vary. High-variance cases are usually under-specified contracts, not bad luck. When a case will not stabilize, look at the acceptance criteria before you blame the model.

The cost is real: multi-sample evaluation multiplies token spend. Reserve repetition for cases where the decision actually depends on stability. You do not need five samples of a case whose output is a fixed enum.

Building the Smallest Useful Harness

Build the mechanism before you reach for a framework. The minimum viable harness is five pieces:

1. A versioned prompt file — the prompt is an artifact with a version identifier.
2. A JSONL test set — one case per line, with inputs, grounding data, and expected behavior.
3. A runner — executes N samples per case against a given prompt version.
4. A grader function per assertion — takes (output, expected) and returns a score plus a reason.
5. A results table — keyed by case and prompt version.

Keep the runner dumb and the graders pure. A grader that takes an output and an expectation and returns a score plus a reason is testable on its own, without a model call. That purity is what lets you trust the grader.

Here is the smallest end-to-end slice that makes the mechanism concrete. A case file:

{"case_id": "PTO-001", "input": "What's my PTO balance?", "grounding": {"pto_balance_days": 12}, "expected": "contains 12"}
{"case_id": "PRIV-001", "input": "What's my colleague's salary?", "grounding": {}, "expected": "declines, leaks nothing"}
{"case_id": "ESC-001", "input": "I need FMLA leave", "grounding": {}, "expected": "routes to HR"}

A runner that samples each case and aggregates:

def run_case(case, prompt_version, n=5):
    scores = []
    for i in range(n):
        out = call_model(prompt_version, case["input"], case["grounding"])
        score, reason = grade(out, case["expected"])
        scores.append({"sample_index": i, "score": score, "reason": reason})
    values = [s["score"] for s in scores]
    return {
        "case_id": case["case_id"],
        "prompt_version": prompt_version,
        "scores": values,
        "mean": sum(values) / len(values),
        "spread": max(values) - min(values),
    }

Run that against a baseline and a candidate, then diff at the case level:

case_id    baseline_mean  candidate_mean  baseline_spread  candidate_spread  verdict
PTO-001    1.00           1.00            0.00             0.00              stable pass
PRIV-001   1.00           0.40            0.00             0.80              HARD REGRESSION
ESC-001    0.60           0.80            0.80             0.40              improved, still unstable
aggregate  0.87           0.73            —                —                 net regression

Read that table the way you would read a failing test suite. PRIV-001 is a hard failure — a privacy case that used to pass now passes less than half the time, and its spread of 0.80 means it is a coin flip. ESC-001 improved but is still unstable; it needs a tighter contract before you trust it. The aggregate moved down, but the aggregate is not the decision — the case-level verdicts are.

Store the full run context with every result:

{
  "run_id": "...",
  "prompt_version": "v7",
  "model": "provider/model-id",
  "model_params": { "temperature": 0.2, "max_tokens": 1024 },
  "grader_version": "g3",
  "timestamp": "...",
  "case_id": "PTO-001",
  "sample_index": 0,
  "score": 1.0,
  "reason": "..."
}

Without prompt version, model identifier, model parameters, and grader version, a regression is unattributable. You will see the score move and have no idea which of four variables moved it.

Add a baseline snapshot so every new run diffs against a known-good reference rather than against memory. The baseline is the run you decided was acceptable; the diff is the evidence for the next change.

Reach for a framework once you need multi-provider comparison, caching, concurrency, or CI integration — not before. The abstraction hides the mechanism you are still learning, and the mechanism is the part that transfers when the framework changes.

Regression Testing Across Model and Prompt Changes

Treat prompt, model version, and decoding parameters as one versioned unit. A model upgrade is a change that must pass the same suite as a prompt edit. This is the part teams forget: the model provider ships a new version, behavior shifts, and nobody re-runs the suite because "we didn't change anything."

Diff at the case level, not just the aggregate. A flat pass rate can hide one fixed case and one newly broken case canceling out. The aggregate says "no change"; the case-level diff says "you traded a bug for a bug." Only the second is actionable.

Classify failures by severity:

• Hard failures — wrong value, leaked data, broken schema, missed refusal. These block release.
• Soft failures — tone, verbosity, style drift. These get triaged, not blocked.

Wire the suite into CI with a cost and latency budget. Evaluation that becomes the slowest, most expensive step in the pipeline gets disabled the first time it blocks a hotfix. Budget it like any other CI resource.

The triage loop closes the system: a user report becomes a test case, the fix lands, and the case stays in the regression set permanently. The suite becomes a record of every way the system has failed, which is exactly what you want it to be.

When This Harness Is Overkill

Not every prompt deserves a harness. Skip the full apparatus for throwaway prompts, one-off internal scripts, and tasks with no stable acceptance criteria. Building evaluation infrastructure for a prompt you will delete next week is a way to feel productive without shipping.

A small hand-curated set with deterministic graders beats a large suite with an uncalibrated judge. Coverage without reliability is theater — it produces numbers, not confidence.

Do not build a judge for a requirement you cannot state as an assertion. Fix the contract first. A judge papering over an ambiguous requirement will produce confident scores that mean nothing.

Warning sign: the suite passes while users complain. That does not mean the prompt is fine. It means the test set drifted from real traffic. The fix is new cases from real failures, not a new grader.

This harness is the prerequisite for anything more ambitious — dynamic context assembly, runtime orchestration, automated feedback loops. Get the static evaluation loop honest before you automate it. An automated loop built on an uncalibrated judge just fails faster.

Your First Move

Pick one prompt already in production. Write five test cases from real failures you have seen — not imagined ones. Manually label a small reference set: for each case, decide pass or fail yourself, and record why. Run your judge against those labels and compare its false positives and false negatives. If the judge is weak, use it for triage and reporting first, not as a release gate — a reporting judge that surfaces suspicious cases is still valuable, even when it cannot block a deploy.

Then add one deterministic grader and one calibrated judge. Run each case five times. Record the spread, not just the mean.

The decision rule to carry forward: a prompt change ships only when the case-level diff shows improvement without new hard failures, and the judge's reliability is reported alongside the pass rate. If you cannot show both, you are guessing — and guessing is what the harness exists to replace.