Validation vs Verification vs Evaluation in AI Systems

The release gate is green. Schema validation passed, the golden set passed, the benchmark score ticked up, and the guardrails logged zero trips. Two days later the agent issues a refund to an account it never verified, and the incident review opens with the sentence every engineer dreads: but the evals passed.

They did pass. They just never tested the claim the team believed they tested. That gap is not a tooling failure. It is a vocabulary failure, and it costs more than most teams admit.

Why "The Eval Passed" Proves Almost Nothing

Watch what happens in a typical review. Someone asks whether the new prompt is safe to ship. Someone else says the evals are green. The room relaxes. Nobody asks which evals, what they inspected, or what claim their passing actually supports.

That is the weak mental model: evaluation treated as one activity that produces one confidence level. Under that model, every check is interchangeable evidence. A schema validator, a benchmark score, and a passing regression suite all collapse into the same green light. The dashboard looks like a single dial labeled quality, and it is lying to you.

The stronger model is a container. Evaluation is the umbrella term for the whole assessment activity. Inside it sit distinct evidence types — verification, validation, testing, benchmarking — and each one answers a different question about a different artifact. A check that answers one question cannot be cited as evidence for another. Full stop.

Here is the anchor criterion I use for the rest of this article: for any check, name the claim it can support before you cite it. If you cannot finish that sentence in one line, the check is either redundant or mislabeled, and either way it is not evidence yet.

This matters more in agent loops than it ever did in traditional software. A loop carries trajectory state — tool selection, ordering, retrieval, revision — and most of that state never appears in the final output. Output-only checks inherit a coverage hole that unit tests never had to think about. We will come back to that, because it is where the taxonomy stops being pedantry and starts predicting production failures.

Three Questions, Three Evidence Types

Before comparing terms, fix the decision axis. Each term answers a different question about a different artifact, and the artifact determines what the check can prove.

Verification asks whether the system conforms to its specification — the requirements, contracts, and constraints we wrote down. It is evidence about conformance to a written standard.

Validation asks whether the system meets intended use and user expectations under realistic operating conditions. It is evidence about fitness for purpose, and it does not care how elegant your spec was.

Evaluation is the broader assessment activity against criteria, with or without executing the artifact. It is the container. Verification, validation, testing, and benchmarking live inside it rather than beside it.

The classic framing is worth keeping because it compresses the distinction cleanly: verification asks are we building the system correctly, validation asks are we building the right system.

The nesting is the part teams get wrong. If evaluation is the container, then "we evaluated it" is not a claim — it is a category error until you name the sub-activity. Saying you evaluated something is like saying you measured it. Measured what, with what instrument, against what threshold?

Read the last two columns together. Every row has a claim it can support and a claim it cannot, and the failure mode is almost always citing the check in the wrong column.

The Model-Level Trap: Validation Data Is Not Validation

The single most common collision in this vocabulary is the word validation itself. It means two unrelated things depending on whether you are talking about training or about systems, and the collision quietly corrupts release decisions.

In training, the validation set is a held-out split used to tune and select. It is a data-splitting convention. Its job is to give you a selection signal without leaking test data into the model. That is all it is.

The test split exists to estimate generalization. Reusing the validation split as your test split silently converts a selection signal into a reported result. The number goes up, the number means less, and nobody notices because both splits are called "eval" in the training script.

System-level validation is a different claim entirely: fitness for intended purpose under realistic conditions, including the operational design domain. A model with a strong held-out score can still fail system validation because the deployment context, the input distribution, or the user expectations were never in the split. The split measured the weights. It said nothing about the system.

When someone says "validation," ask which artifact is being validated — the weights, the pipeline, or the deployed system. The word alone does not tell you, and the answer changes what the evidence proves.

I have watched this collision survive code review because both sides were using the same word for different artifacts. The training engineer meant the split. The product engineer meant fitness for purpose. Neither was wrong, and the release shipped on a misunderstanding.

What Each Check Can and Cannot Prove

Here is the working mapping. For each common check in LLM and agent systems, the evidence type it produces, the claim it legitimately supports, and the over-claim it invites.

Schema and type validators prove structural conformance to a contract. They say nothing about correctness, usefulness, or intent. A perfectly shaped JSON object can contain a fabricated refund amount. The validator will pass it every time.

Unit and integration tests prove specified behavior on specified inputs. This is verification evidence, and it is bounded by the completeness of the specification. Tests cannot cover behavior nobody wrote down, which is exactly where agent systems spend most of their time.

Golden and regression sets prove stability against known cases. They detect drift and regressions. They do not prove fitness for unseen use, and treating a green golden set as a generalization claim is a category error.

Benchmarks produce a comparable score under agreed test conditions. Contamination, saturation, and leaderboard gaming mean a score is a result, not a guarantee. A benchmark tells you how the system performed on that benchmark. It does not tell you how it will perform in your deployment, and the gap between those two claims is where most benchmark-driven decisions go wrong.

LLM-as-judge rubrics scale assessment of qualities that resist exact matching. They require calibration against human labels and ongoing monitoring for drift. An uncalibrated judge is a confident narrator, not a measurement instrument.

Runtime guardrails and self-check loops enforce process compliance at inference time. They can prove a step ran. They cannot prove the step's output was true. A forced verification step that completes on schedule has demonstrated that the loop executed, not that the verification found anything.

The falsification column is the one I would actually use in a review. If you cannot name the cheapest experiment that would break your over-claim, you are not holding evidence. You are holding a feeling with a green badge.

Trajectory Checks Change the Evidence Map

Agent loops break the output-only assumption that most teams inherit from traditional software testing. An agent makes dozens of internal decisions — tool selection, ordering, retrieval, revision — before it emits a final answer. A correct output can mask broken reasoning: lucky hallucinations, irrelevant retrievals that happen not to change the answer, skipped steps that the final text never reveals.

Consider a support agent that must verify customer identity before processing a refund. The final output — refund processed — looks identical whether or not the identity step happened. Every output check passes. The process claim fails, and the process claim is the one with the compliance exposure.

This is where the taxonomy earns its keep. Trajectory-level checks verify the process contract — did identity verification happen before the refund? Output checks validate the result — was the refund correct? These are different claims and they need different evidence. A passing output check is not evidence about the process, and a passing trajectory check is not evidence about the result.

Ground-truth annotation for trajectories requires domain expertise and is expensive. Budget it deliberately rather than assuming output checks cover it. The failure path to plan for is specific: a loop that terminates on a plausible-looking output while skipping a required step passes every output check and fails the process claim, and it will do so silently until someone audits the trace.

If you want the loop mechanics underneath this — state, action, observation, feedback, termination — that is a separate article. For evidence purposes, the point is narrower: the loop adds artifacts that output checks cannot see, and those artifacts carry claims.

Assigning Checks to Claims in Practice

Turn the taxonomy into a habit with a short procedure. For each check in your suite, write the claim it supports in one sentence. If you cannot, the check is either redundant or mislabeled. Then look for the three common misassignments:

• Citing a benchmark as validation evidence.
• Citing a schema check as correctness evidence.
• Citing a passing test suite as proof of intended use.

Each one swaps a claim the check can support for a claim it cannot. The fix is not more checks. The fix is relabeling what the existing checks prove.

Then ask the coverage question: which claims in your system have no check at all? Those are the gaps. The checks that already pass are not the interesting part of the audit.

The cost tradeoff is real and worth stating plainly. Verification is cheap and automatable. Validation is expensive — it needs realistic conditions, representative data, and stakeholder judgment about intended use. Evaluation strategy is the discipline of deciding where to spend each. Teams that treat all three as one budget end up over-investing in cheap verification and under-investing in the validation that would have caught the incident.

And the counterweight: do not over-engineer. For a narrow, well-specified deterministic component, verification alone is sufficient. Do not build a validation apparatus for a claim nobody is making. The taxonomy is a tool for assigning evidence, not a mandate to produce all three types everywhere.

A check is not evidence until you can name the claim it supports and the artifact it inspected. Everything else is a green light with no destination.

The Rule to Carry Forward

Before any check is cited as evidence, name the claim it supports and the artifact it inspected. Verification speaks to conformance. Validation speaks to fitness for purpose. Evaluation is the container that holds both, plus testing and benchmarking, and it is not itself a claim.

Here is the next move, and I would do it this week rather than someday. Pick one production incident — ideally one where the evals were green. Name the claim that actually failed. Then identify which evidence type was missing when the team believed they had it. That single exercise will tell you more about your check suite than another dashboard ever will, because it forces the question the green light was hiding: what did we actually prove, and about what?