Evaluator Design and Feedback Contracts: Graders, Judges, Disagreement, and Targeted Retry

An evaluator is only as useful as the set of distinct repair actions its output can select.

An agent fails a task. The loop retries. The second attempt fails the same way, because the evaluator returned 0.42 and nothing else. The loop had no idea what went wrong, so it did the only thing it could: re-ran the expensive path and re-sampled the same failure. That is not a feedback loop. It is a coin flip with a budget.

I have watched this pattern burn more agent budget than any model limitation. The fix is not a better judge model. It is a better contract between the evaluator and the loop that consumes it. This article is about designing that contract: what the evaluator consumes, what it emits, how to route its output into targeted repair, and how to know whether the evaluator itself is trustworthy.

We will assume you already have a loop with a state/action/observation contract. If you do not, the mechanism here still applies, but you will need that substrate first. The question this article answers is narrower and harder: given a loop, how do you build the feedback signal that makes the loop repairable instead of merely repetitive?

The Score Is Not the Signal

A scalar score collapses distinct failure causes into one indistinguishable bit of information. "The output is bad" and "the output is bad because it cited a source that does not exist" produce the same 0.3. The loop cannot tell them apart, so it cannot repair them differently.

The retry-cost asymmetry makes this expensive. A blind retry re-runs the full task path — retrieval, planning, generation, tool calls — and re-samples from the same distribution. If the failure was caused by a missing retrieval step, re-running the whole task does not fix it. It re-rolls the dice and hopes the failure was stochastic. When the failure is structural, the dice are loaded.

The reframe I want you to hold for the rest of this article: the evaluator is a classifier over failure classes, and the loop is the consumer of that classification. The evaluator's job is not to grade. It is to select a repair action. Grading is a side effect.

Here is the contrast that matters. A pass/fail judge emits one bit:

{ "pass": false }

A judge designed as a control signal emits a structured diagnosis:

{
  "verdict": "fail",
  "failure_class": "unsupported_claim",
  "evidence_span": "revenue grew 40% in Q3",
  "confidence": 0.82,
  "repair_scope": "final_artifact"
}

The second output tells the loop three things the first cannot: what kind of failure this is, where it lives, and how much of the work is implicated. Those three facts are what make targeted retry possible.

Invariant: Every field in the evaluator's output must map to at least one distinct downstream action. If a field cannot change what the loop does next, it is decoration. Delete it.

This invariant is the spine of the article. Hold it while we work through inputs, rubrics, routing, and disagreement.

What an Evaluator Actually Consumes

Most evaluator failures I have debugged were not judge-model failures. They were input-contract failures. The judge was asked to assess something it was never given, or given something it could not use.

An evaluator consumes four input classes:

Reference-free evaluation judges quality without a known-correct answer. It can legitimately claim things like "this claim is not supported by the provided context" or "this response does not address the question." It cannot claim "this answer is wrong" unless the rubric supplies the ground truth. Reference-based evaluation can make correctness claims, but only against the reference it was given — and only if the reference is itself correct.

The trajectory-versus-artifact distinction is where teams get sloppy. Intermediate steps can be correct while the final artifact is wrong (the agent reasoned well and then formatted badly). The artifact can be correct while the trajectory is wrong (the agent got lucky, or used a tool it should not have). If your evaluator only sees the final output, you cannot distinguish these, and your repair action will be wrong half the time.

Context budget is a design constraint, not an afterthought. What you omit from the judge prompt becomes an unmeasured failure mode. If you do not pass the retrieved context, the judge cannot detect unsupported claims. If you do not pass the task specification, the judge cannot detect instruction drift. Every omission is a blind spot you have chosen.

Rubrics That Produce Classes, Not Verdicts

A rubric that asks "is this output good?" produces a verdict. A rubric that produces classes decomposes the task into independently checkable criteria, each with its own failure mode.

Start by separating hard constraints from soft criteria. They demand different repair paths:

• Hard constraints: schema validity, policy compliance, factual grounding, tool-call well-formedness. These are binary. A violation is a defect, not a quality gradient.
• Soft criteria: tone, conciseness, helpfulness, style. These are gradients. A miss is a preference, not a defect.

Mixing them in one holistic question is how you get a judge that says "the output is mostly fine" and a loop that does not know whether to regenerate or re-retrieve.

Force structured output. A fixed failure-class enum plus a required evidence field pointing at the offending span:

from enum import Enum
from pydantic import BaseModel

class FailureClass(str, Enum):
    SCHEMA_VIOLATION = "schema_violation"
    UNSUPPORTED_CLAIM = "unsupported_claim"
    INSTRUCTION_DRIFT = "instruction_drift"
    INCOMPLETE_ANSWER = "incomplete_answer"
    TONE_MISMATCH = "tone_mismatch"
    NONE = "none"

class Evaluation(BaseModel):
    verdict: str
    failure_class: FailureClass
    evidence_span: str | None
    confidence: float
    repair_scope: str

The evidence_span field is not optional decoration. It forces the judge to point at the specific text that triggered the classification. A judge that cannot produce a span is guessing, and you want to know that.

Free-text critique is the trap here. It reads well to humans and is nearly unusable as a control signal. You cannot route on prose. You can route on an enum. If you want the prose for human review, emit it alongside the structured fields — never instead of them.

Failure mode: Rubric criteria that overlap so heavily that the judge's class assignment is effectively random. If "incomplete" and "instruction drift" describe the same output, the judge will pick one arbitrarily, and your routing table will fire the wrong repair half the time. Test your rubric by asking whether two humans would assign the same class to the same failure.

Deterministic Checks Before Judges

Run cheap, exact checks first. Reserve model-based judgment for what only a model can assess.

Deterministic validators — schema checks, type checks, unit tests, regex, tool-call well-formedness — are exact, cheap, and reproducible. They should gate the pipeline. A model-based judge should only see candidates that could plausibly pass.

The cost and latency ordering matters. A judge call is expensive and adds variance. If a parser could have rejected the output, using a judge to verify it adds cost and noise without adding information. Worse, it adds a failure mode: the judge might pass something the parser would have caught.

The reverse failure is subtler and more common in teams that over-trust deterministic checks. A schema validator can confirm the output is well-formed JSON. It cannot confirm the JSON contains a true claim. Using a deterministic check as evidence for a semantic claim it cannot support is a category error — it produces a green checkmark on a broken output.

The ordering rule: deterministic gates first, model judgment second, human review for what neither can settle.

Designing the Feedback Contract

The feedback contract is the interface between evaluator and loop. Treat it as an explicit schema with defined semantics, so the loop can act on it deterministically.

Required fields:

• verdict: pass or fail, against the rubric.
• failure_class: from a closed enum. Closed, not open. An open set cannot be routed.
• evidence_pointer: the span, step, or artifact region implicated.
• confidence: a marker of uncertainty, not a probability you should trust numerically.
• repair_scope: the key field. Does the failure implicate the final artifact only, a specific step, the plan, or the retrieval context?

Repair scope is what makes targeted retry possible. A final_artifact scope means regenerate the output. A step scope means re-run one step. A plan scope means replan. A retrieval scope means re-retrieve. Each scope maps to a different cost and a different blast radius.

Version the contract. Changing the failure-class enum changes loop behavior, so treat it as a breaking interface change. If you add a class, you must add a routing entry. If you remove a class, you must remove its routing entry. An enum change without a routing change is a silent bug.

Failure mode: Letting the evaluator emit an instruction that the loop executes verbatim. "Rewrite the second paragraph to be more concise" turns the judge into an unaccountable planner. The evaluator should report what is wrong and where. The loop decides how to fix it. Keep observation and prescription separate.

Routing Failures to Targeted Retries

The routing table is the mapping from failure class to repair action. Build it by hand. Do not delegate it to the model.

The point of the table is that repair actions differ in kind. Regenerating the artifact, re-running one step, revising the plan, re-retrieving, escalating to a human, and abstaining are all distinct moves. Localized repair preserves completed work. Whole-task retry discards it and re-samples the same failure.

Budget interaction is where this gets interesting. Targeted retries are cheaper per attempt, but they can loop if the repair action cannot address the class. If unsupported_claim triggers re-retrieval and the retrieval source does not contain the needed fact, you will re-retrieve forever.

The termination condition must be per-class, not just total. Cap retries per failure class so a single unrepairable class cannot consume the whole budget:

retry_budget = {
    "schema_violation": 2,
    "unsupported_claim": 3,
    "instruction_drift": 1,
    "incomplete_answer": 2,
}

When a class exhausts its budget, escalate or abstain. Do not fall back to blind retry.

Failure mode: A failure class with no repair action in the table. This silently degrades to blind retry — the loop sees a class it cannot route and does the only thing left. Audit your enum against your routing table every time either changes.

When Judges Disagree

Disagreement is diagnostic information, not noise to be averaged away.

Sources of disagreement are worth naming because each implies a different fix:

• Rubric ambiguity: two criteria overlap, so the judge picks arbitrarily. Fix the rubric.
• Position or ordering effects: the judge favors the first or last option presented. Randomize order.
• Verbosity and length bias: longer outputs score higher regardless of quality. Control for length or penalize it explicitly.
• Task complexity: genuinely hard cases where reasonable judges differ. This is real ambiguity, not a bug.
• Genuine borderline cases: the artifact sits on the decision boundary. Escalate.

The distinction that matters: disagreement caused by an underspecified rubric is a bug you can fix. Disagreement caused by a genuinely ambiguous artifact is a signal you should route to human review.

Aggregation choices have real tradeoffs:

Averaging hides the signal. A 50/50 split and a confident 50 are different states and should route differently. A split means the rubric is ambiguous or the artifact is borderline. A confident 50 means the judge is uncertain about a case it should probably escalate.

Calibration practice: hold a small labeled set, measure judge agreement against it, and re-check after any rubric or model change. Without a labeled set, you have no way to know whether your judge is measuring the task or measuring itself.

Failure mode: Tuning the rubric until the judge agrees with you on the examples you already looked at. This overfits the evaluator to your sample. The judge will agree with you on the cases you inspected and fail on the ones you did not.

Evaluating the Evaluator

Before your evaluator gates agent behavior, you need to know whether it is trustworthy. Measure two error directions separately:

• False pass: a bad output accepted. The loop terminates on a broken result.
• False fail: a good output rejected. The loop burns budget repairing something that was fine.

Their costs are asymmetric. A false pass ships a defect. A false fail wastes compute. Which is worse depends on your application, but you must measure both, because an evaluator can be accurate overall while being useless on the class that matters most.

Build a small hand-labeled set from real trajectories, including known-bad cases. Synthetic examples do not capture the failure modes your agent actually produces. Pull from production traces.

Track per-class precision. An evaluator that is 95% accurate overall but 40% accurate on unsupported_claim is dangerous if that class is the one you care about. Aggregate accuracy hides this.

Monitor drift in production. Failure-class distribution shifts are an early signal that the task, the model, or the inputs changed. If instruction_drift starts firing at three times its baseline rate, something upstream moved.

Keep the judge model version pinned and re-validate on upgrade. Judge behavior is not stable across versions. A rubric that worked on one model version can silently change its class assignments on the next.

When Not to Build a Judge

The decision boundary is worth stating plainly, because model-based evaluators are expensive to build and validate.

Skip the judge when the property is exactly checkable. Schema, types, unit tests, and deterministic business rules are exact, cheap, and reproducible. A judge adds variance and cost to a problem that does not have any.

Skip it when the artifact has a single correct answer that a reference comparison can settle. Semantic equivalence checks against a reference are cheaper and more reliable than open-ended judgment.

Skip it when you cannot afford to validate the evaluator itself. An unvalidated judge in a control loop amplifies its own errors. It does not just mislabel — it mislabels and then triggers the wrong repair, which produces a new output the same judge evaluates. Errors compound.

Prefer human review or abstention when the failure class is high-stakes and the judge's per-class precision is unmeasured. Abstention is a legitimate output. A loop that knows when to stop and ask is more useful than one that confidently repairs the wrong thing.

Decision rule: If the property is exactly checkable, use a deterministic check. If it is semantic and you can validate the judge, use a judge. If it is semantic and you cannot validate the judge, use a human or abstain.

The First Move

Take one existing agent failure. Write down the distinct repair actions you would actually take to fix it. Not the categories — the actions. "Re-retrieve and regenerate." "Replan and re-execute." "Reformat the artifact." "Escalate to a human."

That list is your failure-class enum. Derive it from the repairs, not from a generic quality rubric. The repairs are what the loop can do; the classes are the labels that select them.

Then run the evaluator against a handful of real trajectories and check whether each emitted class maps to a repair action that changes the next attempt. If a class fires and the loop does the same thing it would have done anyway, the class is not doing work.

The decision rule that closes the loop: if a field in the evaluator's output cannot change what the loop does next, delete it. Every field you keep is a field you must maintain, version, and validate. Keep the ones that earn their place by selecting a repair. Delete the rest.