Coding Agent Harnesses: Workspaces, Shells, Artifacts, Tests, and Long-Horizon State

The agent writes the function, re-reads it, decides it looks correct, and stops. The code is wrong. This is the most common failure in autonomous coding, and it is not a model failure. It is a harness failure.

You already know what a harness is: the model-external, editable layer that supplies tools, context, execution, and feedback around a language model. This article skips the definition and builds the thing. The goal is a runtime where the agent cannot declare victory without reality answering back.

Why the Model Is Rarely the Bottleneck

The dominant failure pattern in long-horizon coding is not that the model cannot write plausible code. It is that the model writes plausible code, reviews its own output, and stops. Self-review is biased toward the first solution the model produced. Reading your own work and confirming it looks fine is not verification. It is a mirror.

The harness is the collection of model-external components you can edit: system prompt, tool descriptions, tool implementations, middleware, execution constraints, and feedback loops. Harness engineering is the practice of designing those components deliberately rather than accepting defaults.

The evidence that this matters is hard to dismiss. In one published experiment, a coding agent moved from outside the top 30 to the top 5 on a terminal-based coding benchmark after changing only the harness — same model, same tasks, different system prompt, tools, and middleware. The model did not get smarter. The runtime around it got better at converting model output into checkable state changes.

That is the invariant to hold onto: the harness must convert model output into observable, checkable state changes. If the agent can produce a final answer without the harness ever running a command, reading a file, or executing a test, you have built a chat interface with extra steps.

One caution. Specific model behaviors and benchmark scores are version-dependent. A harness technique that fixes a failure on one model version may be unnecessary on the next. Treat harness design as a response to observed failure modes, not as a permanent architecture.

The Five Surfaces of a Coding Agent Runtime

Before writing code, map the surfaces. A coding agent runtime has five:

The data flow is a loop, not a pipeline:

task spec → workspace → shell actions → test output → artifact/state update → next decision
     ↑                                                                              |
     └──────────────────────────────────────────────────────────────────────────────┘

Each surface feeds the next. The workspace is where the agent acts. The shell is how it acts. Tests are how it learns whether the action worked. Artifacts are how it remembers across context boundaries. State is how the run survives a crash or a reset.

The deliverable is the post-acceptance filesystem state, not the conversation transcript. This distinction matters more than it sounds. If the agent believes the transcript is the deliverable, it will stop when the transcript looks complete. If it believes the filesystem is the deliverable, it will keep working until the filesystem is correct.

The Run Lifecycle: One Trace Through All Five Surfaces

The five surfaces are not a checklist. They are a state machine. Here is the canonical transition for one step:

1. Model proposes action (e.g., "run pytest tests/test_parser.py")
2. Harness validates action against policy (is the command allowed?)
3. Harness executes action in workspace
4. Workspace changes (files written, deleted, modified)
5. Verifier records evidence (test output, exit code, diff)
6. Harness commits step result to run record
7. Next context resumes from recorded state

The run record is the contract between surfaces. A minimal version:

{
  "run_id": "run-abc123",
  "step_id": 7,
  "task_revision": "spec-v3",
  "action": {
    "type": "shell",
    "command": "pytest tests/test_parser.py",
    "cwd": "/workspace"
  },
  "workspace_delta": {
    "modified": ["src/parser.py"],
    "created": [],
    "deleted": []
  },
  "evidence": {
    "exit_code": 1,
    "stdout_ref": "artifacts/step-7-stdout.txt",
    "stderr_ref": "artifacts/step-7-stderr.txt",
    "assertion_failures": ["test_parse_empty: expected 0, got None"]
  },
  "status": "failed",
  "checkpoint_committed": false
}

This record answers the questions a recovery system needs: what was attempted, what changed, what evidence was produced, and whether the step completed. Without it, resumption is guesswork.

Workspace Design: The Deliverable Is the Filesystem

The workspace is not a scratch pad. It is the product. Every layout decision either helps the agent act cleanly or creates ambiguity about what counts as done.

Separate deliverable from scratch. Many evaluation environments constrain where output files must land. An agent that assumes free-form file placement will write the right content to the wrong path. Give it an explicit scratch directory and an explicit delivery path. When the agent treats them as the same space, it will eventually clean up a file the evaluator needs.

Mount harness components as files at fixed locations. System prompt, tool descriptors, tool implementations, middleware, and memory should be inspectable files, not buried configuration. This makes the harness editable without redeploying the agent loop. It also makes failures debuggable: you can diff the prompt, read the tool implementation, and trace the middleware logic as text.

Byte-level equivalence matters. For DSL outputs, config files, and script generation, "looks equivalent" is not equivalent. Whitespace, encoding, and line endings change behavior. If the task produces a config file, the test should compare bytes, not semantics. Tell the agent this explicitly. Otherwise it will produce a semantically correct file with a trailing newline that breaks the parser.

Isolate each rollout. A fresh sandbox per run prevents shell side-effects from leaking between tasks. If task A leaves a compiled binary or a modified environment variable, task B inherits a corrupted starting state. Isolation is not optional for evaluation; it is the precondition for reproducible results.

Failure mode: The agent creates a temporary render, verifies it looks correct, then deletes it as cleanup. The evaluator needs that file. The agent treated scratch space and deliverable space as interchangeable. Fix this with a publish-state rule in the system prompt and a guard in the shell.

Shell Tools: The Action Interface and Its Guards

The shell is where most agent capability lives. It is also where most agent damage happens. Treat it as a contract, not a passthrough.

A shell tool that executes any command the model produces is a liability. A shell tool that intercepts destructive commands and returns a message naming the protected resource is a guardrail. The difference is stateful awareness.

Here is the pattern. When the shell observes a command that would delete or overwrite a file the evaluator needs, it intercepts:

PROTECTED_PATHS = {"/app/output.ppm", "/app/config.yaml"}

def run_shell_command(cmd: str) -> str:
    for path in PROTECTED_PATHS:
        if would_delete_or_overwrite(cmd, path):
            return (
                f"Command blocked: {path} is a protected deliverable. "
                f"It must remain on disk after this run completes. "
                f"If you intended to clean up scratch files, use /tmp/scratch/ instead."
            )
    return execute(cmd)

The agent receives a message naming the protected path, acknowledges it, and finishes without rerunning the cleanup. The verifier finds the correct file on disk.

This works because the guard returns a message the model can reason about. It does not silently fail. It does not throw an opaque error. It names the resource, explains why it is protected, and suggests an alternative.

Publish-state rules. Add a rule to the system prompt that names the post-acceptance filesystem state as the deliverable. This tells the agent that intermediate renders, debug logs, and temporary files are not the product. The product is what remains on disk when the run ends.

Scratch-directory rules. For tasks with constrained delivery layouts, tell the agent where scratch files go. Without this, it will use the working directory as scratch and pollute the deliverable surface.

Idempotency and error signaling. What the shell returns shapes the next model decision. A command that fails silently teaches the agent nothing. A command that returns a clear error message with the relevant path and failure reason gives the agent something to fix.

When not to guard. Over-guarding turns the shell into a maze. If the agent spends more turns fighting the harness than solving the task, you have built a worse agent. Add guards only after you observe a specific destructive or self-defeating command. Start with zero guards. Add one when you see the failure.

Tests as the Feedback Loop, Not a Checkbox

Self-review is biased toward the first plausible solution. Tests are a primary automated falsification surface — not the only one, but the one that most directly answers "does this code do what the spec requires?"

The test loop has four phases:

1. Plan and discover. Read the task, scan the codebase, build an initial plan that includes how to verify the solution.
2. Build with verification in mind. Implement the plan. Build tests if they do not exist. Cover happy paths and edge cases.
3. Verify. Run tests. Read the full output. Compare against the original specification, not against your own code.
4. Fix. Analyze errors. Revisit the spec. Fix the issue.

The critical instruction is step 3: compare against the specification, not against your own implementation. An agent that compares test output to its own code will rationalize mismatches. An agent that compares test output to the task description will find real bugs.

Test output is not a pass/fail gate. It is signal for hill-climbing. A failing test with a clear assertion message tells the agent what to fix. A passing test with no output tells the agent nothing about edge cases it did not cover.

Other verification surfaces. Tests are not the only cheap falsification mechanism. Static analysis, type checkers, schema validators, and diff reviews catch different classes of error. A type checker catches interface mismatches that a unit test might miss. A schema validator catches malformed output that a test might not exercise. Use the cheapest surface that catches the failure class you are seeing.

Cost tradeoff. Running tests inside the loop costs time and tokens. A test suite that takes 30 seconds per run will consume significant wall-clock time over a multi-hour session. A test suite that takes 5 minutes per run may exceed the turn budget. Profile your test runtime and decide what runs on every iteration versus what runs at checkpoints.

Failure mode: The agent writes a test that passes because it tests the implementation, not the specification. The test asserts that add(2, 3) == 5 because the implementation returns 5. It does not test add(-1, 1) == 0 or add(0, 0) == 0. The test is a mirror, not a check.

Artifacts and Handoff Across Sessions

When a run exceeds one context window, the conversation cannot carry the full state. Structured artifacts carry what the conversation cannot.

An artifact is a durable domain output or handoff object: a plan file, a progress log, a task list, a diff summary, a test report. The artifact is the handoff mechanism between sessions and between specialized agents.

Decompose the build into tractable chunks. A multi-hour build is not one task. It is a sequence of features, each with its own verification. Hand off artifacts between chunks rather than raw conversation history. A plan file that lists completed features, pending features, and known issues is more useful than a transcript of the last 50 turns.

Context reset versus compaction. These are different strategies for managing context growth:

A reset gives the agent a clean slate at the cost of needing a handoff artifact rich enough to resume. Compaction preserves continuity but keeps accumulated context, including mistakes and dead ends.

Model-version dependence. Some models exhibit context-anxiety behavior: performance degrades as the context window fills, even when the relevant information is still present. For these models, context resets are necessary. Other models handle long contexts without this degradation, reducing the need for resets. Do not assume the behavior is universal. Test your model.

File-based planning artifacts. A markdown plan file that survives a cleared or compacted context is the simplest handoff mechanism. It does not require a database. It does not require a custom serialization format. It is a file the agent reads at the start of each session and updates as it works.

Failure mode: The handoff artifact is too thin. The next session re-derives decisions the previous session already made, or contradicts them. The fix is to include not just what was done but why: the constraints discovered, the approaches rejected, and the open questions.

Long-Horizon State and Recovery

A multi-hour run will eventually hit a failure: a timeout, a crash, a network error, a model that goes off the rails. The question is not whether recovery happens but whether it happens correctly.

Artifacts versus checkpoints. An artifact is a durable domain output. A checkpoint is a runtime commit record that points to the artifacts and workspace state needed for resumption. A plan file is an artifact. The run record that says "step 7 completed, plan file updated, tests passed" is a checkpoint. You need both, and they serve different purposes.

The checkpoint consistency invariant. A checkpoint may advertise a completed step only after the workspace outputs and verification evidence it names are durable. The checkpoint must carry a step identifier, the task revision it was built against, the observed outputs, and a commit status. If you persist "tests passed" before the relevant file is durable, you have created a checkpoint that lies.

Recovery semantics. When a step fails, you need to know which category it falls into:

Replay semantics. A deterministic replace-at-path operation can be idempotent when atomicity and ownership are defined. A file write that replaces the entire file at a known path is replay-safe. A file write that appends is not. A database insert may not be. A shell command that modifies external state is not. Design your steps so that replay is safe, or track which steps have committed.

The ambiguous crash case. The agent writes a file, then crashes before the checkpoint commits. On resume, the file exists but the run record says the step failed. You have three options: reconcile (check the filesystem, confirm the file matches the expected output, mark the step complete), retry idempotently (re-run the step, which overwrites the file with the same content), or require human review (flag the ambiguity and stop). Pick one policy and apply it consistently.

Checkpoint granularity. Frequent checkpoints cost tokens and latency. Sparse checkpoints lose more work on failure. The right granularity depends on the cost of redoing work versus the cost of checkpointing. For a task where each step takes 30 seconds, checkpoint every step. For a task where each step takes 10 minutes, checkpoint every few steps.

Failure mode: A state reset path that the guard still treats as overrideable. The agent finds a way to reset state that bypasses the protection. Recovery silently discards accepted work. The fix is to make the guard aware of reset paths, not just delete commands.

Observability and Iterating on the Harness

Models are largely black boxes. Their inputs and outputs are visible in text space. That is the improvement surface.

Compress the optimization space. A harness has many knobs: system prompt, tools, middleware, skills, sub-agent delegation, memory systems. Do not tune all of them at once. Pick three: system prompt, tools, and middleware. These are the highest-leverage changes.

Read traces to classify failures. When a run fails, the trace tells you where. But the same symptom — wrong code, skipped verification — can have multiple causes. Use diagnostic signals to distinguish them:

If the agent never read the file it needed, changing the system prompt will not help. Fix the search tool or the context assembly. If the agent planned a sequence that could not work, adding a guard will not help. Fix the planning prompt or the task decomposition.

Every harness change should be a falsifiable prediction. "Adding a publish-state rule will fix the cleanup failure" is falsifiable. "Improving the prompt" is not. Write down what you expect to change before you change it.

Keep the model fixed while changing the harness. If you change both, you cannot attribute the effect. Run the same tasks with the same model and different harness configurations. Compare the results.

Caution on benchmarks. Benchmark gains are results under agreed test conditions. They are not a guarantee of production reliability. A harness that scores well on a benchmark may fail on your specific tasks, your specific codebase, or your specific failure modes. Use benchmarks as a signal, not as proof.

A Minimal Harness You Can Build First

The smallest useful harness is a workspace, a shell, and one test command. That is enough to force reality to answer.

Build order:

1. Workspace + shell + one test command. Skip multi-agent orchestration. Skip artifact handoff. Skip checkpoints. Get the single loop reliable first.
2. Add guards only after you observe a specific destructive command. Start with zero guards. Add one when you see the failure.
3. Add artifact handoff only when a run exceeds one context window. If the run fits in context, you do not need artifacts.
4. Add checkpoints only when a run is long enough that losing it hurts. If a failed run costs 5 minutes, do not build checkpoint infrastructure. If it costs 2 hours, build it.

Decision boundary. A thin harness works when the task is short and the model is strong. A thick harness is necessary when the task is long, the environment is unseen, or verification is expensive. The harness earns its complexity only when it converts model output into observable, checkable state changes.

When not to build a custom harness. If an existing agent runtime already exposes the knobs you need, tune it before writing your own. The harness is not the product. The task completion is the product.

Your Next Step

Start with one existing coding-agent run. Instrument it with traces. Run it on a task that takes at least 30 minutes. When it fails — and it will — classify the failure into one of four categories: search, planning, verification, or state. Change exactly one harness knob to fix it. Run again. The harness earns its complexity one failure at a time.