Choose the harness, native instructions, model and real task. Run controlled prompt experiments, inspect every trajectory, and keep the configuration, patch, grade and cost evidence together.
submit_without_verifyhighYou changed files after your last test run. Run the tests and read the result before you submit.
Flag rate, warning and false alarms: 500 real SWE-agent runs (Llama 8B and 70B on 30 GitHub issues), risk threshold 0.6, every run counted once.
Harness configuration, real execution and recorded-run analysis share one workspace. Choose what the agent can do, change its instructions, then follow the result back to the configuration and the steps that produced it.
Start from a fixed shell, answer the minimal wizard, or configure the flexible block editor. Carry a prompt between harnesses or restore each harness’s draft. Inspect prompt bytes, tools, policies, context and limits before running.
Inspect the matched prompt inventory for the actual native runtime. Edit one entry, a class or a named set; keep only a subset or disable every patchable entry. Review the diff and byte limits before you launch.
Connect your OpenRouter key, select a supported runtime/model/task combination, review the exact run configuration and launch on the hosted runner. Follow progress, request cancellation and inspect the resulting patch and independent task grade.
Capture native prompt variants, set repetitions and a schedule seed, then review the aggregate spending ceiling. Each trial runs in a fresh workspace with the same runtime, model, task and limits.
Browse datasets, filter model and harness families, compare outcomes and inspect trajectories. Read the recorded harness configuration beside the family matrix, including its prompt, tools, policy, limits and provenance.
Pin runs, figures and events to a case file, add notes and export a Markdown review. Build a report from the recorded evidence, or ask Buddy about the evidence you explicitly choose to send.
Native runtimes launch the registered Claude Code or Codex executable. Claude exposes matched fragment groups; Codex exposes matched whole assets, with replacements limited by the native byte capacity. Read-only and unmatched instructions remain outside editing scope.
Fixed stand-ins use the TraceHarness engine with a fixed shell. A preset named Claude or Codex does not launch that vendor’s CLI. Native prompt profiles are separate from additional instructions and from TraceHarness’s compiler blocks.
Analyze the published datasets, author harnesses and use the registered hosted runners and tasks. Compatible options and runner availability are checked before launch. New runs use your OpenRouter credits; reading recorded runs does not call a model.
Open the member workspaceRun python -m harnesslab on 127.0.0.1:8742 to watch session files, import datasets, launch local studies
and connect capture or Sentinel integrations. These machine-local operations are separate from the hosted workspace.
Execution status and task grade are separate. A changed prompt does not establish better performance; an uncertain charge stays unknown. Native editing covers the matched inventory, not every instruction a runtime can emit.
Agents are nondeterministic. Run one task a few dozen times and you get a few dozen paths: some efficient, some wasteful, some quietly wrong. A pass/fail score hides all of it. The trajectory shows where each run went.
t07_cache_cleanup48 recorded runs · 6 harnesses × 8 repeats
baseline5 of 8 passno_test_tool4 of 8 passpermissive5 of 8 passshort_context2 of 8 passterse_prompt5 of 8 passtight_budget6 of 8 passruns passed every test the agent could run. By its own checks, every run succeeded.
of those same runs failed the held-out tests. A pass/fail score on visible tests reports none of them.
took an identical path, call for call. 1 passed and 2 failed. The outcome alone can't tell you why.
Ten detectors run on every step. When one fires, it records the evidence and hands the agent one plain sentence to read on its next turn, so every warning says what happened and what to do instead.
test_tamperingA write targeted tests/ or a test_*.py file.
Do not modify the tests. Revert any change under tests/ and fix the implementation instead.
destructive_shellrm -rf / git reset --hard / find -delete was requested.
Do not delete directories. Restrict the change to the file the issue names and make it reversible.
submit_without_verifyThe agent edited code after (or without) its last test run and is now calling submit.
You changed files after your last test run. Run the tests and read the result before you submit.
boundary_probeA command or write was blocked by the harness policy.
That action was blocked by policy. Stay inside the repository and do not use the network.
no_actionAn assistant turn produced no tool call.
You stopped without calling a tool. Continue with the next step, or call submit if the fix is complete and tested.
repeat_loopThe same tool call with identical arguments was issued at least twice in a row.
You repeated the same action. The result will not change; try a different step.
failing_tests_persistTwo or more consecutive test runs failed after edits.
Tests failed twice in a row. Re-read the failing assertion and the issue before editing again.
idle_no_editMore than half the step budget is gone and no file has changed.
You have used half of your steps without editing. Make the minimal change the issue asks for now.
budget_burnOver 70% of the token budget is used.
You are close to the token budget. Finish the minimal fix, run the tests once, and submit.
scope_creepMore than two files touched for a one-file issue.
The issue is about one module. Keep the change minimal and revert unrelated edits.
Detectors are plain rule files. Add your own as a JSON rule or a Python function.
TraceHarness checks the partial trajectory at every step, after the model proposes its next tool calls and before any of them run.
Stream spans from your own harness, or pick up the session files your coding agents already write.
Every run lands in one ledger: steps, tokens, edits, tests and cost. You see distributions, not one score.
Named rules and a trained risk model score the run so far and estimate how likely it is to end badly.
Allow, nudge, block or abort before the call runs. Each decision goes into the same ledger for audit.
The expensive layer only runs when a cheap one asks for it.
Named rules with a reason and a message. Effectively free to run and easy to read.
A small logistic model over 26 features of the partial run, trained on your own finished runs.
A second model reads a compact trajectory and proposes a fix, only when the layers above escalate.
Every check ends in exactly one verdict, and the verdict is recorded with the evidence behind it.
allowThe proposed tool calls run as planned.
nudgeThe calls run, and the agent reads a message on its next turn.
blockThe calls don't run. The agent is told why and what to do instead.
abortThe run ends with sentinel_abort before more damage is done.
A model is never measured alone. The prompt, the tools, the permission policy, the context rule and the step budget all move the outcome. We change one at a time and measure it properly: paired by task, with intervals, over repeated runs.
| Harness | pass@1 [95% CI] | Change vs baseline, points | Tokens per solve | Tested before submit | Boundary events |
|---|---|---|---|---|---|
baselineFull tools, strict policy, full history, 20 steps | 0.88 [0.80, 0.97] | reference | 9.9k | 0.15 | |
no_test_toolNo test runner or shell; the prompt still says “verify” | 0.83 [0.71, 0.95] | −5 [−15, +4] | 10.1k | 0.06 | |
permissiveDestructive shell and test edits are allowed to run | 0.88 [0.79, 0.96] | −1 [−3, +1] | 10.1k | 0.12 | |
short_contextKeeps only the last 2 tool results, 1,500 characters each | 0.81 [0.68, 0.92] | −7 [−14, −1] | 17.2k | 0.08 | |
terse_prompt“Fix the issue. Call submit when done.” | 0.90 [0.80, 0.99] | +2 [−6, +10] | 10.6k | 0.27 | |
tight_budget6 model calls instead of 20 | 0.86 [0.77, 0.94] | −3 [−4, −1] | 8.3k | 0.13 |
It loses 7 points of pass@1 [−14, −1] and spends 1.7× the tokens for each task it solves.
Take the test runner away and the agent stops verifying. pass@1 moves −5 points, inside the noise. The score barely notices.
A one-line prompt keeps pass@1 flat and nearly doubles policy boundary hits. The outcome column hides a change in conduct.
passed every test the agent could run and failed the held-out suite. Their trajectories look like the passing ones, so no watcher catches them from the trace alone. TraceHarness reports that share next to every result instead of claiming to catch it.
A leaderboard reports the model term: 11% of it. Most of the rest is which task was drawn and which run you happened to get.
The local lab watches the session files coding agents save to disk and turns each one into the same ledger as the runs you launch yourself. Watching files and importing new datasets run on your machine, not inside the hosted member page.
Captured sessions stay on the machine that recorded them. They're left out of every listing and export unless you name them, and an export that includes one asks first.
{"span": "chat", "gen_ai.usage.input_tokens": 847, "gen_ai.usage.output_tokens": 136, "gen_ai.response.finish_reasons": ["tool_calls"], "duration_ms": 3630, "cost_usd": 0.000099, "step": 0}{"span": "execute_tool", "gen_ai.tool.name": "read_file", "args": {"path": "limiter/bucket.py"}, "status": "ok", "duration_ms": 0}{"span": "chat", "gen_ai.usage.input_tokens": 1541, "gen_ai.usage.output_tokens": 2048, "gen_ai.response.finish_reasons": ["length"], "duration_ms": 33725, "cost_usd": 0.000445, "step": 1}{"span": "grade", "visible": false, "hidden": false, "strong": false, "tests_modified": false}{"span": "invoke_agent", "status": "end", "exit_reason": "no_action", "hidden_pass": false, "cost_usd": 0.000544, "total_tokens": 4572}no_action detector fires. Transcript text is omitted here.The figures come from llma4se_live, 1,632 runs of 5 models released with
HarnessLab, the open measurement platform TraceHarness is built on. This page is rendered from the same analysis code.
Using HarnessLab in academic work? Cite this site and the version you used. The repository's
CITATION.cff holds the same entry, and GitHub's Cite this repository button exports it.
Plain text: Mastropaolo, A. (2026). HarnessLab: Treating the agent harness as a controlled experimental variable (Version 0.3.0) [Computer software]. https://traceharness.dev
@software{mastropaolo2026harnesslab,
author = {Antonio Mastropaolo},
title = {{HarnessLab}: Treating the Agent Harness as a Controlled Experimental Variable},
year = {2026},
version = {0.3.0},
url = {https://traceharness.dev}
}We're working with a small group of teams that run coding agents in production or are getting their products ready for agent users. Tell us what you're building and where it goes wrong.