An Eval for Coding Agents, and the Three Bugs I Found in It
A harness that gives a model a coding task, grades the fix against hidden tests in a sealed container, and reports whether the model solves it on every try, not just one. The most useful thing it produced was a list of three ways it was wrong.
- My role
- Designed it and directed the build. Claude Code wrote most of the code.
- Timeline
- Jun to Jul 2026
- Tools
- Claude Code, TypeScript, Docker, mini-swe-agent, Vitest
What this is
At Cadre we had to pick models for coding work, for our own projects and for the organizations we advised. Public leaderboards test someone else's tasks, graded someone else's way. So in June 2026 I built a small test harness of our own.
It is private and unfinished. It runs end to end on 14 tasks, which is too few for a ranking I'd publish. What I can show is how it works and the three bugs I found in it. Each one would have skewed the results without any error to warn me.
How a run works
The model gets a task.
Each task is a small TypeScript codebase with a planted bug or a missing feature, plus a written issue. I wrote all 14. The tests that grade the fix stay hidden, in the style of SWE-bench.
It works inside a locked container.
The agent is mini-swe-agent, the same for every model. It runs its commands in a Docker container that can see only its own copy of the code.
Only source changes count.
Before grading, a filter strips any change to tests or build settings out of the model's fix.
A second container grades it.
It has no network. The hidden tests run against the fix there.
Every task runs five times per model.
I report two numbers: solved at least once (pass@k) and solved all five times (pass^k). On the first task, one model solved 4 of its 5 tries. That is 100% on the first number and 0% on the second. It's the model you don't want running unattended.
A task has to pass two checks before I use it. The reference fix must pass the hidden tests, and six agents that do nothing or cheat must all fail them.
Each model call reserves its cost in a ledger before it's made, and a paid run needs a written plan and a budget ceiling.
The three bugs
1. The agent could read the answer key
On the first run of a harder task, the agent's shell could reach my whole machine. One attempt searched it, found the task's folder, and applied the reference fix and the hidden tests.
How I found it. That run was marked as submitted, but its fix was empty. The task folder itself had been changed, and the leaked tests were then copied into every later run. 18 of the 20 runs had scored as solved. I threw out all 20, about $23 of spend.
The fix. The agent now runs in a container that can see only its own copy of the code. Task folders are cleaned before every run. And the run stops if a task's original files have changed.
2. The grader accepted a fix that fixed nothing
How I found it. A fresh review of work I had already marked complete turned up a hole. A fix that never touched the bug could still score as solved, by pointing the test settings at a fake test that always passes. The review proved it with a real run.
The fix. The source-only filter in step 3, and more cheating agents in the checks every task has to pass. An agent that does nothing has to fail every task, or the task is broken.
3. The cost cap decided the result
I capped each run at $2.50. On the next run, two models hit the cap on all 10 of their attempts. Nine were still scored as solved, but I was now measuring what a model can do for $2.50, which is a different question.
The fix. I threw out those 10 runs, about $25, and ran them again with a $4.00 cap. At $4.00 one of the two models finished on its own, and the other scored the same as before.
Where it stands
I spent about $250 in total, then froze paid runs. Each one now needs my sign-off. The one exception is a cheap difficulty check: the top model alone, three tries on a new task.
Eight of the 14 tasks have been run with a real model. The hard part turned out to be the tasks. In that three-try check, one task stumped the top model, which solved it on 1 of 3 tries. I wrote four more that were meant to be that hard, and it solved each of them on all three tries. A task that separates the top models is hard to write, and I won't publish a ranking until there are enough of them.
What I took from it
An eval is software, and it has bugs like any other. Two of these three would have given me a confident, wrong ranking. I don't trust a harness now until an agent that does nothing fails every task and an agent that cheats gets caught.