coding-eval
An offline harness that replays a coding agent's attempt in a sandbox and reports correctness, regressions, scope and a failure label, keeping measured facts apart from the agent's own claims.
- Year
- 2026
- Kind
- Evaluation · Python
- Status
- Shipped
A green or red test result hides how a coding agent failed. This harness replays a recorded attempt at a repo task in a throwaway directory and reports, separately, whether the target tests now succeed, whether anything that passed before broke, how much of the repo was touched, and a failure label from a two-level taxonomy. Every label prints the evidence that fired it.
Every number is tagged with where it came from: the sandbox ran it, it was recounted from the recording or the diff, or the agent claimed it. A claimed number can never move a verdict. In the shipped corpus, 20 of 21 attempts end by claiming they solved the task. Five did. Thirteen failures that a single success score would count as the same zero carry eight different outcome classes.
The 21 attempts are hand-authored fixtures, not recordings of a real agent, so the report prints counts and refuses percentages. Isolation is a subprocess in a temp directory, not a security boundary: a patch that fakes both the test report and the exit code can get past the check. Code quality, patch security and real-repo scale are not measured.
