The Eval Harness: Grading Outcomes, Not Self-Reports

Grade the outcome, never the self-report

The eval harness is the regression gate that keeps the platform genuinely buildable by agents, not just theoretically buildable. It runs realistic cold-start build tasks against a seeded workspace and grades what actually landed in the workspace afterward — never whether the agent's own narration claimed success. A "solver" is agent-written code executed through code mode against the real tool registry; in CI without a model, hand-scripted solvers exist purely to prove the graders themselves work, and with a model wired in, the identical harness scores model-generated code — only the solver changes, the grading never does.

The one signal that must always be zero

A trial only counts as ok when the solver ran cleanly, every declared grader passed, and zero duplicate resources were created. Duplicate detection — apps or workflows sharing a normalized name that shouldn't have been created twice — gets equal billing with functional correctness, because accidentally spawning a duplicate resource on every retry is treated as the single most damaging field failure this whole harness exists to catch.

Results are tracked as pass@k (did the solver succeed at all, across k attempts) versus pass^k (did it succeed reliably, every time) — a real reliability signal, not just "it worked once."

Continue