What Building Ledger Eval Taught Me

Before I could use Ledger to select better agents, I had to check whether it could reliably distinguish correct code from plausible code. Building a working application and building a useful benchmark turned out to require different kinds of scrutiny.

Ledger is a small synthetic accounting repository. Its eight tasks cover exact monetary arithmetic, CSV negatives, journal validation, currency separation, point-in-time reports, account hierarchies, reconciliation, and a Decimal refactor. The domain gives me explicit contracts to test without using private client code or financial records.

The benchmark construction findings come from the Ledger Eval v1 pilot notes. The run-history interpretation below is checked against the September 13–14, 2026 result records and request logs, inspected on October 2. These are G000 evaluations, not evidence of scaffold evolution.

The first task concerns large values that carry cents. Converting them through float arithmetic can lose exactness.

The balance implementation had 100% branch coverage. I replaced its exact accumulator with float arithmetic, and none of the original canonical tests failed.

Every branch had run. The tests still hadn't checked the property that mattered.

The discriminating exact-cent tests became part of canonical. For this benchmark, canonical needs to assert the invariants that the task mutations deliberately violate. Otherwise, a passing score can tell me more about a gap in the tests than about the agent's solution.

L07 asks the agent to rebuild reconciliation from a written specification. The original description included a tiebreak chain but didn't clearly say whether matching happened row by row or across all candidate pairs.

Canonical used global greedy selection. Row-order greedy selection was also a defensible reading of the prompt. An agent could choose it and fail while reasonably believing it had followed the instructions.

I clarified the contract: selection is global and greedy, and it deliberately does not solve an optimal assignment problem. A separate test distinguishes those behaviors when greedy selection strands a match that an optimal assignment could preserve.

That makes the unusual requirement explicit. It also makes a failure more interpretable: did the agent satisfy the written contract, rather than guess the algorithm I happened to implement?

L08 asks for a cross-cutting move from float to Decimal arithmetic. Its original six failing tests looked like a reasonable grading surface.

A locality audit found a shortcut. Converting Money, balance accumulation, and report totals was enough to pass those tests while leaving float arithmetic in journal aggregation, validation, and reconciliation comparisons.

The benchmark was accepting a partial implementation.

I added three behavioral checks: 0.10 + 0.20 - 0.30 must balance exactly, journal currency aggregation must produce exact 0.30, and reconciliation must distinguish adjacent large cent-bearing values that collapse through float conversion.

The resulting canonical-v6 has 186 tests. L08 has nine fail-to-pass tests and 177 pass-to-pass tests. Those counts describe the final grading surface; they don't establish that every possible incomplete refactor is caught.

The initial 24 G000 attempts produced 14 scored passes and ten failures associated with socket or transport timeouts. Some interrupted attempts had already produced correct candidate code.

I couldn't use that raw score as a clean measure of coding capability.

The ten transport-invalid attempts were retained and replaced for capability measurement using the internal Spark serving path. The replacements didn't reproduce the same transport-error pattern. They did include benchmark timeouts, request-budget exhaustion, and incorrect completed solutions. Those remain capability-valid failures under the defined task constraints.

The retained pilot result was 20/24, or 83.3%, across three capability-valid attempts per task. It combines 14 original external-route runs at max reasoning with ten internal-route replacements at high reasoning. It is not a homogeneous high-reasoning baseline. The score uses a different set of attempts from the original 24-attempt operational record; the ten transport-invalid attempts remain in its 34-attempt history. Their replacement doesn't make those reliability failures disappear or identify one serving component as their cause.

The construction work also exposed a multi-currency fixture that combined USD and EUR into one meaningless total. Canonical was corrected to render separate currency sections. The reference implementation needed auditing too.

All eight final tasks passed the benchmark validation suite. The GLM 5.3 reference also completed: 21/24 capability-valid attempts passed, while L06 account hierarchy remained 0/3. That reference used two serving profiles whose equivalence I haven't established. Its completion doesn't settle L06's fairness or turn the mixed DeepSeek pilot into a controlled baseline.

Validation checks the task mechanics. Before using these results to select descendants, I still need to review L06's integration contract and compare parent and child under one fixed execution profile.

The baseline's failure pattern is the subject of What G000 Calibration Actually Shows. For the experiment that needs this evaluator, start with Evolving an Agent From Three Tools.

Sources: writeup/01-ledger-pilot.md; result.json and logs/model.jsonl from pilot-001 and pilot-001-reasoning-strong. Workspace revision a2467a7; G000 scaffold 8ac83ae; run records pin evaluator 1e38a43 and Ledger canonical d87bf7d. Local run artifacts inspected October 2, 2026.