What G000 Calibration Actually Shows

The run records give me three scores on Ledger Eval v1: 18/24 for the low-reasoning control, 20/24 for the DeepSeek pilot, and 21/24 for the GLM reference. The request logs change how I interpret that ordering. The pilot mixes reasoning settings and serving routes, and the reference mixes serving profiles too.

I initially described the 20/24 result as a normalized high-reasoning baseline and the strong reference as still planned. Those descriptions were wrong. This entry corrects them using the September 13–14, 2026 result records and request logs, inspected on October 2.

These runs evaluate G000 under different configurations. They don't demonstrate a self-created successor or scaffold improvement.

The DeepSeek runs use the alias deepseek-v4-flash, identified in the design notes as DeepSeek-V4-Flash-0731. The retained samples are:

• DeepSeek pilot: 14 original external-route attempts at max reasoning, plus ten internal-route replacements at high reasoning. Together they scored 20/24.
• Low control: 24 internal-route attempts at low reasoning, scoring 18/24.
• GLM 5.3 reference: 19 external-route attempts using glm-5.3-cloud, plus five through local Ollama using glm-5.3:cloud. Both profiles recorded high reasoning. Together they scored 21/24.

The pilot and low control ran on September 13. The GLM reference ran on September 13–14. The records pin the same G000 scaffold, Ledger Eval v1, evaluator revision, and canonical revision across these cohorts. Changing the model for the reference is calibration, not agent evolution.

The pilot's raw history contains 34 attempts, including ten transport-invalid attempts. The reference contains 29 attempts, including five transport errors. Its 24 valid attempts comprise 23 completed runs and one budget-exhausted run. The low control has 24 transport-valid attempts, including two benchmark timeouts.

Benchmark timeouts and budget exhaustion remain capability-valid outcomes under the defined constraints. Transport-invalid attempts remain in the operational record but are excluded from these capability scores.

A full pass requires both the intended fail-to-pass behaviors and the protected pass-to-pass behaviors to pass.

Totals: low 18/24 (75.0%); pilot 20/24 (83.3%); GLM 21/24 (87.5%).

The pilot's two additional passes relative to low are both on L07 reconciliation. But the 8.3 percentage-point aggregate difference cannot be attributed solely to reasoning effort: reasoning and serving route both vary across the pilot sample.

Three attempts per task also don't establish statistical significance or a stable improvement rate. These are repeated trials on eight tasks, not 24 independent kinds of coding work. I haven't established equivalence between the two GLM serving profiles, so its result is a completed reference pilot with that limitation, not a clean upper anchor.

All three low-reasoning L07 attempts passed 28 of 29 fail-to-pass checks and preserved all 157 pass-to-pass checks. The missing behavior differed: attempts one and three missed unexpected CLI runtime-error handling; attempt two missed accepting a string path in the bank CSV parser.

The repeated 28/29 score therefore doesn't identify one repeated failure of matching semantics. It records different integration edge cases that each prevented a full pass.

The pilot's L07 replacements used high reasoning on the same internal serving route as the low control and passed 2/3. That subset is a more interpretable reasoning comparison than the mixed aggregate. It is suggestive, with only three trials per setting. The GLM reference passed L07 3/3.

Account hierarchy failed all three valid attempts in every cohort. The GLM attempts fixed 3/11, 3/11, and 0/11 target behaviors while preserving all 175 regression checks. Those were completed attempts, not transport failures.

The candidate diffs make the failure less simple than an intrinsic model limit. Several DeepSeek attempts added parallel prefix-total helpers while leaving the existing balance lookup and rollup paths unchanged. Some GLM attempts also left parts of the original rollup path unchanged.

The prompt describes the desired hierarchy behavior without naming the existing balance helper or the CLI switch expected by scoring. Visible code provides rollup interfaces and report-flow clues, but the consistency between the prompt, integration points, and graded contract warrants review.

I can't conclude from these results that L06 is invalid or that the models are incapable of hierarchy reasoning. I can conclude that the submitted integrations didn't satisfy its grading contract. That distinction matters before L06 becomes selection pressure for descendants.

The next measurement is a fresh homogeneous parent baseline, followed by a candidate under the same model, reasoning setting, serving profile, runtime, budgets, and scorer. The existing mixed pilot cannot stand in for that controlled comparison.

I also need the L06 fairness review. If it exposes a defect, the correction should receive a new benchmark version rather than silently changing the target after observing scores.

Partial usage telemetry and mixed profiles also limit an efficiency comparison. I don't have a reliable matched token-cost ratio to attach to these scores. Earlier max-reasoning timeout observations don't supply a controlled low/high/max comparison either.

The scaffold-evolution question remains open. These runs tell me where to inspect the agent and evaluator before testing a successor; the benchmark construction lessons explain why I want those checks.

Sources: result.json and logs/model.jsonl from pilot-001, pilot-001-reasoning-low, and pilot-001-reasoning-strong; L06 candidate diffs, task prompt, and visible code. Run records pin G000 scaffold 8ac83ae, evaluator 1e38a43, and Ledger canonical d87bf7d. Workspace revision a2467a7; local run artifacts inspected October 2, 2026.