Evolving an Agent From Three Tools

I've spent a lot of time building machinery around coding agents. This experiment starts from the other end: read, write, bash, and a loop. I want to find out whether a fixed model can propose useful changes to that scaffold, and whether an independent evaluator can show that those changes helped.

I call the seed agent G000. It doesn't begin with a planning system, memory, subagents, a patch tool, or automated context summarization. It can inspect files, replace file contents, and run shell commands. That is enough to search a repository, edit code, and run tests, but the workflow is largely left to the model.

The pre-pilot journal was written on September 12, 2026. This entry pairs that design with the September 13–14 run records, checked on October 2. Those runs evaluate G000 under different configurations; they don't demonstrate improvement by a self-created successor.

The object I want to evolve is the scaffold around the model: its prompts, control flow, intermediate state, tool-use conventions, retry behavior, and context management.

The model weights stay fixed. Within an evaluation comparison, the task revisions, scoring rules, and relevant execution settings also need to stay fixed. Changing the model or benchmark at the same time would make it harder to tell what a scaffold mutation accomplished.

The initial model recorded in the design is DeepSeek-V4-Flash-0731, self-hosted on a DGX Spark cluster, with temperature 1.0 and top-p 0.95. The pre-pilot plan used maximum reasoning. The completed DeepSeek pilot mixes max and high reasoning across serving routes. An internal-route low control and a GLM 5.3 reference also ran. These are configuration comparisons using G000, not successor agents.

I want meaningful code changes proposed by the model rather than random edits. A candidate might introduce task-state tracking, a better editing strategy, a verification step, or a recovery policy. Those are possibilities to test, not features that have already emerged.

The experiment separates three responsibilities:

• self-creation contains the agent scaffold and fixed runtime.
• ledger-bench contains the Ledger application, task snapshots, prompts, and grading contracts.
• bench-eval contains the generic execution, validation, and scoring machinery.

The execution environment lets the agent attempt a task. A separate clean-room scoring environment evaluates its candidate. The agent's final message is not the success criterion.

Scoring checks both fail-to-pass tests, which represent the behavior the task asks it to repair, and pass-to-pass tests, which protect behavior that already worked. Fixing the target failure while breaking the rest of the application should not count as success.

The first benchmark has eight tasks, with three capability-valid attempts per task. Each retained cohort therefore has 24 outcomes, plus the traces needed to inspect how the agent got there. The DeepSeek pilot and GLM reference each mix execution profiles, so a cohort isn't necessarily one uniform configuration. Transport failures stay in the operational record and are classified separately from coding-capability outcomes.

A higher score on one small benchmark would be a starting point. I'd also want to know whether the change repeats across trials, preserves correctness while reducing work, and transfers to tasks that weren't used to select it.

That's why the planned loop includes an archive of candidates and their lineage. Each candidate should preserve its parent, mutation, diff, evaluation artifacts, and selection decision. Keeping more than one useful candidate may matter if different changes help on different kinds of tasks.

Before that loop starts, I need a fitness signal I can interpret. The pilot was stronger than I expected, and the benchmark exposed some problems of its own. I describe those in What Building Ledger Eval Taught Me. What G000 Calibration Actually Shows records the completed pilots, their mixed profiles, and why I still need a homogeneous parent/child comparison and an L06 contract review.

The question that started this experiment is still open. I have evidence about the seed agent and the evaluator. Evidence of scaffold evolution comes later, if an independently evaluated successor earns it.

Sources: writeup/00-pre-pilot.md; result.json and logs/model.jsonl from pilot-001, pilot-001-reasoning-low, and pilot-001-reasoning-strong. Workspace revision a2467a7; G000 scaffold 8ac83ae. Local run artifacts inspected October 2, 2026.