Harness Handbook: Finding What an Agent Must Change

A diagnosis can name the behavior that failed and still leave a coding agent searching the wrong files. Harness Handbook builds a behavior-centered route from a change request to every source location that may need to move.

Suppose an agent repeatedly loses its last valid plan after a tool timeout. The diagnosis sounds precise: checkpoint state before retrying. In a toy harness, that may be one function. In a production system, the same behavior can cross a tool schema, runtime middleware, state-reset logic, a fallback path, session recording, and documentation. Fix the obvious branch and the agent may still fail somewhere colder.

That is the problem this paper isolates. Software repositories are organized by files, functions, and modules. Change requests are expressed as behavior: preserve state across compaction, mask secrets before recording, or keep two shell tools consistent. The missing translation between those two views is behavior localization—finding every implementation site involved in the requested behavior.

The missing step between diagnosis and an edit

Agentic Harness Engineering and Self-Harness begin with failure traces, mine weaknesses, and propose harness changes. Harness Handbook studies the step between the weakness and the patch. Once the diagnosis says “retry state becomes stale,” how does the planner find initialization, resets, reads, writes, mirrored implementations, and documentation without opening an entire repository?

Ordinary search helps when the request and code use the same words. It is weaker when a behavior is distributed or when its rare path has a different name. A repository map has the opposite problem: it can show the file tree without explaining which pieces cooperate to produce one runtime behavior. The paper reorganizes that implementation knowledge around what the harness does.

This is a prerequisite for a bounded edit. The loop cannot honestly claim to “change one thing” if the thing is a behavior implemented in six places and it edits only three.

How the handbook turns behavior into source locations

The system combines deterministic static analysis with organization assisted by a large language model (LLM). Static analysis extracts files, functions, signatures, source ranges, call edges, and named external boundaries. A model then groups those fixed facts into runtime stages and writes the behavioral explanations. It may explain the graph, but it may not invent or rewrite source identities.

The resulting handbook has two connected views:

• An L1–L3 document tree. L1 describes the whole system, L2 describes an execution stage, and L3 links that stage to a concrete function, source region, or file.
• A state-register view. It records state that crosses stages and lists where each register is read and written.

The planner navigates those views with Behavior-Guided Progressive Disclosure, or BGPD. It starts with the behavior, narrows to relevant stages, follows shared state and call relations, then opens the current repository to verify every candidate location before producing an edit plan.

That last boundary matters. The handbook is compressed, model-assisted memory. The repository is evidence. If a locator no longer resolves after a change, the system freezes that entry and excludes it from localization rather than guessing where it moved.

How the map survives the code changing

A generated architecture document normally starts decaying as soon as the first patch lands. Harness Handbook treats synchronization as part of the modification workflow. Every non-empty diff triggers an update. The system reparses changed source, aligns old and new units using fingerprints or file hashes, and refreshes only the affected entries when the larger stage structure remains valid.

If the structure no longer fits, it rebuilds the relevant organization using the same leaf mode and stored configuration. Unparsed or unclassified content stays visible in a coverage record instead of receiving a confident fictional description.

This gives the paper its strongest connection to persistent agent improvement. The map is not a static explanation prepared for one edit. It is state that changes with the artifact and can guide the next round. Yet synchronization alone is not learning: the paper still needs a protected performance comparison before it can claim that this persistent state makes an agent better at improving itself.

What the experiment tested

The evaluation isolates planning. A read-only planner powered by DeepSeek-V4-Pro receives one of 30 behavior-driven modification requests for each of two open-source harnesses. The baseline explores the repository directly. The assisted arm receives the handbook as a navigable skill and must verify handbook locations against the same repository snapshot. Requests, model, tools, decoding settings, and planning contract stay fixed.

The request set is evenly divided among three kinds of localization problem. Query requests alter an existing rule. Cross-file requests add a capability that must pass through several layers. Search-hostile requests hide relevant code in mirrored implementations, fallback branches, or cold paths. Difficulty labels separately capture whether the change touches one obvious site or follows indirect coupling across the system.

Three LLM judges—GPT-5.5, Opus 4.8, and DeepSeek-V4-Pro—score localization, scope control, and reasoning. The paper also compares predicted files and symbols with independent reference plans generated by GPT-5.5 and Opus 4.8. Planning token use measures the cost visible to the planner, including handbook and source context.

What the results actually show

Handbook assistance improved planning while consuming fewer planner tokens. On Codex, its overall plan win rate was 38.3% against 28.3% for direct repository exploration. On Terminus-2, it was 45.6% against 26.7%. These are win rates with ties, so the two arms are not expected to add to 100%.

Agreement with the stronger reference planners improved in every reported recall, precision, and F1 comparison. F1 combines precision and recall into one score. Across both file and symbol granularity, its gain ranged from 5.0 to 18.8 percentage points. Complete misses—plans with zero overlap against the reference—never increased and fell by as much as 25.9 points.

The direction also held across all three request types. The largest gains appeared where the paper's idea should matter most: scattered implementation sites, cross-module behavior, and code paths that keyword search struggles to expose. That consistency is more informative than any one headline number.

What the results do not show

The experiment stops at the plan. Its judges explicitly do not grade the final diff or execution correctness. We therefore do not know whether the additional locations produced patches that compiled, passed regression tests, improved task performance, or avoided new failures. Better localization is a plausible cause of better edits, but this paper does not measure that link.

The ground truth is also softer than a human-validated patch set. Large language models build the answer keys, generate the reference plans, and judge plan quality. Using multiple models and leakage-safe pristine source reduces dependence on one judge, but agreement with a stronger model is not the same as demonstrated code correctness.

The empirical base contains two harnesses and 60 designed requests, spanning six Python files at one end and 2,267 Rust files at the other. It does not establish transfer to other languages, architectures, real issue distributions, or repositories the handbook builder was not tuned around. Terminus-2 also begins with a trusted execution-stage skeleton, while Codex uses a coarser file-level representation inferred from the repository.

Finally, the reported cost reduction covers planner tokens. The paper does not provide an end-to-end comparison that includes initial handbook construction, static-analysis adapters, semantic organization, and repeated resynchronization. The map can make each plan cheaper while still requiring substantial infrastructure to create and maintain.

Where it fits in recursive self-improvement

Harness Handbook is not itself a recursive self-improvement (RSI) result. It does not diagnose a failure from task traces, execute a candidate, evaluate the modified harness, or promote a winner into the next round. The authors say that closing this autonomous loop is their next step.

Its contribution is narrower and still important. In the loop taught in Chapter 3, it belongs between diagnosis and proposal:

This addition becomes more valuable as the mutable artifact grows. A prompt needs no call graph. A production harness may have thousands of files, mirrored tool surfaces, and state that crosses several execution stages. At that scale, incomplete localization becomes a distinct failure mode: the loop understands the bug, writes a reasonable patch, and still leaves part of the old behavior alive.

What we should carry into the book

1. Name behavior localization as its own step. Diagnosis identifies a likely mechanism; localization connects that mechanism to every source site a complete edit must consider.
2. Organize mutable systems by behavior as well as files. A runtime behavior can cross schemas, state, control flow, fallback paths, and documentation.
3. Track shared state explicitly. Read-and-write registries expose distant dependencies that a top-down code read can miss.
4. Use progressive disclosure to protect context. Start with the system and stage, then reveal only the implementation detail the request needs.
5. Treat generated maps as memory, not evidence. Verify every locator against current source before planning an edit.
6. Resynchronize after every accepted diff. Improvement memory that drifts away from the artifact can make later rounds confidently worse.
7. Keep the evidence boundary visible. Planning metrics cannot substitute for compiled patches, regression tests, held-out task performance, or recursive gain.

Glossary