Harness-of-Harness and HoH-lite: persistent development loops
Harness-of-Harness coordinates repeated planning, coding and independent QA around an evolving project. The project artifact and evidence persist between iterations while existing coding harnesses perform the work.
Page reviewed 4 Oct 2026 Source and review policy
Current implementation availability
Section titled “Current implementation availability”HoH-lite is now available. The project README at ae7cc6fb announces it on 24 September 2026 and links the lightweight implementation. The earlier “coming soon” statements in the source history below reflect the 4 September cutoff and are superseded for availability.
Public source improves inspectability; it does not independently reproduce the paper’s measurements or establish production readiness. This review checked the repository and release-status evidence, not a multi-day execution.
Persist the handoff before another development iteration
Section titled “Persist the handoff before another development iteration”The current HoH repository describes stable planner, developer and read-only QA roles operating on one evolving workspace. HoH-lite became available on 24 September; availability does not establish independent reproduction of the paper’s benchmark gains.
requirements → planner brief → developer artifact → read-only QA evidence ↑ │ └──────── committed handoff ───────┘Before the next iteration, preserve the artifact revision, brief revision, accepted capabilities, failed checks, environment/version coordinates and unresolved blockers in a handoff. Otherwise a restarted planner can confuse “developer said complete” with “QA verified complete,” or inadvertently schedule the same rejected change again.
| Handoff result | Next iteration’s treatment |
|---|---|
| QA verifies capability on exact artifact | Preserve it as a regression obligation |
| QA finds deterministic failure | Plan one bounded correction against retained evidence |
| QA cannot run required check | Keep capability unverified; record missing prerequisite |
| Process dies during development | Recover artifact/execution state before new work |
Start a trial on an isolated project with explicit budget and stop criteria. Compare artifact quality and evidence retention across iterations, rather than only counting loops. A UHP task server could execute a development step, but the handoff and project-acceptance policy remain outside its wire contract. The published Vanilla/HoH@3 table below keeps its original models, metrics and authors’ claims unchanged.
What persists between loops?
Section titled “What persists between loops?”Requirements + prior evidence ↓ Planner ↓ prioritized work Developer ↓ changed project QA tester ↓ verified behavior + remaining gapsNext iteration reads the project and evidence againThe research project separates these roles so development can build on demonstrated progress. The unit of improvement is the software artifact; the concept does not require rewriting the underlying harness or creating a new interoperability protocol.
Evaluate it for your project
Section titled “Evaluate it for your project”Before adopting a loop, identify its stop conditions, budget, retained state and acceptance criteria. Determine which actor can change the artifact, which checks are independent, and how a failed iteration is recovered without losing prior verified work. Inspect the HoH-lite instructions for its actual runtime dependencies instead of inventing a generic one-command installation.
Use the author-published results as hypotheses about the tested harness/model combinations. Do not extrapolate a game’s iteration count into a promise about an unrelated application. UHP can supply an external execution boundary in a separately implemented system; no standard UHP-to-HoH binding or released HarnessRouter integration is established here.
Try the published HoH-lite workflow
Section titled “Try the published HoH-lite workflow”The HoH-lite README at ae7cc6fb requires Python 3.12+, Godot 4, and the selected harness CLI and credentials. Its bundled GameCraft example also needs an external benchmark checkout, Node.js/npm, bubblewrap, Xvfb, xdotool and FFmpeg for MCP and replay. Follow that README’s editable installation, local .env setup and gameloop-godot-mcp-vendor prepare step before running a task. An unrelated application needs its own benchmark/project adapter; this example does not make arbitrary projects runnable unchanged.
From hoh-lite/, after setup, replace the benchmark and task paths with real local inputs:
gameloop --config configs/harnesses/codex-gpt-5.5.json \ --bench /path/to/benchmark \ --task tasks/your-task \ --run-id first-trial \ --attempts 1The published runner accepts --attempts 1 to limit the trial to one outer iteration; that iteration still invokes the configured roles and can incur model usage. This command is source-checked, not executed here.
| After the trial | What to inspect or preserve |
|---|---|
| Compare with the paper | Read the run’s reproducibility.json: observed tool versions, source/benchmark revisions, role bindings and configuration hash. The paper used Codex CLI 0.142.5; the default profile accepts the installed CLI rather than guaranteeing that version. |
| Continue a previous run | Keep the same --task, pass --resume-from runs/first-trial, and choose a new --run-id. Automatic resume selects the latest completed valid loop. |
| Recover after interruption | Preserve the old run, including partial work. Automatic resume leaves the interrupted loop there; it does not claim that partial work passed QA or became the new seed. |
Reusing an existing run ID fails before its receipts or artifacts can be overwritten. Inspect the retained evidence before starting another loop; a fresh run ID preserves provenance, but does not itself establish acceptance.
Historical source notes — baseline reviewed 4 Sep 2026
The following evidence preserves the earlier review and its links. “Current”, “stable” and “main” inside this section refer to that historical cutoff; they are not new release or compatibility claims. Use the guidance above for the current scope.
Status first
Section titled “Status first”Harness-of-Harness (HoH) is a research framework for long-horizon autonomous software development introduced in arXiv preprint 2609.01481, submitted on 1 September 2026. The project builds on existing coding-agent harnesses rather than defining a new agent protocol. Its public experiments use Codex with GPT-5.5 (high), OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3 across GameCraft-Bench, FrontierSWE and ProgramBench.
The current public repository and project page expose the paper, benchmark results, demos and development trajectories. The repository explicitly says HoH-lite, a lightweight implementation of the core workflow, is coming soon. This guide therefore treats HoH as published research and architecture evidence, not as a generally released software package or independently replicated production system.
What Harness-of-Harness actually does
Section titled “What Harness-of-Harness actually does”HoH keeps an evolving software project and an evidence record across repeated iterations while coordinating three stable responsibilities:
- Project Planner — reads the requirements and prior evidence, then writes a prioritized development document without modifying the artifact.
- Developer — implements that development document in the shared project workspace while preserving already verified behavior.
- QA Tester — independently inspects and tests the updated artifact, recording verified behavior and remaining gaps while staying read-only with respect to the artifact.
The resulting artifact state and evidence state become the input to the next iteration. The paper describes this as a planning–coding–testing loop with small verifiable increments, independent evaluation, structured outputs, progressive disclosure of deliverables/tools/skills and versioned project history.
The architectural distinction is important: HoH does not require the base coding-agent harness to become a new protocol endpoint. It repeatedly uses an existing harness as the execution substrate while the project evolves from loop to loop.
Requirements / PRD │ ▼Project Planner │ development plan ▼Developer on an existing coding-agent harness │ updated artifact ▼Independent QA Tester │ evidence + unresolved gaps └───────────────┐ │ next iteration ▼ Project PlannerPublished benchmark evidence
Section titled “Published benchmark evidence”The project reports Vanilla versus HoH@1–3 results for three harness/model pairs. The table below reproduces only the published Vanilla and HoH@3 endpoints so the direction and scale of the measured change stay visible without turning the guide into a leaderboard.
| Harness + model | Metric | Vanilla | HoH@3 | Reported absolute gain |
|---|---|---|---|---|
| Codex + GPT-5.5 (high) | GameCraft-Bench Overall | 49.58 | 71.52 | +21.93 |
| Codex + GPT-5.5 (high) | FrontierSWE Dominance | 44% | 71% | +27 pp |
| Codex + GPT-5.5 (high) | ProgramBench Pass Rate | 60.41 | 66.50 | +6.09 |
| OpenCode + DeepSeek-V4-Pro | GameCraft-Bench Overall | 26.90 | 48.98 | +22.08 |
| OpenCode + DeepSeek-V4-Pro | FrontierSWE Dominance | 25% | 44% | +19 pp |
| OpenCode + DeepSeek-V4-Pro | ProgramBench Pass Rate | 45.27 | 57.56 | +12.29 |
| Pi + MiniMax-M3 | GameCraft-Bench Overall | 42.16 | 58.78 | +16.62 |
| Pi + MiniMax-M3 | FrontierSWE Dominance | 35% | 64% | +29 pp |
| Pi + MiniMax-M3 | ProgramBench Pass Rate | 35.83 | 52.68 | +16.85 |
Across all reported benchmark settings, the paper summarizes an average relative gain of 52.25% and a maximum relative gain of 82.86% after three iterations. Those are author-reported aggregate results for this experimental design, not universal performance guarantees for the underlying harnesses.
Multi-day case study
Section titled “Multi-day case study”The project also reports Fusepoint, a single-player narrative first-person shooter built from an empty workspace containing only a product requirements document. The project page says HoH ran for more than 70 iterations without a human in the loop and produced a human-playable game with implemented core mechanics, storyline, visuals and audio.
This case is significant because the unit of autonomy is not one coding turn or one benchmark patch. HoH attempts to preserve verified progress and evidence over many repeated development cycles, making long-horizon artifact continuity itself part of the harness-level architecture.
The public repository links the Fusepoint trajectory so the development history can be inspected. The project also reports another game, Mournlight, at more than 80 loops, but the paper’s principal public multi-day example is Fusepoint.
HoH vs harness self-evolution
Section titled “HoH vs harness self-evolution”The name can be misleading. HoH is not primarily a system for rewriting the underlying coding-agent harness. The paper positions it around continual improvement of the software artifact while operating on existing harnesses. Planner, developer and tester roles, runtime conditions and the base harness/model pairing are held stable within an experimental run while the artifact and evidence evolve.
That makes HoH different from work whose objective is to optimize prompts, tools, skills or execution logic inside the harness itself. The “harness-of-harness” layer is an orchestration envelope around repeated harness executions, not evidence that the base harness has become self-modifying.
HoH, UHP and FrontierHarness answer different questions
Section titled “HoH, UHP and FrontierHarness answer different questions”| Layer | Primary question | Unit being controlled or measured |
|---|---|---|
| Harness-of-Harness | How can repeated harness executions improve one software artifact across many autonomous iterations? | Planner/developer/QA loop, project artifact and evidence state |
| UHP | How can a client execute a selected complete harness through one interoperable HTTP contract? | Client ↔ UHP server ↔ harness task/session execution |
| FrontierHarness Eval | How differently do frozen harness/model/runtime configurations perform on the same benchmark tasks? | Harness + model + runtime configuration |
These layers can be complementary without being interchangeable. HoH provides a long-horizon orchestration strategy; UHP provides an external execution contract; FrontierHarness provides comparative evaluation evidence. None of the three automatically establishes the other two.
Why it matters for harness engineering
Section titled “Why it matters for harness engineering”HoH adds a useful architectural pattern to the emerging harness-engineering vocabulary: continual artifact improvement through evidence-grounded repeated harness execution. The key design choices are not simply “run the coding agent longer.” The framework explicitly separates planning, implementation and independent QA, carries evidence forward, constrains work to bounded increments, and preserves versioned history so later loops can distinguish verified capability from unresolved work.
For interoperability work, the lesson is narrower. A protocol such as UHP can standardize how a harness is addressed and executed, but long-horizon project orchestration still needs its own policy for iteration boundaries, evidence retention, role separation, acceptance criteria and recovery. HoH is one research design for that layer; it does not alter the UHP specification.
Evidence limits
Section titled “Evidence limits”- The currently exposed primary artifact is an arXiv preprint plus project/repository materials; this guide does not treat it as a standards-track document or certification.
- The authors state that HoH-lite is not yet publicly released, so complete implementation-level reproducibility is not available from a packaged HoH runtime at this cutoff.
- The published benchmark covers three harness/model pairs and three software-development benchmarks; it does not establish behavior across arbitrary harnesses, models or domains.
- The benchmark and case-study results are author-published. No independent replication was verified in this research pass.
- The reported gains measure final artifact quality under the HoH experimental setup. They do not measure UHP conformance, protocol interoperability or native UHP adoption.
Related pages
Section titled “Related pages”Read harness composition and subagent delegation for child-agent and cross-harness runtime topologies, FrontierHarness Eval for controlled harness benchmarking, and UHP conformance for protocol-contract verification. The individual Codex, OpenCode and Pi pages track current integration/upstream evidence separately from the versions and models used in the HoH paper.