FrontierHarness Eval: read the benchmark before choosing a harness
FrontierHarness Eval compares harness configurations while holding a model and task set fixed. Use it to form a testable selection hypothesis, then evaluate your own workload and deployment constraints.
Page reviewed 4 Oct 2026 Source and review policy
What this evidence can answer
Section titled “What this evidence can answer”The public repository at e837a70b, rechecked 4 October 2026, still describes v1.0: 30 tasks, 12 configurations, 9 harnesses and 360 runs using Kimi K3. Those are frozen experimental coordinates, not a ranking of today’s latest software.
A practical reading order is: benchmark definition, version snapshot, scoring method, task-level results, then headline chart. A configuration that looks attractive on average may fail the particular task family your product needs.
Read one-task differences before declaring a winner
Section titled “Read one-task differences before declaring a winner”In the frozen 30-task result, 66.7% represents about 20 passes and 63.3% about 19. The gap is roughly one task, not evidence that the first configuration will dominate every deployment. The published normalized results and benchmark definition retain one canonical evaluation per cell; this guide does not add confidence intervals or repeated-trial evidence that the study did not publish.
frozen aggregate → relevant task slice → failure evidence → local hypothesisFor a repository-maintenance product, first identify which task families involve the same edit/build/test sequence as your work. Compare where configurations disagree, then inspect verifier failures and costs for those cells. A fast result on an unrelated task family should not outweigh failure on your required workflow.
| Selection question | Evidence to retain |
|---|---|
| Is the pass difference concentrated? | Per-task outcomes, including failures |
| Does low cost survive unsuccessful attempts? | Published failure-inclusive cost basis |
| Are results repeatable? | New repeats, seeds and variance in a separate study |
| Does operational policy fit? | Permissions, recovery and artifact requirements |
The last two rows need new evidence from your deployment; the frozen leaderboard cannot supply it. Keep that new comparison separate from both the original result set and protocol conformance.
Turn the result into an evaluation plan
Section titled “Turn the result into an evaluation plan”| Before choosing | Record |
|---|---|
| Workload fit | Which benchmark tasks resemble your real work, and which do not |
| Version fit | Harness build, model, provider route, tool setup and runtime |
| Cost basis | Whether failures are included and how cache usage is priced |
| Success evidence | Artifact/verifier result, not only the agent’s final message |
| Repeatability | Repeats or seeds, timeouts and missing/failed runs |
| Operational fit | Permissions, recovery, artifacts and observability |
Keep the published measurements unchanged when a project ships a new version. If you run a new comparison, give it a separate result set and record the changed conditions; do not splice fresh cells into the frozen table and retain its original label.
This guide has not rerun the 360 trials or independently reproduced the rankings. UHP conformance asks whether an execution server implements a protocol; HarnessRisk asks a safety question. Neither is established by high task pass rate.
Historical source notes — baseline reviewed 3 Sep 2026
The following evidence preserves the earlier review and its links. “Current”, “stable” and “main” inside this section refer to that historical cutoff; they are not new release or compatibility claims. Use the guidance above for the current scope.
Status first
Section titled “Status first”FrontierHarness Eval v1.0 is a frozen coding-agent harness benchmark launched by Runta on 1 September 2026. Its public definition fixes one model — Kimi K3 through Fireworks — across 9 harnesses, 12 configurations, 30 software-engineering tasks and 360 evaluations. The benchmark repository publishes the task definitions, normalized results and harness-version snapshot used for the run.
The central result is architectural rather than protocol-level: changing the harness while holding the model, task set and execution environment constant produced materially different pass rates, cost, cache behavior and wall-clock time. The published pass rates span 50.0% to 66.7%, while the source’s effective-cost-per-pass values span about 17.5×, from $1.05 to $18.34.
What the benchmark actually measures
Section titled “What the benchmark actually measures”FrontierHarness v1.0 uses a controlled execution design intended to expose harness effects without changing the model between configurations:
- 30 tasks: 21 Terminal-Bench 2.1 tasks and 9 DeepSWE v1.1 tasks.
- 12 configurations across 9 harnesses: DeepSeek Harness is evaluated in four presets; the other harnesses have one configuration each.
- 360 cells: one canonical evaluation for each task/configuration pair.
- One model: Kimi K3, served through Fireworks.
- Fresh runtime state: each formal trial starts from a restored Runta golden checkpoint with the same vCPU, memory, disk size, disk contents and memory state.
- Deterministic pass/fail scoring: task verifiers determine success.
- No warm benchmark rehearsals: formal tasks are not used for debugging; warm and smoke labels are excluded from the frozen definition.
- Comparable cache accounting: first-turn cache reads are repriced consistently across harnesses.
The launch article also says all 12 configurations for a task run in the same job and that the benchmark uses one attempt per cell rather than best-of-N retry selection.
Frozen harness versions
Section titled “Frozen harness versions”The public version snapshot was captured on 22 August 2026. That matters because several projects have released newer builds since then. The leaderboard should therefore be read as a controlled historical experiment, not as a comparison of each project’s latest stable release on 3 September.
| Harness / configuration | Frozen version |
|---|---|
| Codex | 0.148.0 |
| Claude Code | 2.1.237 |
| DeepSeek Harness — Creator | 0.1.0-rc.8 |
| DeepSeek Harness — Minimal | 0.1.0-rc.8 |
| DeepSeek Harness — PTC | 0.1.0-rc.8 |
| DeepSeek Harness — Standard | 0.1.0-rc.8 |
| Pi | 0.84.2 |
| Oh My Pi | 17.4.0 |
| Kimi Code | 0.37.2 |
| Exo Harness | 0.1.0 |
| OpenCode | 1.18.19 |
| Hermes | 0.20.4 |
Published v1.0 results
Section titled “Published v1.0 results”The table below uses the public benchmark’s pass rate, effective_cost_per_pass, typical median cache-hit rate and median runtime. These are results for the exact frozen configurations above, not universal properties of the harness products.
| Configuration | Pass rate | Effective cost / pass | Cache, median cell | Median time |
|---|---|---|---|---|
| Codex | 66.7% | $3.47 | 88.0% | 6m 43s |
| DSH Creator | 63.3% | $3.28 | 84.3% | 6m 44s |
| Claude Code | 63.3% | $18.34 | 67.8% | 9m 38s |
| Pi | 60.0% | $2.43 | 79.4% | 7m 33s |
| DSH PTC | 60.0% | $4.58 | 87.2% | 7m 44s |
| DSH Standard | 60.0% | $3.46 | 86.5% | 6m 17s |
| Oh My Pi | 56.7% | $4.75 | 82.2% | 6m 46s |
| Kimi Code | 56.7% | $3.65 | 88.0% | 7m 56s |
| DSH Minimal | 56.7% | $4.72 | 84.6% | 5m 41s |
| Exo Harness | 53.3% | $1.05 | 70.3% | 6m 17s |
| OpenCode | 50.0% | $3.24 | 78.4% | 6m 27s |
| Hermes | 50.0% | $2.90 | 85.9% | 6m 58s |
The source reports 209 successes and 151 failures across the 360 cells. Codex leads pass rate, DSH Minimal leads median runtime, and Exo Harness has the lowest effective cost per pass in the frozen dataset.
How to read the cost numbers
Section titled “How to read the cost numbers”FrontierHarness deliberately prices failures into the efficiency picture. Its normalized dataset exposes effective_cost_per_pass, while the launch article describes the same headline values as cost per pass and discusses the cost of failed work rather than filtering failures away.
That distinction matters. A configuration can fail cheaply, solve fewer tasks and still have attractive spending characteristics for retry-heavy workflows; another can solve more tasks but spend much more on long unsuccessful trajectories. Pass rate and cost therefore answer different questions.
The source also warns that cache-hit percentage is not itself cost. One highly cached but very long trajectory can still consume more tokens and money than a short cache miss.
Important limitations
Section titled “Important limitations”FrontierHarness is unusually useful because it controls several confounders, but it does not isolate the harness as an abstract independent variable.
- Single-model benchmark. Every configuration uses Kimi K3. The result measures each harness’s interaction with that model and gateway path, not how the harness performs across model families.
- Software-engineering scope. The 30 tasks are terminal/software-development tasks. The project explicitly says v1.0 may not generalize to other knowledge work.
- Frozen old versions. The version snapshot was captured on 22 August, while the public launch occurred 1 September. Current releases can differ materially.
- Harness-model fit remains a confounder. Runta specifically notes that Claude Code’s high cost may reflect its interaction with Kimi K3’s implicit cache behavior rather than a harness-only defect.
- One formal attempt per cell. The benchmark intentionally avoids retry/best-of-N effects; production systems may choose different retry policies.
- Operator-published evidence. Runta designed, executed and hosts the benchmark. The public task definitions and normalized results make the work inspectable, but this guide does not treat it as independent certification of any harness vendor.
The project’s stated v1.1 direction is a broader harness × model matrix, which would make model/harness interaction effects more directly observable.
FrontierHarness vs UHP conformance
Section titled “FrontierHarness vs UHP conformance”| Question | FrontierHarness Eval | UHP conformance |
|---|---|---|
| Primary purpose | Compare task quality and execution efficiency across harness configurations | Verify implementation of the UHP client/server contract |
| Unit under test | Complete harness + model + gateway/runtime configuration | UHP server behavior at a declared conformance class |
| Main outcomes | Pass/fail task result, cost, cache behavior, runtime | Protocol checks for discovery, tasks, streaming, sessions, files, sharing and related semantics |
| Version sensitivity | Frozen harness/model/runtime snapshot | Pinned UHP version and conformance-suite shape |
| Cross-harness ranking | Yes, within the benchmark design | No |
| Native UHP adoption evidence | No | Only when a project actually implements/claims the UHP contract with evidence |
HarnessRouter’s published UHP conformance evidence and FrontierHarness’s HarnessRouter-adjacent harness measurements therefore belong to different evidence classes. They can complement one another, but one must not be used as a proxy for the other.
Why it matters for harness engineering
Section titled “Why it matters for harness engineering”FrontierHarness provides public quantitative evidence for a point often hidden by model-only benchmarks: the harness is a measurable part of the system. Tool-loop behavior, context management, cache interaction, prompting, stopping conditions and runtime policy can change cost and outcomes even when model weights and task state are held constant.
For UHP, that strengthens the case for keeping protocol interoperability separate from performance evaluation. A protocol can make different harnesses addressable through one contract; it does not make those harnesses behaviorally equivalent. Interoperability answers “can the client run it through the same contract?” while a harness benchmark asks “how does this complete configuration perform on this workload?”
Related pages
Section titled “Related pages”Read UHP conformance for protocol verification, harness composition for internal orchestration topologies, and the individual Codex, Claude Code, DeepSeek Harness, Pi, OpenCode and Hermes pages for current upstream/integration status that should not be confused with FrontierHarness’s frozen 22 August version snapshot.