FrontierHarness

One model. Twelve harnesses. Thirty real agentic tasks.

Explore the live results →

Same model. Different harness. Very different outcome.

We ran the same Kimi K3 model through 12 coding-agent harnesses on the same 30-task benchmark. The harness alone changed task success, cost, cache behavior, and time.

The leaderboard is only the start. Failed runs, total cost per pass, cache behavior, and task-level results are available in the interactive report.

What is in this repository

.
├── benchmark.json              # Public benchmark definition
├── metadata/
│   ├── difficulty.json         # Difficulty assignments and source methodology
│   └── harness-versions.json   # Harness versions used for the run
├── results/
│   ├── eval-data.json          # Normalized aggregate and task-level results
│   └── scatter-data.json       # Pareto chart coordinates and frontier data
└── tasks/<task>/
    ├── instruction.md          # Prompt shown to every harness
    └── task.toml               # Public task metadata and environment definition

The repository intentionally contains results and task definitions only. Internal infrastructure, credentials, runtime identifiers, private evidence, solutions, and deployment configuration are not included.

Benchmark design

• 30 tasks: 21 Terminal-Bench tasks and 9 DeepSWE tasks
• 12 harnesses: Claude Code, Codex, four DSH modes, Exo Harness, Hermes, Kimi Code, Oh My Pi, OpenCode, and Pi
• One model: Kimi K3, served by Fireworks
• 360 cells: one canonical result for every task × harness pair
• Deterministic scoring: verifier-based pass/fail outcomes
• Comparable cost: first-turn cache reads repriced consistently across harnesses

See benchmark.json for the public benchmark definition and results/eval-data.json for the complete normalized result set.

Use the data

jq '.harnesses[] | {name, successful, effective_cost_per_pass}' results/eval-data.json

Every task directory contains the exact public instruction and task metadata used by the benchmark.