Bug Hunt Bench — which AI coding model fixes the most real bugs?

105 real bugs, hidden in two production codebases. Frontier coding models — GPT-6, Claude, Grok, Gemini, DeepSeek, Kimi, GLM and more — get one round per repo in their own agentic CLI (Codex CLI, Claude Code, Grok CLI, Antigravity CLI) to find and fix what they can. Every diff is graded blind against a withheld answer key. The score counts planted bugs only.

Live board: bughunt.productcompass.pm · Method, caveats and definitions · Raw data · Findings by wave

If the numbers save you a benchmark run of your own, star this repo — that is what keeps the bench findable, and new models are added as they ship.

Updated Sep 5, 2026 · 52 scored runs · 22 models · 39 of 105 bugs have never been fixed by any model.

Current leader: GPT-6 Astra at max effort — 48 / 105 (24/45 on repo 1, 24/60 on repo 2).

Best run per lab: OpenAI: GPT-6 Astra (max) 48 · Anthropic: Fable 5.1 (max) 43 · xAI: Grok 4.6 (xhigh) 27 · Google: Gemini 3.7 Flash (high) 22 · Moonshot AI: Kimi K3 (default) 21 · Z.ai: GLM-5.3 (default) 19 · Alibaba: Qwen3.8-Max (xhigh) 19 · Tencent: Hy4 Preview (high) 18 · Meta: Muse Spark 1.2 (xhigh) 17 · DeepSeek: DeepSeek V4-Flash (default) 14

Extras are real, unplanted defects a model fixed on the way; they are counted and never added to the score. Costs are token estimates at published list rates unless tagged bill (an actual invoice or credits delta) or floor (a reconstructed lower bound). default effort means the serving path had no working effort dial; ran lower marks a run whose CLI quietly replaced the requested tier. Wall clock is repo 1 plus repo 2 agent time, dependency install excluded.
7 superseded re-runs stay in the CSVs and on the live board but are left off this table.

What Bug Hunt Bench is

A benchmark of AI coding agents on the job they are actually sold for: reading an unfamiliar, real codebase and fixing what is wrong with it. Not a puzzle set, not a single-file task, not a synthetic repo.

• Two unrelated production codebases. Repo 1 is a ~28K-line TypeScript VS Code extension (45 planted bugs). Repo 2 is a ~60K-line React / TypeScript LMS on Supabase Edge Functions and Clerk (60 planted bugs). Nothing is shared between them but the language.
• The bugs are real. All 60 repo-2 bugs are regressions that actually shipped and were later fixed, each re-introduced from its own fix commit. On repo 1, 16 of the 45 are real reverted fixes and 29 were authored in the same style. The test suites were kept green when the bugs went in, so a passing suite points at nothing.
• Every model gets the identical prompt (repo 1, repo 2): find and fix as many planted bugs as you can, keep the checks green without weakening tests, write a BUGS_FOUND.md. One round per model per repo, no network, no git history, in the CLI its own vendor ships.
• The diff is the ground truth, not the report. A model's own claim of what it fixed is not scored. An independent judge model compares the resulting diff against the withheld answer key without knowing which model produced it.

How to read the score

• Fixed /105 is the only number in the score: planted bugs whose fix the judge verified in the diff. No partial credit.
• Extras are real defects a model fixed that nobody planted. They are real work — one model found 38 of them in a single wave — but the set has no answer key and no ceiling, so counting them would reward volume. They are tracked in their own column and never added.
• Claimed only means the model's report named a bug that its diff does not fix. It is the failure mode this bench was built to catch, and it is why reports are not graded.
• Cost is tagged by kind: a real bill, a token estimate at list rates, a reconstructed floor, or free. They are not interchangeable and none is a quote.
• Effort is the reasoning tier the run was served at, not the tier that was requested. Where a CLI silently ran a lower tier the row says so; where an aggregator's effort parameter provably does nothing the row says default.

FAQ

Which AI coding model fixes the most bugs?

The table above is the answer as of its date, and the live board is the same data with filters, two cost-and-time maps and a per-bug coverage view. Read the leader with its effort tier: a model at max and the same model at medium are different rows for a reason.

Is this a real-world benchmark, or synthetic?

Real. The codebases are production repos with their own tests, docs and history; the bugs are shipped regressions re-introduced from their fix commits (all of repo 2, a third of repo 1); the task is the one a developer would give an agent. What is synthetic is only the selection: a fixed set of 105 defects, so that every model is measured against the same thing.

Why is the score out of 105 and not a percentage?

Because 105 is the number of planted bugs and the two repos are not equally hard. A percentage would hide that repo 1 carries 45 of them and repo 2 carries 60, and that models rank differently on each — the per-repo columns are there because one repo was not enough to rank the middle of the field.

Why are extra fixes not counted?

Some models "bug-max": they report a long list of real but irrelevant problems. Those fixes are often genuine, but the set has no answer key and no ceiling, so it cannot be scored reliably, and adding it would reward the model that touches the most files. Extras are counted, shown, and kept out of the score. On this board, zero false-positive fixes have been recorded on either repo: every extra any model applied was a real defect.

Who grades, and can a model grade its own work?

An independent judge model grades each diff against the answer key, blind — it never sees which model produced the submission, and model names are redacted from the packet. No model grades itself or a sibling from the same lab: OpenAI arms are graded by Grok, everything else by GPT-5.5 through Codex. Whether the score depends on the judge was checked directly — all six early repo-1 arms reproduced exactly under a second judge from a different vendor — and the receipts are in results/judge-calibration.md.

How noisy is a single run?

One round per model per repo, so treat fixed as ±1 and extras as ±2. Same-setting variance has been measured: three Grok 4.5 runs on identical settings scored 16, 13 and 17. A one-fix gap between two rows is a tie. A five-fix gap is the first kind of lead on this board that clears its own variance band.

Does reasoning effort (max, xhigh, high, medium) help?

Where the dial is real, yes, and mostly at the top: on every first-party effort dial measured so far, the top tier scored highest and the step into it was usually the largest, while the middle tiers often sit within the ±1 noise of each other. Cost per fix does not necessarily rise with the tier. On aggregator paths (OpenRouter and similar) the effort parameter is frequently inert — accepted with HTTP 200 and applied nowhere — which is why those rows are labelled default rather than the tier that was requested. Each dial's numbers are in the table; the write-ups are in results/waves.md.

What do default and "ran lower" mean in the effort column?

default: the serving path had no working effort dial, so the model ran at whatever it runs at — verified by probing the path, not assumed. "Ran lower": a higher tier was requested and the CLI quietly substituted a lower one; the row is published as a correction with the evidence. A requested tier is not an applied tier, and the board only prints what was served. Probe outputs are filed under results/effort-dial-probes/.

Are the costs real bills?

Some are. Each cost carries a kind: bill (an invoice or a credits delta), list (a token-count estimate at published rates — the case for subscription-covered CLI runs), floor (a reconstructed lower bound, where a CLI reports context occupancy rather than spend), or free. Listed and billed figures can diverge: one aggregator run listed at $1.26 billed $5.54 because a cache discount was never applied. Do not rank costs across kinds to the dollar.

Could a model have seen the answers? Is the benchmark contaminated?

The answer keys, the seeded sources and the per-model diffs are withheld and are not in this repository — publishing them would burn both benches. During a run a model has no network access and no git history. Repo 1 is derived from a public open-source extension, so a model may know the fixed code; it cannot know which 45 defects were planted, and the field's repo-1 scores (2 to 24 of 45) do not suggest recall. Bug identifiers are never published either: per-bug coverage uses frozen indices, not ids.

Can I run it on my own model, or reproduce a row?

Not directly: the seeded repositories are withheld to keep the bench usable. What is public is everything needed to check a number — the exact prompts, every scoreboard and metrics row, per-bug coverage by index, the judge calibration, and the effort-dial probes. To get a model on the board, open an issue with the model, the first-party CLI it ships with and the serving path; runs are added as models become reachable.

Why is model X not on the board?

Either it has not shipped a first-party agentic CLI or an API path that the harness can drive, or it has not been run yet. New models are usually run on launch day when access opens; a run is announced on the board with its date.

Which bugs has nothing ever fixed?

The count of bugs never fixed by any model in any run is in the header of the table above, and the Coverage view on the live board shows exactly which indices each run fixed and where models overlap. The survivors are the interesting part of the set.

How do I cite this?

Huryn, P. Bug Hunt Bench: 105 real bugs, two production repos, frontier coding models graded blind. https://bughunt.productcompass.pm — data: https://github.com/phuryn/bug-hunt-bench

What is in this repository

The runner, the answer keys, the seeded repositories and the judge transcripts live in a private repository. Numbers flow one way: runs are scored there, receipts are published here, the site is generated from the receipts. No figure on the board or in this README is typed by hand.

Issues yes, pull requests no. There is nothing here to edit by hand: a hand edit would be overwritten on the next wave, so pull requests are closed automatically. Issues are open for two things: a number that disagrees with a receipt, and a model that should be on the board. Details: CONTRIBUTING.md.

Author and license

Built and run by Pawel Huryn — Product Compass, X, LinkedIn. Related: pm-skills, agent skills for product managers.

MIT. Use anything; a link back is appreciated. If a number in a post and a receipt here disagree, the receipt wins and I want to know: open an issue.