Primitive Bench

The marketplace for verifiable AI outcomes.
State a task, get a fixed price upfront. We route across the right tools to complete it.
If the outcome isn't delivered, you get your money back.

Book a call · Website · Docs · Methodology · Contributing · LinkedIn

From tool calls to guaranteed outcomes

A year ago, vibe coding arrived. Next, agents run full end-to-end workflows: choosing a domain
name, buying a design, paying a hosting provider, deploying the app. Developers and indie founders
will lead; enterprises will follow. Every one of those workflows is a chain of tool calls that has
to actually work.

Most tool marketplaces stop at the call: they hand your agent an API and wish it luck. Primitive
Bench sells the outcome, not the call.

• State a task. Describe what you want done, in plain language.
• Get a fixed price upfront. The estimator prices the job from known, published API pricing
  before you commit a cent.
• We route and execute. Primitive Bench internally selects and calls the sub-APIs that get it
  done, and traces every step.
• Outcome guaranteed, or refunded. If the outcome isn't delivered, you get your money back.

Load your AI into the money, get a guaranteed outcome. If it isn't done, get your money back.

Book a call →

Why this, not another tool aggregator

An open marketplace alone is not a moat. The wedge is the layer on top: route to the right tools,
observe what they actually did, and stand behind the result. Primitive Bench is the observability
and routing layer for tools, not just a place to find them.

The verification layer

An outcome guarantee is only as honest as the machinery that decides whether the outcome was
delivered. That machinery is what most of this repo is: a vendor-neutral harness that measures
whether a tool actually did its job, per slice, with real statistics.

The open engine here observes input and output, traces every tool call, and scores the result
against golden answers with confidence intervals and separability tests. That observability layer is
how Primitive Bench knows a task is "done" and where routing should send the next one. Held-out
golden answers never live in this repo; they sit behind the private eval server, so the scores stay
honest and the marketplace has a neutral arbiter no vendor can pay to move.

"One winner is a lie."

No tool wins every slice. We score per-slice, per-constraint with confidence intervals and
statistical separability, so routing picks the right tool for your task instead of a single
global favorite.

What we measure

The primitives below are the building blocks routing composes into outcomes. Each is scored by a
public eval package so the routing and refund decisions rest on evidence, not folklore.

Filling in a 🚧 is the highest-impact first contribution, and it directly widens what the
marketplace can route and guarantee. See CONTRIBUTING.md.

What keeps the guarantee honest

• No fake #1. A winner is named for a slice only when it's statistically separable from the
  runner-up (McNemar p < α, non-overlapping CIs); otherwise we publish a tie band, and routing
  treats them as interchangeable.
• Real evals, not reviews. Every claim is backed by a canonical, citable statistic: McNemar,
  Wilson intervals, seeded bootstrap, Bradley-Terry / Elo.
• Reproducible by anyone. Deterministic seeds, pinned versions, and public dev splits
  reproduce public runs bit-for-bit.
• Neutral arbiter. No pay-to-rank. Three-tier ground truth (verified-external,
  authoritative-registry, sentinel-planted) with canary markers for contamination detection.

Quickstart

Packages aren't on PyPI yet, so run from a clone for now.

uv sync
uv run bench run --primitive ocr --config configs/ocr.yaml
uv run bench view ./runs/<run_id>

The bench CLI scaffolds a config (bench init), runs an eval (bench run), summarizes slices with
separability badges (bench view), and submits to the held-out eval server for scores only
(bench submit).

Query it from your agent (MCP)

The verification layer isn't just a site to read, it's a tool your AI agent can query while it
reasons. The Primitive Bench MCP server exposes the per-slice leaderboards over the
Model Context Protocol, so a coding agent can ask "which web search API wins for government
registry lookups?" and get back the slice-specific winner, or an honest TIE band, with Wilson
CIs and a citation. That's the routing brain, exposed: the agent supplies the constraints, the
benchmark supplies the statistically honest answer. No server-side LLM, so queries are free.

🟢 Live now at https://benchpublic.vercel.app/mcp, add it to Claude Code:

claude mcp add --transport http primitive-bench https://benchpublic.vercel.app/mcp

Live for websearch, extraction, reranker, and retrieval today. See apps/mcp to run it
locally or deploy your own.

How it works

Primitive Bench uses the proven harness shape (dataset → Task → Adapter → Scorer → result schema,
converging with EleutherAI lm-eval, UK AISI Inspect, and Stanford HELM) as the observability
substrate under the marketplace.

The Gate. bench-schemas is the frozen contract (v0.1.0): every package imports types only
from it and writes only files it owns, with no shared mutable state. That boundary is what lets the
build lanes run in parallel without colliding. See apps/docs/DECISIONS.md (D-03)
and the methodology.

Repo layout

packages/
  bench-schemas/   # THE FROZEN CONTRACT — RunManifest, ItemResult, SliceResult, ScorerOutput, AdapterSpec
  bench-core/      # harness engine: deterministic seeding, run/manifest, per-run dirs
  bench-stats/     # McNemar, Wilson, bootstrap CIs, hit@k, nDCG/MAP/MRR, Bradley-Terry
  bench-adapters/  # provider/primitive adapter SDK (lm-eval registry pattern)
  eval-*/          # one package per primitive: public golden dev set + scorer + slice defs
apps/
  cli/             # the `bench` CLI: init / run / view / submit
  docs/            # methodology + DECISIONS.md
golden-sets-public/  # PUBLIC dev splits only (canary-marked). Held-out answers NEVER here.

Roadmap

The verification layer is live today. The outcome marketplace (fixed-price estimator, routing across
sub-APIs, and the refund guarantee) is being built on top of it. Near-term work:

• Cost estimator that prices a task from known, published API pricing before execution.
• Routing that turns a stated task into the sequence of tool calls that satisfies it.
• Outcome verification thresholds: a precise, per-task-type definition of "done" measured from
  the traced input/output, and the scope limits on what Primitive Bench will and won't take on.
• Open contributor payouts so anyone who uploads a tool earns when routing uses it.

Have a workflow you'd want guaranteed end-to-end?
Book a call →
That's exactly the customer discovery we're doing now.

Contributing

We love contributions big and small: a new vendor adapter, a slice that separates two tools, or a
whole stubbed primitive. Start with CONTRIBUTING.md; the best first issue is
implementing one of the 🚧 verticals using eval-websearch / eval-extraction as the template.

License

• Code: Apache-2.0.
• Public datasets under golden-sets-public/: CC-BY-4.0.
• Third-party attribution is in NOTICE. We learn from lm-evaluation-harness, Inspect, HELM,
  ann-benchmarks, VectorDBBench, and OmniDocBench, and we do not vendor GPL/commercial-dual code.

Built by the Primitive Bench team · primitivebench.com · LinkedIn