Dokimi — agent measurement framework

Documentation: https://intel.github.io/dokimi/ — guides, plus a
CLI reference and an
API reference generated from the code.

dokimi — Greek δοκιμή, "test, proof". The name is the method: an
artifact does not get trusted because it was written or because it is popular,
it gets trusted because it passed a test that could have failed.

Documentation, skills, MCP doc sources and agent persona prompts are cheap to
write and easy to over-claim. This repository is the framework that measures
whether any of them actually helps a coding agent: change one thing, hold
everything else fixed, run the same question or task set, and report the delta
together with its p-value.

The measurement is the deliverable. An artifact ships with the scorecard that
earned it — a deliberate contrast to popularity-ranked marketplaces, which
structurally bury niche HPC and numerical-accuracy tooling.

The framework is vendor-neutral: any model reachable through LiteLLM, any
docs, skills or tasks.

What it measures

1. Does it help? Enabled vs disabled arms

An arm is one context condition. Every arm runs the same question or task
set against the same model, and the report gives the per-arm delta against the
baseline with paired statistics. Arms are given as specs (dokimi arms run --arms …):

The same arms drive question answering (LLM-as-judge with judge-validity
checks), executable Docker coding tasks through Codex, Claude Code, Harbor or
BenchFlow harnesses (dokimi tasks run), and file/line localization
(dokimi explore run). Results roll up per subject (dokimi subjects run):
a skill, MCP server, doc source, profile, bundle or repository, each with
a scorecard.

2. What does it add? Novel vs known content

An arm delta says whether an artifact helps; it does not say which parts.
dokimi novelty analyze --artifact skill:<path>|text:<path> splits the
artifact into knowledge atoms and probes each one against the model with and
without the artifact. Each atom comes out added (new to the model),
redundant (the model already knew it), degraded (the artifact broke
something the model knew) or still missing (stated but not picked up).
text: takes any file, so the same analysis runs on a documentation page or
a repository's AGENTS.md. The output is per (artifact, model) pair, with
shortening candidates (novelty shorten) and a check that the artifact can be
selected among competitors at all (novelty routing). See
skill-novelty.md.

3. Repositories: scan the guidance, then measure agents working in them

A repository is a subject in its own right, described by a repo-eval.yaml
manifest it owns:

• dokimi repo scan (L0, no LLM): do the links, paths, identifiers, commands
  and build labels the guidance names exist in the tree?
• dokimi repo reach (L1): which guidance files does a given harness actually
  load?
• dokimi repo run: run an agent per (task, arm, model, repetition) inside the
  repository's image, grade each run with the repository's own tests, and
  dokimi repo report the per (task, arm, model) table. With the guidance:*
  arms this measures what the repository's agent guidance does to the agent's
  work; dokimi repo compare puts each arm against a baseline arm, paired by
  task.

See the repository evaluation contract.
LLM-free static checks on the documentation itself (dokimi run) round this out.

Repository layout

README.md                          This overview
Makefile                           Delegates targets into dokimi/
dokimi/                            The framework (self-contained Python project)
LICENSE  NOTICE                    Apache-2.0 + attribution (repo-wide)
.github/workflows/                 CI (runs inside dokimi/)

All code, tests and CI live in dokimi/. The
evaluation content that gets measured — skills, executable tasks, golden
question sets — lives in its own repository and is passed in by path, so the
framework can measure artifacts it does not own.

Reviewing a skill or a repository

The usual request is "use dokimi to review this skill" or "review this
repository's agent setup". This section is for whoever runs that request, a
person or a coding agent. Run the steps in order: each one costs more than
the step before it, and the first ones are free.

Setup, from dokimi/:

cd dokimi
pip install -e .                 # installs the `dokimi` command

The novelty and arms steps take --model and --provider; arms run also
needs --judge-provider (it does not fall back to --provider for the judge);
tasks run and repo take --model only. The provider is one of openai,
anthropic, amazon-bedrock, google-vertex or openrouter.
Credentials are read from the environment: OPENAI_API_KEY,
ANTHROPIC_API_KEY, OPENROUTER_API_KEY, AWS_REGION with
AWS_BEARER_TOKEN_BEDROCK for Bedrock, and Google application-default
credentials (GOOGLE_APPLICATION_CREDENTIALS, with VERTEXAI_PROJECT and
VERTEXAI_LOCATION) for Vertex. The judge must be a different model from the
one under measurement, because a model that grades its own answers inflates
the scores. novelty refuses a self-judge; arms run does not check, so name
--judge --judge-model <other> --judge-provider <judge-provider> yourself.

A skill

• Competitors are the other skills an agent sees next to this one when it
  picks a skill to load. For a catalog skill they are its siblings in the
  same catalog: --catalog <dir> takes every other skill under it, and
  --competitors skill:<path> … names them one by one. Routing asks whether
  this skill wins the user requests meant for it. If you only have the one
  skill, skip step 2 and say so in the review: a number measured against
  invented competitors measures nothing.
• Queries are 5–10 requests written the way a user would phrase them.
  Write them without looking at the skill's own headings.
• Stop after triage (step 4) and report. The standard and study
  tiers cost 5–20 times more, so run them only when whoever asked agrees to
  the bill. Pass --max-usd to novelty steps: the run stops once it has
  spent that much (calls in flight can overshoot), writes no artifact when the
  cap trips, and stops at the first call to a model with no known price. Set
  the cap above the expected dollar spend: the triage example below was about
  $1.80. arms run and tasks run have no cap, so size them first: questions
  times arms, or tasks times harnesses. One measured example: triage on a
  19k-token skill with a mid-size model was 103 calls, about $1.80; study on
  the same skill was 1,900 calls, about $14.50.
• At triage with --include-resources, the 25-atom cap takes atoms in
  document order, and the entry point comes first: the flag reaches
  references/ only if SKILL.md has fewer than 25 atoms. For such skills,
  use --tier standard (60 atoms), or add --all-atoms to the tier command for
  a full sweep; run --estimate-only first, it costs more.
• Step 5 needs evaluation content: a golden question set for 5a, executable
  tasks for 5b. arms run takes no tasks. That content lives outside this repository
  (external-content.md). Without it, the
  review covers selection and content, not outcomes, and should say so.

The full path with reading notes: skill-novelty-walkthrough.md.

A repository's agent setup

With no repo-eval.yaml in the repository, steps 1 and 2 check the default
guidance globs (<repo>/**/AGENTS.md, <repo>/**/CLAUDE.md,
<repo>/.github/copilot-instructions.md, <repo>/.github/instructions/*.md)
and say so.
To choose the files for scan, write a guidance-only manifest outside the tree and pass
it with --manifest-file:

schema: repo_manifest.v1
id: my-repo
guidance:
  globs: ["**/AGENTS.md", "**/CLAUDE.md", ".github/copilot-instructions.md"]
  also_scan: ["CONTRIBUTING.md"]

The id must be lowercase: start with a letter or digit, then use only letters, digits, ., _ and -.
reach always adds the harness's own guidance file names (for example CLAUDE.md, CLAUDE.local.md or AGENTS.md).

A guidance-only manifest has no tasks, so step 4 refuses it.

How to read the results:

• Scan: fail counts against the guidance and each one still needs a
  look. Harness tool names (run_in_background) and placeholders
  (<name>, caseNN_*) can show up as fail too. unresolved also needs
  a look: it can be a path the scan could not check, or one whose directory
  is gone.
• Reach: a guidance file nobody loads cannot help anyone. Run reach
  before you judge the guidance's content. loaded comes from the harness
  itself, not from the harness's documentation, so a result that contradicts
  the docs is a finding.
• Step 4: without tasks there is no outcome measurement. The review
  should say so rather than infer one from steps 1–3.

The contract a repository codes against: repo-evaluation-contract.md.

Developing dokimi

make benchmark-run    # delegates into dokimi/ (see its README)
make test             # runs the dokimi test suite

For the framework's quickstart, CLI surface, and architecture, see
dokimi/README.md.

Where to start reading

The Documentation links at the top go to the published site and its CLI and API
reference. The hand-written pages can also be read on GitHub under
dokimi/docs/.

License

Licensed under the Apache License, Version 2.0; see NOTICE
for attribution.