config-drift-checker

Your agent conventions are code. This is their CI. The rules your team taught its coding
agent (CLAUDE.md or AGENTS.md, skills, hooks) decide how your software gets written now, and
everything underneath them moves without asking: Claude Code alone ships about 25 releases a
month, and the model behind an alias changes server-side with no changelog
(it already has, silently, for weeks).
This turns those rules into eval cases, runs them on every PR and every release against a pinned
baseline, and tells you the moment something stops working: when, why, and what moved.

One suite, three agents. First-class on Claude Code (skills, hooks, release canaries, the
whole drift machinery); the same cases also run through OpenAI's Codex and Google's Gemini CLIs,
with live passing runs on all three, experimental labels on the newer two until full published
comparisons.

Proof it works: we broke our own setup, and the repair skill fixed it.
A skill's trigger description rewritten the way a careless PR would: the suite fell 1.00 → 0.56,
the tripwire case read 0.00, and the report named the cause itself: the skill was discovered but
never invoked, so fix the trigger wording, not the packaging. Then the repair skill restored the
behaviour on its own and proved it with a green re-run, for $0.28. Both artifacts are unedited.

See it live

Quick start

One command in the repo whose setup you want protected:

claude plugin marketplace add jameskomo/config-drift-checker && claude plugin install config-drift-checker@jameskomo && claude "/config-drift-checker:setup"

Five minutes: it finds your CLAUDE.md, skills and hooks, writes starter eval cases from them,
smoke-runs them, and writes .cdc.yml plus the GitHub workflow. Prefer never opening Claude at
all? node <plugin-root>/tools/cdc-bootstrap.mjs . runs the same setup headlessly, and
--no-agent scaffolds everything for $0 (blank starter case included) so you fill in the prompts
yourself. Add one secret
(CLAUDE_CODE_OAUTH_TOKEN from claude setup-token to run on a Pro/Max subscription at no extra cost, within its usage limits, or
ANTHROPIC_API_KEY), push, done.

Try the two free checks on any plugin, nothing to install. Each is a single file with no
dependencies (read it first, it's short):

curl -fsSLO https://raw.githubusercontent.com/jameskomo/config-drift-checker/v1/config-drift-checker/tools/skill-lint.mjs && node skill-lint.mjs .        # your SKILL.md files
curl -fsSLO https://raw.githubusercontent.com/jameskomo/config-drift-checker/v1/config-drift-checker/tools/suite-doctor.mjs && node suite-doctor.mjs .    # your eval suite vs the installed Claude Code; add --fix

Neither starts a model run. What they check.

Already have a suite in the claude plugin eval format? One step:

- uses: jameskomo/config-drift-checker/action@v1
  with: { plugin-dir: . }

(@v0 keeps working; both moving tags point at the same latest release.)

What you get

Every row is shipped and tested; where a public receipt exists, it's linked.

Detect: know the moment behaviour moves

Diagnose: red comes with answers, not homework

Repair: and prove the fix

Operate: it runs itself, and reports to you

Across agents: one suite, three CLIs

How this relates to claude plugin eval

Claude Code ships an eval runner, and it's good: it runs your cases, grades them, generates
starter cases with init, and writes a report. We build on it, not beside it: cases are in that
exact format, the Action prefers the official runner (bundled fallback for older versions), and a
claude plugin eval . --json out.json result feeds our diff, report and drift index directly.

One sentence: their command answers "does my plugin work right now on my machine"; this answers
"did anything stop working since the baseline, across every release, without me watching".

Trust

Stable since v1.0: the v0/v1 moving tag never breaks your workflow; breaking changes mean
a new major with an upgrade note in CHANGELOG.md.
Zero npm dependencies (Node builtins only), 246 tests that run against a fake claude with no API
key, every third-party action pinned to a verified commit SHA, CodeQL on every push. Runs on your
runner with your key; nothing is sent to us, because there is no us to send it to. Details in
docs/security.md.

What's here

config-drift-checker/   the plugin: skills (setup · run · write-case · repair) + the tools
  tools/                shim runner · diff · classify · report · dashboard · coverage · watch · gate · promote · trace-keeper · fleet · bootstrap · drift-bisect · drift-digest
  test/                 node --test suite, fake claude, npm test
action/                 composite GitHub Action: gate → run → diff → store → PR → repair → alert
examples/komo-stack/    a full example suite with .cdc.yml and baseline results
docs/                   user guide · architecture · eval format & runner · runbook · security

Hosted tier (waitlist)

Self-hosting is free forever, that never changes. A managed tier (we run the canaries, fleet
dashboards and alerts; you get the PRs and the pages) gets built when enough teams want it:
join the waitlist,
two questions, no commitment. Discussions are open
for everything else.

Documentation

Start with the user guide (.cdc.yml reference here); full index in docs/.

Licence

FSL-1.1-Apache-2.0: free to use, modify and self-host; not to be offered as a competing
commercial service; each release becomes Apache-2.0 two years after publication.