harness-audit

Cut what your coding agent loads before you type, and point it straight at what it needs.

Português (Brasil)

harness-audit is an Agent Skill that audits the harness of a software project (CLAUDE.md, AGENTS.md, GEMINI.md, rules, skills, hooks, docs and Obsidian notes), restructures it so agents start every session with a short map, measures the result, and installs guardrails so the project does not drift back into a mess.

It works with Claude Code, Codex, Cursor and Antigravity CLI (the successor of Gemini CLI), with or without an Obsidian vault.

What it did on real repos

Session start, read from /context in a fresh session:

The finding that paid for the first audit was not size, it was silence. AGENTS.md had reached 349 KB against Codex's 32 KB read cap (project_doc_max_bytes), so the agent was receiving roughly the first tenth of it. No error, no warning, nothing in any log. On the second repo the same cap cut the file at 19.1%, mid-sentence, and CLAUDE.md had grown to 603,471 bytes, which meant nothing with a 200k window could open the project at all.

The honest ceiling: task success was 8/8 before the audit and 8/8 after. This does not make an agent smarter and never will. What moved was time (40 min to 7 across four tasks), operator interventions (2 to 0) and side effects (6 to 0).

The three pilots are written up as open issues, including a third one where the after-benchmark turned out to be invalid, and why.

Contents

• The problem
• What the skill does
• Commands
• Quick start
• Installation
• How it keeps the project organized
• What gets installed in your project
• Compatibility
• Obsidian
• Measuring the results
• Lint codes
• Limits
• FAQ
• Contributing
• Sources

The problem

Every token an agent loads before your first prompt competes for the model's attention. In real projects that layer grows quietly:

• instruction files turn into encyclopedias;
• the same paragraphs live in CLAUDE.md, AGENTS.md and GEMINI.md;
• @imports pull whole documents into every session;
• rules without scope load even when you never touch that code;
• dates and "currently working on" notes break prompt caching;
• notes pile up in Obsidian with no index an agent can use.

The research points in one direction. Model accuracy degrades as context grows (Chroma, Context Rot, 18 models tested). Instruction-following drops when files carry too many rules (HumanLayer). Redundant or auto-generated AGENTS.md files can lower task success and raise cost (ETH Zurich evaluation). Fewer, better-placed instructions work better than more instructions.

What the skill does

1. Interviews you briefly. Detects which agents the project uses and whether there is an Obsidian vault, then confirms with you.
2. Measures a baseline. Tokens loaded per agent, per layer (always-on, conditional, on demand), plus runtime numbers from Claude Code and Codex transcripts.
3. Scores the harness on 8 dimensions and writes a change plan grouped by risk.
4. Applies only what you approve, on a separate git branch. It relocates content instead of deleting it.
5. Installs a maintenance layer: a placement map, an automatic keeper skill, hooks for each agent, a git pre-commit check and optional CI.
6. Measures again and writes a before/after report. Budgets are locked so they can only shrink.

Commands

Running /harness-audit with no argument starts diagnose. apply refuses to run without an approved plan.

In agents without slash commands, just ask: "run the harness-audit skill in diagnose mode".

How changes are approved

Quick start

# 1. Install the skill for Claude Code
git clone https://github.com/fabianmartinelli-fm/harness-audit.git
cp -r harness-audit/skills/harness-audit ~/.claude/skills/

# 2. In your project, with everything committed
claude
> /harness-audit diagnose

Review .harness/reports/plan.md, approve, then run /harness-audit apply and /harness-audit verify.

Installation

Requirements: Python 3.9+ (standard library only) and git.

Claude Code

cp -r skills/harness-audit ~/.claude/skills/          # personal, all projects
# or
cp -r skills/harness-audit .claude/skills/            # this project only

Keep the disable-model-invocation: true line in SKILL.md: it hides the skill from the model's context until you call it.

Claude apps (Settings > Skills upload)

Download harness-audit.zip from the latest release and upload it. The release zip contains a single SKILL.md with standard frontmatter, as the upload requires. Use it where Claude can reach your local project files.

Codex, Cursor, Antigravity

Build the packages yourself

python3 tools/build_dist.py

Creates dist/harness-audit.zip and dist/harness-keeper.zip (upload-ready, validated) and dist/claude-code/ (filesystem copies with Claude Code extensions).

How it keeps the project organized

Writing instructions is not enough, because instructions are probabilistic. The skill combines three layers, following the guides and sensors model described by Birgitta Böckeler on martinfowler.com.

flowchart LR
    A[Agent edits a file] --> B{post-edit hook}
    B -- problem --> C[Agent gets the exact fix]
    C --> A
    B -- ok --> D[Agent finishes the turn]
    D --> E{stop hook}
    E -- index or log outdated --> C
    E -- ok --> F[git commit]
    F --> G{pre-commit + CI}
    G -- fails --> C
    G -- ok --> H[Organized project]

One source of truth: AGENTS.md is canonical, CLAUDE.md imports it with @AGENTS.md, and scoped rules and skills live in .harness/ and are generated into .claude/, .cursor/ and .agents/ by sync.py.

What gets installed in your project

your-project/
├── AGENTS.md                  map + routing table (canonical for all agents)
├── CLAUDE.md                  @AGENTS.md + Claude-specific lines
├── .harness/
│   ├── config.json            agents, docs root, budgets, placement folders
│   ├── PLACEMENT.md           where each thing goes
│   ├── rules/                 canonical scoped rules
│   ├── skills/harness-keeper/ canonical keeper skill
│   ├── scripts/               lint, sync, index, hooks, measurement, code sensor, tree survey (vendored)
│   ├── reports/               baseline, plan, after, HARNESS-REPORT.md
│   └── budgets.lock.json      ratchet
├── docs/                      (or your Obsidian folder)
│   ├── index.md               generated catalog
│   ├── log.md                 append-only history
│   └── decisions/ plans/ runbooks/ references/ architecture/ product/ raw/
├── .claude/settings.json      hooks            .claude/rules, .claude/skills   (generated)
├── .cursor/hooks.json         hooks            .cursor/rules, .cursor/skills   (generated)
├── .codex/hooks.json          hooks            .agents/skills                  (generated)
└── .agents/hooks.harness.example.json          Antigravity hooks, to adapt

Scripts are copied into the project, so teammates and CI do not need the skill installed.

Compatibility

Gemini CLI stopped serving individual accounts in June 2026; projects still using it are handled by the Antigravity adapter.

Models. The skill depends on the agent, not on a specific model. Use a frontier model for diagnose and apply (long, multi-step work with many file edits). check is script-driven and runs fine on smaller models. The heavy lifting is done by deterministic Python scripts, so results are consistent across agents.

Obsidian

During diagnose the skill scans the repo, parent folders and common locations (Documents, iCloud, Dropbox, OneDrive) for .obsidian folders and asks you to confirm the vault and the project folder. Three layouts are supported:

Notes get frontmatter that Obsidian Properties and Dataview also read. The index is generated from that frontmatter. Markdown links with relative paths are preferred over bare [[wikilinks]], which agents cannot resolve reliably. Details in references/obsidian.md.

Measuring the results

Want to know if it works on your project before starring? Run the full cycle and read the numbers.

/harness-audit diagnose      # baseline saved to .harness/reports/baseline.json
/harness-audit apply
/harness-audit verify        # after.json + comparison table

First pilot: a real project

Measured on fabianmartinelli.com, a Next.js 16 site with Supabase, MDX content and three locales, used daily with Claude Code and Codex. Same model (Opus 5) and same effort level in both rounds.

Context loaded before the first prompt, read from /context in a fresh session:

Static inventory, which the scripts read without a running session: Claude Code 9,288 → 7,815 tokens, Codex 4,729 → 3,257 tokens. Those numbers apply to every project on that machine, not only this one.

Four real tasks, run twice, in fresh sessions, before and after:

Read this honestly. Task success was 8/8 in both rounds: the agent already solved everything before the audit, so the skill did not make it smarter, and it never will. What changed is how it got there:

• time dropped about fivefold, because the agent went straight to the right file instead of exploring;
• operator interventions went from 2 to 0;
• side effects went from 6 to 0. In the first round the agent wrote five files into an external Obsidian vault, committed in the middle of a task and left a dev server running. In the second round it asked first and wrote nothing;
• decisions got grounded. Asked for a new section, the agent read the design rules first and refused to use a card because a recorded decision forbids cards on that site. It also refused to invent testimonials, and said so.

Context spent per task fell only 8%. If your projects are mostly read-heavy work like this one, expect the win in time, consistency and blast radius rather than in tokens.

Second pilot: a repository that had gone unmaintained

Same skill, a very different starting point: a large Next.js product repo (site, admin, client portal, several products) with ~2,800 tests, a git submodule, 25 worktrees and several sessions landing PRs in parallel. Two agents, Claude Code and Codex.

The finding that paid for the audit was not the token count. Codex was reading about 10% of AGENTS.md: the file was 349 KB against a 32 KB cap, and the truncation is silent, with no error and nothing in any log. Months of instructions written for an agent that never received them. On top of that, 92.5% of AGENTS.md was a literal copy of CLAUDE.md (3,031 of its 3,276 non-blank lines), so the project had two sources of truth maintained by hand and one of them was read half-way.

125 rules moved into docs/rules/ with frontmatter. Nothing was deleted: superseded content kept status: superseded, and 10 sections that existed in both entry files with different content were moved out and marked draft for a human to reconcile, not merged silently. After the audit: typecheck clean, 2,777 tests passing, production build green.

The two pilots together are the honest range. On a tidy project the context win is small and the payoff is speed and blast radius. On a project that had grown unchecked for months the context win is most of the window. Run diagnose and read your own numbers before deciding what this is worth to you.

For a stronger test, agree on 3 to 5 real tasks during diagnose. verify measures on its own what it can measure on its own — context, lint, entry-file sizes and Codex read coverage — and for the behavioural half it presents the prompts and asks you to run them in a fresh session, because a subagent inherits the context of the session that called it and would measure the wrong harness. A smaller context with worse task results counts as a regression.

Please share your numbers in a results issue. Real reports calibrate the default budgets for everyone.

Lint codes

Run it anytime: python3 .harness/scripts/lint.py (--json, --staged, --strict, --update-lock).

Limits

• Token counts from files are estimates (about 4 characters per token). Runtime numbers come from Claude Code and Codex transcripts; Cursor and Antigravity numbers are typed in by you.
• Antigravity hook names and schema change between agy versions. An example file is installed for you to adapt; until then, pre-commit and CI are the guaranteed checks.
• Cursor User Rules live in the app settings and cannot be read from disk.
• Hooks for notes in an external vault run locally; git and CI cannot see those files.
• Vendors change loading rules often. Each agent reference records when it was checked.

FAQ

Will it delete my docs?
No. Superseded content gets status: superseded. apply works on a branch that you review before merging.

Does it touch ~/.claude, ~/.codex or ~/.gemini?
Only measures them if you allow it. Changes there need explicit approval per file.

Do I need Obsidian?
No. Without a vault, knowledge lives in docs/.

Can I use it on a brand new project?
Yes. apply sets up the structure and guardrails from the start.

Why is harness-keeper a separate skill?
It is small and loads automatically during normal work. The audit is heavy and runs only when you call it. apply installs the keeper into each project for every agent.

Contributing

Bug reports, agent adapters and real-world results are welcome. See CONTRIBUTING.md. Run bash tests/smoke.sh before opening a pull request.

If the skill saved you context or a cleanup afternoon, a star helps other people find it.

Sources

• Anthropic: Effective context engineering for AI agents; Claude Code docs (memory, skills, hooks)
• OpenAI: Harness engineering: leveraging Codex in an agent-first world
• Andrej Karpathy: llm-wiki (April 2026)
• Chroma: Context Rot: How Increasing Input Tokens Impacts LLM Performance
• HumanLayer: Writing a good CLAUDE.md; Dex Horthy on research, plan, implement
• Philipp Schmid: Writing a Good AGENTS.md (ETH Zurich evaluation)
• Manus: Context Engineering for AI Agents: Lessons from Building Manus
• Birgitta Böckeler: Harness engineering for coding agent users (martinfowler.com)

License

MIT. Built by Fabian Martinelli at FM Solutions.