AI Readiness Analyzer

Deterministic AI-readiness scoring for GitHub Copilot and Claude Code repository configuration.

Point it at a repository. Get back a reproducible score, a letter grade, and a
ranked list of concrete fixes — computed entirely by static analysis, with
zero model calls in the scoring path. Same commit in, byte-identical report
out, every time.

Status: v0.3.0 "AgentCompass" — pre-1.0. All eight scoring pillars are
live with a catalog of 104 rules, platform sub-scores, waivers,
terminal/JSON/Markdown/SARIF reports, and a clone-free web UI. The original
design document is plan.md; the v0.2.0 refactor plan is
plan-v2-fable.md and the web-app plan is
plan-v3-codecompass.md; the full generated rule
catalog is docs/RULES.md. See
CHANGELOG.md for exactly what's built.

Why this exists

Repositories increasingly ship configuration meant for AI coding agents —
copilot-instructions.md, CLAUDE.md, AGENTS.md, SKILL.md files, custom
agents, path-scoped instructions, hooks, MCP servers. Whether any of it
actually works is usually invisible until an agent silently fails to load a
skill, ignores a 900-line instructions file, or never triggers a skill because
its description is too vague.

AI Readiness Analyzer answers "is this repo actually ready for an AI agent?"
the same way a linter answers "does this code compile" — deterministically,
offline, and with a specific file and line number for every issue.

What it checks

Eight pillars, 104 rules (full catalog):

Quickstart

git clone https://github.com/YoavLax/agent-compass.git
cd agent-compass
python -m venv .venv
# Windows: .venv\Scripts\activate   |   macOS/Linux: source .venv/bin/activate
pip install -e .

airx analyze /path/to/some/repo

PATH can also be a remote repository — a GitHub owner/repo shorthand or any
git clone URL (https://, ssh://, git@host:...). It's shallow-cloned to a
temp directory for the analysis and removed afterwards; requires git on PATH.

airx analyze YoavLax/agent-compass
airx analyze https://github.com/YoavLax/agent-compass.git --ref main

AI Readiness Analyzer — /path/to/some/repo

Overall score: 61.1/100   Grade: D
Platforms:     copilot 62.5   claude 61.1   parity delta 1.4

Pillars:
  foundation      86.4%   (presence 100.0%, quality  77.3%, weight 20, 9 rules)
  skills          99.8%   (presence 100.0%, quality  99.6%, weight 15, 37 rules)
  ...

Findings (20):
  [error  ] skills.name.dirname-match   .github/skills/deploy/SKILL.md
            Name 'deployer' does not match parent directory 'deploy'. VS Code/Copilot silently fails to load this skill.

Top fixes (estimated score gain):
  1. +4.8  [additive      ] verify.test-command.documented
     Document the repository's test command in an entry point so agents can verify their work.

CI usage

airx analyze . --format json -o report.json     # canonical machine output
airx analyze . --format sarif -o airx.sarif     # GitHub code scanning
airx analyze . --format md                      # PR comment / job summary
airx analyze . --min-score 70 --fail-on error   # quality gate
airx analyze . --fail-level 3                   # gate on maturity level instead of raw score
airx compare baseline.json report.json          # exit 1 on regression

Exit codes: 0 passed · 1 gate failed · 2 input/config error · 3 internal error.

Configuration (.airx.yml)

airx init scaffolds it:

profile: standard        # or: minimal, enterprise (weight profiles)
min_score: 70
fail_on: error
fail_level: 3            # optional maturity-level gate (1-5); unset disables it
ignore:
  - skills.compat.unverified
waivers:
  - rule: skills.present
    reason: "Domain knowledge lives in an internal plugin marketplace."
    expires: "2027-01-01"
    approved_by: platform-team

Waived rules score as satisfied but stay visible in the report. Waiver expiry
is only evaluated against an explicit date (--today 2026-07-29 or
AIRX_TODAY) — the scoring path never reads the clock, so output stays
reproducible.

AgentCompass — the web UI

AgentCompass — your compass for AI-agent-ready repos. Paste a public GitHub
repository URL into the browser, get the full report: overall score and grade,
Copilot/Claude platform bars, per-pillar breakdown, filterable findings, and
the ranked top fixes.

A Copilot / Claude Code / All toggle scopes the report to one agent
harness, so a Claude Code-only (or Copilot-only) team isn't scored against
rules for a harness they don't use. The selection carries over on re-analysis
and is reflected in the shareable URL (?platform=claude).

The scan is clone-free: one GitHub Trees API call lists every file in the
repository, and only the files the rules actually read — classified AI
artifacts, the four probe files, skill directories — are fetched (kilobytes,
not the repo). Name-only rules see the complete listing, content rules see
real files, and the whole snapshot is pinned to a single commit SHA. Nothing
is persisted; each request is self-contained.

Run it

docker compose up          # then open http://localhost:8080

Or without Docker:

pip install -e ".[dev]"                     # server deps (or ".[web]" for runtime only)
cd web && npm install && npm run build      # → web/dist
cd .. && STATIC_DIR=web/dist uvicorn airx_server.app:app --port 8080

Server configuration

All configuration is environment variables:

API

• POST /api/analyze with {"source": "<github url or owner/repo>", "ref": null}
  — or {"path": "<relative path>"} in local-path mode — returns the canonical
  JSON report plus a meta block (source, ref, resolved_sha,
  listed_files, fetched_files, duration_ms). Errors come back as
  {"error": {"code", "message"}} with 400/404/413/422/429.
  An optional "platform": "copilot"|"claude"|"all" field scopes scoring to
  one platform's rules (default "all"), mirroring the CLI's --platform
  flag; the applied value is echoed back as the report's top-level platform key.
• POST /api/analyze/stream — the same analysis and the same request body,
  streamed as NDJSON: one {"type":"progress","phase","done","total"} object
  per line while the work runs, then a single terminal
  {"type":"result","report":{...}} or {"type":"error","error":{...}}. The
  counts are measured, not estimated — phase walks
  resolving → listing → fetching → linked → scoring, and done/total in the
  fetch phases are real file counts — which is what lets the web UI draw a
  progress bar that tracks the run. The report on the result line is identical
  to what /api/analyze returns for the same commit; use the plain endpoint for
  scripting, where one request and one status code is the simpler contract.
• GET /api/health — liveness. GET /api/version — {version, local_mode}.

Private repositories

The online scan only reaches public GitHub. For private code, self-host
AgentCompass next to your repositories: mount them read-only into the
container, set ALLOW_LOCAL_PATHS=true and LOCAL_REPOS_ROOT, and analyze by
relative path — the analysis itself never touches the network. A Helm chart
for Kubernetes deployments lives at deploy/helm/agentcompass; see
deploy/README.md for both setups.

How it works

path → fs.scan            deterministic, symlink-free traversal
     → discovery          declarative artifact patterns (skills, agents, prompts,
                          instructions, hooks, MCP, settings — see src/airx/patterns.py)
     → probe              repo facts: test/build/lint evidence, CI, hygiene
     → rules/*            94 pure functions, one per check, in a versioned registry
     → scoring            presence/quality split per pillar, platform sub-scores,
                          profiles, waivers, grade banding
     → report/*           terminal | json | markdown | sarif + ranked remediation plan

Every rule is a pure function of its input. There are no model calls, no
network access, and no wall-clock or environment dependence anywhere in the
scoring path — see plan.md §3 for the determinism contract and
tests/test_determinism.py for its enforcement.

The scoring model, briefly

Each pillar splits into a presence score (does the relevant artifact
exist at all?) and a quality score (how good is it?), combined as
0.4 × presence + 0.6 × quality. This makes the score resistant to gaming in
both directions: deleting every skill scores worse than having one flawed
skill, and duplicating a mediocre skill doesn't inflate the score (it's an
average, not a sum). Rules that don't apply are removed from both numerator
and denominator; a pillar with nothing applicable at all is excluded from the
weighted overall rather than scoring a vacuous 100%.

Any error-severity finding caps the overall grade at C, regardless of the
arithmetic score — and error severity is reserved for objective,
spec-verifiable failures (a skill that silently fails to load, a committed
credential), never for style heuristics. The cap never upgrades an
already-worse grade.

Each grade also carries a named, 1–5 maturity level — the same underlying
number, just framed for adoption tracking and CI gating (--fail-level N)
rather than as a score:

The level is derived from the final (possibly error-capped) grade, never the
raw score — it can't claim a higher maturity than the grade itself allows.

Every rule is tagged by platform, so the report also carries separate
copilot and claude scores and their parity delta — a rich AGENTS.md
with no CLAUDE.md bridge shows up as a Copilot/Claude gap, not just a
buried warning.

Commands

airx analyze PATH [--format terminal|json|md|sarif] [-o FILE]
                  [--html [FILE]]
                  [--profile minimal|standard|enterprise]
                  [--platform copilot|claude|all]
                  [--min-score N] [--fail-on error|warning|never]
                  [--fail-level 1-5]
                  [--ignore PREFIX]... [--no-waivers] [--today YYYY-MM-DD]
                  [--ref BRANCH|TAG|COMMIT]         # remote PATH only
airx rules        [--format terminal|json|md]     # the catalog; generates docs/RULES.md
airx compare      OLD.json NEW.json               # regression diff for CI
airx init         [--force]                       # scaffold .airx.yml

PATH is a local directory, a GitHub owner/repo shorthand, or any git clone
URL — remote repos are shallow-cloned to a temp directory and cleaned up
after analysis.

--html [FILE] additionally writes a self-contained, offline HTML report
with collapsible sections (pillars, findings by severity, top fixes, waivers,
inventory) — default path airx-report.html when no FILE is given.

GitHub Action

Run the analyzer in CI without publishing or installing anything yourself —
the composite action installs ai-repo-analyzer straight from the pinned
uses: ref, so a tag (e.g. @v1) is the only version contract:

- uses: actions/checkout@v4
- uses: YoavLax/AI-Repo-Analyzer@v1
  with:
    path: .              # default: "."
    profile: standard    # minimal | standard | enterprise
    fail-on: error        # error | warning | never
    min-score: 70         # optional overall-score gate
    fail-level: 3          # optional maturity-level gate (1-5)

Outputs: score, grade, maturity-level, maturity-label, exit-code. A
Markdown report is also written to the job summary by default
(job-summary: false to disable). The step's exit code mirrors airx analyze's own gate (§8.4), so a failing score turns the job red the same way
a failing test would.

Not yet implemented

airx fix, duplication detection, and nested-monorepo aggregation — see
plan.md §12 and plan-v2-fable.md §1 for
sequencing.

Contributing

See CONTRIBUTING.md — in particular, the section on the
determinism contract, which every rule must preserve.

Security

See SECURITY.md for the threat model and how to report a
vulnerability.

Core Team

Author	Author
@YoavLax	@bechor25

Credits

The SKILL.md validation rules and their thresholds are vendored from
AgentEval (MIT licensed). The rule
catalog is derived from the published
Agent Skills specification, the
Claude Code documentation,
and GitHub Copilot's custom-instructions guidance
— see plan.md §15 for the full bibliography.

License

MIT © 2026 Yoav Lax, Bechor Simhaev