skill-optimizer

Benchmark and self-optimize SDK, CLI, and MCP guidance so every agent model can use your tool reliably.

skill-optimizer runs your SDK / CLI / MCP docs against multiple LLMs, measures whether they call the right actions with the right arguments, and iteratively rewrites your SKILL.md / docs until a floor score is met across every model.

Quickstart

git clone https://github.com/bucurdavid/skill-optimizer
cd skill-optimizer
npm install
export OPENROUTER_API_KEY=sk-or-...

# Scaffold config for your surface type (sdk | cli | mcp)
npx tsx src/cli.ts init cli

init cli creates a skill-optimizer/ directory with:

• skill-optimizer.json — the main config (task generation enabled by default)
• cli-commands.json — command manifest template (used as fallback if code-first discovery finds nothing)

init sdk creates skill-optimizer.json only. init mcp creates skill-optimizer.json + tools.json.

Open skill-optimizer/skill-optimizer.json and fill in these fields:

For CLI and MCP surfaces: if code-first discovery yields nothing, edit the companion manifest (cli-commands.json or tools.json) with your real commands/tools — the config already points to it as a fallback.

Tasks are generated automatically from your discovered surface — you don't need to write them manually.

Then run the benchmark:

npx tsx src/cli.ts run --config ./skill-optimizer/skill-optimizer.json

How it works

1. Discover callable surface (SDK methods / CLI commands / MCP tools) via tree-sitter or a manifest.
2. Scope the surface with target.scope.include / target.scope.exclude globs.
3. Generate tasks — one prompt per in-scope action, coverage-guaranteed.
4. Benchmark — every configured model attempts every task; static evaluator checks action calls + args.
5. Verdict — PASS/FAIL against two gates (per-model floor, weighted average).
6. Optimize — mutate SKILL.md / docs inside allowedPaths, re-benchmark, accept only if both gates hold, rollback if not.
7. Recommendations — on FAIL, one critic call summarizes what to improve manually.

Configuration reference

All configuration lives in a single skill-optimizer.json file.

target fields

benchmark fields

optimize fields (all optional)

Annotated example config

{
  "name": "my-mcp-project",
  "target": {
    "surface": "mcp",
    "repoPath": ".",
    "skill": "./SKILL.md",
    "discovery": {
      "mode": "auto",
      "sources": ["./src/server.ts"]
    }
  },
  "benchmark": {
    "format": "pi",
    "apiKeyEnv": "OPENROUTER_API_KEY",
    "models": [
      { "id": "openrouter/anthropic/claude-sonnet-4-6", "name": "Claude Sonnet", "tier": "flagship", "weight": 2 },
      { "id": "openrouter/openai/gpt-4o-mini",          "name": "GPT-4o mini",   "tier": "mid",      "weight": 1 }
    ],
    "taskGeneration": {
      "enabled": true,
      "maxTasks": 20,
      "seed": 1
    },
    "output": { "dir": "./benchmark-results" },
    "verdict": {
      "perModelFloor": 0.6,
      "targetWeightedAverage": 0.7
    }
  },
  "optimize": {
    "model": "openrouter/anthropic/claude-sonnet-4-6",
    "apiKeyEnv": "OPENROUTER_API_KEY",
    "thinkingLevel": "medium",
    "allowedPaths": ["SKILL.md"],
    "requireCleanGit": true,
    "maxIterations": 5,
    "minImprovement": 0.02,
    "reportContextMaxBytes": 16000
  }
}

Interpreting the verdict

Every benchmark run produces one of two verdicts: PASS or FAIL.

Two gates must both be satisfied for a PASS:

• benchmark.verdict.perModelFloor (default 0.6): every model must pass at least this fraction of tasks. A single model below the floor fails the run, regardless of the average.
• benchmark.verdict.targetWeightedAverage (default 0.7): the weighted average score across all models must reach this threshold.

benchmark.models[].weight (default 1.0): heavier-weighted models count more toward the weighted average. Use higher weights for flagship models you care most about.

The optimizer only accepts a mutation when:

1. the weighted average improves by at least minImprovement, AND
2. no model that was above the floor drops below it.

Exit codes: 0 = PASS, 1 = FAIL — usable directly in CI pipelines.

Scope & coverage

Control which actions are benchmarked with target.scope:

• target.scope.include (default ["*"]): glob patterns for actions to include.
• target.scope.exclude (default []): glob patterns for actions to exclude.

The * wildcard matches any sequence of characters including dots and slashes — it is not limited to a single path segment.

Examples:

• "Wallet.*" — includes all Wallet methods
• "*.internal*" — excludes anything with "internal" anywhere in the name
• "get_*" — includes only getter actions

Task generation is coverage-guaranteed: every in-scope action gets at least one task. If the first generation pass misses any, a targeted retry runs (max 2 iterations). If coverage still fails, an error names the uncovered actions and suggests either fixing SKILL.md guidance or adding them to scope.exclude.

Cost notes

Rough LLM spend per run:

• Baseline benchmark: N models × M tasks LLM calls.
• Optimizer iteration: 1 mutation call + N models × M tasks re-benchmark per iteration.
• Recommendations: 1 critic call, only on FAIL verdict.

No per-failure LLM calls — feedback is deterministic (structured failure details + patterns + passing/failing diffs).

Dependencies

The optimizer's coding agent is powered by @mariozechner/pi-coding-agent — a small OSS wrapper around OpenRouter that handles agent sessions and tool loops. Models are accessed through OpenRouter — you need one API key for everything.

Troubleshooting

Missing OPENROUTER_API_KEY: Set it in your shell before running:

export OPENROUTER_API_KEY=sk-or-...

Dirty git: The optimizer requires a clean git state in the target repo (requireCleanGit: true by default). Commit or stash uncommitted changes before running.

maxTasks < scope_size: benchmark.taskGeneration.maxTasks must be >= the number of in-scope actions. Run npx tsx src/cli.ts --dry-run --config ./skill-optimizer.json to see the count without making LLM calls.

Empty scope: target.scope.include matched nothing. Check your glob patterns — remember * matches everything including dots.

Legacy skill-benchmark.json: Rename it to skill-optimizer.json. The loader will tell you if it finds the old name.

Contributing

See CONTRIBUTING.md.