mcp-eval-gate

A CI regression gate for MCP servers. It calls your server's tools, compares the results to a saved baseline, and fails the build when the output gets worse, even if the schema did not change.

Read the write-up: MCP server testing in CI: the bug no schema check can see.

Why

An MCP server's tool names and input schemas can stay identical while what a tool returns
quietly gets worse: a refactor breaks a code path, a dependency bump changes behavior.
Nothing crashes, the agent using it just gets worse answers.

Other tools cover parts of this. The official Inspector's CLI runs scripted single-run
assertions. mcp-server-diff compares declared schemas and says it does not test output
correctness. A few other early tools record golden outputs and diff them too, for example
vexyo and
cisco-open/mcptoolkit-test.
mcp-eval-gate is another take on golden-output regression.

How it works

1. You describe how to reach your server and list the tool calls that matter, with the output
   you expect from each, in a golden_set.yaml.
2. run --update-baseline calls every tool and records the results as the baseline, once,
   while everything is known to be good.
3. run calls the same tools again and compares against the baseline. It prints a table and a
   diff, and exits 1 if anything failed or got worse, so any CI job can gate on it.

Install

Requires Python 3.12+.

pip install mcp-eval-gate

# only if you use match_type: judge
pip install "mcp-eval-gate[judge]"

# or run it without installing
uvx mcp-eval-gate --help

Quickstart

mcp-eval-gate init                   # scaffolds golden_set.yaml, which runs as is against the
                                     # official MCP reference server (needs Node.js)
mcp-eval-gate run --update-baseline  # first run: record the baseline
mcp-eval-gate run                    # later runs: gate on regressions

Then point server at your own server and replace the cases. --update-baseline refuses to
record while any case fails, since a baseline should hold known-good output.

Golden set

server:
  command: node
  args: ["dist/index.js"]
  # or, for an HTTP server instead of stdio:
  # url: http://localhost:3000/mcp

cases:
  # substring match
  - id: get-weather-nyc
    tool_name: get_weather
    tool_args:
      city: "New York"
    match_type: contains
    expected_output: "New York"

  # exact match
  - id: cancel-subscription
    tool_name: cancel_subscription
    tool_args:
      immediate: true
    match_type: exact
    expected_output: "cancelled"

  # scored by an LLM against a rubric (needs ANTHROPIC_API_KEY)
  - id: retention-policy-answer
    tool_name: search_docs
    tool_args:
      query: "data retention policy"
    match_type: judge
    judge_criteria: "Answer must state data is retained for 90 days"
    min_judge_score: 0.8

examples/golden_set.yaml is a copy you can edit.

Case fields

A tool call that returns an error always fails its case.

Exit codes

Run as an MCP tool

The package also installs mcp-eval-gate-mcp, an MCP server over stdio with one tool,
run_eval_gate(config_path, baseline_path, update_baseline). Add it to an MCP client's config:

{
  "mcpServers": {
    "mcp-eval-gate": { "command": "mcp-eval-gate-mcp" }
  }
}

Without installing, use uvx:

{
  "mcpServers": {
    "mcp-eval-gate": { "command": "uvx", "args": ["--from", "mcp-eval-gate", "mcp-eval-gate-mcp"] }
  }
}

Validation

validation/ has two runs against real servers, each with the actual command
output committed and steps to reproduce it. No paid API calls.

• A one-line regression planted in the official MCP reference server, caught with exit
  code 1 and a real diff.
• A real bug the official filesystem server shipped (garbled
  text when a multi-byte character straddled a read boundary), caught by running the commit
  before its upstream fix against a baseline from the fixed commit. The
  write-up
  walks through it.

Both are small. They show the gate works end to end on real code, not how often this class
of bug occurs.

Limitations

• Matching is exact, substring, or an LLM judge. There is no normalization for volatile
  output such as timestamps or ids, so output that varies between runs can't be compared
  reliably.
• It checks tool outputs only, not schemas or protocol conformance.
• match_type: judge has only been tested against a stub client, not the live Anthropic API.

Feedback

If you maintain an MCP server, I'd like to know what you would want to test and what got in
the way. Open an issue with the
feedback form.

Development

uv sync --extra judge --dev
uv run pytest --cov=src --cov-report=term-missing
uv run ruff check src tests

License

MIT