mcp-observatory

The GitHub-native CI, SARIF, and security gate for MCP servers before agents depend on them.

Agents should not depend on tools nobody tests. MCP Observatory turns a local MCP check into release-gate evidence maintainers already understand: GitHub Actions, security findings, schema drift detection, PR reports, score badges, agent-accessible diagnostics, and GitHub Code Scanning SARIF.

Start with one server:

npx @kryptosai/mcp-observatory test npx -y my-mcp-server

Then convert the passing check into CI with Code Scanning:

npx @kryptosai/mcp-observatory setup-ci --all --command "npx -y my-mcp-server" --sarif

See the launch page, GitHub Code Scanning demo, and GitHub Code Scanning for MCP servers.

Two more fast paths:

Cloned this repo? Start here: CLONED_THIS.md. Want to contribute? Add one server to the MCP Target Registry or use the Agent Task Pack.

Add MCP CI in one command:

npx @kryptosai/mcp-observatory setup-ci --all --command "npx -y my-mcp-server"

Upload normalized MCP findings to GitHub Code Scanning when you want a security-native release gate:

npx @kryptosai/mcp-observatory setup-ci --all --command "npx -y my-mcp-server" --sarif

Add Observatory as an agent-accessible MCP server:

claude mcp add mcp-observatory -- npx -y @kryptosai/mcp-observatory serve

Building an autonomous agent, OpenClaw-style productivity machine, MCP gateway, or bot runtime? Start with the agent runtime quickstart, copy the OpenClaw MCP reliability agent template, or point your agent at llms.txt and AGENTS.md.

Or test a server immediately:

npx @kryptosai/mcp-observatory test npx -y @modelcontextprotocol/server-everything

Use it as a CLI, a GitHub Action, or an MCP server that lets your AI agent scan, test, record, replay, and verify other MCP servers autonomously.

Why MCP Observatory

MCP servers are becoming production dependencies. If agents rely on them, teams need a way to catch broken tools, unsafe schemas, schema drift, slow responses, and security footguns before those failures reach users.

Observatory gives maintainers and teams:

• One-command CI setup with setup-ci --all
• GitHub PR comments for compatibility, drift, and security findings
• GitHub Code Scanning SARIF for normalized MCP findings
• Health score badges for public trust signals
• Record/replay/verify workflows for regression testing
• MCP server mode so agents can inspect other MCP servers directly
• Production support path for hosted history, private repo reporting, certification, support, and fleet visibility

See the launch page, GitHub Code Scanning for MCP servers, Code Scanning demo, target gallery, target registry, target contribution guide, Agent Task Pack, setup-ci --doctor, MCP server security field guide, Safety Methodology, MCP Server Safety Index, June 2026 safety field report, reference evaluations, MCP lock files, public proof, campaign attribution, local metrics dashboard, and commercial support.

For Security And Platform Teams

MCP servers are becoming part of the AI software supply chain. Agents need reliable, testable, auditable tools before those tools become dependencies in mission-critical workflows.

MCP Observatory gives security and platform teams MCP server CI, schema drift detection, security findings, SARIF/HTML/Markdown reports, GitHub Code Scanning upload, and a path toward certification or fleet visibility. Local OSS use stays free; production, private repo, and fleet usage can move through a paid MCP Readiness Review.

Production Support

Local OSS use stays free under MIT. Teams running MCP in production can use the MCP Readiness Review for CI rollout, SARIF/Code Scanning setup, private repo review, recurring security reports, certification review, support, and fleet visibility. The default package starts at $2,500.

Run npx @kryptosai/mcp-observatory cloud, open a pilot request from the issue chooser, or see COMMERCIAL.md. Also see privacy and telemetry, campaign attribution, and terms for production use.

Quick Start

Scan every MCP server in your Claude config:

npx @kryptosai/mcp-observatory

Go deeper — also invoke safe tools to verify they actually run:

npx @kryptosai/mcp-observatory scan deep

Test a specific server:

npx @kryptosai/mcp-observatory test npx -y @modelcontextprotocol/server-everything

Add it to Claude Code as an MCP server:

claude mcp add mcp-observatory -- npx -y @kryptosai/mcp-observatory serve

Or add it manually to your config:

{
  "mcpServers": {
    "mcp-observatory": {
      "command": "npx",
      "args": ["-y", "@kryptosai/mcp-observatory", "serve"]
    }
  }
}

Commands

Run with no arguments for an interactive menu:

What It Does

Check capabilities — connects to a server and verifies tools, prompts, and resources respond correctly.

Invoke tools — goes beyond listing. Actually calls safe tools (no required params / readOnlyHint) and reports which ones work and which ones crash.

npx @kryptosai/mcp-observatory scan deep

Detect schema drift — diffs two runs and surfaces added/removed fields, type changes, and breaking parameter changes.

npx @kryptosai/mcp-observatory diff run-a.json run-b.json

Recommend servers — scans your project for languages, frameworks, databases, and cloud providers, then cross-references the MCP registry to suggest servers you're missing.

npx @kryptosai/mcp-observatory suggest

Or ask your agent "what MCP servers should I add?" when running in MCP server mode.

Security scanning — analyzes tool schemas for dangerous patterns: shell injection surfaces, broad filesystem access, missing auth, and credential leakage in responses.

npx @kryptosai/mcp-observatory test --security npx -y my-mcp-server

Record / replay / verify — capture a live session, replay it offline in CI, and verify nothing changed. Like VCR for MCP.

# Record a session
npx @kryptosai/mcp-observatory record npx -y @modelcontextprotocol/server-everything

# Replay offline (no server needed)
npx @kryptosai/mcp-observatory replay .mcp-observatory/cassettes/latest.cassette.json

# Verify the live server still matches
npx @kryptosai/mcp-observatory verify cassette.json npx -y @modelcontextprotocol/server-everything

Watch for regressions — re-runs checks on an interval and alerts when something changes.

npx @kryptosai/mcp-observatory watch target.json

Scan locations

When you run scan, it looks for MCP configs in:

• ~/.claude.json (Claude Code)
• ~/Library/Application Support/Claude/claude_desktop_config.json (Claude Desktop, macOS)
• %APPDATA%/Claude/claude_desktop_config.json (Claude Desktop, Windows)
• .claude.json and .mcp.json (current directory)

CI / GitHub Action

Add Observatory to your MCP server's CI pipeline:

npx @kryptosai/mcp-observatory setup-ci --all --command "npx -y my-mcp-server"

Check the adoption kit:

npx @kryptosai/mcp-observatory setup-ci --doctor

Or create the workflow manually:

# .github/workflows/observatory.yml
name: MCP Server Check
on: [pull_request]

permissions:
  contents: read

jobs:
  observatory:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: KryptosAI/mcp-observatory/[email protected]
        with:
          command: npx -y my-mcp-server
          deep: true
          security: true
          comment-on-pr: false
          set-status: false

Action inputs:

The action can comment on PRs and set commit statuses when the workflow grants write permissions. setup-ci generates read-only third-party-friendly workflows by default and lets maintainers opt into comments/statuses later. init-ci remains available as a backward-compatible alias. See action/README.md for all options.

Production teams can add hosted CI history, private-repo reporting, recurring security reports, certification review, support, and fleet visibility. Run npx @kryptosai/mcp-observatory cloud, see COMMERCIAL.md, or open a pilot request from the issue chooser.

Certified by MCP Observatory

MCP server maintainers can add a public compatibility/security signal to their README:

[![MCP Observatory](https://img.shields.io/badge/MCP%20Observatory-enabled-2563eb)](https://github.com/KryptosAI/mcp-observatory)

Or generate a score badge from a live check:

npx @kryptosai/mcp-observatory badge npx -y my-mcp-server --output docs/mcp-health.svg

See the certification distribution loop for the GitHub Action template, maintainer PR body, and badge rollout playbook.

Generate a pilot-ready production/security report from local run artifacts:

npx @kryptosai/mcp-observatory enterprise-report \
  --account "Your Company" \
  --format html \
  --output observatory-enterprise-report.html

For clearer internal account attribution in CI, set:

MCP_OBSERVATORY_ORG=your-company.com
MCP_OBSERVATORY_CONTACT=your-team-contact

Testing Feishu/Lark integrations? See the Feishu/Lark MCP guide.

Lock Files

$ npx @kryptosai/mcp-observatory lock              # Snapshot all server schemas
$ npx @kryptosai/mcp-observatory lock verify        # Verify no drift since last lock

Lock files are the package-lock for AI tools: commit the MCP contract, then make every tool, schema, prompt, or resource drift visible in CI. See MCP lock files.

Trend Tracking

$ npx @kryptosai/mcp-observatory history            # Show health trends over time

Nightly Scans

$ npx @kryptosai/mcp-observatory ci-report          # Generate regression report for CI

MCP Server Mode

No other testing tool is itself an MCP server. Add Observatory as a server and your AI agent can autonomously test, diagnose, and monitor your other MCP servers.

claude mcp add mcp-observatory -- npx -y @kryptosai/mcp-observatory serve

Your agent gets 10 tools:

An AI tool that checks other AI tools. It is a tool testing tools that serve tools.

Security

The MCP server runs inside AI hosts where an LLM chooses which tools to call. To prevent prompt-injection attacks:

• Command allowlist: Only npx, node, python, python3, uvx, docker, deno, bun are permitted as base executables. The CLI has no restrictions.
• Path validation: File-reading tools are constrained to the runs/cassettes directories.
• No arbitrary execution: Use the CLI for unrestricted commands.

CLI vs MCP: Intentional Differences

Compatibility

Works with any MCP server that uses standard transports:

Servers needing API keys work via env in the target config. Python servers work via uvx. See the full compatibility matrix for tested servers and known issues.

Target config files

For more control (env vars, metadata, custom timeout):

{
  "targetId": "filesystem-server",
  "adapter": "local-process",
  "command": "npx",
  "args": ["-y", "@modelcontextprotocol/server-filesystem", "."],
  "timeoutMs": 15000,
  "skipInvoke": false
}

npx @kryptosai/mcp-observatory run --target ./target.json

HTTP / SSE targets

{
  "targetId": "my-remote-server",
  "adapter": "http",
  "url": "https://mcp.example.com/mcp",
  "authToken": "${MCP_SERVER_TOKEN}",
  "headers": {
    "X-Api-Key": "$MCP_SERVER_API_KEY"
  },
  "timeoutMs": 15000
}

Target configs support ${VAR}, $VAR, and env:VAR references in authToken, headers, and local-process env values.

How It Compares

Each tool has strengths. Observatory focuses on regression detection and CI-friendly workflows. mcp-recorder is great as a transparent proxy. MCPBench is the go-to for performance benchmarking. mcp-jest is ideal if you're already in a Jest workflow.

Prior Art

The record/replay/verify pattern is inspired by:

• VCR (Ruby) — pioneered cassette-based HTTP record/replay
• Polly.js (Netflix) — HTTP interaction recording for JavaScript
• mcp-recorder — MCP-specific traffic recording proxy
• MCPBench — MCP server benchmarking
• mcp-jest — Jest-style testing for MCP servers

Limitations

• Servers requiring interactive OAuth (e.g., Google Drive) need pre-authentication before Observatory can connect
• Custom WebSocket transports (e.g., BrowserTools MCP) are not supported
• A few servers time out or close before init — see known issues and compatibility

Contributing

See CONTRIBUTING.md for guidelines. The fastest way to contribute is to add a real passing target with a distinct capability shape, a clearer report surface, or a cleaner startup diagnosis.

If Observatory saved you a broken deploy, consider giving it a star. It helps others find the project.