toolright
Health Gecti
- License — License: MIT
- Description — Repository has a description
- Active repo — Last push 0 days ago
- Community trust — 23 GitHub stars
Code Uyari
- fs module — File system access in action.yml
- process.env — Environment variable access in fixtures/env-probe-server.ts
Permissions Gecti
- Permissions — No dangerous permissions requested
Bu listing icin henuz AI raporu yok.
Does your agent pick the right tool? Test MCP tool descriptions against real models and keep only the rewrites that measurably help.
🎯 Toolright
Does your AI agent pick the right tool?
Audit MCP tool descriptions for confusable look-alikes, measure accuracy with real models, and keep only the rewrites that help.
Quick Start ·
Why Toolright? ·
Live Results ·
GitHub Action ·
CLI Commands ·
Contributing
Status: v0.0.3, pre-release. 112 automated tests pass (unit, real stdio and HTTP MCP transports, security, a fuzz run). Tier 2 has been run live end-to-end with Ollama (
qwen2.5:3b), and Tier 3 cloud provider testing has been verified live on Groq (openai/gpt-oss-120b). Disambiguation tuning on synthetic and real servers demonstrates measured accuracy improvements on held-out test splits (see Live results so far).
Quick Start
Prerequisites
- Node.js >= 20.0 (download)
- An MCP server to audit (or an existing JSON snapshot from
toolright list --json) - (Optional) Ollama for Tier 2 local evaluations or an API key (Groq, Anthropic, OpenAI, Gemini) for Tier 3 cloud runs.
1. Install or Run Instant
Install globally via npm or pip (or run zero-install via npx):
# npm
npm install -g toolright
# or pip (Python launcher for the same CLI)
pip install toolright
# or zero-install
npx toolright <command>
2. Tier 1: Instant Audit (No API key, No model)
Catch confusable tool look-alikes, missing parameter docs, and pushy superlative claims instantly:
# Audit an MCP server command directly
toolright lint -- node server.js
# Or audit an exported tools snapshot
toolright lint --tools fixtures/company-tools.json
8 tools checked (similarity: lexical, threshold 0.45)
Confusable pairs (most likely cause of wrong picks):
warn find_customer and lookup_customer look similar (62%) and nothing tells the agent when to use which.
warn close_item and resolve_item look similar (54%) and nothing tells the agent when to use which.
warn remove_member and deactivate_member look similar (51%) and nothing tells the agent when to use which.
warn send_alert and send_notice look similar (48%) and nothing tells the agent when to use which.
3. Tier 2 & 3: Measure Real Accuracy & Tune
Evaluate how models actually behave and automatically rewrite weak descriptions with held-out validation gating:
# Evaluate selection accuracy with real prompts
toolright eval --tools server-tools.json -m groq:openai/gpt-oss-120b
# Automatically tune weak descriptions with held-out safety gating
toolright tune --tools server-tools.json --rewriter groq:openai/gpt-oss-120b -m groq:openai/gpt-oss-120b
Why Toolright?
When an LLM agent fails, the bug is often not in the model or the code—it is in your tool descriptions.
Building MCP servers today leaves tool descriptions to human intuition:
- Two tools sound alike (
find_customervslookup_customer), causing wrong selections. - Superlatives ("always use this tool for search") cause greedy misroutes.
- No usage guidance tells the agent which tool beats another.
Toolright treats tool descriptions as optimizable code:
⚡ Tier 1: LintInstant static analysis
|
📊 Tier 2: EvalEmpirical accuracy scoring
|
🎯 Tier 3: TuneHold-out gated rewriting
|
Live Results So Far
What real models did. Tested on local models (qwen2.5:3b), cloud frontier models (openai/gpt-oss-120b on Groq), and the official filesystem server.
| Run | Server / Fixture | Result |
|---|---|---|
eval, qwen2.5:3b, 38 hand-written prompts x 1 |
fixtures/confusing-server.ts (5 tools) |
95% correct (36/38, 95% CI 83-99%), 0 API errors. Both misses were inside the confusable pairs lint flagged (list_issues vs search_issues, get_user vs find_user). |
tune, picker & rewriter qwen2.5:3b, 2 runs per prompt |
fixtures/company-server.ts (8 tools in 4 confusable pairs) |
2 rewrites accepted, accuracy improved from 84% (27/32) to 91% (29/32, 95% CI 76-97%) on held-out test prompts. Validation accuracy gained on find_customer (88% -> 91%) and lookup_customer (91% -> 97%). 2 candidates rejected (remove_member, deactivate_member). |
eval, groq:openai/gpt-oss-120b, 64 prompts x 1 |
fixtures/company-server.ts (8 tools in 4 confusable pairs) |
96% correct (49/51, 95% CI 87-99%) via Groq cloud API. Correctly resolved 6 out of 8 tools with 100% accuracy. |
eval & tune, groq:openai/gpt-oss-120b, 84 prompts |
@modelcontextprotocol/server-filesystem (official 14 tools) |
Baseline accuracy 60% (12/20) on held-out test prompts with severe confusion on read_file vs read_text_file. A rewrite candidate for read_text_file improved validation accuracy (76% -> 78%) but was rejected by the non-regression safety gate because it caused regressions on write_file. |
Key Learnings & Guarantees
- Non-regression safety gate: Candidate rewrites that increase accuracy overall are strictly rejected if they cause adjacent tools to regress.
- Accurate failure mode detection: With easy prompts where models achieve 100% baseline accuracy, Toolright detects lack of headroom and skips unnecessary rewrites.
- The gate measures accuracy, not truth: Read every proposed diff. Use a strong model for
--rewriterto ensure descriptions stay factual.
Features
1. Interactive HTML Visual Reports
toolright eval and toolright tune generate standalone, dark-mode visual reports (report.html and eval.html):
- Visual accuracy badges & Wilson confidence intervals
- Interactive per-tool progress bars
- Top confusable wrong-pick matrices
- Side-by-side diff viewers for accepted vs rejected rewrites
2. Mine Prompts from Real Logs (ingest-logs)
Don't rely only on synthetic prompts. Extract actual user prompts from OpenAI and Anthropic JSONL chat transcripts:
toolright ingest-logs production-transcripts.jsonl --tools tools.json -o real-prompts.json
3. Toolright as an MCP Server (toolright serve)
Expose Toolright directly to your AI coding agents (Claude Desktop, Cursor, Cline, Roo Code):
toolright serve
Provides the lint_tools tool so agents can audit and improve tool descriptions while building MCP servers.
GitHub Action CI
Fail the build or automatically post PR comments when an MCP tool description is confusable, missing, or vague:
Create .github/workflows/toolright.yml:
name: Toolright Audit
on: [pull_request]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: ab-rar-6024/[email protected]
with:
tools: "fixtures/company-tools.json"
fail-on: "error"
comment-on-pr: "true"
github-token: ${{ secrets.GITHUB_TOKEN }}
CLI Commands
| Command | Description | Cost |
|---|---|---|
toolright list [server...] |
Print the tools an MCP server exposes | Free |
toolright lint [server...] |
Tier 1: Check for confusable pairs, superlatives, and schema gaps | Free |
toolright gen [server...] |
Generate direct & near-miss test prompts for human review | Model calls |
toolright eval [server...] |
Measure model selection accuracy across held-out splits | Model calls |
toolright tune [server...] |
Baseline, rewrite weak descriptions, and gate improvements | Model calls |
toolright ingest-logs <file> |
Extract evaluation prompts from real conversation transcripts | Free |
toolright serve |
Run Toolright as an MCP server over stdio | Free |
Supported Providers
Specify any provider with --model <provider>:<model-id>:
| Provider | Model Spec Example | Environment Variable |
|---|---|---|
| Ollama (Local) | -m ollama:qwen2.5:3b |
None required (OLLAMA_HOST optional) |
| Groq | -m groq:openai/gpt-oss-120b |
GROQ_API_KEY |
| Anthropic | -m anthropic:claude-3-5-sonnet-latest |
ANTHROPIC_API_KEY |
| OpenAI | -m openai:gpt-4o |
OPENAI_API_KEY |
| Gemini | -m gemini:gemini-1.5-pro |
GEMINI_API_KEY |
| Custom OpenAI-compatible | -m custom:<model> |
TOOLRIGHT_BASE_URL & TOOLRIGHT_API_KEY |
| Heuristic (Offline stand-in) | -m heuristic |
Free / Offline keyword matcher |
Security & Privacy
An MCP server is untrusted code, and so is model output. Toolright protects your development environment:
- API Keys are scrubbed: Never written to reports, never sent over unencrypted HTTP, and never passed to MCP server child processes.
- Hostile server defenses: Hard timeouts, tool count caps (max 500), and giant payload truncations prevent hanging the CLI.
- Terminal safety: Strips escape sequences and bidirectional Unicode overrides from names and descriptions.
- Non-destructive: Toolright proposes description diffs in
descriptions.diff; it never modifies your server code directly.
See SECURITY.md for details.
Roadmap
- Run on 5-10 popular MCP servers and publish before/after benchmarks (started:
@modelcontextprotocol/server-filesystemtested) - Verify Tier 2 live with Ollama
- Verify Tier 3 cloud providers live (verified with Groq)
- GitHub Action (
action.yml) with automated PR comments - Ship Toolright itself as an MCP server (
toolright servewithlint_tools) - Prompts mined from real agent logs (
toolright ingest-logs) - Standalone visual HTML reports (
report.htmlandeval.html) - PyPI and npm automated release workflows on git release (
.github/workflows/release.yml) - Negative control evaluation (measuring whether models trigger tools on non-tool conversational prompts)
Contributing
We welcome contributions of all kinds! Start by running the test suite:
git clone https://github.com/ab-rar-6024/toolright.git
cd toolright
npm install
npm test # 112 tests pass: unit, integration, security; no keys needed
npm run typecheck
npm run build
See CONTRIBUTING.md for guidelines.
License
Toolright is open-source software licensed under the MIT License.
Yorumlar (0)
Yorum birakmak icin giris yap.
Yorum birakSonuc bulunamadi