LLM Endpoint Doctor

English | 简体中文

Give it an LLM relay URL and API key. Get the correct base URL, supported
protocols, models, streaming status, protocol-specific tool compatibility, and
ready-to-copy client config.

Status: usable V0.2 MVP. The CLI has zero runtime dependencies.

When do you need it?

You have been given an LLM relay, a self-hosted model service, or an
OpenAI-compatible API. Basic chat appears to work, but you do not know whether
the endpoint can support a real agent or automated workflow.

Typical situations include:

• You want to build an agent but do not know whether the relay supports a full
  tool-call continuation.
• You want to connect Codex, Cline, OpenCode, or another coding agent but do not
  know whether it needs Chat Completions or the Responses API.
• You are preparing data labeling, batch classification, or structured
  extraction and need to establish that basic generation and streaming work.
• You do not know whether the Base URL needs /v1, or whether you received a
  complete endpoint URL.
• Normal chat succeeds, but the model cannot continue after a tool result is
  returned.
• You want client-side MCP but do not know whether the API has the required
  function-calling behavior.
• A relay claims OpenAI compatibility without saying which protocols and models
  actually work.
• The same model works through an official API but returns unexplained 400, 404,
  or streaming errors through the relay.

LLM Endpoint Doctor performs automatic discovery and small capability probes
before you commit to an integration. It does more than ask the endpoint to reply
to “hello”: it tests whether the endpoint can support practical applications,
agents, and automation.

What it answers

• Does the base URL need /v1?
• Is the input a base URL or a full /chat/completions / /responses endpoint?
• Does the relay support OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages?
• Can /models be read? Which text model should be tried?
• Does SSE streaming actually emit recognizable events?
• Can each protocol complete a full function/tool loop?
• Is client-side MCP likely to work, and through which protocol?
• Does the Responses endpoint pass basic, streaming, and full Codex agent checks?

“Client-side MCP likely” only means a normal function-call loop succeeded. It does
not claim support for OpenAI hosted MCP, Anthropic MCP Connector, web search, file
search, computer use, or other provider-hosted tools.

The current release also does not validate long-run JSON Schema / Structured
Outputs stability or generate dedicated Cline and OpenCode configuration
templates. Those capabilities remain on the roadmap and are not presented as
verified by the current report.

Requirements

• Node.js 20 or newer
• No npm install is required to run from the repository

Install

Install the CLI globally:

npm install -g llm-endpoint-doctor
endpoint-doctor --version

Or run it without a global install:

npx llm-endpoint-doctor probe https://api.example.com

Agent-friendly local JSON

Let Codex or another agent create a local configuration from only the base URL:

endpoint-doctor init https://api.example.com --model gpt-5

This creates .endpoint-doctor.local.json:

{
  "baseUrl": "https://api.example.com",
  "apiKey": "PASTE_API_KEY_HERE",
  "apiKeyEnv": null,
  "model": "gpt-5",
  "quick": false,
  "aggressive": false,
  "timeoutMs": 15000,
  "maxCandidates": 4,
  "requirements": [],
  "output": "report.json"
}

The user only needs to replace PASTE_API_KEY_HERE. The agent can then run:

endpoint-doctor probe

The default local file is auto-detected and is included in .gitignore. It is a
plain-text secret file: never commit, upload, print, or paste it into a chat. For
CI or shared machines, leave apiKey as the placeholder and set apiKeyEnv to
the name of a secret environment variable instead.

Use endpoint-doctor.example.json as a committed template. A different local
path can be selected explicitly:

endpoint-doctor probe --config ./private/relay.json

When a local config supplies the key, a different positional URL is rejected so
the credential is not silently sent to another endpoint.

Direct CLI usage

export API_KEY='sk-...'
endpoint-doctor probe https://api.example.com --key-env API_KEY

To run directly from a source checkout instead:

node ./bin/endpoint-doctor.mjs probe https://api.example.com --key-env API_KEY

The tool may make several small billable model requests. Use --quick to perform
only model and basic protocol discovery.

Useful options:

--config FILE            Read local JSON configuration
--quick                  Skip streaming and tool-loop tests
--aggressive             Try /api/v1, /openai/v1 and other extra paths
--model MODEL            Do not auto-select from /models
--require LIST           Require capabilities and return exit code 2 if missing
--json                    Print machine-readable JSON
--out report.json        Save a reusable report
--timeout 30000          Per-request timeout in milliseconds
--max-candidates 4       Limit URL candidates

Automation requirements

Agents and CI can require exact capabilities instead of parsing human output:

endpoint-doctor probe --config .endpoint-doctor.local.json \
  --require openai-responses,openai-responses-stream,openai-responses-tools

Exit code 0 means every requirement passed. Exit code 2 means the report was
produced but one or more required capabilities are missing. Supported names:

openai-chat                 openai-chat-stream       openai-chat-tools
openai-responses            openai-responses-stream  openai-responses-tools
anthropic                   anthropic-stream         anthropic-tools
codex-basic                 codex-stream             codex-agent
client-mcp                  any-stream               any-tools

codex-agent is strict: Responses basic generation, SSE streaming, and the
Responses function/tool continuation must all pass. A tool loop working only
through Chat Completions no longer makes the endpoint Codex agent-compatible.

Generate configuration

endpoint-doctor config report.json --target codex
endpoint-doctor config report.json --target curl
endpoint-doctor config report.json --target openai-js

Codex config is generated when a working Responses API endpoint is detected.
If full Codex agent compatibility did not pass, the generated config contains a
warning instead of presenting the endpoint as fully verified.

Tool-loop test

The test gives the model one harmless function:

get_magic_number() -> 42

The model must call it and then return 84. This catches gateways that accept
tools but fail when a tool / function_call_output result is sent back.

Security

• Prefer local JSON, --key-env, or apiKeyEnv instead of --key in shell history.
• .endpoint-doctor.local.json is ignored by Git but contains a plain-text key.
• Keys are never written into diagnostic JSON reports.
• The CLI does not save full request headers.
• A configured key cannot be reused with a different positional URL.
• Tests use tiny prompts but may still incur upstream charges.
• The tool does not modify client configuration files.

Current scope

Included in V0.2:

• URL normalization and /v1 discovery
• OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages
• model listing with Bearer and Anthropic authentication attempts
• plain generation, SSE streaming, and complete function/tool continuation
• protocol-scoped tool and agent readiness
• agent-friendly local JSON with automatic discovery
• machine-enforced capability requirements and exit codes
• human and JSON reports
• Codex, curl, environment and OpenAI JS configuration output

Not included yet:

• OS keychain credential storage
• Codex skill, plugin, or MCP server packaging
• hosted MCP execution
• provider-hosted web_search, file_search, computer or code interpreter
• Chat Completions ↔ Responses proxy mode
• GUI, key rotation, billing, routing or load balancing

Development

npm test
npm run check

Tests run against a local mock relay implementing all three supported protocols.
Both commands are cross-platform.

Contributing and security

See CONTRIBUTING.md before opening a pull request. Report
vulnerabilities through the process in SECURITY.md, and never put
a live API key or private endpoint in a public issue.

Roadmap

1. Add OS keychain-backed profiles and keep plain-text local JSON as an explicit fallback.
2. Add a local STDIO MCP server and a Codex skill, then package both as a plugin.
3. Add diagnostic signatures for lost reasoning content and malformed streamed tool calls.
4. Add optional hosted-tool probes that are off by default.
5. Add a budget-limited local broker for agent-driven dataset generation.

License

MIT