nuphus-mcp

Desktop automation MCP server — computer use for any AI agent. See the screen, control windows/mouse/keyboard, and drive Chrome over the Model Context Protocol (stdio). Desktop & browser automation need no API key; OCR runs locally; vision plugs into your own vision LLM (OpenAI-compatible, BYOK).

nuphus-mcp is a lightweight, cross-platform desktop automation MCP server
that exposes desktop + browser automation as standard MCP tools. It speaks
JSON-RPC 2.0 over stdio — no daemon, no network service, one binary. Claude
Desktop, Cursor, VS Code, Copilot, or any MCP client can connect and
immediately control the screen, windows, keyboard/mouse, and Chrome —
computer use for any AI agent — desktop & browser automation need no API
key; local OCR is built in; vision works with your own vision LLM
(OpenAI-compatible, BYOK).

🇨🇳 Mainland China mirror: this repo is mirrored on
Gitee for fast in-China access
(Chinese docs served by default there). 中文文档

┌──────────────────┐   stdio JSON-RPC   ┌──────────────────────┐
│  Any MCP Client  │  ───────────────►  │      nuphus-mcp      │
│  (Claude/Cursor/ │  ◄───────────────  │  desktop-api crate   │──► screen/window/mouse/keyboard
│   Nuphus itself) │  single-line JSON  │  nuphus-browser crate│──► Chrome (CDP)
└──────────────────┘                    └──────────────────────┘

Features

• 36 MCP tools (15 desktop + 21 browser) — screenshots, window control,
  mouse/keyboard, Chrome CDP automation, and more — see TOOLS.md /
  TOOLS.zh-CN.md for the full reference.
• Desktop automation: screen size, screenshot (BMP/base64), window list,
  window activate/screenshot/move/resize/info, mouse click/drag/scroll/position,
  keyboard input/hotkey, clipboard write/clean — implemented on the
  desktop-api crate (xcap + Win32, no Tauri dependency).
• Computer vision pair: desktop_vision (BYOK — send a screenshot to your
  own vision model via an OpenAI-compatible API) + desktop_perceive (local
  OCR with PaddleOCR, models auto-downloaded on first run; optional YOLO icon
  detection). Used together they give AI agents both semantic understanding
  and pixel-precise coordinates — the battle-tested vision→perceive flow from
  the Nuphus desktop app. See TOOLS.md for BYOK env vars, model
  setup, and the recommended flow.
• Browser automation: navigate, snapshot (accessibility tree with @N
  refs), click, type, exec, scroll, extract, screenshot, evaluate,
  back/forward, wait_for, cookies get/set/import, upload, tabs, downloads —
  implemented on nuphus-browser (chromiumoxide CDP).
• Zero-cost stdio: no HTTP server, no daemon. The process reads
  single-line JSON from stdin and writes responses to stdout.
• Safety-first: destructive tools are annotated per the MCP spec; optional
  strict-confirm mode; path validation for screenshot/upload.

Repository Layout

nuphus-mcp/
├── Cargo.toml                  # workspace root
├── TOOLS.md / TOOLS.zh-CN.md   # 36-tool reference
├── crates/
│   ├── nuphus-mcp/             # MCP Server (this repo's product)
│   ├── nuphus-browser/         # Browser automation core (CDP)
│   └── desktop-api/            # Desktop control core (vendored)
└── ...

Prerequisites

• Rust toolchain (stable) — build from source with Cargo.
• Chrome or Edge — required for browser tools. The server auto-detects an
  installed browser; if none is found, browser_* tools return a clear error.
• Windows recommended for full desktop control — see Platform Support below.

Platform Support

API Keys & Local Models

Vision — BYOK, OpenAI-compatible

desktop_vision uses your own vision model through an OpenAI-compatible
Chat Completions endpoint. Nothing is required unless you call this tool — and
when it is not configured the tool returns a clear error instead of silently
failing.

Perceive models (local, auto-downloaded)

desktop_perceive runs PaddleOCR locally with ONNX Runtime. The first call
downloads the OCR models automatically into %APPDATA%\Nuphus\models (or
NUPHUS_MODELS_DIR). Download failures return a clear error with manual
instructions. YOLO icon detection (icon_detect.onnx) is an optional
enhancement and is not auto-downloaded. See
TOOLS.md → Vision & Local Models.

All other tools need no API key.

Install & Run

Install via npm (recommended — all platforms, prebuilt binaries):

npm install -g @nuphus/nuphus-mcp

The nuphus-mcp meta package installs the prebuilt binary for your platform
automatically (Windows x64/arm64, macOS arm64, Linux x64/arm64) and puts the
nuphus-mcp command on your PATH. No Rust toolchain needed:

nuphus-mcp   # stdio MCP server

Build from source (requires the Rust toolchain):

cargo build --release -p nuphus-mcp
# binary at target/release/nuphus-mcp(.exe)

The server reads newline-delimited JSON from stdin and writes JSON-RPC
responses to stdout. Logs go to stderr.

# quick smoke test
echo '{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test"}}}' | nuphus-mcp

MCP Client Configuration

Point any MCP client at nuphus-mcp. After npm install -g @nuphus/nuphus-mcp the command is on your PATH; otherwise use the absolute
path to the binary (nuphus-mcp / nuphus-mcp.exe).

Claude Desktop — claude_desktop_config.json:

{
  "mcpServers": {
    "nuphus-mcp": {
      "command": "nuphus-mcp",
      "args": []
    }
  }
}

Any MCP client (generic mcpServers JSON):

{
  "mcpServers": {
    "nuphus-mcp": {
      "command": "nuphus-mcp",
      "args": [],
      "env": {}
    }
  }
}

Supported MCP methods: initialize, notifications/initialized, ping,
tools/list, tools/call.

Demo

A self-contained stdio client that walks through
initialize → tools/list → tools/call:

cargo build -p nuphus-mcp
cargo run -p nuphus-mcp --example demo

Tests

cargo check --workspace
cargo test -p nuphus-mcp          # protocol + security + vision + models tests (28)

Safety

This server can physically control the machine it runs on. Read
SECURITY.md and the Safety Annotations section of
TOOLS.md before deploying. Recommended: run with
--confirm-write (or NUPHUS_MCP_CONFIRM_WRITE=1) so write tools require an
explicit "confirm": true argument.

License

MIT