screenshot → guess coordinates → click → hope      ✗  slow, fragile, moves your mouse
snapshot   → act on #ref       → verify the diff   ✓  what this kit does

• One command per client. Works with Claude Code, Claude Desktop, Codex,
  Cursor, Gemini CLI, opencode, pi and DeepSeek Harness. It is a stdio MCP
  server with no runtime dependencies beyond pyobjc and Pillow.
• Accessibility first. Each element comes back with a stable #ref, its
  role, its label and exact screen geometry. Re-observing returns only the
  diff. Electron and Chromium apps get their full tree turned on
  automatically.
• Background input. Clicks, keys, scrolls, drags and Unicode typing are
  posted to the target process, so the user can keep working. CJK, emoji and
  accents arrive intact, and the clipboard is left alone.
• Verified, never optimistic. Every result separates action_sent from
  verified. Failures say whether it is safe to retry, whether to
  reobserve, or whether to never retry.
• Safe by default. There is a built-in deny list for password managers
  and system auth. The kit refuses to type into a password field that holds
  Secure Event Input, redacts secure fields, and blocks lock, log-out and
  force-quit chords. It also offers a dry-run mode, a size-only audit log and app
  allow/deny lists.
• Vision only when needed. Set-of-mark screenshots draw numbered marks
  linked to refs, and on-device Apple Vision OCR returns screen coordinates.
  Both are fallbacks for canvas, game and custom-drawn UI.

Contents

• Quick start
• Tools
• How an agent should use it
• Safety and privacy
• CLI reference
• Other agent integrations
• Jev semantic guards (optional)
• Related projects
• Known limitations
• Repository layout

Quick start

Requirements

• macOS 12 or later, Python 3.10 or later.
• Grant two permissions to the app that runs your agent (your terminal, Claude
  Desktop, Cursor, and so on). macos-cu doctor tells you exactly which app
  is missing which permission.
  • Accessibility: needed for reading the tree, AX actions and posted input.
  • Screen Recording: needed for screenshots and OCR (without it every
    frame is black).

Claude Code

Pick one of these:

# A. As a plugin: MCP server, skill and /macos-doctor command
/plugin marketplace add Sur-Cai/macos-computer-use-kit
/plugin install macos-computer-use@macos-computer-use-kit

# B. As a plain MCP server
claude mcp add --scope user macos-computer-use -- uvx macos-computer-use-kit mcp

The plugin finds macos-cu on your PATH and otherwise falls back to uvx
or pipx run. Nothing has to be pre-installed except uv
or pipx.

Every other client, with one command

pipx install macos-computer-use-kit          # or: pip install / uv tool install
macos-cu setup claude-code                   # also: claude-desktop, codex, cursor, gemini, opencode
macos-cu setup skill                         # copy the agent skill to Claude, Codex and opencode
macos-cu doctor                              # permissions, displays, OCR, policy

setup is idempotent and supports --dry-run. JSON configs are merged, so
your other servers are preserved, and a .bak copy is written before the
first change. --read-only registers only the observation tools.

{
  "mcpServers": {
    "macos-computer-use": {
      "command": "uvx",
      "args": ["macos-computer-use-kit", "mcp"]
    }
  }
}

Codex (~/.codex/config.toml):

[mcp_servers.macos-computer-use]
command = "uvx"
args = ["macos-computer-use-kit", "mcp"]

GUI apps do not inherit your shell's PATH. Use absolute paths
(which uvx) if the client cannot find the command. macos-cu setup print
prints the exact argv for your machine.

Tools

The MCP server exposes 18 tools. Observation tools carry readOnlyHint, so
clients can auto-approve them. To trim the list, use mcp --read-only,
--tools a,b or --exclude-tools c.

How an agent should use it

macos_doctor                                   once: permissions + display layout
macos_snapshot app="Notes"                     → #a1b2c3d4 AXButton 28x28 @screen[812,64] New Note …
macos_act ref="a1b2c3d4"                       → {"ok":true,"action_sent":true,"verified":true}
macos_type app="Notes" text="周会纪要 ✅"        → Unicode events, clipboard untouched
macos_snapshot app="Notes" diff_against=<path> → only what changed

The rules behind the loop, which the bundled skill teaches:

1. Observe semantically. Snapshot before you screenshot.
2. Act on the element, not the pixel. Prefer menu paths and AX actions,
   then a click at center_screen, then keyboard shortcuts.
3. Verify, don't sleep. Use a snapshot diff or macos_wait.
4. Never retry blindly. If action_sent is true, re-observe before
   repeating anything that can't safely run twice. A second "Send" is a
   second message.
5. stale_ref means re-observe. It never means "pick the nearest element".

Coordinate spaces

Displays placed left of or above the primary have negative coordinates.
That is normal, and macos_doctor prints the layout.

Safety and privacy

Everything runs locally. The kit makes no network calls, except the optional
Jev guard, which only runs when you configure a key and call it.

Local files stay private to your account:

• Snapshot caches and the audit log live in ~/.cache/macos-computer-use/,
  or wherever MACOS_CU_CACHE_DIR points.
• They are created with 0700/0600 permissions.
• Snapshots are pruned to the newest 20 (MACOS_CU_SNAPSHOT_KEEP).
• Screenshots and OCR captures are written only where you ask (out=).
  Temporary OCR captures are deleted.

CLI reference

Every command prints JSON (or compact text for ax tree/find/snapshot), so an
agent can drive the kit from any shell. Exit codes are stable: 0 ok,
2 usage or permission, 3 not found / stale ref, 4 capture failed,
5 target changed, 6 refused by policy, 7 timeout.

macos-cu ax snapshot --app Finder --interactive
macos-cu ax press --app Finder --ref 9d2261d7
macos-cu menu select --app Safari --path "File > New Private Window"
macos-cu input type --app Notes --text "你好, world 👋"
macos-cu input click --window-id 12345 --x 171 --y 28 --expect "<pid:wid:x:y:w:h>"
macos-cu shot annotate --app "System Settings" --out /tmp/marks.png
macos-cu ocr --app Preview --text "Total"

Other agent integrations

The MCP server covers most clients. There are native bridges for two
harnesses that prefer their own tool format. Both are thin wrappers over the
same CLI and call it with argv arrays (shell: false).

Install the CLI first (pipx install macos-computer-use-kit). If an app
launched from the Dock can't find it, set MACOS_CU_BIN=/abs/path/to/macos-cu.
All artifacts share one version number and are released together. See
packages/pi and packages/dsh
for details.

Jev semantic guards (optional)

This is the only feature that needs a key. Right before an irreversible step,
a small model gives calibrated judgments: is this still the intended
recipient? does the field hold the intended text? what blocks the action?
Code decides whether to proceed, and the model may only suggest the two
recoveries that cannot send anything.

echo '{"task":"send the report to Alice",
       "expected":{"recipient":"Alice","message":"Q3 numbers"},
       "observed":{"chat_title":"Bob","input_text":"Q3 numbers"}}' | macos-cu jev guard
# {"answers":{"right_target":0.02,"input_ok":0.98,"blocker":"wrong_target"},"decision":"switch_target"}

The key is read from TYPESAFE_API_KEY or ~/.config/typesafe/api_key
(https://console.typesafe.ai/keys). Question design is covered in
skill/reference/jev-best-practices.md.

Related projects

Computer use is a busy space. These are the projects worth knowing, with star
counts as of September 2026. Pick the one that fits.

Where this kit fits. It is pure Python on pyobjc, so there's no binary to
notarize: uvx runs it anywhere. Its focus is agent-loop reliability. That
means verified results with retry semantics, stale-ref detection, snapshot
diffs, and a deterministic safety policy. The optional Jev guard adds a
calibrated check before irreversible steps. The design follows the patterns
mature computer-use agents converged on, reimplemented from scratch.

Known limitations

• macOS only. AX, CGEvent, ScreenCaptureKit and Vision are macOS APIs.
• Custom-drawn UIs (games, canvases, some chat apps) expose little or no
  accessibility data. Use annotate, ocr and coordinate clicks there, and
  verify visually.
• Most apps accept background input, but a few only accept typing or pasting
  while frontmost. Use macos_app action=activate first.
• The user may be working at the same time. One extra verification before an
  irreversible step is cheap.

Repository layout

src/macos_computer_use/   the CLI + MCP server (single source of truth)
skill/                    the agent skill (canonical; synced by scripts/sync-skill.sh)
plugins/claude-code/      Claude Code plugin (MCP launcher, skill, /macos-doctor)
.claude-plugin/           marketplace manifest for `/plugin marketplace add`
packages/pi/              pi package (skill + native tools)
packages/dsh/             DeepSeek Harness bundle
tests/                    unit tests: policy, chords, diffs, OCR geometry, MCP protocol
tools/*.py                legacy script shims over the same modules

See CONTRIBUTING.md for conventions,
PUBLISHING.md for release steps and
CHANGELOG.md for the changelog.

License

MIT