Nova

A Computer Use implementation in Rust. Nova is a Model Context
Protocol server that gives an LLM agent
control of the macOS desktop: screenshots, mouse, keyboard, scrolling,
window/app introspection, and the clipboard — the "computer use" capability,
built natively in Rust rather than wrapping a Python/JS automation stack.

Built directly on Apple's ScreenCaptureKit, CoreGraphics (CGEvent), and the
Accessibility APIs — a single self-contained binary, no runtime to install.
Connect it to any MCP client (Claude Desktop, an agent runtime, your own) over
stdio or Streamable HTTP.

Tools

Requirements

• macOS 15+ (a transitive dependency, apple-metal, needs the macOS 15 SDK / Xcode 16+).
• Screen Recording permission — for screenshot / ocr / list_windows.
• Accessibility permission — for posting mouse/keyboard events and Set-of-Mark.

Both are granted to the nova binary itself, not the app that launches it —
see Permissions & code signing. On a dev
build you must also sign nova, or the grant breaks on every rebuild.

Run

cargo run                      # stdio transport (default)
cargo run -- --http            # Streamable HTTP on 127.0.0.1:3100
cargo run -- --http --addr 0.0.0.0:8080

The Swift runtime that ScreenCaptureKit links is located via an LC_RPATH
baked in by build.rs, so no DYLD_* environment variable is needed for
cargo run/cargo test or the standalone binary.

Install

macOS 14+ (Apple Silicon or Intel — the release is a universal binary). Pick one:

# Homebrew (recommended) — puts `nova` on your PATH
brew install bigduu/tap/nova

# npx — no install; great for MCP configs
npx -y @bigduu/nova --version

# Prebuilt tarball — download, unquarantine, drop on PATH
curl -fsSLO https://github.com/bigduu/Nova/releases/latest/download/nova-vX.Y.Z-universal-apple-darwin.tar.gz
tar xzf nova-*.tar.gz
xattr -dr com.apple.quarantine ./nova   # only needed for the raw download
sudo mv nova /usr/local/bin/

# From source
cargo build --release      # produces target/release/nova

The release binary is ad-hoc signed, not notarized. Homebrew and npm
install it without a Gatekeeper prompt; only the raw tarball download needs
the one-time xattr -dr com.apple.quarantine above.

Use it from an MCP client

Claude Desktop (or any stdio MCP client) — add Nova to the client's MCP
config. For Claude Desktop that's
~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "nova": { "command": "npx", "args": ["-y", "@bigduu/nova"] }
  }
}

With brew install (or a binary on PATH) the command is simply "nova"; from
source, use the absolute path to target/release/nova.

Restart the client; the Nova tools then appear to the agent. Grant Screen
Recording and Accessibility — see
Permissions & code signing (as a subprocess,
the responsible process is usually the host app or your terminal, so grant
that, not only the binary).

HTTP clients — run Nova as a server and connect over Streamable HTTP:

nova --http                       # 127.0.0.1:3100/mcp
nova --http --addr 0.0.0.0:8080   # reachable on the LAN

First calls. Call read_ui (optionally read_ui(window="<name>")) to list the
numbered actionable elements as text, then click_mark(number=N) to activate one —
no screenshot, no coordinates needed. Take a screenshot when you need to see the
screen (verify a visual result, read a rendered view) or screenshot(marks=true) when
you want the picture alongside the same numbered list; drop to zoom_region / ocr
for surfaces with no Accessibility tree (canvas, games). All pointer tools use the
pixel space of the most recent screenshot, so the loop is: read_ui → act → read_ui
(or screenshot) to confirm.

Permissions & code signing (macOS)

Two non-obvious things about how macOS grants Nova its Screen Recording and
Accessibility permissions:

Nova is its own permission subject — grant the nova binary, not the launcher.
As a standalone CLI (not inside an .app bundle), Nova does not inherit the
Screen Recording grant of its parent process (your terminal, Claude Desktop, or a
host app). Add the binary itself: System Settings → Privacy & Security → Screen
Recording (and Accessibility) → + → ⌘⇧G → enter the path to
target/release/nova, then enable it. A headless subprocess can't trigger the
permission prompt, so adding it by path is the reliable way.

Sign it, or the grant breaks on every rebuild. cargo build produces an
ad-hoc, linker-signed binary whose code-signing identity is a content hash
(nova-<hash>) — it changes every build, so macOS treats each rebuild as a brand-
new app and the grant stops applying. Sign with a stable self-signed identity so
the grant persists:

cargo build --release
./scripts/dev-codesign.sh --release   # re-sign after EVERY build

The first run creates a Zenith Nova Code Signing identity in your login keychain
(click Always Allow once if codesign prompts) and signs the binary with a
fixed identifier (com.zenith.nova). Grant the two permissions once (above);
later rebuilds, re-signed with the same cert, keep the grant.

Troubleshooting — screenshot fails with a "wedged" / "busy" capture error.
All captures (and window enumeration) run in ONE shared per-user daemon
(nova --capture-daemon, flock-elected, socket /tmp/nova-capture-<uid>-<hash>.sock),
because replayd keys clients by executable path — two same-binary
ScreenCaptureKit clients evict each other's XPC identity and wedge every new
stream start. The daemon kills itself if a capture exceeds its 8s watchdog,
and the client auto-recovers: kill+respawn the daemon, then (second failure)
SIGKILL all nova capture processes and killall -9 replayd — wedges self-heal
without manual action. If they don't: nova --selftest (probes ScreenCaptureKit
in a sacrificial subprocess, then the daemon path) and read
/tmp/nova-capture-worker.log (step trace) + /tmp/nova-capture-daemon.log
(daemon stderr). Manual remedy = kill the processes holding streams
(pkill -f -- --capture-daemon), NOT replayd: plain killall replayd is a
no-op (replayd ignores SIGTERM), and even killall -9 replayd doesn't cure a
wedge while a stream-holding client survives — it just reconnects and re-wedges
the fresh replayd.

Coordinate grounding

A general LLM judging pixel coordinates off a downscaled screenshot is the main
source of mis-clicks — so the primary path avoids pixels entirely.

• read_ui (AX-first, no image) — list the actionable UI as text ([N] role "label", plus a field's value) straight from the Accessibility tree, then
  click_mark(number=N). No capture, no image tokens, works in the background;
  this is the default way to find and click things. Screenshots are for when you
  need to see the screen. Needs Accessibility permission; an app with no tree
  comes back empty (fall back to screenshot / ocr).
• Set-of-Mark screenshot — screenshot(marks=true) (the default) draws
  numbered boxes over the same actionable elements and returns the SAME [N] list
  PLUS the picture, so click_mark(number=N) works identically — use it in place of
  read_ui when you also need the image. Needs Accessibility permission; degrades
  to plain coordinates without it.

When a screenshot is needed, all click/move/scroll tools work in the pixel space
of the last screenshot — the server remembers that frame and maps clicks back to
the real screen, so the model just "clicks what it sees":

• screenshot(window: "<name>") — capture a single window (substring of its
  title or app name) instead of the whole display. Smaller, sharper image → less
  context and far less downscaling → better precision. Later clicks map into
  that window.
• zoom_region(x, y, w, h) — magnify a rectangle of the last screenshot at
  native resolution (capturing only that rectangle). For reading small targets
  on surfaces that expose no Accessibility tree (canvas, games, custom views),
  where coordinates are the only option. A labeled coordinate grid is overlaid
  so the model reads positions straight off the axes.

Testing

The suite is split into fast, hermetic tests (run by default) and side-effecting
end-to-end tests (opt-in, #[ignore]d).

Default — unit + hermetic integration tests

cargo test

Runs everything that has no side effects and needs no special permission:

• unit tests for coordinate scaling, the key/char keystroke maps, combo parsing,
  batch (de)serialization, and MCP tool registration;
• tests/e2e_interaction.rs — screenshot→logical coordinate mapping (via
  CGDisplay, no permission needed) and a non-destructive clipboard
  round-trip (snapshots and restores the clipboard).

This is what CI runs (see .github/workflows/ci.yml, macOS runner).

End-to-end tests (#[ignore]d)

These either post real input events (they move the cursor, click, scroll, or
type into the focused window) or require Screen Recording permission, so they
are excluded from cargo test and must be opted into. Run them on a desktop
session where that's acceptable:

# all of them
cargo test -- --include-ignored

# or a single one
cargo test --test e2e_input mouse_move_roundtrips_through_cursor_position -- --ignored

mouse_move_roundtrips… is the strongest input check: it proves events
actually reach the window server and that the screenshot→logical coordinate
conversion is correct, all click paths share the same posting mechanism.

Run e2e_capture_worker single-threaded (-- --ignored --test-threads=1):
the tests share one daemon/socket.

list_applications_returns_app_bundles (in e2e_input) is not ignored — it
only reads Spotlight and is tolerant of a Spotlight-less CI host.

Lint & format

cargo fmt --all -- --check
cargo clippy --all-targets

Releasing (maintainers)

A version tag drives everything via .github/workflows/release.yml: it builds
the universal binary (arm64 + x86_64, ad-hoc signed), smoke-tests it, and
attaches it to a GitHub Release. When their secrets are present it also updates
the Homebrew tap and publishes the npm wrapper; without them those steps skip,
so the GitHub Release still ships.

# bump the version, then tag it (the tag must match Cargo.toml)
git tag v0.1.0
git push origin v0.1.0

One-time setup for the optional channels:

• Homebrew — gh repo create bigduu/homebrew-tap --public, then add a
  fine-grained PAT with Contents: write on that repo as the nova-repo secret
  HOMEBREW_TAP_TOKEN. (packaging/homebrew/nova.rb is the formula template.)
• npm — publish under the @bigduu scope; add an npm automation token as the
  secret NPM_TOKEN. The package (npm/) downloads the matching release binary
  on install.

License

MIT © bigduu