mac-local-vision (macvis)

Zero-token, on-device vision for AI agents and E2E tests — built as a 100% Pure
Swift single binary on Apple's native Vision and FoundationModels frameworks.
No Node, no Python, no runtime dependencies: the OS is the dependency.

• Zero-Token OCR — extract text/layout locally, never spending cloud vision tokens.
• Fast E2E targeting — find returns the exact pixel center of a word for click/assert.
• QR/barcode scanning — barcode decodes every symbology Vision supports (QR, Code128, EAN, PDF417, ...) in one call.
• QR generation — make-qr writes a scannable QR code PNG (CoreImage, no Vision needed).
• On-device semantic ask (Beta) — multimodal reasoning via Apple Foundation Models (macOS 27 Beta).
• Local face sorting — cluster photos by person without uploading anything.

Tiny & fast

A ~240 KB stripped single binary (the frameworks are the OS, nothing is bundled — no
Node, no Python, no node_modules), and every call is a fresh ~0.3 s end-to-end —
process launch plus recognition, no daemon to keep warm:

_{Best of 5, on a 1080×2400 screenshot (19 lines, Korean + English), Apple M4.}

Versus shipping that screenshot to a cloud vision API: no network round-trip, no vision
tokens, no per-call cost, and nothing leaves the machine.

ask (multimodal LLM inference, needs macOS 27 (Beta) + Apple Intelligence) is a
different kind of fast — still no cloud round-trip, but the cost is real generation
time, not process launch:

_{Small sample, MacMini-M4, macOS 27 Beta.}

Requirements

Apple Silicon Mac, macOS 26+. No other dependencies — Vision and FoundationModels
ship with the OS. Building the ask (multimodal) path additionally needs the macOS 27 SDK
(Xcode 27); everything else builds and runs on macOS 26.

Install

Apple Silicon, macOS 26+.

Homebrew

brew install junmo-kim/tap/macvis

mise

mise use -g github:junmo-kim/mac-local-vision      # add @0.1.0 to pin a version

Both fetch the same prebuilt binary — it's ask-enabled, so the Vision commands run on macOS 26+
and ask lights up on macOS 27. Then run macvis doctor to see what's available here.

Direct download — grab macvis-<version>-macos-arm64.tar.gz from
Releases, then:

shasum -a 256 -c macvis-*-macos-arm64.tar.gz.sha256   # verify the download
tar xzf macvis-*-macos-arm64.tar.gz
sudo mv macvis /usr/local/bin/                        # or ~/.local/bin on your $PATH

The binary is ad-hoc signed (not notarized), so a directly-downloaded copy may trip Gatekeeper —
approve it under System Settings → Privacy & Security. (Homebrew / mise installs avoid this.)

From source (Swift 6.2+ toolchain / Xcode 26):

swift build -c release && cp .build/release/macvis /usr/local/bin/

Status

sort-faces accuracy: faces are grouped by an image feature print over the face
crop (Apple exposes no public face-embedding API). This reliably groups near-duplicate
/ same-session faces, but is a weak identity signal across large pose/lighting/age
changes — tune --threshold (distances are in the output). Not a person-recognition DB.

ask is Beta because it rides on a pre-release Apple stack — macOS 27 (Beta) and the new
Foundation Models multimodal API. The call — session.respond { prompt; Attachment(image) } —
matches Apple's official WWDC26 Foundation Models session
and builds against the macOS 27 SDK. On a real macOS 27 Beta boot, both the error path
(Apple Intelligence off → apple_intelligence_not_enabled, exit 71) and real end-to-end
inference are verified: accurate answers, full --stream output (not deltas), and ~1-7 s
latency depending on image complexity and answer length. Apple Intelligence itself is gated to
eligible, internal-boot installs, so ask tracks the platform: it graduates from Beta as macOS 27
ships. The rest of macvis (ocr / find / sort-faces / mcp) is stable on macOS 26 today.

Build & test

swift build -c release       # binary at .build/release/macvis
swift test                   # pure logic + ask plumbing + Vision OCR fixtures

Building the ask path against real macOS 27 APIs needs the Xcode 27 SDK; the
current target compiles on macOS 26 with ask behind @available(macOS 27, *)

• runtime availability guards.

Usage

macvis ocr ./receipt.png                                # extract text
macvis find ./screen.png --target "Submit"              # pixel center of a word
macvis find ./screen.png --target "결제하기"             # non-Latin works too (locale-aware)
macvis barcode ./ticket.png                              # scan every QR/barcode symbology
macvis qr ./ticket.png                                   # scan for QR codes only
macvis make-qr "https://example.com" --out ./qr.png     # write a scannable QR PNG
macvis ask ./design.png --prompt "main theme color?"    # Beta — needs macOS 27 (Beta)
macvis doctor                                           # which modes work here

find filters at --min-confidence 0.3 by default (ocr keeps everything, default 0.0);
lower it for blurry or headless renders.

Output is YAML by default (--format json for jq). Data goes to stdout; logs and
structured errors go to stderr. Exit codes distinguish bad args (64), permanent ask
ineligibility (70), and retryable failures (71).

Sample output

find returns the exact click point (x,y = center) plus the bounding box:

$ macvis find ./screen.png --target "Submit"
found: true
x: 456
y: 66
left: 380
top: 48
width: 152
height: 36
confidence: 1
text_found: Submit

--format json for jq:

$ macvis find ./screen.png --target "Settings" --format json
{"found":true,"x":126,"y":69,"left":38,"top":48,"width":176,"height":42,"confidence":1,"text_found":"Settings"}

barcode scans every QR/barcode symbology in one call and returns each code's payload,
symbology, and pixel box; code_count: 0 (not an error, exit 0) when none are found:

$ macvis barcode ./ticket.png
image_width: 1080
image_height: 2400
code_count: 1
codes:
  - payload: "https://example.com/ticket/abc123"
    symbology: qr
    x: 512
    y: 300
    left: 480
    top: 270
    width: 64
    height: 64
    confidence: 1

Restrict to specific symbologies with --symbology qr,code128 (comma-separated); an unknown
symbology name is a structured bad_request/unknown_symbology error, exit 64.

qr is barcode narrowed to QR only, enforced server-side — there's no --symbology flag
to override it. Same output shape as barcode, just always QR-scoped:

$ macvis qr ./ticket.png
image_width: 1080
image_height: 2400
code_count: 1
codes:
  - payload: "https://example.com/ticket/abc123"
    symbology: qr
    x: 512
    y: 300
    left: 480
    top: 270
    width: 64
    height: 64
    confidence: 1

make-qr is the write counterpart to barcode/qr — it encodes text into a scannable QR PNG
via CoreImage (CIQRCodeGenerator), not Vision, so it works on any Mac regardless of
Vision/Apple Intelligence availability. Give --out to write a file and get its path back;
omit it to get the PNG as base64 in image_data instead (for remote/MCP callers with no local
filesystem access):

$ macvis make-qr "https://example.com" --out ./qr.png
path: ./qr.png
width: 250
height: 250
correction_level: M

--correction-level L|M|Q|H trades code density for damage tolerance (default M); --size N
sets the per-module pixel magnification (default 10), not the overall image side length — the
reported width/height are the image actually produced, since module count depends on payload
length and correction level. An unknown correction level is bad_request/invalid_correction_level,
exit 64.

doctor reports what runs here, plus the locale-derived OCR languages and (once ask is
available) which languages it's ready to answer in right now:

$ macvis doctor
ocr: available
find: available
sort-faces: available
barcode: available
ask: "unavailable: needs_macos_27_sdk"
ocr_languages:
  - ko-KR
  - en-US
ask_languages: []

ask_languages mirrors ask's own status — empty whenever ask itself is unavailable
(including on macOS 26, where the text-only model can be ready while image-based ask isn't),
and the full set of ~24 supported languages once ask reports available. Once ready, ask
isn't limited to the device's system language — it answers fluently in whichever of those
languages the prompt is written in. There's no flag to force a different response language,
though: Apple's API doesn't expose one, so ask follows the prompt's language rather than a
--lang-style override.

Use from an LLM agent

Two equivalent integrations — both run the same VisionService engine, so the output is identical.

MCP server (stdio) — macvis mcp speaks stdio JSON-RPC and exposes ocr / find / barcode /
qr / make-qr / doctor as tools (plus ask when the binary is built with the macOS 27 multimodal path):

{ "mcpServers": { "mac-vision": { "command": "/path/to/macvis", "args": ["mcp"] } } }

MCP server (HTTP) — macvis serve runs an HTTP JSON-RPC endpoint for non-Mac or remote nodes
that cannot launch a local process. Defaults to 0.0.0.0:9090; use --host / --port to restrict:

macvis serve --host 127.0.0.1 --port 9090   # loopback only (safest)
macvis serve                                  # all interfaces — warns on startup; restrict with a firewall

Configure remote nodes to use type: http with the Mac's LAN address:

{ "mcpServers": { "macvis": { "type": "http", "url": "http://192.168.x.y:9090/mcp" } } }

Remote callers pass images as base64 in the data field instead of path. There is no built-in
authentication — treat this as a filesystem read service on your LAN and secure accordingly.

Claude Code skill — this repo ships one in skills/macvis/.
Symlink it into your skills directory:

ln -s "$PWD/skills/macvis" ~/.claude/skills/macvis

The agent then invokes macvis on natural-language triggers (read text from an image, find a
UI element's coordinates, …). The skill calls macvis from your PATH, so install it first
(see Install).

Architecture

A single VisionService seam executes every request and is shared by the CLI and the MCP
server, so both produce identical output. Calls run in-process — Vision OCR has no heavy
model to keep resident and the MCP server is already long-lived, so there's no daemon to
manage.

License

MIT © 2026 Kim Junmo