mobile-agent

mobile-agent drives Android devices, Android emulators, iOS Simulators and physical iPhones for AI agents, and is designed around the latency of each tap-and-observe step. A daemon keeps a live mirror of every attached screen (its windows, accessibility tree and latest frame), updates it from the platform's change events and frame callbacks, and answers snapshots, waits and settle checks from that mirror instead of reading the device and sleeping on every request. On Android, a single-dex agent pushed over adb runs as the shell user under app_process and holds the device's UI automation role: no APK is installed and no instrumentation runs. On the iOS Simulator the driver works from the host: input goes through the Simulator's dtuhidd HID service, frames come from the display's IOSurface, and the accessibility tree comes from a Rust reader that the daemon spawns into the Simulator, with no XCTest runner and nothing installed. A physical iPhone's tree can be read only from an XCTest runner, so there a Rust runner built in this repository, started through the phone's test manager and kept for the session, serves the tree and text entry over the same device protocol, while the driver does the rest from the host: input through the developer disk image's dtuhidd, frames from screenshots it polls, and the phone's services over a CoreDevice tunnel it opens in its own process, without root or a helper. The mobile-agent CLI follows agent-browser's command grammar, @eN refs and JSON envelope, so prompts and skills written for agent-browser carry over where devices have the same concepts, and the mobile-agent-mcp server gives an MCP client the same tools, refs and replies (Agent surfaces).

Status. Experimental and pre-release: nothing is published to crates.io or as a binary yet, so mobile-agent is built from source (Build), and main is the version CI builds and tests on macOS and Linux. The drivers rely on private Apple and hidden Android interfaces, which makes this development tooling, never something to ship inside an app (Status and limitations). It is under the MIT License (LICENSE, THIRD-PARTY-NOTICES.md); CONTRIBUTING.md says how a change is made and checked, SECURITY.md how to report a vulnerability, and CHANGELOG.md what has changed.

Supported platforms

CI (.github/workflows/ci.yml) validates and runs no benchmark; the benchmarks have a workflow of their own, which never runs per push (How measurements are made), and so do the device-backed tests, .github/workflows/device-tests.yml, weekly and on dispatch, which can narrow a run to one job, one package and a test filter: the workspace's tests on a fresh iPhone 17 Simulator of the xcode-27 image and on an API 36 emulator on Linux, with the fixtures installed, the gating variables set and MOBILE_AGENT_HOST_GATES=off, under which the tests' millisecond bounds, measured on a quiet developer Mac, are printed rather than judged (design D1 of openspec/changes/pass-4-fixes). ci.yml runs just validate in three jobs: Linux, which also builds the Android agent and the Compose fixture; macOS with Xcode 26 on GitHub's macos-26 image; and macOS with Xcode 27 on its xcode-27 image. A fourth, Audit, lints the workflows with actionlint, audits them with zizmor, its online audits included, checks the dependencies' advisories, licenses, bans and sources with cargo-deny (deny.toml), and runs the publish scan and openspec validate --all --strict. Before them, a job classifies a pull request with scripts/ci-changes.sh: one that changes documentation alone, Markdown anywhere, docs/, openspec/, the licenses and the issue templates, runs Audit alone, the Linux and macOS jobs skipped by their if, so that their required checks read skipped, which passes them; a push to main and a dispatch run every job. Every job starts without permissions and reads only what it needs, every action is pinned to a full commit SHA, no checkout keeps its credentials, and the tools come from lockfiles or from releases pinned with their SHA-256 (docs/research/open-sourcing.md). Each macOS job keeps the image's default Xcode when it has that major version and otherwise selects the newest Xcode_<major>*.app. The macOS jobs build and test the iPhone driver (crates/ios, on its fake device) and the runner's crate (agent-ios, on the host); nothing runs against a phone, the device tests skipping without MOBILE_AGENT_IOS_UDID, and the signed runner bundle is not built there, since just agent-ios-build skips with SKIP_AGENT_BUILD=1 or without MOBILE_AGENT_IOS_TEAM, as just agent-build skips the Android agent with SKIP_AGENT_BUILD=1. The spikes and every measurement ran on Xcode 27.0 with CoreSimulator 1171.7 and iOS 26.5, and so did the Simulator integration tests, which the device tests also run on the xcode-27 image's iOS 27.0. Nothing has run on Xcode 26, where input would be touch only, through the older Indigo transport (crates/simulator/README.md, Compatibility).

Build

Requirements:

• Rust. Install rustup. It takes the toolchain from rust-toolchain.toml: the version pinned there, with clippy, rustfmt and the aarch64-apple-ios-sim target. Run rustup toolchain install in the checkout, as CI does, to install them before the first build, or when cargo reports that the pinned toolchain is not installed; rustup show active-toolchain there names the toolchain in use.
• macOS, for the Simulator. Xcode 26 or 27, selected with xcode-select, with an iOS Simulator runtime; and libjpeg-turbo, brew install jpeg-turbo, which the frame encoder links statically from Homebrew's Apple silicon prefix (.cargo/config.toml; set TURBOJPEG_LIB_DIR and TURBOJPEG_INCLUDE_DIR for another prefix).
• The Android agent. The daemon's build compiles the agent with agent-android/build.sh (kotlinc and R8, no Gradle) and embeds the jar, so it needs a JDK 17 or newer with java and jar on PATH (tested with Temurin 21), the Kotlin compiler, kotlinc on PATH or KOTLIN_HOME set to the compiler's home, the directory holding bin/kotlinc and lib/ (tested with 2.4.20), and an Android SDK at ANDROID_HOME, by default $HOME/Library/Android/sdk, with platforms;android-36 and a build-tools package (CI installs 37.0.0). On Linux, set ANDROID_HOME.
• Android at run time. The SDK's platform-tools, whose adb server the driver talks to on port 5037, and for an emulator, the emulator package and an AVD.
• The iPhone's runner. agent-ios/build.sh, which just agent-ios-build runs, builds the runner for aarch64-apple-ios, the second target rust-toolchain.toml names, links it into a copy of Xcode's XCTRunner.app with the iPhoneOS SDK's clang and signs it with codesign for the team MOBILE_AGENT_IOS_TEAM names, with no xcodebuild project, into the git-ignored agent-ios/dist/ (agent-ios/README.md, Building and signing). It needs Xcode 27.0's iPhoneOS SDK, the only one tried, and the team's identity and profile (Quick start). The daemon installs that bundle from the workspace it was built in, or the one MOBILE_AGENT_IOS_RUNNER_APP names, when the phone's runner is missing or says hello with another version (crates/daemon/README.md, The physical iPhone driver).

From the checkout:

cargo build --release                      # target/release/mobile-agent, mobile-agent-daemon and mobile-agent-mcp
export PATH="$PWD/target/release:$PATH"    # the CLI starts the daemon it finds beside itself

• cargo build --profile profiling builds the release code with line-level debug information and nothing stripped, into target/profiling/, for a sampling profiler such as Instruments' Time Profiler, which then names inlined functions and lines; release builds strip the debug information (spike S15, docs/spikes/s15-profiles.md).
• SKIP_AGENT_BUILD=1 cargo build --release builds without the Android toolchain. That daemon deploys an agent only from a jar that MOBILE_AGENT_JAR names at run time; MOBILE_AGENT_JAR set at build time embeds that jar instead of building one.
• The Simulator's tree reader is cross-compiled for aarch64-apple-ios-sim during the build and embedded in the daemon, behind the reader feature of crates/simulator, which is on by default. The daemon spawns it into each booted Simulator it attaches. Without that Rust target the build warns and embeds no reader, and tree reads fall back to the slower host translator, marked degraded; doctor shows which one a build has.
• Contributors: every dependency install goes through Socket Firewall. Install it with npm i -g sfw, then prefix installs with it: sfw cargo fetch, sfw cargo add, sfw cargo install just, sfw npm install -g @fission-ai/openspec. just validate runs the checks a change must pass: build, clippy with warnings denied, format check, the publish scan (just check-public: no machine path, device identifier, team id, email, host model or private term in a tracked file, and gitleaks' rules when it is installed), the workflows' lint and audit (actionlint and zizmor, when installed), the dependencies' check (cargo deny, when installed), tests, the checks of the repository's own scripts, the agent build, the iPhone runner's build, the Kotlin codec cross-check, the fixture scripts' checks with fake build tools, the fixture builds (with the SwiftUI and Expo fixtures' device builds when MOBILE_AGENT_IOS_TEAM is set) and openspec validate. The tests that need a Simulator, an emulator or an iPhone skip unless their environment names one (crates/daemon/README.md, crates/simulator/README.md, crates/ios/README.md).

Quick start

The daemon attaches to targets that are already running; it boots nothing. For Android, put the SDK's tools on PATH, start an adb server, and start an emulator with its gRPC control endpoint, whose port is the console port plus 3000, or connect a device with USB debugging on and its prompt accepted:

export ANDROID_HOME="${ANDROID_HOME:-$HOME/Library/Android/sdk}"
export PATH="$ANDROID_HOME/platform-tools:$ANDROID_HOME/emulator:$PATH"
adb start-server
emulator -avd <name> -grpc 8554 -grpc-use-token      # in a terminal of its own

Without the control endpoint, taps and screenshots go through the agent and there is no frame stream; doctor and status say how to enable it on a running emulator: adb -s emulator-5554 emu grpc 8554. Emulator 37.1.11 serves the endpoint even when started without -grpc (only -ports leaves it off), so to take the agent's path on an emulator that serves it, as a physical device does, start the daemon with --emulator-controller off or MOBILE_AGENT_EMULATOR_CONTROLLER=off; status and doctor report the setting. For the iOS Simulator, boot one in Simulator.app or with xcrun simctl boot <name or UDID>.

For a physical iPhone, the daemon registers its driver only when MOBILE_AGENT_DRIVERS names ios (MOBILE_AGENT_DRIVERS=ios drives iPhones alone): its listing checks the runner's signing identity and probes each phone through lockdown, and a phone whose lockdown does not answer holds the listing for 10 s, so a daemon that drives no phone pays for neither (crates/daemon/README.md, Drivers). The rest are the maintainer's steps, done once by hand, since nothing here signs in to an account or registers a device (agent-ios/README.md, Requirements):

• On the Mac: a paid development team's Apple Development identity in the login keychain and an iOS development provisioning profile for the App ID dev.mobileagent.* that includes the phone, both made in Xcode; the team's identifier exported as MOBILE_AGENT_IOS_TEAM; and the runner built with just agent-ios-build.
• On the phone: Developer Mode on (Settings > Privacy & Security, then the restart it asks for), the Mac trusted, the phone unlocked on USB, and "Enable UI Automation" approved with the passcode when a runner's start asks for it. XCTest waits 60 s for the approval, and the driver then reports automation_not_enabled. The SE asked again from time to time, also with Settings > Developer > Enable UI Automation turned on (2026-10-10): one approval lasted the eight hours of task 9.6's runs, and the next start, about an hour after them, asked again (task 9.9).

devices lists each phone in its readiness state, ready or the first step it lacks (not_trusted, developer_mode_off, disk_image_unmounted, device_locked, no_signing_identity, unsupported_ios), and doctor names the step, with its ios-signing, ios-runner and ios-devices checks and the iPhone driver's rows in the compatibility table: the supported range, then each call site with the iOS and Xcode it was validated on (crates/ios/README.md, Compatibility). The driver mounts the developer disk image when the phone lacks it and installs the runner itself.

Settings ships with every device, so it needs no fixture app. Type, using the refs your own listing prints:

mobile-agent doctor                         # toolchains, adb server, emulators, Xcode compatibility
mobile-agent devices                        # connected targets and their state
mobile-agent open com.apple.Preferences     # on Android: mobile-agent open com.android.settings
mobile-agent snapshot -i                    # actionable elements, each with a ref
mobile-agent click @e11                     # a ref from that listing, General in the run below
mobile-agent wait --settle
mobile-agent --json snapshot -i
mobile-agent close                          # stop the app and release the target
mobile-agent shutdown                       # stop the daemon; it returns once its targets have stopped

With several targets connected, give open a --platform android|ios, or a --device <id> when targets share a platform; the session then keeps using the target it holds. The first command starts the daemon, which exits after an hour without requests (--idle-timeout). shutdown replies once the daemon has stopped its targets, an Android agent or a Simulator's reader, and removed its socket, and names a target that had not stopped within 5 s in data.notStopped. mobile-agent status reports the daemon, its targets and, for each attached target, the paths its input, frames and tree take.

A run on an iOS 26.5 Simulator, trimmed where … stands. In doctor, ⚠ marks a warning, which does not fail it; sfw and just matter only to contributors. Ref numbers are never reused, and open lists the screen while it waits for the app, so the first listing need not start at e1. The degraded verdict is the quality check at work: Settings' search field holds two identical Dictate buttons that only their position tells apart.

$ mobile-agent doctor
✓ daemon               running, pid 11976, version 0.1.0+640310.18d934c42feb802c
✓ drivers              simulator, android
✓ xcode                Xcode 27.0 (27A266a) at /Applications/Xcode.app/Contents/Developer
✓ simulator-runtimes   iOS 26.5
✓ android-sdk          $HOME/Library/Android/sdk, adb 1.0.41 (37.0.1-15733141)
✓ adb-server           running on port 5037, protocol 41
✓ emulator-control     gRPC control: emulator-5554 on port 8554
⚠ sfw                  sfw is not on PATH
    hint: install it with npm i -g sfw; every install goes through Socket Firewall
✓ just                 just 1.58.0
✓ xcode-compatibility  29 validated combinations, listed below

Xcode compatibility:
  Xcode 27.0 (27A266a)  Xcode 26.0 up to but not including 28  validated  validated on Xcode 27.0; the CI matrix also builds on Xcode 26
  CoreSimulator 1171.7  1155.4 or later for dtuhidd  validated  input through dtuhidd
  …
$ mobile-agent devices
B0647CDD-575F-45E9-9398-F6F5AFF23C1F  ios      simulator         iPhone 17           booted
7B292E1E-7A29-41A5-9C69-2ABA1CEB3683  ios      simulator         mobile-agent-spike  booted
emulator-5554                         android  android-emulator  sdk_gphone64_arm64  device
$ mobile-agent --device 7B292E1E-7A29-41A5-9C69-2ABA1CEB3683 open com.apple.Preferences
✓ Opened com.apple.Preferences on 7B292E1E-7A29-41A5-9C69-2ABA1CEB3683
$ mobile-agent snapshot -i
- heading "Settings" [ref=e9]
- button "Apple Account, Sign in to access your iCloud data, the App Store, Apple services, and more." [id=com.apple.settings.primaryAppleAccount, ref=e10]
- button "Ready for Apple Intelligence" [id=com.apple.settings.followUpItem.com.apple.icloud.gm, ref=e24]
- button "General" [id=com.apple.settings.general, ref=e11]
- button "Accessibility" [id=com.apple.settings.accessibility, ref=e12]
…
- button "Screen Time" [id=com.apple.settings.screenTime, ref=e19]
(more items below, scroll down to reveal)
- searchbox "Search" [ref=e21]
  - button "Dictate" [id=Dictate, ref=e22]
  - button "Dictate" [id=Dictate, ref=e23]
⚠ degraded snapshot (sibling_collision): e22, e23 are identical siblings told apart only by position; take a new snapshot right before acting on one of them
$ mobile-agent click @e11
✓ Done (hit button "General", dispatch 0.1 ms, first frame change 346.0 ms, first tree delta 445.4 ms)
$ mobile-agent wait --settle
✓ Settled after 1054.8 ms: frames changed last
$ mobile-agent --json snapshot -i
{"success":true,"data":{"origin":"com.apple.Preferences","readMs":0.073,"refs":{"e25":{"name":"Settings","role":"button"},"e26":{"name":"General","role":"heading"},…},"removedRefs":["@e9","@e10",…,"@e24"],"replicaRevision":7,"servedFromReplica":true,"snapshot":"- button \"Settings\" [id=BackButton, ref=e25]\n- heading \"General\" [ref=e26]\n…","target":"7B292E1E-7A29-41A5-9C69-2ABA1CEB3683","verdict":{"backend":"simulator-reader","degraded":false,"fallback":false,"reasons":[]}},"error":null}
$ mobile-agent close
✓ Closed com.apple.Preferences and released 7B292E1E-7A29-41A5-9C69-2ABA1CEB3683
$ mobile-agent shutdown
✓ Daemon stopped

The Android half was verified on the API 36 emulator on 2026-09-28, with a release build made as above (trimmed):

$ mobile-agent doctor
…
✓ android-sdk          $HOME/Library/Android/sdk, adb 1.0.41 (37.0.1-15733141)
✓ adb-server           running on port 5037, protocol 41
✓ emulator-control     gRPC control: emulator-5554 on port 8554
…
$ mobile-agent devices
emulator-5554  android  android-emulator  sdk_gphone64_arm64  device
$ mobile-agent --platform android open com.android.settings
✓ Opened com.android.settings on emulator-5554
$ mobile-agent snapshot -i
- button "Search Settings" [id=search_action_bar, ref=e1]
(1 more item above, scroll up to reveal)
- button "Network & internet Mobile, Wi‑Fi, hotspot" [ref=e3]
…
(11 more actionable items below, scroll down to reveal)
$ mobile-agent close
✓ Closed com.android.settings and released emulator-5554
$ mobile-agent shutdown
✓ Daemon stopped

Agent guide

mobile-agent help agent prints a compact form of this section for an agent's prompt (crates/cli/src/agent-guide.txt), and every command's --help ends with examples and states its defaults. The JSON examples below come from a daemon serving fake targets (crates/daemon/README.md, The fake driver): a launcher, a Settings app with one Wi-Fi switch and the fixture app, whose timings are not a device's.

mobile-agent snapshot -i       # 1. list actionable elements with their refs
mobile-agent click @e3         # 2. act on a ref from that snapshot
mobile-agent wait --settle     # 3. let the screen settle
mobile-agent snapshot -i       # 4. snapshot again before the next action

Output. --json (or MOBILE_AGENT_JSON=1) prints one line per command in agent-browser's envelope: success, data, error, then code and warning only when set. Every failure exits with status 1 and carries a snake_case code; in JSON mode its message travels in the envelope on stdout, without --json it goes to stderr. A warning marks a success not to take at face value: a degraded snapshot, an open whose screen did not settle, a settle that timed out, a click on an element partly but not mostly in view, a click whose tap landed on another element than the one it named, a fill or a type whose field showed nothing to read back, a type whose field holds other text than the keys added or that found no field with the focus, an open that met a dialog of system UI in front of the app, a shutdown that left a target stopping.

{"success":false,"data":null,"error":"e4 is stale: the button \"Fixture\" it named is no longer on the screen. Take a new snapshot and use its refs","code":"stale_ref"}

Snapshots. A line reads - role "name" [attributes, ref=eN]: value. snapshot -i lists only elements with refs: actionable elements and named headings, cells and list items. snapshot lists every element, texts and system bars included, and -d N limits the depth. A line in parentheses, such as (12 more actionable items below, scroll down to reveal), or (more items below, scroll down to reveal) when the scroll range gives no count, stands for content scrolled out of view, and (2 more actionable items under the keyboard, close it to reveal) for what lies wholly under the keyboard, which snapshot -i leaves out; an element partly under it carries visible=35%, the share of it that shows. What a dialog hides is left out; snapshot -i also leaves out what a tap reaches none of, as behind a header, under a footer or under a view drawn over it whole, counting it in its list's line with what covers it named as a refused action names it, as in (6 more actionable items above, 3 of them behind the header, scroll up to reveal), and both listings mark what a tap reaches part of with visible=, whatever covers the rest, snapshot keeping what a tap reaches none of at visible=0% and its lines counting only what it leaves out, so that the covered counts of snapshot -i are the elements snapshot shows at visible=0%; a view listed before a list across its top is its header, named or not. The keyboard covers what it draws: on iOS the band across the top of its window that the keyboard keeps empty for the input accessory view of the field in focus, which the app draws in its own window, as Safari draws its address bar above the keys, covers nothing, and a keyboard window that draws no keys, as while a hardware keyboard is connected to a Simulator, covers nothing and gets no line; iOS's edit menu covers only the strip it draws on. Every tab of a tab bar has a ref, the selected one marked selected. In data, snapshot holds the listing, refs maps eN to {role, name}, removedRefs names the refs the previous snapshot listed that are gone, origin is the app in front, and verdict gives the backend, whether the read is degraded or came from a fallback, and the reasons with advice (fallback_backend, truncated, sparse_tree, sibling_collision, agent_warning, disconnected); a degraded verdict also sets warning. readMs is how long the daemon took to answer, device reads included, and servedFromReplica is true when the mirror answered without reading the device. Windows that send no change events, every Simulator and iPhone window and Jetpack Compose windows on Android, are re-read after each injected action, when frames change and on a 1 s poll, an iPhone's every 10 s while its frames hold still, so a snapshot right after an action can wait for that read.

Delta snapshots. snapshot --delta answers with agent-browser's delta object, relative to this session's last delta snapshot. Its kind is full (the whole listing, and always the first time), unchanged, or delta, which carries ref changes (add, remove, or replace of the role or name) and a treeChange splice (startLine, deleteCount, lines) that rebuilds the new listing from the previous one. Without --json a delta prints as lines, as the server's results carry it: a header naming the lines of the last delta snapshot's listing it replaces, counted from 1, then the lines that take their place as the listing prints them, and removed refs: @e5, @e6 for the refs it removed, as delta (revision 34 from 33): lines 3 to 5 replaced by and Bold Text's three lines with checked=true after a flip of that switch in Settings (action-verdicts). The whole listing comes back when the delta would be 70% of its size or more, or when the options, the replica or the app in front changed; --full asks for it. diff snapshot instead compares the screen with this session's last snapshot element by element.

{"success":true,"data":{"origin":"com.android.settings","readMs":0.123,"replicaRevision":5,"servedFromReplica":true,"snapshot":{"baseRevision":1,"kind":"unchanged","revision":2},"target":"emulator-5554","verdict":{"backend":"fake","degraded":false,"fallback":false,"reasons":[]}},"error":null}

Refs. Write @e3; ref=e3 and e3 work too, and a number is never reused. A ref names one control for as long as it stays on screen: its element and, for an element without an identifier, also its accessible name, or else its role and its place among its siblings, so a field whose label changes once it holds text keeps its ref. When the element goes, even briefly (another screen, a relaunch, a sheet that hides the screen beneath it), or an element without an identifier changes both its name and its place, the next snapshot gives it a fresh ref. An action or a get on the old one acts on the element the app rebuilt in its place, its twin, the one of its role and identifier, else of its role, name and bounds, and names both refs, first in its line and in the reply's resolved, as a fill by the ref of Maps' search field, whose own tap put the field in a new node, answered ✓ Done (@e381 now @e395, read back "Lisbon", typed), the session acting on the twin's ref from then on; without a twin it fails with stale_ref (action-verdicts; reports/12.3-action-verdicts.md). Finding the twin reads nothing and costs 0.1 to 0.2 ms on a Simulator; such a fill takes its 1.6 to 2.5 s with or without one, in the read after its focus tap, its keys and its read-back (android-pass-fixes). A resync after an event sequence gap, a reconnection or a fall back to another tree backend rebuilds the mirror and makes every earlier ref stale. A ref from another session's snapshots fails with unknown_ref; an element under a dialog, or with less than 8 points a side left from under a view drawn over it on iOS (a fixed footer, a floating button, a sticky header, a disabled button), a bar or the keyboard, fails with element_covered naming what a tap at its centre would land on, and one with nothing a tap reaches in view after a fresh read with element_offscreen. Refs never wait.

Acting and reading.

Selectors. Where a ref goes, a selector works too: #id or id=... for a test identifier, text=... or bare text for a name or value that contains the text, text==... for one that is exactly the text, role=.... A selector names any element the screen's tree holds, in view or not, one on screen and not covered first, and waits up to the timeout only while it holds none; an action on a match that is covered or out of view is refused at once with element_covered or element_offscreen, as on its ref.

Clicks and their timing. click (alias tap) replies once the tap's first frame change and first tree delta have arrived, or after 200 ms without a change, at most 1 s; after the first tap of a new input session it waits up to 1.5 s for the first change and 2.5 s in all. The reply's spans are milliseconds on the daemon's clock, from the dispatch: dispatchMs until the target accepted the input, frameChangeMs until the first frame whose hash differs, treeDeltaMs until the first tree delta, and watchedMs for how long the watch lasted. frameChangeMs and treeDeltaMs are absent, never 0 or null, when the signal did not come: the tap changed nothing that way, or the target has no frame source (Android devices, and emulators without the controller or with --emulator-controller off). The reply's hit names the element the tap landed on as the replica held it at dispatch, by its role, its name and the session's ref, and the line names it before the spans, ✓ Done (hit button "General", dispatch 0.1 ms, ...), with (no ref) after an element the session holds no ref for, as a plain text, hit text "Address" (no ref) on Safari's address field (android-pass-fixes), or none where no element a listing lists lies under the point; when the element hit is neither the one the click named, nor part of it, nor what holds it, as when a view of system UI comes down over it between the snapshot and the tap, a warning names both and the tap has gone in: the tap at (201.14, 164.19) landed on button "placeholder-button-5", not on @e14 (button "placeholder-button-1"): the layout moved under the point or a view lies over it; take a snapshot to see what the tap did (action-verdicts). hit is the replica's view, not what the app did with the tap: a list still gliding takes a tap as a stop while the reply names the row under the point (reports/12.3-action-verdicts.md, finding 1), which is why scroll answers once the content has come to rest. click makes no stroke: it taps the centre of the largest part of the element that shows from under what covers it, at least 8 points on each side, with a warning when that part is not most of the element, as scroll --until judges it; one with nothing a tap reaches in view gets its window read again once, then is tapped where it is or refused with element_offscreen (close the keyboard, or scroll --until it); a covered element is refused with element_covered before any read, so a tap never lands on the view over it (rn-app-fixes).

{"success":true,"data":{"accepted":true,"clicked":"@e2","dispatchMs":0.054,"frameChangeMs":0.003,"hit":{"name":"Fixture","ref":"@e2","role":"button"},"ref":"@e2","treeDeltaMs":0.064,"watchedMs":6.068,"x":204.0,"y":120.0},"error":null}

Settle and waits. wait --settle resolves once neither the tree nor the frames have changed for the quiet window (--quiet, default 200 ms), at most --timeout (default 3000 ms). Within a second of the session's last click, fill, type, press, swipe or scroll it first waits for that action's first effect, a change of the tree or the frames, until a second after the action, then for the quiet window, and the reply's input names the action (action-verdicts). On a Simulator, whose windows send no change events, the settle once quiet reads the tree the frames changed and settles again from a change that read finds, so an animation that pauses longer than the quiet window before its last step, as Safari's More menu did before its items, does not end it with the tree unread (task 3.5 of action-verdicts). endedBy names the signal that changed last: tree, frame, both (within 20 ms of each other), none (nothing changed since the last action, or the screen was already quiet; after an action less than a second before, nothing changed within a second of it, which the line says as nothing changed within 1 s of the click), or timeout. A timeout is still a success: settled is false, stillChanging names what kept changing, and warning says so. --tree-only ignores frames, for video. On an Android target without frames, the settle also reads a few elements' bounds again every 50 ms and does not end while the platform still slides or scales a window in, and after a stroke that leaves a list at its end it holds for 500 ms while the list springs back. wait <ref or selector> waits for an element (--state visible|hidden|enabled on screen, attached|detached in the tree, in view or not) and wait --text <text> for text; both name what they found as the listing names it, ✓ Found button "Sign in" @e4, with the ref the full listing gives an element that bears one, and ✓ Found text "Welcome, agent" for a plain text, which bears none; both fail with wait_timeout and a summary of the screen, which, as an element_not_found's, becomes the session's last snapshot, so its refs can be acted on. open waits for the app's first tree, past any launch splash, and for its screen to settle after the launch's first effect, then checks that the app is in front: when another holds the front once the app's screen showed and settled, as the home screen takes it back from a running app launched while it goes home, it launches the app again, and fails with not_in_front naming what is in front when the app still is not (action-verdicts); it replies settled, elapsedMs and firstTreeMs; when a dialog of system UI stands in front of the app, as a permission alert, it returns at once, unsettled, naming the dialog in alert and in a warning, alert "Allow “Maps” to use your location?" stands in front of com.apple.Maps; answer it before you act on the app: a snapshot lists its buttons, as it does for one that came in front while the screen settled. On the fixture's endless animation: wait --change waits for the effect of an action that may still be arriving: it returns once the listing in the options of the session's last snapshot has changed from what that snapshot showed, at once when it changed before the call, with endedBy naming what moved (tree for the listing, frame or both) and the delta snapshot --delta gives, which becomes the session's last snapshot; a change the listing does not show, or frames that move alone, as an animation's, leave it waiting, --frames ending it at a frame change too, for a canvas or a map, with the delta unchanged, and a listing that holds still fails with wait_timeout at the wait's timeout.

{"success":true,"data":{"elapsedMs":3000.329,"endedBy":"timeout","frameChanges":175,"settled":false,"stillChanging":["frame"],"treeChanges":0,"waited":"settle"},"error":null,"warning":"the screen did not settle within 3000 ms: frame changes kept arriving"}

Scrolling. scroll down 600 moves the content 600 points with the finger, and the platform's fling carries it on: each stroke lifts moving, as a swipe's does, and the reply comes once the screen has come to rest after the last, at most 3 s after it, movedOnMs saying how long the content moved on, ✓ Done (2 strokes, at rest 2341.3 ms after the last), since a list still gliding takes a tap as a stop; on the Simulator a list moved on for 0.7 to 1.0 s where the glide met its end and for 2.3 to 2.5 s on a list of 85 rows (action-verdicts, design D6). A stroke must be longer than a tap, more than 8 points on Android and 10 on iOS, or the command fails with stroke_too_short. Strokes keep off the keyboard and the system bars: each goes through what they leave of the screen, or of the element -s names, since a keyboard takes a stroke that comes down on its keys for typing (task 10.32: Gboard glide-typed "by" into the focused field at every stroke). When the keyboard leaves too little for a stroke, the scroll fails with covered_by_keyboard before it touches the screen. scroll down 600 --until '#row-150' scrolls until the element is mostly in view: its centre and at least half of it inside what its list shows, less the bars over it. Each stroke comes to rest before the finger lifts, so the list moves only as far as the finger, and is at most three quarters of what the list shows, so the element cannot pass through the list between two looks; the screen settles after each stroke. The reply's visible is the share of the element's bounds in view, from 0 to 1. The scroll fails with element_not_found once the content stops moving or no stroke along its axis brings more of the element in, and waits at most --timeout.

Timeouts. Waiting commands take --timeout <ms>; otherwise the session's default applies (set timeout <ms>), else --default-timeout or MOBILE_AGENT_DEFAULT_TIMEOUT, else 10000 ms. The maximum is 600000 ms. open's timeout bounds its launch, the app's first tree and the settle after it together, so an Android launch still running at it fails with deadline_exceeded, am start -W had not finished launching com.android.chrome by open's deadline, 9999 ms after the launch began; the app may still be starting: open it again with a longer timeout, or take a snapshot to see what is in front, and a longer --timeout gives it more (android-pass-fixes).

Batch. batch runs commands in order, given as quoted strings or as a JSON array of argv arrays on stdin, and --bail stops at the first failure. With --json it prints one array of {command, success, result, error} entries, with code on failures, and exits 1 when any entry failed. Here the third command never ran:

$ mobile-agent --json batch --bail 'get text #settings-wifi' 'click @e99' 'snapshot -i'
[{"command":["get","text","#settings-wifi"],"error":null,"result":{"text":"Wi-Fi"},"success":true},{"code":"unknown_ref","command":["click","@e99"],"error":"@e99 is not a ref from session \"default\"'s snapshots of emulator-5554; take a snapshot and use the refs it prints","result":null,"success":false}]

Sessions and targets. One daemon serves every session. --session <name> (default default) keeps target claims, refs, diff baselines and default timeouts apart. A command claims its target for its session, and another session is refused with target_claimed, naming the holder in data.holder, unless it passes --force; close releases the claim and stops the app the session opened, and close --keep releases it and leaves the app running, so the next session finds it as this one left it. cancel ends the session's command in flight, such as a wait another shell or client left running, which then fails with cancelled: a tap or tree read already under way finishes first, and cancel replies once the command has ended. With several targets and no --device or --platform, a command fails with ambiguous_target and lists them in data.candidates. Android allows one UI automation owner per device, so while the daemon holds one, Maestro, Appium and uiautomator dump cannot run against it; attaching while another tool holds the role fails with automation_conflict naming the likely holder, and --force takes it over.

iPhones. A phone on USB is a target once MOBILE_AGENT_DRIVERS names ios (Quick start). A command on a phone that is not ready fails with its readiness state as the code, and devices and doctor name the step. The tier is slower: a tree read of a 500-cell screen takes about 0.5 s and frames come 14 to 19 a second, so settle and waits take longer. Text goes in through the phone's runner, typed (keys) or set (direct), so the pasteboard is left alone; text whose method is left to the daemon is typed into an empty field and set into one that holds text. A runner that stops answering is started again: the call it failed fails with runner_stalled, which can be repeated, and a second stall in the session, or a runner that cannot start again, is disconnected, after which the next command attaches again (crates/daemon/README.md, The physical iPhone driver).

Error codes.

The full lists are in crates/daemon/README.md (Errors) and crates/cli/README.md (Output).

Agent surfaces

Two surfaces drive the same daemon, and a session one of them leaves is where the other finds it.

• The CLI, mobile-agent, for a shell, a script, a harness and the skills written for agent-browser: it follows that tool's grammar, @eN refs and JSON envelope, one process per command at about 5 ms (Agent guide; crates/cli/README.md).
• The MCP server, mobile-agent-mcp, for a client that calls tools over the Model Context Protocol, Claude Code, Claude Desktop, Codex or Cursor: a stdio server the client starts, with one tool per action an agent drives, 17 with help, under agent-browser's names where the concepts match (snapshot, click, fill, type, press, swipe, scroll, screenshot, wait, get, open, close) and the daemon's elsewhere (devices, status, doctor, diff_snapshot), input schemas from the request types, results as the CLI prints them, the same refs, codes and sessions, and the agent guide behind its help tool. It starts the daemon beside itself on the first call, as the CLI does, acts in a session of its own and releases its target with the app running when it exits (crates/mcp/README.md).

Take the CLI where the agent runs commands, the server where the client speaks MCP and would otherwise wrap the CLI in a shell tool. Both come out of cargo build --release, and the server takes the CLI's environment (MOBILE_AGENT_SOCKET_DIR, MOBILE_AGENT_DRIVERS for a phone, the settings file) from the entry that starts it.

Claude Code. claude mcp add --transport stdio mobile-agent -- /path/to/target/release/mobile-agent-mcp, with --scope project for a .mcp.json the repository shares, or the entry in a JSON file passed as --mcp-config, the form the acceptance used:

{"mcpServers": {"mobile-agent": {"command": "/path/to/target/release/mobile-agent-mcp", "env": {"MOBILE_AGENT_DRIVERS": "simulator"}}}}

The permission rule for every tool of the server is mcp__mobile-agent__*, which print mode (claude -p --allowedTools) needs, since it cannot ask.

Claude Desktop. The same entry under mcpServers in claude_desktop_config.json (Settings, Developer, Edit Config), with the binary's full path, since the app does not read a shell's PATH:

{"mcpServers": {"mobile-agent": {"command": "/path/to/target/release/mobile-agent-mcp", "args": [], "env": {}}}}

Codex. In ~/.codex/config.toml; the approval mode is what the acceptance found codex exec needs, since it runs with approvals off and otherwise fails every call to a tool not annotated read-only with "MCP tool call requires approval, but approval policy is never", while snapshot, get, wait and the other read-only tools run:

[mcp_servers.mobile-agent]
command = "/path/to/target/release/mobile-agent-mcp"
default_tools_approval_mode = "approve"   # "writes" asks before each acting tool in the interactive client

Codex gives a call 60 s, which the server's bound of 55 s stays under, and shows the model a result's structuredContent alone when there is one, so leave --structured off there (docs/research/agent-surfaces.md, How the clients consume servers).

Cursor. The same JSON as Claude Code's in the project's .cursor/mcp.json or in ~/.cursor/mcp.json.

The server's flags, each in crates/mcp/README.md:

• --tools read-only serves devices, status, doctor, snapshot, diff_snapshot, get, wait, screenshot and help alone, for a client that may only look; it also passes Codex's writes approval mode without a prompt.
• --call-timeout <seconds> bounds every call, 55 by default: a waiting tool's timeout is capped at it less 2 s, a call that names none keeps the daemon's default of 10 s, progress goes to the client every 5 s, and a call past the bound is cancelled in the daemon.
• --structured adds an outputSchema to each tool and a structuredContent to each result, for a client that reads JSON rather than the text.

Acceptance. A fresh Claude Code session given only the server's card and help opened the SwiftUI fixture on the Simulator, signed in, waited for the change, scrolled to a row, opened its detail and took a screenshot in 21 turns and 19 tool calls with none failed, $1.03 and 89 s; Codex did the same in 1 turn and 16 calls in 37 s (reports/12.1-mcp-acceptance.md).

Real apps. A first pass through the server on 2026-10-20, Claude Code on claude-opus-5-5 with one fresh claude -p per task, did nine tasks on the Simulator's Settings, Calendar, Maps, Safari and Photos and the emulator's Settings, Clock, Dialer and Chrome: 116 turns, 104 calls and 9 failed on iOS, and 58, 50 and none on Android; its notes and transcripts are kept privately. The real-app-fixes change fixed what it met on iOS: rows on screen left out of the listings, a timeout raised to the bound's 53 s on every call that takes one, type without a read-back, switches a plain tap does not flip, and replies that named less than the screen held. The second pass, on 2026-10-21 with those fixes and on claude-fable-5-1, the host's default by then, did three of its iOS tasks again, Settings' Bold Text on and off, an event in Calendar and example.com from Safari's address bar, in 52 turns and 46 calls with none failed, against the first pass's 87, 79 and 8 on the same tasks, and found three gaps the change then fixed: get value by the identifier a switch shares with its cell, fill's read-back of a field the app replaces as the text goes in, and Safari's focused address field counted under the keyboard (reports/12.2-real-apps.md). A third pass on 2026-10-22, the nine tasks again with those fixes and on the same model, completed each, with 2 failed calls of 63 on iOS against the first pass's 9 of 104, and 1 of 64 on Android against none of 50, and met replies that did not say what happened, which the action-verdicts change answers: a click names the element its tap landed on, a settle right after an action waits for its effect or says that none came, a ref whose element the app rebuilt stands for its twin, open leaves the app in front or fails with not_in_front, and a delta prints as lines. Replayed through the server, Maps' fill by the ref its snapshot gave filled the twin and named it, Safari's settle right after the Enter that loads a page named what changed last, Settings came to the front after press home and open five times in five, and Bold Text's flips came back as delta lines; Photos' click right after a scroll named the row it hit and opened nothing, the list still gliding, which a scroll that answers once the content has come to rest closed, the same click then opening the album three times in three, and Safari's More menu listed no element right after a settle 2 times in 9, which a settle that reads the tree its frames changed before it ends closed, the menu then listed whole right after it 20 times in 20; a fresh run of the Photos task took 7 turns and 5 calls with none failed, against the third pass's 16 and 14 (reports/12.3-action-verdicts.md). A fourth pass on 2026-10-23, the nine tasks again on the same model, left iOS with no failed call and Android with 4 of 63, Chrome's open failing with a would-block error from adb after 10 s on an emulator that launched apps in seconds, which the android-pass-fixes change answers: a launch takes open's own deadline and fails past it naming the launch, and a tap that lands on a plain text names it with (no ref). Replayed through the server, Chrome's cold open opened in 1.5 to 1.8 s 28 times in 28 in the emulator's first minutes after a boot, a tap at the point of Safari's address bar answered hit text "Address" (no ref) 6 times in 6, and a fresh run of the Chrome task took 13 turns and 11 calls with 1 failed, against the fourth pass's 22, 20 and 3, its failed call an open whose launch the emulator, six minutes after its boot, took 18.9 s over, which the model opened again with a timeout of 40 s (reports/12.4-android-pass.md).

Measured

The harness judges each gate at p95, and "under" is strict (crates/bench/README.md, Gates).

iOS Simulator (reports/10.2-simulator.md, reports/10.2-simulator/report.md): an iPhone 17 Simulator on iOS 26.5, Xcode 27.0, commit ec567d8, 50 measured iterations.

The warm snapshot comes from the mirror's replica; in the driver's suite, reading the SwiftUI screen from the device took 149.9 to 178.4 ms, under a load that the suite's own Simulator boots raised to between 100 and 250.

Android emulator (reports/10.1-emulator.md): an API 36 arm64 AVD on Android Emulator 37.1.11 started with -gpu host, commit 60e528a, 50 measured iterations. With the controller on (-grpc 8554 -grpc-use-token, reports/10.1-emulator-grpc/report.md), taps and frames go through the emulator's gRPC controller; with it off (-ports 5554,5555, reports/10.1-emulator-agent/report.md), taps go through the agent and there is no frame stream.

• Both runs drove the CLI through a shim (reports/10.1-emulator-grpc/bench-android-cli.rs) that task 10.10 has since made unnecessary. It adds a process start to every wall-clock sample: without it, mobile-agent --version took 5.0 and 5.3 ms at p95 (reports/10.1-emulator.md, finding 9).
• The runs' own reports judged tap to frame change against the 50 ms gate of the time, and failed it. Task 10.13 restated the gate for host rendering, which these values pass (docs/research/emulator-frames.md). The controller-off report, made before task 10.10, counts that series as failed where the harness now reports it as not measured.
• open on a cold-booted emulator, the host quiet (android-pass-fixes, 2026-10-24, Settings, Clock and Chrome): in the emulator's first minutes after the boot, a cold open costs the app's launch, as adb shell am start -W alone reports it (Settings 0.2 to 0.3 s, Clock 0.2 s, Chrome 0.65 to 0.85 s), plus about 0.1 to 0.5 s for the first tree, Settings' the longest for its first read; a warm one takes 0.1 to 0.4 s to its first tree (firstTreeMs), and the settle after it adds 0.2 to 1.4 s to the reply. The AVD's 2 GB, which the emulator runs at 2,560 MB, and -memory 4096 gave the same times, and launches of 1 to 3.6 s came now and then on both, so an open much slower than these is the emulator's when am start -W alone is as slow. A run's report records the emulator's memory beside its AVD (crates/bench/README.md, Reports). From about six minutes after a cold boot on, the same emulator launched apps in seconds, at either memory and with the daemon stopped: am start -W alone took Chrome 2.3 to 7.8 s, Settings 2.7 to 5.7 s and Clock up to 5.1 s through twenty minutes, and one launch of Chrome 18.9 s by Android's own count, past open's default 10 s, so that open failed with deadline_exceeded and opened Chrome with a timeout of 40 s (reports/12.4-android-pass.md, finding 1).

Physical Android device (reports/10.3-physical-android.md, reports/10.3-physical-android/agent/report.md): a Nexus 5X on Android 8.1 (API 27), commit 9769aa0 with agent 0.4.3, 20 measured iterations after 2 warm-up. A device has no frame stream: taps and screenshots go through the agent. The tap acknowledgement and warm snapshot gates are those the task named, taken from the android-driver spec's 50 ms "Tap acknowledged" bound and the Android warm snapshot gate, through a gate file (reports/10.3-physical-android/gates.json).

• Cold attach misses its gate on this phone: app_process alone takes 1.1 to 1.4 s to start there, and two of Expo's 20 opens did not settle within 10 s.
• Agent 0.4.2 and earlier could not run on Android 8.0 to 9: they called three platform members those levels lack (task 10.3, agent 0.4.3).
• On Android 8.1 the Compose fixture's launch focuses its first field and raises the keyboard over the tab bar, so the Compose list-scroll and big-screen flows failed their golden checks. Expo's list-scroll failed its check at the list's end.

Physical iPhone (reports/11.1-ios-device.md, The gates on the fixed run; reports/11.1-ios-device/a6-swiftui-primitives-fixed/report.md, reports/11.1-ios-device/e2-expo-primitives-fixed/report.md): an iPhone SE (3rd generation) on iOS 26.6.2, Xcode 27.0, the release build at commit 9652e5e with the fixes of group 9 of physical-ios (task 9.6), the runner 0.2.0 already on the phone, 50 measured iterations after 3 warm-up. Input goes from the host through the disk image's dtuhidd, and frames are screenshots polled on DVT and CoreDevice. The gates are the ios-device class's defaults (crates/bench/README.md, Gates).

• The first run, on 2026-10-10 at commit f8e840b before group 9's fixes, passed the same four gates (reports/11.1-ios-device.md, Gates).
• Cold attach starts a daemon, lists the phone, opens the tunnel and the runner's session and brings the fixture up. SwiftUI's p95 is four attaches of 3.1 to 4.3 s among 49 of 2.0 to 2.4 s, in a series that raised the host's load to 27.2 itself; the phase it repeats, a5, failed the gate at 9,205.7 ms, four of its attaches taking 5.4 to 29.5 s while the host held process launches.
• The warm snapshot is answered from the daemon's replica, as on the Simulator: all 200 of the fixed run were, the handler taking about 0.5 ms. Its 1 s limit was set from the runner's read of the same 500-cell screens, 450 to 650 ms, which no warm snapshot waits for, and the gate table shows the reads beside the replica's answers (reports/11.1-ios-device.md, Snapshot reads on physical iPhones). The first snapshot after a relaunch waits for a read the daemon begins as open returns: 766.1 and 457.8 ms at p50 on SwiftUI and Expo.
• Tap dispatch counts the 50 ms the tap holds the finger down. Tap to frame change runs from the dispatch to the arrival of the first screenshot whose pixels below the status bar moved, at 14 to 19 captures a second: 272.7 and 154.0 ms at p50 on SwiftUI and Expo, against 72.8 and 47.7 ms on the Simulator, whose frames stream at 30 a second.

Against the baselines: how many times faster mobile-agent's p50 was on each reference flow. "Not timed" means the tool failed the flow's golden check, so the harness did not time it. mobile-agent's flows had no failed or flaky iteration in the runs below; a baseline's failure and flake rates follow its factor where they are not zero.

The login rows were measured again on 2026-10-07, after fill began reading the field back (task 10.23), together with the text-entry series (reports/10.26-remeasure.md); the other rows are the published runs'. On the emulator, mobile-agent's two series ran once more that day, after the read-back's 200 ms hold was kept for typed text alone (task 10.30), and its login factors there are taken against the baselines of the earlier run (reports/10.26-remeasure.md, 10.30 rerun). On the Simulator mobile-agent's two series ran once more, after the mirror began reading a field's window again as soon as a pasted text's frame changes it (task 10.29), and its text entry and SwiftUI login once more, after a paste began putting its text on the pasteboard without devicectl (task 10.31); the login's fills are typed and read back as before, so its Simulator rows stand (reports/10.26-remeasure.md, 10.29 and 10.31 reruns). A text entry of 36 characters took 199.2 and 387.0 ms at p50 on the Simulator (SwiftUI, Expo), which pastes it, against 376.3 and 555.4 ms before task 10.31, 545.5 and 718.3 ms before task 10.29 and 249.3 and 228.3 ms before the read-back, and 43.0 and 234.9 ms on the emulator (Expo, Compose), which sets it directly, against 23.8 and 26.1 ms before the read-back and 224.1 and 240.0 ms with the hold on every fill. At the median the read-back is 21.2 ms of it on the emulator's Expo fixture, whose fields report a change at once, 218.1 ms on Compose, whose window is read again after the action, and 92.4 to 100.4 ms on the Simulator. The Simulator's entries fall in two clusters about 250 ms apart, the slower waiting for a re-read after the emptied field's caret blinked, and its p50 lands in either.

iOS Simulator (reports/10.2-simulator.md, Baselines; login, reports/10.26-remeasure.md):

Android emulator, controller on (reports/10.1-emulator.md, Baselines; login, reports/10.26-remeasure.md, 10.30 rerun):

Physical Android device, the login flow alone (reports/10.3-physical-android.md, Login against the baselines; 20 measured iterations):

Physical iPhone, the fixed run: each flow through mobile-agent, agent-device and WebDriverAgent in one phase, with the harness's hand-offs (reports/11.1-ios-device.md, The flows on the fixed run; 50 measured iterations). A factor under 1 means the baseline was faster.

• At p50 mobile-agent took 6.3 and 5.8 s on login, 30.6 and 28.4 s on list-scroll and 4.6 and 3.4 s on big-screen, on SwiftUI and Expo; WebDriverAgent 5.9 and 5.7 s, 11.1 and 10.4 s and 3.5 and 3.8 s; agent-device 11.3 and 10.8 s on login and 3.6 and 3.9 s on big-screen; and mobile-agent on the Simulator 4.1 and 3.9 s, 16.3 and 15.0 s and 2.5 and 2.8 s.
• What the comparison measures: mobile-agent's flows settle and verify after every step, as an agent does. A click waits for its first changed screenshot, then for a read of 450 to 650 ms, and scroll --until makes 22 to 25 resting strokes on the SE's 667-point screen, each followed by a settle and a read. WebDriverAgent's flows fire their requests and check at the end: its list-scroll flings the list in 9 swipes and polls for row-200 (reports/11.1-ios-device.md, Flows).
• agent-device's list-scroll failed its golden check on both fixtures, its scroll --until spending its fixed 12-pass budget before row-200 came into view (reports/11.1-ios-device.md, finding 9). The harness skips Maestro on iPhones with the note that 2.11.0 refuses them, its device driver unfinished (the host has 2.10.0, whose unfinished driver lacks that guard), and maestro-runner was not installed. In the first run mobile-agent's runner connection was lost beside the baselines in every phase but Expo's big-screen (reports/11.1-ios-device.md, finding 5), so the comparison is the fixed run's.

What each tool costs the host and the device, memory and CPU over ten login flows and a minute idle, is in reports/10.20-resources.md: per flow, mobile-agent's processes used 0.35 CPU-s on the Simulator against agent-device's 2.75 and Maestro's 18.82, and on the emulator 0.28 on the host and 0.05 in the guest against 0.62 and 0.22, and 13.45 and 1.82. With an iPhone attached and idle, the daemon took 10.4% of a core and usbmuxd 7.6% in the first run, its screenshots and its 1 s poll together, and 4.7% and 7.7% in the fixed run, with the poll of a still screen at 10 s and DVT's screenshot failing on the phone; the phone's own share was not measured (reports/11.1-ios-device.md, The idle poll).

The list-scroll flow reaches row-200 with one scroll --until, as an agent would, since task 10.17; before, the harness ran its own loop of settles, checks and plain strokes. p50 of the flow before and after (reports/10.17-list-scroll.md, with the Simulator's after column from the 10.2 run), the emulator with its controller on:

Host costs, from the host microbenchmarks (reports/micro/baseline.json; reports/micro/README.md says how to run and read them): what each host-side path costs without a device, measured by just micro on 2026-10-08 at commit 7b161de on the same Apple M1 Max under macOS 26.6.2, with rustc 1.98.1 and criterion 0.8.2, from a 1-minute load of 2.46 that the quiet-host guard passed. Each value is the median of criterion's 100 samples, in microseconds; allocations are per iteration, one pass over the input, where the benches of protocol, mirror and daemon count them. The mirror's paths run on its scripted fake target, whose read hands over a copy of the screen where a driver would decode one, and the socket's on a daemon serving a fake emulator. The 1,049-node screen is the Expo fixture's Big tab as spike S1 read it on the emulator, reconstructed, since the repository holds no capture.

• The socket's span runs from writing the request out to the reply parsed, the host's whole path less the CLI process; most of its work is in the daemon's process, whose allocations the bench does not count. The baseline's other 39 benchmarks cover the remaining inputs and cases: the golden sample and the 1,000 requests, to_vec and write_frame, the list and Bluesky fixtures, full listings and listings taken again, reads with one change, with a window added or removed and after a resync, scroll_viewport, uncovered, waits, and the React Native screen's settle with frames.
• The daemon's Simulator frame thread runs that sequence without the JPEG: it downscales and hashes each frame it reads, at most 30 a second, and encodes a JPEG only for a screenshot (crates/daemon/src/devices/simulator.rs). Spike S2's pipeline encoded every frame; at 60 frames a second the sequence with the JPEG would take 13.4% of a core, where S2 measured 20.9% for its whole pipeline (docs/spikes/s2-simulator-frames.md, Verdict).
• Beside spike S5, which encoded its 500-node tree in 26.9 µs and decoded it in 59.1 µs on its own model types, built with fat LTO and one codegen unit and with each decoded value dropped outside the timed region (docs/spikes/s5-wire-format.md, Rust on the host): the suite's 500-node snapshot has S5's shape on the shipped crate's types, 21,340 bytes against the 21,410 of S5's own tree on those types (docs/spikes/s8-bench-harness.md, Input), and its decode drops the message inside the timed region, under the workspace's release profile. Built with S5's profile, on a quiet host at a 1-minute load of 2.45, the same benchmarks gave 25.4 µs encode, 25.6 µs write_frame, 27.4 µs to_vec and 119.6 µs decode, against 54.4 µs encode and 123.8 µs decode in the first baseline, taken the same day (task 3.2 of the microbenchmarks change): the encode's gap to S5 is the build profile, and the decode's, twice S5's under the same profile, the shipped node decoder.
• The baseline was measured again on 2026-10-11 at commit cb9dd0a (task 9.6 of physical-ios), from a 1-minute load of 3.11 after 90 s of the guard's wait: every benchmark came within the threshold of the values above, from −4.6% (mirror/snapshot/full-list) to +2.7% (daemon/wire/reply-snapshot-big), the median 0.0%, with no allocation count changed. It was measured once more on 2026-10-21 at commit 6757f39 (task 5.2 of real-app-fixes), under macOS 27.0.1, from a 1-minute load of 3.59: against the run of 2026-10-11, the suites whose code had not changed between, the protocol's, the Simulator's frame pipeline and the emulator's frame key, ran 4.1 to 4.3% slower at the median; the mirror's listings, delta and diff took 1 to 11 allocations and 600 to 8,392 bytes more, from the commits between, their times from −3.8% to +1.8%; the request's parse took 4 allocations against 8; and daemon/socket/snapshot-interactive-big took 868.2 µs against 773.8 µs, +12.2%, while an A/B of the daemon before the host rule of real-app-fixes and with it read −1.5% (its design D2). It was measured again on 2026-10-30 at commit d7930f8 (task 3.1 of profiling), under macOS 27.2, from a 1-minute load of 3.33, with 74 benchmarks: against the run of 2026-10-21, the reply's building and writing took 35.8 µs and 12 allocations where its line alone took 94.8 µs and 21, daemon/socket/snapshot-interactive-big 592.3 µs, −31.8%, after the fixes of that change; the mirror's listings right after a read 505 allocations more on the 500-cell screens, the ref table keeping each issued ref's identifier for Mirror::twin, their times within the threshold; and the suites whose code had not changed, the protocol's, the Simulator's frame pipeline and the emulator's frame key, 2.7 to 7.8% faster, a median of 5.7% on the protocol's. reports/micro/baseline.json holds that run's values; the table keeps the first run's, but for the rows marked †, which are that run's: the benchmarks the profiling change added, and the reply's, which since its task 2.3 builds the reply from a mirror's listing as well as writing its line.

Spike measurements. The spikes' throwaway programs, kept privately since the repository was made public (docs/spikes/README.md), measured these before the crates existed, on one host (Apple M1 Max, macOS 26.6.2, Xcode 27.0, CoreSimulator 1171.7, iOS 26.5, an API 36 arm64 emulator); the spike verdicts table in openspec/changes/archive/2026-10-07-mobile-automation-foundation/design.md summarizes them. They are not harness results.

How measurements are made

crates/bench (mobile-agent-bench, run with just bench) drives the release CLI as an agent does, one --json process per command, on the reference fixtures of fixtures/CONTRACT.md (SwiftUI and Expo on the Simulator, Expo and Compose on Android), with 3 warm-up and 50 measured iterations per series and nearest-rank p50, p95, maximum, failure and flake rates. The same login, list-scroll and 500-node flows run through Maestro 2.10.0 (on Java 21.0.12) and agent-device 0.21.14 (on Node 25.8.0) as baselines, as the runs' setups record (reports/10.1-emulator.md, reports/10.2-simulator.md), and through maestro-runner when it is installed, each timed by wall clock around the tool's process. Each run writes report.json and report.md, committed under reports/, and exits non-zero when a gate fails.

On a physical iPhone the harness drives the same CLI with MOBILE_AGENT_DRIVERS=ios (crates/bench/README.md, Physical iPhones). The phone ran twice, on 2026-10-10 (tasks 7.1 to 7.3 of physical-ios) and, after group 9's fixes, on 2026-10-11 (task 9.6), whose numbers the tables above give: on USB and external power, unlocked, Auto-Lock at Never, which no tool reads, so the run states it (--auto-lock-off), and the runner already installed. Before each phase the harness installs the fixtures' device builds, which their own scripts sign for the team (DESTINATION=device fixtures/<fixture>/build.sh), through CoreDevice's devicectl rather than through the tool under test, and it records the phone's model, iOS version, battery, runner and input and frame paths as the daemon reports them. The baselines on the phone are WebDriverAgent 16.14.1, built from the copy in Appium's XCUITest driver 12.16.0, re-signed for the team, started through the phone's test manager by the harness's own helper and attached to Appium 3.8.0, and agent-device 0.21.14, which builds and runs its runner with xcodebuild and so needs the Mac's DevToolsSecurity on (sudo DevToolsSecurity -enable, the maintainer's step, for that baseline alone); Maestro is skipped, since 2.11.0 refuses physical iPhones (crates/bench/README.md, Baselines). A phone has one automation session, so each tool's release ends its runner app there and the hand-off checks the phone's processes. The host's endpoint security inspects process starts and at times held them for seconds to minutes, and every CLI command is a process of its own, so each phase first waited for a quiet host, in the fixed run a 1-minute load under 4, no build above 5% of a core and a launch of /usr/bin/true within 1 s in two checks 60 s apart; the report names the phases run again and the series that still carry load or a launch hold (reports/11.1-ios-device.md, Method, findings 4 and 10).

just micro (mobile-agent-bench micro, crates/bench/README.md, Microbenchmarks) times the host-side paths without a device, through the suites in the crates' benches/ under criterion 0.8.2 at its default settings: 3 s of warm-up, then 100 samples over about 5 s per benchmark, of which it keeps the median, the median's 95% interval and the sample count, beside the allocations and bytes per iteration that the benches of protocol, mirror and daemon count. It measures only on a quiet host, a 1-minute load under 4 and no build, emulator or Simulator process above 5% of a core, and otherwise exits 2 or, with --wait, waits for one. It compares the run with the committed baseline, reports/micro/baseline.json, and exits 1 on a regression: a median more than 5% slower than the baseline's and above the 95% interval the baseline recorded around its own, or any rise in allocations or bytes, which compare exactly. The threshold is twice criterion's run-to-run coefficient of variation on a quiet host, with a floor of 5%; spike S8 measured that coefficient at 1.15%, so the floor holds (docs/spikes/s8-bench-harness.md). A baseline is refreshed only by a run that passed the guard, --save-baseline, and committed with the change that moved it. ci.yml compiles the suites, since just lint covers all targets, and runs none of them: a hosted runner's timings are not this host's, so CI compares two refs on one runner instead, below.

Profiles attribute a path's time to its frames, where the microbenchmarks time paths chosen in advance: spike S15 recorded the daemon on the Simulator and the emulator, the CLI's start and the MCP server's round trip with xctrace's Time Profiler and Allocations on the profiling build (Build), found this code's CPU under 3% of every command that reads or acts, and ranked the hot spots design D3 of openspec/changes/archive/2026-10-30-profiling fixed, each shown first by a benchmark and measured after by micro --ab (docs/spikes/s15-profiles.md).

Benchmarks run in CI too, but never on a push or a pull request: .github/workflows/ci.yml validates, and .github/workflows/benchmarks.yml measures, on a schedule, Sundays at 02:17 UTC, and on a manual dispatch that picks the suites (micro, harness or both), the two refs and the harness's iterations (crates/bench/README.md; design D1 of openspec/changes/ci-benchmarks). Its microbenchmark jobs compare two refs on one runner, micro --ab: the base checked out into a worktree of its own, both built there and run in three rounds with the sides alternating, each benchmark judged by the relative difference of the two medians, a head more than 10% slower than base, or any rise in allocations or bytes, failing the job; the protocol, mirror, daemon, android and MCP server's suites run on an Ubuntu runner and the Simulator's frame pipeline on a macOS one. A scheduled run's base is the commit the previous scheduled run measured, so it compares the week's changes. Its harness jobs run the reduced plan, run --plan ci --gates none: one fixture per platform at 10 measured iterations after 2 warm-up, SwiftUI on a Simulator of GitHub's xcode-27 image, whose Xcode 27 the Simulator driver's text input needs, and Compose on an API 36 x86_64 emulator with KVM and software rendering, beside Maestro 2.10.0 and agent-device 0.21.14 installed at pinned versions; mobile-agent is judged there by how many times faster it ran each flow than each baseline in the same run, not by the gates. Each job writes its tables to the job summary and keeps report.json and report.md as an artifact for 90 days, and a scheduled run prints its numbers beside the previous scheduled run's, shown and never judged, since the two runs had different machines.

What CI does not measure: the gates, whose limits stay with the measured hosts above, since a shared runner of 2 to 4 cores is another machine; physical devices, which need a self-hosted runner and, on iOS, someone at the phone for its approval prompt; WebDriverAgent, whose adapter drives physical iPhones only; and a threshold of its own, until spike S12 measures the runners' noise and sets the 10% it starts from. The workflow has run twice, both on dispatch: on 2026-10-30 the microbenchmarks (run 41034955381), its Linux job green and its macOS job failing its A/B of main against itself by the runner's noise, which ci-benchmarks task 2.1 measures; on 2026-10-31 the harness (run 41181272250), green on both runners, its emulator then made without a device profile, a 320x640 display at 160 dpi, where the workflow now boots the Pixel 8's, 1080x2400 at 420 dpi. docs/research/ci-benchmarks.md estimates a weekly run of both suites at about 4 hours of runner time and $6.60 in a private repository, the macOS jobs most of it, and nothing in a public one.

Status and limitations

• One physical Android device has run: a Nexus 5X on Android 8.1 (task 10.3, reports/10.3-physical-android.md). Cold attach missed its 1.5 s gate there. With the keyboard up, a scroll stroke that starts over the keyboard reaches it, and Gboard typed the strokes into the focused field. On Compose's 500-cell screen commands waited about 10 s, since Android 8.1's node cache is one for the whole process and every eviction empties it. Devices have no frame stream, so clicks there report no frameChangeMs and settle rests on the tree. OEM Android builds and the levels between 8.1 and 16 are unverified.
• One physical iPhone has run: an iPhone SE (3rd generation) on iOS 26.6.2 under Xcode 27.0 (reports/11.1-ios-device.md). Every path the driver takes is gated on the phone's iOS version; iOS 27, whose dtuhidd takes the host's input only while a display stream runs, the stream that would also give frames, is a later path, and no other iOS version has run.
• An iPhone has no frame stream on iOS 26: the CoreDevice display stream needs iOS 27 (CoreDevice error 9021 on 26.6.2), and AVFoundation's capture of the phone waits on the macOS camera permission for the process that starts the daemon, so it has not been measured. Frames are screenshots polled on DVT and CoreDevice, 14 to 19 a second, so tap to frame change is reported rather than gated, settle's quiet window lasts at least two capture intervals, and the re-read after a tap waits up to 500 ms for the first changed frame (openspec/changes/archive/2026-10-14-physical-ios/design.md, D23).
• What an iPhone's tree costs and lacks: a read of a 500-cell screen takes 450 to 650 ms; an idle phone's still screen is read again every 10 s, which keeps the runner reading about 5% of the time; and the first snapshot after a relaunch waits for a read the daemon begins as open returns, about 670 ms on SwiftUI and 450 ms on Expo (reports/11.1-ios-device.md, Read costs and The idle poll). A view the app marks modal, other than an alert, is not marked, since testmanagerd's reply carries no children count, and XCTest leaves out a row a floating tab bar covers, as SwiftUI's row-12 under the SE's bar, which scroll --until brings out. Text goes in through the runner, typed at 60 characters a second or set, never pasted.
• The phone's developer services failed twice in the day of the fixed run (reports/11.1-ios-device.md, finding 11): DVT's screenshot stopped answering and testmanagerd stalled the runner on the 43rd emptying text set of every session, until a reboot; later DVT refused its process-control channel at attach, and answered again by itself about 1 h 30 min after. A runner that stops answering is ended and started again once, and a screenshot poller whose captures time out is reopened, then left out and named in status, the screenshot coming through CoreDevice; but a phone whose DVT refuses that channel cannot be driven, since open launches through it, and CoreDevice's app service as the launch's fallback is an open question (openspec/changes/archive/2026-10-14-physical-ios/design.md, Open Questions).
• An iPhone needs a person at times: the passcode for "Enable UI Automation" at a runner's start (Quick start), and an unlock, since a locked phone fails with device_locked. One automation session exists per phone, so mobile-agent's runner and another tool's take turns.
• Only Xcode 27.0 and 27.1, iOS 26.5, the API 36 emulator, one API 27 device and one iPhone on iOS 26.6.2 have run on a developer Mac: the gates were measured under Xcode 27.0, and Xcode 27.1, on macOS 27.0.1, ran the MCP server's acceptance, spike S14 and the passes on real apps (reports/12.1-mcp-acceptance.md, finding 9). iOS 27.0's Simulator has run only in CI's device tests, on the xcode-27 image, where its tree reads once the reader bounds the floating bar container iOS 27 types a toolbar, and where tests on the first frame and taps of a newly booted device, the copy back and the translator on a fresh device still fail (design D1 and D9 of openspec/changes/pass-4-fixes).
• The emulator's tap-to-frame gate is p95 under 67 ms, four frames of the AVD's 60 Hz display, and holds for host GPU rendering (-gpu host) with the controller on: the guest's own frame pipeline takes about three frames at p95, and the tool's dispatch about 3 ms (docs/research/emulator-frames.md). The 10.1 run measured 52.5 ms on the Expo fixture and 55.1 ms on the Compose one. Software rendering is not gated; spike S7 measured 146.5 ms at p95 there.
• On a Simulator whose first boot did not come from the driver, the first tree reads can fail for several seconds, until the reader has turned automation mode on (spike S4); retry.
• React Native: reports/10.4-rn-smoke.md compared 17 logged-out React Native screens on the Simulator (Bluesky, Ignite's demo app, the Expo fixture), where no XCTest read reported a sparse root, then Bluesky's signed-in feeds, threads, profiles, notifications, search, chat and settings on the Simulator and the emulator. Signed in, agent-device's XCTest tree came back sparse or timed out on the search screen with the keyboard up and on a 711-node thread, and mobile-agent's snapshots of those screens were whole and not degraded. The same run found that with the keyboard up on the Simulator every element counts as 0% in view, so click warns wrongly and scroll --until, which stops by the same measure, cannot find its element in view (task 10.21).
• React Native apps on the Simulator, after seven rounds of production-app runs, which checked the fixes of openspec/changes/archive/2026-10-15-rn-app-fixes/ on a production React Native app (its sign-in, consent, switcher, feed, creation and members screens, with the Simulator's hardware keyboard connected and disconnected): no wrong tap in the last round and the sign-in flow in 18.4 to 20.9 s typed or filled; what stays open: typed keys on React Native fields are dropped at times under host load (5 of 11 fills in one round), which fill's read-back repairs with a paste in all but the rare lost paste, which fails text_mismatch; a control the app places under the status bar strip reads as in view and its tap is accepted without effect; a refusal behind a dialog names "the modal view", not the dialog's title; texts a button covers whole are absent from the full listing while get finds them; elements wholly under the software keyboard carry no visible=0% mark in the full listing.
• The drivers use private Apple Simulator frameworks, the developer services of Apple's disk image and private XCTest calls on an iPhone, and hidden Android APIs. This is development tooling; nothing of it ships inside an app.
• The MCP server has run once per client, Claude Code and Codex, on the Simulator with the SwiftUI fixture, with no failed call (reports/12.1-mcp-acceptance.md), and with Claude Code on real apps in four passes and two replays (Agent surfaces), one run per task, so that each count is one sample; no call met type's warning for a field that dropped keys or the cap on a call's timeout (reports/12.2-real-apps.md, What was not measured). Claude Desktop and Cursor are configured above and have not run. What the passes left: on the Simulator an animation that pauses longer than about 430 ms, the quiet window, the read of the tree the frames changed and a second window, still ends a settle before its last frame, as Safari's menu closing, which paused 358 to 560 ms, did 4 times in 24 (action-verdicts, design D2); on Android a click answers no change in 203 ms when the tap's effect comes after its 200 ms window, as Settings' Dark theme, the Dialer's first digit and Chrome's search box did, which wait with change then collects; in Clock's time picker in keyboard mode the minute chip is refused with element_covered, the picker's minute input view, which has no ref, lying over it, so a model taps its point from a screenshot; a wait with text right after the Enter in Chrome's address bar can match the omnibox's suggestion, button "Example Domain: example.com", before the page holds the text; Clock names a new alarm's row Alarm, without its time, which is the app's label; a cold launch slower than open's default 10 s, as an emulator's from about six minutes after a cold boot can be, fails with deadline_exceeded and needs a timeout (reports/12.4-android-pass.md, finding 1); and the server has no batch tool, so taps that must come in order take a call and a turn each, as the Dialer's seven digits took seven.
• Safari's web content is not in the iOS tree, so a page's text is read from a screenshot, where Android's Chrome lists its web view; and Safari's address bar, on its start page as on a page, is a plain text without a ref, which snapshot -i leaves out and the full listing names by its identifier, TabBarItemTitle; a tap on it focuses the field and answers hit text "Address" (no ref) (reports/12.2-real-apps.md, Safari and finding 5; reports/12.4-android-pass.md).
• Not built: a CDP-subset endpoint and Android video streaming.
• The OpenSpec change in openspec/changes/archive/2026-10-07-mobile-automation-foundation/ (proposal, design, specs, tasks) is the source of truth for the design, its decisions and the remaining tasks; the microbenchmarks change, archived in openspec/changes/archive/2026-10-08-microbenchmarks/, adds the host microbenchmarks, design D13 to D17, the physical-ios change, archived in openspec/changes/archive/2026-10-14-physical-ios/, the physical iPhone, design D18 to D26, the mcp-server change, archived in openspec/changes/archive/2026-10-20-mcp-server/, the MCP server and the shared request and reply types, design D1 to D12 of its own, the real-app-fixes change the fixes the first pass on real apps called for, design D1 to D7 of its own, the action-verdicts change, in openspec/changes/archive/2026-10-23-action-verdicts/, the replies the third pass called for, design D1 to D6 of its own, and the android-pass-fixes change, in openspec/changes/archive/2026-10-24-android-pass-fixes/, a launch that takes open's deadline, hits named without a ref and the costs the fourth pass called for, design D1 to D4 of its own.
• docs/overview.md answers, with sources, how the design differs from Maestro, agent-device and Appium, where it is and is not faster, what it leaves out, which platforms and versions it supports, which protocol it speaks and how the two agent surfaces, the CLI and the MCP server, sit on the daemon.