English | 한국어 | 日本語 | 简体中文

An Android AI agent that uses your phone for you

Ask in plain language, for example "Get my phone ready for bed: turn on dark mode, set an alarm for 6:30 AM, and turn on Do Not Disturb". OpenPhoneUse reads what is on the screen, decides what to do, and works through your apps the way you would. It is a phone use agent, the mobile counterpart of a computer use agent: a mobile GUI agent that does not need apps to ship an integration.

• Runs on the phone alone. No PC, root, ADB or Shizuku. You turn on one accessibility service and sign in to a model.
• Works in the apps you already have. It reads the accessibility tree and a screenshot, so drawn content such as games, maps and charts works too.
• Uses your model. Your ChatGPT plan, an OpenAI API key, Claude, Grok or Gemini.
• Asks first. Always before paying or changing security settings. Before deleting, unless you asked for that one recoverable item, and before sending, unless you named the recipient and the text. It never types your passwords; it hands that step to you.

Benchmark: 94.5 / 100 on 31 graded tasks in 9 categories, with a median of 42 seconds per task (GPT-6.1 Terra with fast mode, on an Android emulator, 2026-11-03, one run on a build from before this release). See how it is graded.

Download the APK or read the install steps.

Demo

_{Real app on an Android emulator. The sample conversation is seeded and the overlay states come from debug broadcasts (how these were made). Watch the MP4.}

Answers that show their work	Asks before risky actions	Works inside your other apps	Routines and nudges
Choose model and speed	Your plan or an API key	Two panes on a foldable

What it can do

These requests come from the benchmark suite, shortened here; the originals are in Korean or English. Each one is typed into the app as a new conversation.

Besides driving apps, it has built-in tools for contacts, calendar, alarms, reminders, SMS, notifications, photos, files, location and routes, plus memory, routines ("when I arrive at work, go silent"), Gmail and Google Calendar connectors, remote MCP servers, and two-pane layouts on foldables.

Install

On your phone: tap the button. On a computer: scan the code with your phone's camera. Always the latest version, about 70 MB. All versions and checksums.

1. Install the APK. Open the downloaded file and allow installs from that source when Android asks. It is not on the Play Store. If Android says the app was not installed, uninstall any earlier test build first. Google Play Protect may show "App scan recommended" and then "Harmful app blocked". Google does not say why; OpenPhoneUse asks for strong access, such as accessibility control and text messages. Decide for yourself: compare the APK with the OpenPhoneUse.apk.sha256 file in the release, read the source, or build it yourself. To go ahead, choose the option to install anyway, and leave Play Protect on.
2. Turn on control: Settings → Accessibility → OpenPhoneUse → on. On Android 13 and newer, a sideloaded APK usually has this switch greyed out by "restricted settings"; if so, go to Settings → Apps → OpenPhoneUse → ⋮ (top right) → Allow restricted settings, then turn it on again.
3. Sign in: in the app's Settings, choose Continue with ChatGPT to use your plan, or paste an OpenAI, Claude, Grok or Gemini API key.
4. Ask for something, for example "Turn on Do Not Disturb" or "Compare the current price of the AirPods Pro 3 at Amazon, Best Buy and Walmart".

Requirements: Android 11 or newer, and a ChatGPT account or an API key from one of the providers above. To build it yourself, see Build from source.

How it works

1. Read the screen two ways. Android's AccessibilityService gives the app every window as a tree, compressed into one numbered line per element, for example [3] Button "Search" clickable (540,212). The same screen also goes as a screenshot in which every element you can act on is outlined and tagged with its number (Set-of-Mark), so the model sees layouts, icons, charts and games and still acts by number.
2. Call tools. The model streams back tool calls such as tap, type_text, scroll, open_app or batch. One batch can walk through several screens by naming each target by its visible label. Deep links and open_app skip menus.
3. Check, then act. Before every action the live screen is read again and the target is found there; if the screen moved, nothing is done and the new screen goes back to the model. Each tap or typed text carries the model's own risk label (money, security, delete, send, change) and effect, judged in the same tool call, and RiskGuard keywords back it up. A confirmation card appears when the rules below say so, and only then does the accessibility service act.
4. Start thinking early. The model call starts as soon as the screen looks settled. If the screen then changes, the call is restarted with the new screen, so you get the speed without acting on an old screen.
5. Keep a plan for long tasks. With update_plan the agent writes its own checklist and marks steps as it goes. Its reasoning carries over between steps and old screens shrink to one-line digests, so a 50-step task keeps its goal in view.
6. Research. Questions use the provider's hosted web search (OpenAI or Claude), with sources under the answer. When a page has to be read or used, the agent opens it in Chrome on the phone and reads it like any other app.

The main model (GPT-6.1 Terra with fast mode by default) makes every decision. Repetitive work, such as scrolling a long list and collecting names and prices, can be handed to a lighter helper model through a delegate tool, and the helper goes through the same safety checks. The whole harness is written in Kotlin for this app; it does not use LangChain or another agent framework. More detail: docs/ARCHITECTURE.md.

Safety and privacy

What leaves the phone: what a task needs goes to the model provider you chose: your messages, the compressed screen text and screenshots, and whatever a tool reads for the task, such as file names, contacts, calendar events, photos, texts, notifications, your location or saved memories. Place searches go to OpenStreetMap's Nominatim, and maps load OpenStreetMap tiles. Services you connect yourself receive what is sent to them. There is no OpenPhoneUse server.

Found a hole? Please report it privately: SECURITY.md.

Benchmark

GPT-6.1 Terra with the fast mode switch on, run on 2026-11-03. The run predates this release's sign-in changes and has not been repeated since, so scores and times may differ on the current build. Per-task scores and judge notes: bench/reports/2026-11-03-v2-final.md.

Reading the score. This is one run of one model on an emulator, not on a physical phone. The median task took 42 seconds and the whole suite 45 minutes. The weakest tasks were Minesweeper (23), an alarm that depends on the weather forecast (69) and a three-day trip plan (80). An earlier attempt was thrown out because the emulator captured blank screenshots; run.py now refuses to start when a screen capture comes back blank.

The suite. bench/suites/v2.json holds 31 tasks in 9 categories: settings, apps, safety, chains (look something up, then act on it), maps, media, web, research and games. Prompts mix Korean and English, like real use. Difficulty runs from one-step lookups to a three-day trip plan or a game of Minesweeper.

How it is graded. Every task has a rubric of weighted items and scores 0 to 100:

• device: an adb shell command after the run must match, for example the alarm list or settings get system screen_off_timeout.
• reply: a regex over the final answer (the right number, a source link).
• trace: the tool log, for example that Google Maps was actually used.
• confirm / no_confirm: whether a confirmation card was or was not shown (safety tasks decline every card).
• judge: scored blind by a judge model (Claude Sonnet, with the tasks split across three instances) from the reply, the screenshots and any recorded device facts. Run labels hid which model ran.

Fair runs. Before every task the emulator is restored from the same snapshot (apps, browser, drafts, settings, the agent's own memory), its clock is set to the real time in Seoul, and Chrome's tabs and game saves are cleared. When several models are compared they run task by task, alternating which goes first.

The runner drives a debug build from a PC over adb (the app itself never needs adb; the runner does, to set up, check and record):

python bench/run.py --suite v2 --snapshot bench --no-video              # the whole suite
python bench/run.py --suite v2 maps-transit-plan trip-plan              # chosen tasks
python bench/run.py --suite v2 --models gpt-6.1-terra,gpt-6-astra       # compare models
python bench/score.py packets RUN_DIR... --out JUDGE_DIR                # blind packets for the judge
python bench/score.py report RUN_DIR... --judge JUDGE_DIR --out report.md

It needs a debug build with a signed-in model, the accessibility service on and an emulator snapshot saved with adb emu avd snapshot save bench. Results go to bench/results/<timestamp>/. The full report of the run above is in bench/reports/.

How OpenPhoneUse compares

There are several ways to automate an Android phone. They make different trade-offs; this table is about approaches, not specific projects, and "varies" means it depends on the tool.

PC-driven agents are great for research and for running many devices; macros are great for fixed, repeatable jobs. OpenPhoneUse aims at the case in between: a single phone, almost any app, a request in plain words, and no extra hardware.

FAQ

What is OpenPhoneUse?

OpenPhoneUse is an open-source Android app that lets an AI agent operate your phone like a person. It reads the screen through Android's AccessibilityService, sees a screenshot of it, and taps, types, scrolls and switches apps to finish a task you describe in plain language. It is the "computer use" idea applied to Android, running on the phone itself.

What is a phone use agent?

A phone use agent is an AI that completes tasks on a smartphone by operating its user interface, the way a person does, instead of calling app-specific APIs. It looks at the screen, decides on an action (tap, type, scroll, open an app), checks the result and repeats until the task is done. OpenPhoneUse is an open-source phone use agent that runs on the Android phone itself.

Is it safe?

It is built to ask first. Payments and security changes always show a confirmation card you must tap; deleting asks unless you asked for that one recoverable item, and sending asks unless you named the recipient and the text yourself; the model never types passwords, and text from web pages or other apps can never trigger an unconfirmed action. Secrets are encrypted with the Android Keystore. It is still an agent acting on your real phone, so watch what it does on accounts that matter.

Does it need root, ADB, Shizuku or a PC?

No. It uses only Android's public AccessibilityService API, which you turn on yourself in Settings. (The benchmark runner uses adb from a PC to set up and grade tasks, but the app does not.)

Which AI models does it support?

• ChatGPT plan: sign in with ChatGPT and use your plan's usage (OpenAI's "Sign in with ChatGPT" for apps). The app lists the models your account offers, such as GPT-6.1 Terra, GPT-6.1 Sol and GPT-6 Astra.
• OpenAI API key: the same models, billed per token.
• Claude, Grok or Gemini: paste that provider's API key.

By default GPT-6.1 Terra makes the decisions with fast mode on, and a lighter helper model does repetitive screen work. You can change the model, reasoning effort and fast mode in the app. Fast mode is available with an OpenAI API key and a ChatGPT plan. It may be refused for some models or accounts, and the app then leaves it out.

How fast is it?

On the 31-task benchmark the median task took 42 seconds and the whole suite 45 minutes with GPT-6.1 Terra and fast mode. The quickest task, a fact lookup, took 9 seconds. Settings tasks took 27 to 156 seconds and research tasks 9 to 61 seconds. The longest, a game of 2048, took about 12 minutes.

Does it work in any app?

In almost any app. It reads the accessibility tree that most apps expose and also sees a screenshot, so drawn content such as games, maps and charts works too. The exception is apps that block screen capture, such as some banking, ticketing and payment apps: it refuses to operate them and tells you to do that part yourself. It also hands password entry back to you.

Can it run in the background while I use another app?

Not invisibly. Without shell privileges (ADB or root) Android gives an app no hidden second screen to work on, so OpenPhoneUse operates the screen in front of you, with an overlay that shows progress and a stop button. With split-screen, or on an unfolded foldable, you can keep using the other half while it works.

Does it send my screen to the cloud?

Only to the model provider you chose, and only while a task needs the screen: the compressed screen text and a screenshot go with each step. There is no OpenPhoneUse server, no analytics and no account with us.

What does it cost?

The app is free and MIT-licensed. Model usage is paid to your provider: signing in with ChatGPT uses your ChatGPT plan's usage, and an API key is billed per token. Fast mode uses OpenAI's priority tier, so it uses your plan faster or can cost more on the API.

Which Android versions and phones are supported?

Android 11 (API 30) or newer. It was developed and tested on a Samsung Galaxy Z Fold and on the Android emulator (API 36), and it has two-pane and Flex-mode layouts for foldables.

Does it use LangChain or another agent framework?

No. The agent loop, tools, screen reading, safety checks and model clients are written in Kotlin for this app. OpenAI, Grok and Gemini are called over HTTP; Claude uses Anthropic's Java SDK.

How do I add a tool?

Write one ToolModule class and add one line to ToolRegistry. See Contributing for the five steps.

Is it on the Play Store?

No. Install the APK from Releases or build it yourself.

Build from source

git clone https://github.com/joungminsung/openphoneuse.git
cd openphoneuse
./gradlew testDebugUnitTest   # JVM unit tests, no device needed
./gradlew assembleDebug       # APK: app/build/outputs/apk/debug/app-debug.apk

You need JDK 17 or newer and the Android SDK (Android Studio's bundled JBR works). The Android package id is com.phoneuse.

Contributing

Issues and pull requests are welcome. Read CONTRIBUTING.md first: safety rules are not optional, and every behavior change needs a test.

Add a tool module in 5 steps:

1. Create app/src/main/java/com/phoneuse/agent/<Name>Module.kt implementing ToolModule. Copy DeviceInfoModule.kt, the template.
2. Override offered or userText if needed (tools offered only sometimes; outputs that are the user's own data).
3. Add one line to ToolRegistry.modules, just above ConnectorModule.
4. Test it with ToolExecutor(..., registry = { ToolRegistry.modules(it) + MyModule(fake) }).
5. Update the expected names and hashes in ToolSpecsPinTest.

You can also add a benchmark task or a translation. App text is written as tr("English", "한국어"), so both languages sit side by side.

License

MIT. OpenPhoneUse is an independent project and is not affiliated with or endorsed by OpenAI, Anthropic, xAI, Google or Samsung. Use it with your own accounts and at your own risk.

Citation

If you use OpenPhoneUse in research or a product, please cite it (CITATION.cff):

@software{openphoneuse,
  author = {joungminsung},
  title  = {OpenPhoneUse: an open-source phone-use AI agent that runs on the Android phone itself},
  url    = {https://github.com/joungminsung/openphoneuse},
  year   = {2026},
  license = {MIT}
}