OpenLive

The open voice and vision layer for AI agents.

Your AI can think. OpenLive gives it ears, a mouth, and eyes.
Bring your own model, or talk to the coding agents you already use, with the whole
voice loop running on your own machine. An open alternative to ElevenLabs Agents,
Gemini Live, and OpenAI Realtime.

 

 

Demo

https://github.com/user-attachments/assets/065775b0-0a4a-4adf-8fa7-bcf065e6337f

What this is

Wiring an AI into a real conversation is harder than it looks: voice activity
detection, knowing when someone actually stopped talking, streaming speech-to-text,
the model turn, streaming text-to-speech, and barge-in so you can interrupt. Then
camera and screen on top. Hosted platforms rent you that pipeline by the minute and
run it on their cloud.

OpenLive is that pipeline, open and local. The listening, the speaking, and the
watching all run on-device (WebGPU). You bring the brain, and any brain works:

• A model you have a key for. Anthropic, OpenAI, Google, xAI, DeepSeek, Groq,
  Ollama (fully local), and a dozen more. No per-minute audio fees; you pay only
  the model costs you'd pay anyway.
• The coding agent you already use. Claude Code, Codex, Cursor, OpenCode, or
  Hermes, driven locally over the
  Agent Client Protocol (JSON-RPC over stdio),
  under your own login. Talk to your agent, watch it work, answer its permission
  asks by voice.

Whichever brain you pick, OpenLive is the same thing it has always been: the ears,
mouth, and eyes around it. Nothing you say leaves the machine. The only thing that
goes out is the final transcript (plus camera or screen frames if you turn them on),
to whatever brain you picked.

An honest note on architecture: OpenLive is a cascaded pipeline (speech to text to
model to speech), not a full-duplex speech-to-speech model like GPT-Live. That's a
real trade. A speech-native model can overlap talk and listen in ways a cascade
can't, but the cascade is exactly what makes "any brain, all local, no audio fees"
possible.

Features

The core, the ears / mouth / eyes:

• On-device voice loop. Silero VAD, Whisper STT, Smart-Turn end-of-turn, and
  your pick of two TTS engines: Kokoro (28 voices, light) or Supertonic (10 voices,
  44.1 kHz). All of it runs in the app on WebGPU.
• Speak as yourself. Settings → Clone Voice records 5 to 30 seconds of you
  (with a seekable listen-back before anything is saved) and your assistant speaks
  in your voice from then on. Zero-shot cloning (ZipVoice, Apache-2.0) running
  locally, an optional ~208 MB install, deletable anytime. Profiles preview with any
  text, rename, and export/import between machines. Clone only your own voice or
  one you have clear permission to use; impersonation is on you, not the tool.
• It can see. Camera or screen frames ride each turn, and the look tool grabs
  a crisp hi-res frame on demand. A text-only model can borrow a separate vision
  model's eyes.
• Barge-in. Interrupt any time and it stops mid-word, like a real conversation.
• Your assistant, your way. Custom instructions in Settings → General apply to
  every brain, built-in or agent. Speaking speed and spoken progress narration live
  there too.

The integrations that serve it:

• Voice-drive your coding agent. Pick Claude Code / Codex / Cursor / OpenCode /
  Hermes per conversation, pick its project folder, and talk. Model, mode
  (ask / accept edits / bypass), and the agent's other options switch mid-call, all
  reported by the agent itself over ACP.
• Sessions are the agent's own. A call with Claude Code lands in
  ~/.claude/projects/… where claude --resume finds it, and the agent's existing
  CLI sessions show up in OpenLive's History. Resume from either side.
• Permission relay. When the agent wants to run a command or edit files,
  OpenLive speaks the question; answer by voice ("yes" / "no") or tap.
• Narrated progress. Optional: while the agent works in silence, OpenLive speaks
  its plan steps out loud ("Step 2 of 4 — refactor the store").
• Live plans and costs. The agent's working plan renders as a checklist while it
  works, and a context/cost chip tracks the session.
• Manage agents in Settings. Install, sign in, update, and uninstall each
  agent's CLI from the app. Status updates itself while you finish a sign-in in the
  terminal, and if the terminal can't open you get the exact command to run instead.
• Floating mini mode. Shrink to an always-on-top pill that keeps listening while
  you work, with a menu-bar tray and notifications to close the loop.
• A transcript you can use. Agent replies render as markdown with copy buttons
  on code blocks, and the whole conversation exports to a Markdown file.
• Private by design. Audio never uploads. API keys are encrypted at rest
  (AES-256-GCM) and only the last four digits are ever shown.

Screenshots

Home	In a live call
Pre-call setup	Settings — agents
Clone your voice	Mini mode

Why on-device voice matters

The listening and speaking never leave your computer. The only thing that goes out
is the text turn to the brain you picked, the same call you would make from any app.
No audio uploads, no per-minute meter, no lock-in.

That also skips the separate speech-to-text, text-to-speech, and real-time-audio
fees hosted platforms charge on top. You still pay your normal model and vision API
costs, nothing more. With a coding agent as the brain there's nothing extra to pay
at all; it runs under the login you already have.

How it works

mic ─▶ VAD ─▶ streaming STT ─▶ end-of-turn ─▶ your AI ──────────▶ streaming TTS ─▶ speaker
     (Silero)  (Whisper)        (Smart-Turn)  (BYO model, or a     (Kokoro / Supertonic /
                                    ▲          coding agent over    your cloned voice)
                camera / screen ────┘          ACP on local stdio)
                frames (vision)

Everything outside "your AI" runs locally in the renderer. The turn goes over a warm
local WebSocket to a small agent server, which either streams a provider reply or
drives your coding agent's ACP adapter as a child process. The app starts speaking
sentence by sentence while the reply is still being written.

Get started

Just use it: grab the installer from the
latest release, open the app,
paste a model key (or pick the coding agent you already use — install/sign in from
Settings → Agents if needed), and start a call. The voice models download from
Hugging Face the first time you talk — roughly 200 MB with Kokoro, more with
Supertonic or a bigger Whisper — and are cached after that.

Build it from source:

pnpm install
pnpm desktop:dev      # runs the web + agent servers and opens the app window

You can also run it in a browser during development with pnpm dev, then open
localhost:3000. Run the tests with pnpm test.

Repo layout

apps/desktop     Electron shell: spawns the local servers, media perms, window,
                 mini mode, tray + notifications
apps/web         Next.js UI + the on-device voice engine (src/lib/live/*) + /api routes
                 (agents install/auth, history discovery, settings)
services/agent   Hono + ws: the /live WebSocket, the ACP agent driver (acp-agent.ts,
                 supervisor.ts), voice cloning (voice/*), the built-in provider turn loop
packages/shared  the agent registry (single source of agent identity), wire protocol,
                 shared types
packages/harness provider-neutral model adapters, live model listing, cost/effort
packages/db      JSON-file store: encrypted keys, settings, conversations

For how the pieces fit together — the ACP driver, the voice loop, resume, and the
delegate/worker tool flow — see docs/ARCHITECTURE.md.

Contributing

OpenLive is open to contributions. Start with CONTRIBUTING.md for
how to set up, where things live, and how to send a change. Good first issues are
labeled in the tracker.

License

MIT. Use it, change it, ship it.