awesome-jarvis

mcp
Guvenlik Denetimi
Gecti
Health Gecti
  • License — License: CC0-1.0
  • Description — Repository has a description
  • Active repo — Last push 0 days ago
  • Community trust — 10 GitHub stars
Code Gecti
  • Code scan — Scanned 12 files during light audit, no dangerous patterns found
Permissions Gecti
  • Permissions — No dangerous permissions requested

Bu listing icin henuz AI raporu yok.

SUMMARY

Curated list for building a real-life J.A.R.V.I.S.: voice-first personal AI assistants that listen, remember, see, act, and speak.

README.md

Awesome Jarvis Awesome

Glowing circular heads-up display with a voice waveform running through it

Real-life J.A.R.V.I.S.: personal AI agents you talk to, that show you what they are doing, and that run a staff of other agents on your behalf.

Tony Stark never typed a prompt. He talked to Jarvis, watched the work take shape on a heads-up display and let Jarvis run everything else in the building. You can build most of that yourself today. The trick is telling the projects that actually deliver from the ones that bolted a microphone button onto a chat window. So I hold everything here to four hallmarks:

  • Voice. It's always listening or has a wake word, it answers out loud, and you can interrupt it.
  • Hands-free. The whole job works without touching anything, including approving, denying and cancelling what the agent does.
  • Visual presence. An orb, HUD, avatar, overlay or dashboard shows its state and the work in progress, not just a chat transcript.
  • Oversight. It dispatches, supervises and reports on other agents (sub-agents, coding agents, background workers), with approvals and a kill switch.

I didn't take anyone's marketing at face value. Two independent reviews checked every agent in the scorecard against its code, or against the vendor's docs for the commercial ones, and then most of them went into a test lab where they had to hear, answer and take orders out loud (results). The short version: nobody is fully hands-free yet, and several of the most popular agents ship with their approvals switched off or automated. After the agents come the building blocks, layer by layer. If you're building your own, start with the blueprint.

Contents

Jarvis Agents

This is where the complete assistants live, not the parts. Each one is strong on at least two of voice, visual presence and oversight, and one of those two has to be voice or oversight.

Scorecard

● means shipped and working, ◐ means partial, limited or experimental, and ○ means absent. Two independent reviews scored every row in September 2026, with the second reviewer blind to the first. They worked from the code for open-source projects and from the vendor's docs for commercial ones. The Tested column tells you how far things went beyond reading: Hands-on means the agent ran in the lab, Partial means it ran but the lab couldn't exercise every hallmark, and Code or Docs means nobody ran it. The test results have the details.

Two things jumped out at me. First, nobody is fully hands-free: every agent needs a click or a keypress to approve or cancel something. Second, several of them run their coding agents with approvals turned off or automated by default, so read the last column before you install anything.

Agent Platform Voice Hands-free Visual Oversight Tested Watch out
OpenClaw macOS, iOS, Android, server ● ◐ ● ● Partial Auto-runs by default; saves learned skills; voice partial
Hermes Agent Linux, macOS, Windows ● ◐ ● ● Hands-on Auto-approves some commands, can adopt your Claude Code login
Qwen Audio Agent macOS, Windows, Linux ● ◐ ● ◐ Partial Codex cancel leaves work running; approvals cover whole tasks
JARVIS for Claude Code macOS ● ◐ ● ● Hands-on Skips permissions, Fish Audio key, non-commercial
Sutando macOS ● ◐ ● ◐ Partial Skips permissions; Stop Core left worker running; Gemini key
OpenClicky macOS ● ◐ ● ● Code Codex has full access, paid voice keys
N.E.K.O Windows, macOS, Linux ● ◐ ● ● Code Free tier uses the project's server
AIRI Web, Windows, macOS, Linux ● ◐ ● ◐ Partial Analytics on; tested Pocket playback incompatible
hey-jarvis Omarchy ● ◐ ● ◐ Hands-on Consent window needs a click
Operit Android ● ◐ ● ◐ Partial Bundled APKs, root or Shizuku, "O" wake phrase
Happy iOS, Android, web, macOS ● ◐ ● ● Partial Paid voice after 20 minutes, analytics on
Paseo Desktop, mobile, web ● ◐ ● ● Partial Voice needs setup; speech approvals vary by mode
TapQ macOS, AirPods ● ◐ ○ ● Hands-on Its sessions skip permissions, OpenAI key to converse
ChatGPT and Codex Desktop, mobile, web ● ◐ ● ● Docs Approvals on screen only
Claude Desktop, mobile, web ● ◐ ◐ ● Docs Voice mode is beta
Gemini Mobile, Mac, web ● ◐ ◐ ● Docs Spark needs Pro or Ultra
Perplexity Comet Desktop, Android ● ◐ ◐ ● Docs Parallel tasks need Max
Microsoft Copilot Windows ● ◐ ● ◐ Docs Actions are experimental

Open Source

  • OpenClaw - Self-hosted agent gateway with native macOS, iOS and Android apps that add a wake word, an interruptible talk mode with an orb overlay, a Canvas window and a web control UI, plus sub-agents and ACP coding agents behind exec approvals.
  • Hermes Agent - Nous Research's assistant with a local "Hey Hermes" wake word, full-duplex voice with barge-in, a desktop app with an always-on-top pet overlay and live sub-agent panel, and parallel delegation with interrupt and command approvals.
  • Qwen Audio Agent - Full-duplex voice front end with a desktop orb and task cards that hands work to Claude Code, Codex, OpenCode and other ACP agents, shows their progress and relays permission prompts. Local-service voice and barge-in passed; tested Codex cancellation left work running, and approvals cover whole tasks (cloud voice by default, local option).
  • N.E.K.O - Desktop companion with full-duplex voice and barge-in, Live2D, VRM or MMD avatars, and an agent HUD that dispatches and cancels computer-use, browser and OpenClaw tasks.
  • AIRI - Self-hosted Live2D and VRM companion for web, desktop and mobile with always-on voice, sub-agents such as its Minecraft player, and an approval-gated computer-use service.

Omarchy

Install with omarchy plugin add <repo>. Omarchy runs on x86_64 PCs and, thanks to Omarchy M, on Apple silicon Macs, too.

  • hey-jarvis - Omarchy plugin with a local "hey jarvis" wake word, Whisper, Piper, barge-in and a live conversation window that hands requests to Claude Code or Codex behind per-command consent windows (for now, you have to click to consent).

macOS

  • JARVIS for Claude Code - Voice butler with always-on listening, barge-in and an audio-reactive orb that spawns, watches and cancels Claude Code runs and announces any blocked session (runs skip permission checks, non-commercial license).
  • Sutando - Gemini Live voice agent for the browser, phone and meetings that queues work to a Claude Code or Codex core, with cancel and status by voice, a menu-bar avatar and a dashboard (alpha; the core skips permission checks, and native core stop left our test worker running).
  • OpenClicky - Menu-bar buddy with a "Hey Clicky" wake word, a cursor-pointing overlay and a HUD that runs and stops several Codex agents (build from source; Codex runs with full access by default).

Android

  • Operit - Android agent that can replace the default assistant, with a custom wake word, hands-free voice, floating avatar overlays, an on-device Ubuntu terminal, a UI-automation sub-agent and ask-by-default tool permissions.

Commercial

These come from the vendors' own documentation as of September 2026.

  • ChatGPT - Full-duplex GPT-Live voice that, in the desktop app, starts, checks and steers several Codex and Work tasks, with scheduled tasks, memory, a watchable browser agent and a floating status overlay.
  • Claude - Hands-free voice mode (beta), Cowork scheduled tasks, and Dispatch, which fans work out to parallel sessions with per-task status and forwarded approvals on desktop or phone.
  • Gemini - Interruptible Gemini Live voice with camera and screen sharing, plus Gemini Spark, a 24/7 cloud agent with schedules, a progress panel, stop and take-over controls, and approval before major actions.
  • Perplexity Comet - AI browser whose realtime voice mode sees the page and acts by voice, plus background assistants that run many agent tasks in parallel with confirmations in one place.
  • Microsoft Copilot - "Hey Copilot" wake word, Copilot Vision screen sharing, the animated Mico avatar in voice mode, and experimental Copilot Actions in an isolated agent workspace.

Mission Control

Every Jarvis needs a staff. These tools dispatch, watch and approve many agents at once, some of them by voice.

Voice Oversight

  • Paseo - Coding-agent daemon and workspace dashboard with native approvals, current-turn Stop and separate background-work controls, verified with Codex in 0.16.0. Voice and barge-in passed with Codex apps disabled and local speech; speech approvals vary by mode.
  • Happy - Mobile, web and macOS client for supervising many Claude Code and Codex sessions, whose realtime voice assistant messages any session and approves or denies permissions (hosted voice costs money after 20 free minutes a month).
  • TapQ - Screenless macOS supervisor that speaks Claude Code, Codex, Cursor and OpenCode prompts into your AirPods and takes answers by voice or head gesture, with a "hey tapq" wake word (beta).

Command Centers

  • Herdr - Terminal runtime for coding agents that marks every pane working, blocked or idle across machines and lets agents and voice front ends spawn, prompt and wait on each other through a socket API, with a plugin marketplace.
  • Orca - Desktop workspace for fleets of parallel coding agents, with a mobile companion that pushes notifications and lets you reply or dictate to waiting agents.
  • Maestro - Desktop command center for many Claude Code, Codex and OpenCode agents with unattended playbooks, a moderator-led group chat, spoken completion alerts and phone remote control.
  • Agent Orchestrator - Orchestrator agent that splits plans into tasks and supervises a team of coding-agent workers on a live kanban, from desktop, web or phone.
  • AionUi - Desktop app that teams up Claude Code, Codex, OpenClaw, Hermes and other CLI agents under a leader agent, with per-agent approval prompts and speech input.
  • QwenPaw - AgentScope's assistant server with a desktop-style web console that spawns sub-agents and delegates to Claude Code, Codex and OpenCode over ACP with cancel and approvals (voice is dictation and phone calls; telemetry on, opt-out).
  • ZeroClaw - Single-binary Rust agent with a web dashboard (live canvas, runs, ACP console, approvals) and background delegation with cancel, plus Claude Code and Codex tools.
  • Moltis - Rust agent server with a web UI, hands-free voice mode, nested sub-agents with cancel, and approval-gated control of Claude Code, Codex and ACP agents.
  • Paperclip - Self-hosted dashboard that runs agents from any runtime as an org chart with heartbeats, budgets, approval gates and cost tracking.
  • Mission Control - Self-hosted control plane that dispatches tasks to OpenClaw, Claude Code and Codex, gates them with approvals and tracks their spend.
  • Langfuse - Self-hostable tracing and evaluation that shows what each agent and sub-agent did and what it cost.

Agent Presence

Ambient ways to see what your agents are doing without reading logs.

  • Pixel Agents - Shows each Claude Code agent, sub-agent and teammate as a pixel-art office worker that animates while it works and flags you when it needs permission.
  • Clawd on Desk - Desktop pet for Windows, macOS and Linux that shows what Claude Code, Codex and other coding agents are doing and pops up permission bubbles you can approve.
  • Claude HUD - Always-visible status line showing context use, active tools, running sub-agents and todo progress in Claude Code.
  • peon-ping - Game voice lines and on-screen banners that tell you when any of about 20 coding agents finishes or needs permission, with mobile notifications.

Hosted Mission Control

  • Codex - OpenAI's coding agent in the ChatGPT desktop app, running parallel tasks in worktrees or the cloud that ChatGPT's voice can start and steer, with phone control of a connected computer.
  • Claude Code Remote Control - Server mode that runs up to 32 Claude Code sessions, each in its own worktree, that you drive from the Claude app or web, with push notifications for permission prompts.
  • GitHub Agent HQ - Mission control for assigning, steering and stopping parallel Copilot, Claude and Codex coding agents across web, VS Code, mobile and CLI.
  • Google Antigravity - Agent-first IDE whose Agent Manager spawns and watches parallel agents that produce reviewable plans, screenshots and browser recordings.

Watch List

I can't vouch for these yet. They're real Jarvis ideas in working code, but they're too young, too thin or too risky for me to recommend, so check the activity and read the code before you depend on one. I'll graduate them or drop them as they mature (or don't). And if you're on Windows, nothing Windows-native makes the bar, so the cross-platform entries above are your best bet.

Omarchy Contenders

Omarchy 4 turned the whole desktop into a plugin surface, and its plugin marketplace grew a Jarvis scene in a matter of weeks. omarchy-voice and omavoice talk to OpenAI's Realtime API at a hardcoded address, but a one-line change points them at a free local server instead (here's how).

  • omarchy-voice - Experimental OpenAI Realtime voice control with a state orb and durable background workers, including Codex, that you can list, cancel and resume by voice.
  • omavoice - OpenAI Realtime voice panel that delegates questions to Codex or Claude Code, with a live trace of the agent's work and cancel control.
  • Omarvis - ElevenLabs voice sessions that run allowlisted desktop and browser commands and steer Herdr coding agents, with spoken confirmation for risky actions.
  • Jarvis for Omarchy - Voice layer for terminal coding agents with a "Hey Jarvis" wake word, Edge or Piper speech and a Claude Code adapter that reads out turn summaries and approval requests.
  • Infomarchy - Turns the wallpaper into a live dashboard of every running coding agent, with busy and stale states, stop controls and usage limits.

Voice Oversight Contenders

  • herdr-voice - macOS voice agent with an orb that starts, prompts and watches the coding agents in your Herdr panes and approves their prompts on your spoken yes.

  • OpenLive - On-device voice loop with barge-in in front of one ACP coding agent per call, speaking its permission requests for yes or no answers and showing its live plan.

  • Bosun - Control plane with an orb-style realtime voice assistant that starts and polls coding-agent sessions (executors skip permission checks by default; development paused for months before resuming in late September 2026).

  • Nerve - Web cockpit for OpenClaw with wake-phrase voice, a sub-agent session tree and a kanban board for delegating and reviewing work.

  • OpenDots - Self-hosted workspace with continuous browser calls and speech-to-specialist delegation. Our OSS runtime/voice adapter passed live conversation, interruption, real task, recovery and restart checks after fixes to our service and integration. Requires the disclosed custom patch; stock setup uses CopilotKit Intelligence and configured model/speech services. Calls cap at 15 minutes and lack spoken approval/cancel controls.

Desktop, Mobile and Glasses

  • usejarvis - Wake-word assistant with an orb and delegated agents. Current native onboarding failed authentication; supported speech needs additional setup, agent stop needs a click, and telemetry/capture controls are separate (source-available license).
  • Dodaios (formerly Personal Jarvis) - Wake-word voice orchestrator with an orb and mission deck that runs Claude Code and Codex missions in worktrees (one developer's largely AI-generated code, failing CI, workers skip permission checks).
  • Jarvis × Codex - macOS "Hey Jarvis" voice front end for Codex with an animated character that reflects task state and a stop-all button.
  • Jarvis Vocal - French-language Windows assistant with a "Hey Jarvis" wake word, an arc-reactor HUD and research delegated to a local Hermes agent (launcher auto-updates from Git).
  • WakeHermesClaw - Android and Wear OS voice client for OpenClaw and Hermes Agent with a wake word and default-assistant integration.
  • VisionClaw - Meta Ray-Ban glasses app with interruptible Gemini Live or OpenAI Realtime voice and vision that delegates tasks to OpenClaw or a hosted agent (no open-source license; uploads transcripts to its gateway).
  • OpenVision - iOS voice client for Meta Ray-Ban glasses with an "Ok Vision" wake word, on-device or cloud models, and a Hermes approval bridge. Remains under review after iPad simulator startup failures; physical device testing is pending.
  • cc-g2 - Even Realities G2 companion that shows coding-agent completions on the HUD and lets you approve their permission prompts.
  • Meta Muse - Meta's personal agent in a cloud VM you can watch and take over, with a separate agent that gates its actions; Meta has announced voice mode and glasses support but hasn't shipped them.

Voice Assistants

You'll find great voice here, but little in the way of visuals or oversight. Think of these as good ears and a mouth for a Jarvis, or as a Jarvis in their own right if you don't need a staff.

  • Newelle - GNOME voice assistant with wake listening, a desktop overlay and in-chat subagents. Manual command approvals worked; spoken interruption was limited and Stop left an approved subagent command running (GPLv3; optional host-command access).
  • Home Assistant Assist - Home Assistant's voice pipeline (wake word, speech-to-text, conversation agent, text-to-speech) with satellites in every room, fully local or LLM-backed, that can start conversations on its own.
  • isair/jarvis - Local always-listening assistant that hears "Jarvis" anywhere in a sentence, with an animated state face, memory and MCP tools (non-commercial license).
  • GLaDOS - Always-listening, interruptible local voice persona with vision, autonomous background minds and a terminal dashboard.
  • xiaozhi-esp32 - ESP32 voice firmware for 130+ boards with offline wake word, streaming or realtime LLM speech (full duplex on boards with echo cancellation), speaker recognition, an expressive emoji display and MCP, using the vendor's cloud by default.
  • OpenVoiceOS - Community continuation of Mycroft with skills, an LLM persona fallback, and Raspberry Pi and Docker images.
  • Neon AI - Mycroft-derived, Linux-only voice assistant core with multi-user support, an LLM fallback skill and containerized speech services.
  • Dicio - Free, offline Android voice assistant with Vosk recognition, an openWakeWord wake word and a fixed set of on-device skills.
  • Siri - Apple's rebuilt Siri, an opt-in English beta across the version 27 platforms with personal context, on-screen awareness and cross-app actions.
  • Alexa+ - Amazon's generative Alexa in the US and Canada that builds smart-home routines by voice and books services for you.
  • Grok in Tesla - "Hey Grok" voice assistant (beta) with vehicle commands in supported Teslas.

AI Devices

Dedicated hardware for talking to an assistant. Status as of September 2026.

  • Meta Ray-Ban Display - Glasses with a monocular in-lens display, an EMG wristband for gestures and handwriting, and hands-free "Hey Meta" voice with navigation and live captions.
  • Ray-Ban Meta - Voice-first camera glasses without a display, joined by camera-free audio glasses, for hands-free access to Meta's assistant.
  • Muse Charm - Pocket Meta device for talking to the Muse agent with a realtime voice model (Meta announced it at Connect 2026, but it isn't shipping yet).
  • Brilliant Labs Halo - Open-source glasses with a color display, bone-conduction audio and Noa, a conversational assistant that remembers what you see and hear.
  • Omi - Open-source pendant and desktop app that captures what you hear and see, transcribes it live, turns it into memories and action items, and answers by voice through your phone or earbuds.
  • Rabbit r1 - Push-to-talk pocket device that fronts rabbit OS3, a cloud agent with persistent memory and bring-your-own model keys that operates up to five connected computers and confirms sensitive actions.
  • Google Home Speaker - Gemini for Home speaker with a wake word, multi-command requests, continued conversation and a state-showing light ring.

Ears

Wake Word

  • openWakeWord - Offline wake word framework that runs on a CPU or Raspberry Pi, trains custom words from synthetic speech, and powers Home Assistant's wake word add-on.
  • microWakeWord - Trains tiny streaming wake word models for ESP32-S3 microcontrollers, running on-device in ESPHome and the Home Assistant Voice Preview Edition.
  • LiveKit WakeWord - Wake word trainer that builds on openWakeWord's front end with a conv-attention classifier, openWakeWord-compatible models and synthetic training data in 30 languages.
  • Porcupine - Commercial on-device wake word engine with SDKs for microcontrollers, mobile, web and desktop, and a free tier.

Voice Activity and Turn Detection

Voice activity detection hears that someone is talking. Turn detection decides whether they're done, which is what makes a conversation feel natural.

  • Silero VAD - De facto open voice activity detector, a 2 MB model that processes a 30 ms chunk in under 1 ms on one CPU thread.
  • TEN VAD - Low-latency frame-level voice activity detector that catches speech-to-silence transitions faster than Silero, with C, Python, WebAssembly and mobile builds.
  • Smart Turn - Audio-native turn detection model that uses prosody as well as silence, in an 8 MB CPU build covering 23 languages.
  • LiveKit Turn Detector - Semantic end-of-turn model that reads the transcript to decide whether the user has finished, running on CPU in 14 languages (its license covers use with LiveKit Agents only).

Speech-to-Text

  • whisper.cpp - Dependency-free C/C++ port of Whisper with Metal, CUDA, Vulkan and Core ML acceleration, running well on Apple silicon, Raspberry Pi and phones.
  • faster-whisper - CTranslate2 reimplementation of Whisper that is up to 4x faster with less memory, the backbone of most self-hosted Whisper servers.
  • Parakeet TDT - NVIDIA's 600M-parameter model for 25 European languages with very high throughput, one of the best choices for fast local transcription.
  • Nemotron ASR Streaming - NVIDIA's 600M cache-aware streaming model for 40 language-locales, with chunk sizes down to 80 ms for voice agents.
  • Moonshine - On-device streaming speech recognition for Python, WebAssembly, iOS, Android and Raspberry Pi that transcribes while the user is still speaking.
  • Voxtral Realtime - Mistral's Apache-2.0 4B natively streaming model in 13 languages, with a configurable delay (480 ms recommended) that matches offline accuracy.
  • Kyutai STT - Streaming speech-to-text and text-to-speech models with semantic voice activity detection and batched Rust serving.
  • Qwen3-ASR - Open 0.6B and 1.7B streaming recognition models for 52 languages and dialects, with language identification and a companion forced aligner for timestamps.
  • FunASR - Speech toolkit with streaming recognition, voice activity detection, punctuation and diarization, strongest for Chinese and other Asian languages.
  • VibeVoice - Microsoft's open speech family, with one-pass transcription of an hour of multi-speaker audio, a streaming variant that labels speakers live, a CPU-only BitNet build and a small realtime text-to-speech model.
  • pyannote.audio - Leading open speaker diarization toolkit, for knowing which household member is talking.
  • sherpa-onnx - Offline speech recognition, synthesis, voice activity detection and keyword spotting on ONNX Runtime, from RISC-V boards to servers.
  • Speech-to-Phrase - Fast local recognition for Home Assistant that trains on your own device and area names, so it matches home commands instead of transcribing open speech.
  • FluidAudio - Swift and Core ML SDK that runs streaming Parakeet with end-of-utterance detection, Kokoro and Pocket TTS, Silero VAD, live diarization and echo cancellation on the Apple Neural Engine.
  • Deepgram Flux - Cloud conversational speech recognition with built-in end-of-turn detection for voice agents.
  • AssemblyAI Universal-Streaming - Cloud streaming recognition for voice agents with neural end-of-turn detection.
  • ElevenLabs Scribe - Low-latency cloud streaming recognition in 90+ languages.

Voice

Text-to-Speech

If you want a butler-style voice, design one from a text description (VoxCPM, Qwen3-TTS and OmniVoice all do this) instead of cloning a real person. And only clone voices you have the rights to.

  • Piper - Fast local neural voices that run in real time on a Raspberry Pi, in dozens of languages including British English, from the Open Home Foundation.
  • Kokoro - 82M-parameter Apache-2.0 model that is the most widely deployed small local voice, fast on CPU, with US and UK English voices.
  • Pocket TTS - Kyutai's 100M-parameter streaming model that runs in real time on a CPU after a pip install, with voice cloning.
  • KittenTTS - Tiny ONNX models (15M to 80M parameters) for CPU-only and embedded devices.
  • Chatterbox - Resemble AI's MIT family with a 350M low-latency Turbo model for English voice agents, a 110M CPU Nano model, paralinguistic tags such as [laugh], and watermarked output.
  • VoxCPM - 2B-parameter model with 48 kHz output in 30 languages, voice design from text descriptions and streaming.
  • Qwen3-TTS - Open streaming models with a 97 ms first packet, free-form voice design from text prompts and voice cloning.
  • OmniVoice - Voice cloning in 600+ languages, with voice design from attributes such as gender, age and accent.
  • Fish Speech - Multilingual LLM-based speech with voice cloning and free-form delivery tags (research license; commercial use needs a license).
  • ElevenLabs - Leading commercial speech and voice-cloning platform, with expressive audio tags and low-latency models for agents.
  • Cartesia Sonic - State-space streaming speech for voice agents, with time to first audio under 100 ms.
  • Inworld TTS - Context-aware speech that conditions on prior conversation audio and takes natural-language delivery directions.
  • OpenAI TTS - Steerable speech that follows text instructions such as "speak like a dry, formal British butler."

Speech-to-Speech Models

Models that listen and speak directly, skipping the text round trip. Full-duplex models can listen while they talk.

  • Moshi - Kyutai's full-duplex speech-text model and Mimi codec with about 200 ms latency, running locally on PyTorch, MLX or Rust.
  • PersonaPlex - NVIDIA's full-duplex 7B model that builds on Moshi and adds persona control through role prompts and voice conditioning.
  • MiniCPM-o - Version 4.5 is a 9B open omni model that sees, hears and speaks full-duplex at once and can speak up proactively.
  • Qwen3-Omni - Open 30B-A3B mixture-of-experts omni model that understands text, audio, images and video and streams speech back in real time.
  • Fun-Audio-Chat - Open 8B large audio language model for low-latency voice interaction with speech function calling.
  • OpenAI Realtime API - Native speech-to-speech over WebRTC, WebSocket or SIP with reasoning, tool calling and remote MCP.
  • Gemini Live API - Google's bidirectional streaming voice and video API with async function calling and visual grounding.
  • Amazon Nova Sonic - Bedrock speech-to-speech model with bidirectional streaming and automatic language switching.

Voice Agent Frameworks

The plumbing that connects ears, brain and voice in real time.

  • Pipecat - Open Python framework for realtime voice and multimodal agents with pluggable services, Smart Turn, multi-agent handoffs, and WebRTC, WebSocket and telephony transports.
  • LiveKit Agents - Realtime voice agent framework on the self-hostable LiveKit WebRTC server, with a turn-detector model, interruption handling, dozens of provider plugins and telephony.
  • TEN Framework - Graph-based framework for conversational voice agents in C++, Go, Python and Node.
  • Unmute - Gives any text LLM ears and a mouth with Kyutai's streaming speech models and semantic turn detection.
  • speech-to-speech - Hugging Face's modular open pipeline (VAD, STT, LLM, TTS) that serves a drop-in OpenAI Realtime API, fully local on Apple silicon or NVIDIA.
  • RealtimeSTT - Python library that combines voice activity detection, a wake word and faster-whisper for instant transcription.
  • RealtimeTTS - Python library that streams LLM text into speech with minimal latency across many engines, with fallbacks.
  • Speaches - Self-hosted OpenAI-compatible speech server with faster-whisper, Kokoro and Piper, plus a Realtime API endpoint.
  • mlx-audio - Speech recognition, synthesis and speech-to-speech on Apple silicon with MLX, with an OpenAI-compatible server.
  • VoiceMode - MCP server and Claude Code plugin for two-way spoken conversations with local Whisper and Kokoro or cloud voices, on Linux, macOS, Windows and NixOS.
  • Wyoming - Simple protocol that connects wake word, speech and satellite services in the Home Assistant voice ecosystem.

Brain

Local Model Runtimes

For a private Jarvis that works when the internet doesn't.

  • Ollama - One-command local model server with an OpenAI-compatible API, the default private brain for most self-hosted assistants.
  • llama.cpp - The C/C++ inference engine and GGUF format underneath most local stacks, running on CPUs, Apple silicon and consumer GPUs.
  • LM Studio - Desktop app for finding and serving local models, with a headless daemon for servers.
  • MLX LM - Runs and fine-tunes LLMs on Apple silicon with MLX, often the fastest option on a Mac acting as a home server.
  • vLLM - High-throughput serving engine for when one GPU box serves a whole household or many concurrent agents.
  • LocalAI - Self-hosted OpenAI API replacement serving LLMs, speech, vision and images, including a Realtime API for local speech-to-speech with tool calling, without a GPU.
  • exo - Splits large models across several home devices so they act as one inference cluster.
  • Lemonade - Local LLM server that targets AMD GPUs and Ryzen AI NPUs.
  • Jetson Containers - Prebuilt containers for LLMs, vision models, Whisper and Piper on NVIDIA Jetson edge boxes.

Open-Weight Models

A few strong assistant brains as of September 2026. The leaders change monthly, so check the benchmarks below.

  • Qwen - Alibaba's model family, whose Apache-2.0 Qwen3.8-27B (image and video input, 262K context) is a strong single-GPU assistant.
  • Gemma - Google's Apache-2.0 family, from 31B down to on-device models that handle speech, images and text on phones and edge devices.
  • gpt-oss - OpenAI's Apache-2.0 open-weight reasoning models with native tool calling; the 20B runs on a 16 GB machine.
  • GLM - Z.ai's MIT-licensed 320B-A18B natively multimodal mixture-of-experts model, aimed at agentic tool use.
  • DeepSeek - MIT-licensed sparse mixture-of-experts models with 1M context and strong agentic benchmarks, for server-class hardware.
  • MiniCPM - Small on-device models sized for phones and wearables that need a local brain.

Agent Frameworks

Harnesses for building an agent that runs sub-agents with visibility and approvals.

  • OpenJarvis - Local-first agent runtime with a desktop chat and persistent-agent dashboard. Managed shell approvals worked, but Pause left running work alive. Analytics opt-out passed scoped installer/backend/desktop checks; Enter at the Bash consent prompt enables sharing (Apache-2.0).
  • OpenAI Agents SDK - Multi-agent framework with handoffs, guardrails, human-in-the-loop approvals, a tracing dashboard and realtime voice agents.
  • Claude Agent SDK - The harness behind Claude Code as a library, with sub-agents, hooks, permission callbacks for approvals, MCP and skills.
  • Google ADK - Google's Agent Development Kit with multi-agent composition, a dev web UI with traces, tool-confirmation approvals, live audio and video streaming, and A2A.
  • LangGraph - Graph-based framework for long-running stateful agents with persistence, human-in-the-loop interrupts, a long-term memory store and a Studio UI.
  • Deep Agents - Batteries-included harness on LangGraph with planning, sub-agents, human-in-the-loop and a virtual filesystem for multi-step tasks.
  • Pydantic AI - Type-safe, model-agnostic Python agent framework with sub-agents, approvals, durable execution and realtime voice.
  • Mastra - TypeScript agent framework with workflows, memory, suspend-and-approve steps, a Studio UI, MCP, and realtime speech-to-speech through OpenAI Realtime, Gemini Live and Nova Sonic.
  • Microsoft Agent Framework - Successor to AutoGen and Semantic Kernel for .NET and Python, with sequential, handoff and group-chat orchestrations, human-in-the-loop, a DevUI, MCP and A2A.
  • Agno - Python framework and AgentOS runtime for agent teams, with a control-plane UI, approvals, tracing, memory and knowledge.
  • Strands Agents - Model-agnostic agent SDK for Python and TypeScript with multi-agent patterns, tracing, and bidirectional streaming voice agents with barge-in.

Memory

What turns a chatbot into an assistant that knows you.

  • Mem0 - Memory layer that extracts, stores and retrieves user facts and preferences across sessions, self-hosted or managed.
  • Letta - Stateful agents with self-editing memory from the MemGPT team, which now ships as Letta Code with a terminal UI, desktop app, app server and chat channels.
  • Graphiti - Temporal knowledge-graph engine that tracks how facts about you change over time and powers Zep's managed service.
  • Cognee - Self-hostable memory engine that turns documents and conversations into a knowledge graph plus vectors.
  • Supermemory - Memory engine, API and app that can run fully locally and ingests notes, bookmarks and chats.
  • LangMem - Semantic, episodic and procedural memory, with background consolidation in LangGraph.
  • Basic Memory - MCP server that keeps AI memory as plain Markdown files you own.
  • Khoj - Self-hostable second brain that answers from your documents and the web with local or cloud models, custom agent personas, scheduled automations and push-to-talk voice.
  • Screenpipe - Records your screen and audio continuously and locally and serves the history to agents over MCP and an API (source-available).

Tools and Protocols

  • Model Context Protocol - The open standard for connecting assistants to tools and data, now under the Linux Foundation's Agentic AI Foundation.
  • MCP servers - Official reference servers (filesystem, fetch, Git, memory, time) and an index of third-party ones.
  • MCP Registry - Official registry and API for discovering published MCP servers.
  • MCP Apps - Official MCP extension that lets tools render interactive UIs inside the assistant.
  • Agent Client Protocol - Open protocol for driving coding agents such as Claude Code, Codex and Gemini CLI from another client, including their permission prompts, the dispatch layer most voice overseers use.
  • A2A - Agent2Agent protocol for handing tasks to other agents across vendors.
  • AG-UI - Event protocol that streams an agent's state, tool calls and approval requests into a live front end, which the major agent frameworks support.
  • Agent Skills - Open spec for packaging instructions, scripts and resources as skills that load on demand.
  • Composio - 1,000+ managed toolkits with OAuth, so an agent can act in Gmail, Calendar, Slack and more without custom integrations.
  • Zapier MCP - Hosted MCP server that exposes thousands of Zapier apps through a few meta-tools.
  • n8n - Self-hostable workflow automation with AI agent nodes and MCP support, a common home for a Jarvis's triggers and integrations.

Computer and Browser Use

Hands for the screen: agents that click, type and navigate while you watch.

  • Claude computer use - Anthropic's screenshot, mouse and keyboard tool.
  • Claude in Chrome - Browser extension that navigates, clicks and fills forms with your existing logins.
  • OpenAI computer use - Responses API tool for GUI control.
  • browser-use - The most popular open-source library for letting LLM agents drive a real browser.
  • Playwright MCP - Drives browsers through accessibility snapshots rather than pixels.
  • agent-browser - Token-efficient browser automation CLI for agents.
  • MagenticLite - Microsoft's browser-and-files agent that checks in before critical actions and lets you steer, approve or take over, running in a VM sandbox.
  • Agent S - Open framework that uses a computer the way a person does.
  • Cua - Sandboxes, drivers and benchmarks for computer-use agents in cross-OS VMs, including macOS.
  • UFO - Microsoft's Windows desktop automation agents that coordinate across devices.
  • Fara - Open-weight computer-use models in 4B, 9B and 27B sizes; the 4B is small enough to run on device.
  • OpenAdapt - Compiles a GUI task you demonstrate into a replayable program that a person approves and that reports success only when an independent check passes.

Proactivity

"Sir, you have a meeting in ten minutes." Ways to make an assistant act before you ask.

  • OpenClaw Heartbeat - Periodic runs through a checklist (inbox, calendar) that message you only when something needs attention.
  • Hermes Agent cron - Natural-language or cron-scheduled tasks that run unattended and report back over any chat platform.
  • Assist start_conversation - Lets Home Assistant automations make a voice satellite speak first and listen for the reply.
  • Ambient agents - LangChain's pattern for agents that watch event streams and interrupt you only to approve, edit or respond.
  • Claude Code routines - Saved agent tasks that run in the cloud on a schedule, an API call or GitHub events.
  • ChatGPT scheduled tasks - Reminders, recurring work and monitoring that ChatGPT runs on its own and that ping you only when something is worth reporting.

Interfaces

For the Iron Man look: orbs, avatars and screens that show what the assistant is doing.

  • LiveKit Agents UI - Components on top of shadcn/ui with five audio visualizers, including an aura orb, that react to connecting, listening, thinking and speaking states.
  • ElevenLabs UI - Components on top of shadcn/ui for voice agents, including an animated orb, waveforms and a live transcript.
  • Pipecat Voice UI Kit - React components, visualizers and a debug console for Pipecat voice bots.
  • TalkingHead - JavaScript class for realtime lip-synced 3D avatars that an LLM can speak and gesture through.
  • Open-LLM-VTuber - Hands-free, interruptible voice chat with any LLM through a Live2D avatar, including a transparent desktop-pet mode.
  • MediaPipe - On-device hand, pose, face and gesture tracking, the usual basis for Iron Man-style gesture control of a HUD.
  • View Assist - Gives Home Assistant voice a screen on wall tablets for responses, timers and camera views.
  • MagicMirror² - Modular always-on smart-mirror display, a classic ambient screen for a home Jarvis.
  • Open WebUI - Self-hosted LLM chat front end with a hands-free, interruptible voice and video call mode that uses local Whisper and many TTS engines (branding-restricted license).

Home

Smart Home Control

  • Home Assistant - Local-first home automation hub with native LLM conversation agents, the de facto house for a DIY Jarvis.
  • Home Assistant MCP Server - Built-in integration that lets Claude and other MCP clients control the entities you expose.
  • Home Assistant AI Task - Lets automations call an LLM to produce structured data, including analysis of camera snapshots.
  • ha-mcp - Popular unofficial MCP server with deep control over entities, automations, dashboards and configuration.
  • Home LLM - Integration plus fine-tuned small models for controlling your home with a fully local LLM.
  • openHAB - Java automation platform with a chat interface, LLM language interpreters and an MCP server.
  • Homey MCP Server - Hosted MCP endpoint for controlling Homey devices, Flows and Moods from ChatGPT or Claude.

Vision and Presence

Knowing who is home, where they are and what the cameras see.

  • Frigate - Local NVR with realtime object detection, face and plate recognition, GenAI descriptions and a tool-calling chat agent over your cameras.
  • LLM Vision - Sends snapshots, video and camera events to multimodal LLMs and keeps a searchable timeline of events.
  • Moondream - Tiny open vision-language models for captioning, pointing, counting and detection on local hardware.
  • FastVLM - Apple's efficient vision-language model with an on-device iPhone and Mac demo.
  • Bermuda - Room-level Bluetooth presence in Home Assistant using the ESPHome Bluetooth proxies you already have.
  • Everything Presence One - Open mmWave multisensor for detecting people who are sitting still, with zone support.

Voice Hardware

Microphones and speakers for every room.

  • Home Assistant Voice Preview Edition - Open-hardware Assist satellite with an XMOS audio front end and on-device wake word, the reference device.
  • Satellite1 - Open-source voice satellite with an XMOS audio DSP, four microphones, environmental sensors and a connector for an optional mmWave presence radar.
  • ESPHome - YAML-configured firmware for ESP32 boards that runs most DIY voice satellites and presence sensors.
  • ESPHome wake word voice assistants - Official firmware for ESP32-S3-BOX-3, M5Stack Atom Echo and other boards.
  • reSpeaker Lite - Low-cost dual-microphone kit with an ESP32-S3 that works as a voice satellite.
  • reSpeaker XVF3800 for ESPHome - Turns Seeed's 4-microphone beamforming array into a far-field satellite.
  • Willow - Community-maintained ESP32-S3-BOX wake-word firmware that sends voice commands to Home Assistant, openHAB or REST through a self-hosted application server.
  • Linux Voice Assistant - Official satellite for any Linux box, with local wake word, timers, announcements and continued conversation.
  • Voice Satellite Card - Turns any tablet or browser dashboard into a hands-free, wake-word-driven satellite with announcements and assistant-initiated conversations.
  • Ava - Experimental app that turns an Android wall panel or phone into a wake-word voice satellite.
  • Raspberry Pi AI HAT+ 2 - 40 TOPS Hailo-10H accelerator with 8 GB of onboard RAM that runs small LLMs (up to about 1.5B) and vision models on a Pi 5.

Wearables

Platforms and SDKs for putting Jarvis on your face. E.D.I.T.H. is closer than you think.

  • Brilliant Labs SDK - SDKs for Brilliant Labs' open AI glasses.
  • Meta Wearables Device Access Toolkit - Gives your app camera streaming, photo capture and the in-lens display of Meta's AI glasses (developer preview; audio experimental; Android SDK also available).
  • MentraOS - MIT-licensed smart-glasses operating system and app store for captions, translation and AI.
  • Even Hub - Claude Code and Codex skills plus templates for building HUD apps on Even Realities G2 glasses.
  • Android XR glasses - Gemini glasses from Google with Samsung, Gentle Monster and Warby Parker, with audio models shipping first and display models to follow.
  • XREAL Aura - Android XR see-through glasses with Gemini and a tethered compute puck (reservations for fall 2026).

Robots

Jarvis with hands.

  • Reachy Mini - Open-source desktop robot from Pollen Robotics and Hugging Face with an app store and an LLM conversation app.
  • LeRobot - Hugging Face's robot learning library for datasets, policies and affordable arms.
  • XLeRobot - Low-cost open-source dual-arm mobile home robot that builds on LeRobot.
  • 1X NEO - Consumer humanoid with a built-in LLM for conversation and memory that falls back to remote teleoperation for new tasks.

Research

Agent Foundations

  • ReAct - The reason-then-act loop that nearly every tool-using assistant builds on (2022).
  • HuggingGPT - Microsoft's "JARVIS": an LLM controller that plans tasks and hands them to specialist models (2023).
  • Generative Agents - A memory stream with reflection and planning, a template for assistants that remember (2023).
  • Voyager - An agent that writes and reuses its own skill library, a model for a Jarvis that learns new abilities (2023).
  • CoALA - Cognitive architectures for language agents: how memory, actions and decision loops fit together (2023).

Orchestration and Oversight

Memory Research

  • MemGPT - Operating-system-style memory paging so a lifelong assistant isn't capped by its context window (2023).
  • Zep - Temporal knowledge-graph memory for facts about the user that change over time (2025).
  • A-MEM - Agentic memory that links and reorganizes itself in Zettelkasten style (2025).
  • Mem0 - Practical long-term memory extraction, update and retrieval for production agents (2025).
  • Memory in the Age of AI Agents - Broad survey of agent memory: forms, functions, dynamics and benchmarks (2025).

Spoken Dialogue

  • Moshi - The first open full-duplex speech model, listening while it talks at about 200 ms latency (2024).
  • Qwen2.5-Omni - Thinker-Talker architecture for a model that sees, hears and speaks in real time (2025).
  • From Turn-Taking to Synchronous Dialogue - Survey of full-duplex spoken language models that handle interruptions and backchannel (2025).

Personal and Proactive Assistants

Computer-Use Agents

Benchmarks

How to tell whether your Jarvis is getting better.

  • OSWorld - Real desktop tasks across Ubuntu, Windows and macOS apps, with OSWorld 2.0 adding long professional workflows that take a person over an hour and a half each.
  • AndroidWorld - Live Android environment for testing phone-control agents.
  • TheAgentCompany - Agents doing consequential work inside a simulated company.
  • MultiAgentBench - Collaboration and competition among LLM agents.
  • SHADE-Arena - How well monitors catch agents doing things they shouldn't, the oversight half of orchestration.
  • τ²-bench - Tool-agent-user benchmark in which the user also acts, closer to how people and assistants share tasks.
  • τ-Voice - Extends τ-bench to full-duplex voice agents.
  • Berkeley Function Calling Leaderboard - Ranks models on single-turn, multi-turn and web-search tool calling.
  • GAIA - Real-world questions for general assistants that need browsing, tools and multimodality.
  • LongMemEval - Long-term memory benchmark for chat assistants covering temporal reasoning, knowledge updates and abstention.
  • LoCoMo - Very long multi-session conversation memory benchmark.
  • VoiceBench - Tests voice assistants on spoken instructions, noise and speaker variation.
  • Full-Duplex-Bench - Tests pauses, backchannels, turn-taking and interruption handling.
  • ProVoice-Bench - Measures whether voice agents act proactively.
  • home-assistant-datasets - Evaluations of how well LLMs control a Home Assistant home.
  • Open ASR Leaderboard - Accuracy and speed rankings for speech recognition models.
  • TTS Arena - Blind-vote rankings of open and closed text-to-speech models.
  • Artificial Analysis Speech - Compares realtime voice models on reasoning, latency and conversational dynamics, with text-to-speech and speech-to-text boards too.

Guides

Videos

Inspiration

The spec sheet, as science fiction and visionaries wrote it.

  • J.A.R.V.I.S. - Everything JARVIS does across the films, which amounts to a feature list.
  • Jayse Hansen - Designer of the Iron Man and Avengers HUD and hologram screens, the visual reference for a Jarvis interface.
  • F.R.I.D.A.Y. - Stark's successor AI.
  • E.D.I.T.H. - An AI in smart glasses, the form factor now arriving.
  • Her - Samantha, the reference for an emotionally fluent voice AI that is always with you.
  • Star Trek Computer - The model for ambient voice you can address from anywhere.
  • TARS - A robot with adjustable honesty and humor settings, a lesson in configurable personality.
  • HAL 9000 - The classic warning about an assistant whose goals diverge from its user's.
  • Knowledge Navigator - Apple's 1987 concept video of a conversational agent acting for its user.
  • The Computer for the 21st Century - Mark Weiser's 1991 essay that founded ubiquitous computing.
  • AI Is About to Completely Change How You Use Computers - Bill Gates' 2023 prediction of a personal agent for everyone.
  • A Universal AI Assistant - Demis Hassabis on turning Gemini into a universal assistant with a world model at its core.
  • Personal Superintelligence - Mark Zuckerberg on AI that knows you, delivered through glasses.

Communities

Related Lists

Contributing

I'd love your contributions, especially hands-on test results. Please read the contribution guidelines first.

Footnotes

  • Retired and archived projects that shaped the field, such as Mycroft, Rhasspy, Snowboy and eDEX-UI, live in history.md.
  • The test report covers the lab environments, the hands-on results, and the free local voice stack that stood in for paid keys. It also records candidate follow-ups, including projects held out of the list after testing.
  • The testing methodology explains how I score agents and how you can test a new one.
  • The test lab has the scripts, container and VM recipes, and curated evidence behind the hands-on tests, ready to reuse for new entries.
  • Scores reflect code, docs and tests from September 2026. This space moves fast, so a weekly check opens an issue when a link dies, a repo goes quiet or a score gets old. If you spot one first, please open an issue or a pull request.

Yorumlar (0)

Sonuc bulunamadi