Jarvis

An always-on voice assistant for Windows. Say "Hey Jarvis," ask a question, get a
spoken answer — with a 36-tool agentic layer behind it that can search the web, run
code in a sandbox, read a calendar, watch a homelab, and see through a webcam.

Wake-word detection and speech-to-text run locally. Only the transcribed text ever
leaves the machine — with one opt-in exception, background agents,
which are off by default.

mic ──► openWakeWord ──► faster-whisper ──► Claude ──► edge-tts ──► speakers
        (local)          (local)           (agentic    (streamed)
                                            tool loop)

The console window and the optional ambient overlay — a translucent,
click-through HUD that floats over whatever you're working on. Both draw the
same arc reactor, whose rotation frames are pre-rendered and cached so the
animation costs an array index rather than a core.

Roughly 29k lines of Python across 70 modules. The full engineering log — the design
trade-offs, the post-mortems, and the conclusions that turned out to be wrong — is in
docs/MILESTONES.md; its
Start here section picks the six entries worth
reading if you only read six.

The parts worth reading

An agentic tool loop, not a chatbot. Claude is given 36 tools and decides which to
call, chaining them across iterations. Ask "what's the most played movie in my Plex
library?" and it composes four separate MCP calls to get there — there is no
hand-coded path for that question.

Latency is the design constraint. The reply is streamed from the model, chunked
into sentences, and synthesized to speech while the model is still generating, so the
first audible word lands about a second in rather than after the full response. The
system prompt is prompt-cached; per-turn context (speaker identity, current time) rides
a second, deliberately un-cached block, so a change there costs ~20 fresh tokens
instead of invalidating the whole cached prefix.

Least privilege, enforced in code. Not every caller gets every tool. A request
arriving from the phone client or the Discord bridge is served a restricted tool
surface — no shell, no filesystem, no code execution, no self-update. That boundary is
enforced at two independent server-side gates: the tool list is filtered before the
model ever sees it, and the executor refuses denied tools by name if one is somehow
reached. It is never enforced by asking the model nicely in the prompt. Mutating
actions are confirmation-gated, and the gate carries the information needed to evaluate
it — a confirmation you can't assess is theater, not safety.

Arbitrary code gets a boundary, not an allowlist. run_code executes
Claude-written Python in an ephemeral Podman container: no network, no host mount,
CPU/memory/PID caps, hard-killed at 30 seconds. You cannot allowlist arbitrary code, so
it is contained instead.

Everything fails soft. Any single subsystem — TTS, the Plex bridge, the GPU
transcription server, the calendar feed — can fail without taking down the listening
loop. Degrade and log; never crash the thing the user is talking to.

A regression gate, because a bug a test would have caught earns a test.
scripts/run_all_tests.py runs 62 gates — syntax, module wiring, a JS structural
check, and 59 test suites totalling ~1,500 assertions — and must be green before
anything ships. CI runs the same command on every push; the nine gates that need
a native ML toolchain or Windows SAPI are skipped by name, never silently
folded into the pass count.

Capabilities

Multilingual by design: the language is detected per turn, and the reply comes back in
that language, in a matching voice.

Stack

Python 3.12 · openWakeWord ·
faster-whisper · Anthropic SDK (streaming,
prompt caching, tool use) · edge-tts with a
pyttsx3 offline fallback · sounddevice · YOLOv8n · Resemblyzer · Podman ·
websockets · MCP

Windows-native rather than WSL, deliberately: WSL2 audio bridging is not reliable
enough for always-on real-time capture.

Quick start

git clone https://github.com/samonti86/jarvis.git && cd jarvis
python -m venv venv
.\venv\Scripts\Activate.ps1
pip install -r requirements.txt

Copy-Item .env.example .env    # then set ANTHROPIC_API_KEY
python scripts\doctor.py       # verifies deps, config, audio devices
python main.py                 # or: pythonw jarvis.pyw  (silent, no console)

Then say "Hey Jarvis."

doctor.py is read-only and tells you exactly what is missing — it is the fastest
answer to "why won't it start?". An Anthropic API key is the only hard
requirement; every other integration is optional, off by default, and degrades to
"not configured" rather than failing.

• docs/SETUP.md — full walkthrough, including how to obtain each API key
• docs/ENV_VARS.md — every setting, its default, and where it's read

Run the gate with python scripts/run_all_tests.py.

Layout

main.py                 composition root: load config, build subsystems, start threads
src/listen_loop.py      voice path — wake word, barge-in, follow-up/conversation modes
src/turn_runner.py      one turn end-to-end: history, streaming, tool loop, TTS
src/bootstrap.py        subsystem assembly (Plex, announcer, remote console, shutdown)
src/llm.py              Anthropic client, streaming, tool loop, per-origin tool boundary
src/wake_word.py        openWakeWord
src/speech_to_text.py   faster-whisper (local, or offloaded to a GPU host)
src/text_to_speech.py   edge-tts + fallback, sentence-chunked streaming
src/security.py         vision security mode (person detection, challenge/response)
src/*.py                the tool and subsystem modules (70 in total)
tests/*_test.py         the regression suites — everything here runs in the gate
scripts/                operational entry points + hand-run probes; never collected
docs/SETUP.md           deployment walkthrough
docs/MILESTONES.md      engineering log — index + "start here"
docs/milestones/        the log itself, 108 entries across 5 parts

License

Personal project, published as a portfolio piece.