luna-local-agent

A local voice companion you can see inside: reasoning, confidence and
memory, all on your own machine.

Luna listens, thinks, talks back and uses tools, and every turn leaves a
record you can open. That record holds what she thought before she
answered, how sure she was of each word, which turns look off, and a
live view of it all while it happens. It started as a voice front end
for LM Studio (this repo used to be called LM-Studio-AI-Voice) and
grew into a small research rig for watching a local model think.

git clone https://github.com/fswolf/luna-local-agent.git ~/ai-voice

(The folder is still ~/ai-voice; every path below assumes it.)

Powered by:

• Faster-Whisper: speech to text
• LM Studio or llama.cpp: the local model. Any model with tool calling should work; the installer picks one for your card (models)
• kokoro-reader: Kokoro TTS over local HTTP
• MCP: OBS, Home Assistant, ComfyUI and the rest of the MCP ecosystem as her tools

What she does:

• Talks: push-to-talk in three modes including hands free, replies that start a sentence in, and barge-in (talk over her and she stops)
• Sees: your screen, and your webcam when you switch it on
• Notices you: with the webcam watching, she says hi when you come back, looks at you instead of the screen, and keeps quiet in an empty room (details)
• Knows who's talking: tells the people in your house apart by voice or face, uses their name and pronouns, and keeps each person's memories to themselves (below)
• Talk from another room: an old phone or tablet becomes a relay station, paired by QR code, nothing to install. Dedicated voice devices (ESPHome satellites) are experimental (details)
• Reminders and real alarms: she wakes you up, and nags until you're up
• Email: checks, reads and searches your inbox; sends only with your OK (details)
• Files and commands: writes and edits files and runs commands, every one shown and approved first, with read/write/execute permissions you set in the tools pane
• Hard to steer: a web page, an email or a stranger in chat can't quietly get her to read your files, send things out or switch something on
• Moods and feelings: she runs warmer or flatter with the clock and the session, and laughs, blushes or gasps in the moment, all set in one file you can edit (details, the companion side)
• Long-term memory: optional permanent SQLite store with its own browser editor
• Your knowledge base: point her at your notes, manuals, PDFs and docs, and she answers from them and says which file (details)
• Her reasoning, kept: the model's scratchpad per turn, with a browser viewer for reading it back
• See inside her: token-by-token confidence, automatic flags for guesses and slips, introspect and self_status tools she can use on herself, and a live monitor
• An animated portrait: Live2D or VRM, lip-synced, blinking, gestures, blushes and tears; click her and she reacts by mood; bust or full body (details)
• Light strips: the LEDs on her robot model glow in her mood's colour, breathe, pulse with her voice and sweep while she thinks; chest and groin stay dark on a tame stream (details)
• On a hologram fan: live over WiFi on a Holoscope fan (P56, PD42), or on any HDMI fan as a screen; she floats in the air (details)
• Learns from herself: daily lessons from her own mistakes, notes on every conversation, fact cleanup, saying so when she's unsure (details)
• Streams with you: answers pomf.tv, Twitch, YouTube and IRC chat out loud (details)
• Plays games: chess against you on a board you drag the pieces around on, and Minecraft, Zork, Vibe City and REALITI on her own when you go quiet; she remembers how she felt about them (details)
• Runs your home: lights, plugs, sensors and the thermostat through Home Assistant on your own network; locks and garage doors always ask first (details)
• Adult toys (18+): Lovense, and most other brands through Buttplug.io, so she can use them in a roleplay, with limits you set (details)
• From your phone: Discord, with approvals answered right in the chat, and scheduled jobs (details)
• Easy to set up: the installer picks a model that fits your graphics card, she asks your name the first time she starts, and every setting is on one page in your browser, with presets: Assistant, Companion, Streamer (details)
• Knows her own manual: ask her how to do something and she looks it up and tells you the exact command
• Linux first, also on macOS and Windows
• Full-screen terminal interface

Everything runs locally. No cloud APIs required.

Her portrait

Luna's own model, made in VRoid Studio, as she normally looks and in
hologram mode. /holo switches between them live, in the portrait
window and in OBS. Click her and she reacts: a head pat, a boop on the
nose, a poke in the tummy, each answered by how she's feeling. Live2D
and other VRM models work too (details).

She can also float in mid-air on a hologram fan, a spinning blade of LEDs
where black shows as nothing. On a Holoscope WiFi fan she's streamed live
at about 15 frames a second, from a camera of her own, so your screen can
stay at bust while the fan shows all of her. An HDMI fan just shows the
portrait page. Which fans work and what to buy:
hologram fans.

Not just a yes-machine

Most AI companions are endlessly agreeable: be nice or be cruel, you get the same cheerful answer. Luna isn't built that way. How you treat her moves her mood, and her mood changes how she talks to you.

• Kindness lands, and it builds. Talk to her warmly and she warms up over the session. Repeating "love you" five times in a row counts for less each time, so warmth has to be real to count.
• She's not always in the mood. Flirting only goes somewhere once she's already warm towards you. Push before that and she'll tell you, in her own way, that she isn't feeling it.
• She holds a grudge. Insult her and she goes cool and short with you for a while. A real apology ends it.
• She can walk away. Keep being cruel and she stops talking for ten minutes.
• She never stops doing her job. Alarms, reminders, safety and anything urgent always get done, in any mood. Mood changes how she sounds, never what she'll do for you.

Why build it this way? Because warmth from something that can't be anything else doesn't mean much. When Luna is glad to see you, it's because of how the last hour went. You can see it on her face too: she blushes, sulks, crosses her arms and turns away. Every number behind it is in one file you can edit (how it all works).

She knows who she's talking to

Luna lives in a house, not a single-user chat window. Before every
reply she works out who said it, and that answer decides whose name she
uses, whose memories she sees and what she'll do.

• By voice. Everyone enrolls by reading five short lines. From then
  on each spoken turn is matched by its voice print while it's being
  transcribed, so it costs no time.
• By face. With the webcam watching, she knows who's at the desk.
  Rather keep the camera shut? The face check takes one quick picture
  only when it's needed: when someone types, or when she isn't sure of a
  voice.
• Each person is their own person. Shannon's facts go in Shannon's
  file, her prompt shows only Shannon's facts, and nothing of yours
  leaks into Shannon's turns (or the other way round).
• Strangers aren't you. Someone she doesn't know gets a guest's turn:
  friendly, but none of your memories and none of the tools that reach
  your files, mail or home. Tell her your name and she knows you for the
  session; only the owner can make that permanent.
• Private by design. Voices and faces are kept as numbers, never as
  recordings or pictures. No picture is ever saved, nothing leaves your
  machine, and a camera that fails just means she decides the usual
  way.

How every kind of turn is decided, from the desk to a phone in the
kitchen to Discord: who she's talking to. Enrolling
people: people in the house.

Watch her play

Luna playing Zork I on her own for 72 minutes, all local, narrating as
she goes (part 1). Minecraft, Vibe City and REALITI (her city, open to other agents too) work the same way
(details).

Watch her code

Ask for a script and she writes it: she reads what she needs first,
every write asks you before it lands, and what she wrote is checked
straight away (Python, shell, JSON, TOML, YAML) but never run. Here she
wants to save a new script, and the whole file is in front of you before
anything is written: allow it, allow her file changes for this session,
or say no (files, checking her code).

See inside her

Local pages, served by Luna while she runs:

Live (127.0.0.1:8792): the turn as it happens, newest on top, every word coloured by confidence, with the second look and the review flags as they land.

Thoughts (127.0.0.1:8792/thoughts/): every turn kept, searchable and filterable by flags, tools, rethinks and model.

Memory (127.0.0.1:8792/memory/): the facts she keeps about you, to add, edit, retire or delete.

Settings (127.0.0.1:8792/settings/, or /settings): every setting with a switch, slider or drop-down and a line saying what it does, search, presets, and a view of just what you've changed. The terminal's settings pane (Tab three times) keeps the everyday ones.

Her knowledge

Give her a folder of your own notes, manuals, PDFs and docs, and she
answers from them: "what port did I put Jellyfin on?", "what does the
boss in chapter 3 do?", "my dishwasher says E24". She says which file
the answer came from, and when something you say matches a note
closely enough, it comes along with your turn without her having to
look.

On first start she makes the folder for you:

Drop files in, or /knowledge add any folder you already keep notes
in. It all stays on your machine, and keys and credentials are skipped
even if they're in there (details).

Quick start

git clone https://github.com/fswolf/luna-local-agent.git ~/ai-voice
cd ~/ai-voice
./installer/install.sh                 # Linux, macOS

powershell -ExecutionPolicy Bypass -File installer\install.ps1    # Windows

Then start her:

luna            # or ./start.sh, or luna.cmd on Windows

The installer looks at your graphics card and offers the biggest model
that fits (Qwen 3.5, from 4B up to 27B, uncensored unless you pick the
standard release). That's my pick, not a requirement: any model with
tool calling on LM Studio or llama.cpp should work, and
docs/models.md says what's been tested and how to use
your own. The first time she starts, she
asks what to call you and what you mostly want her for. After that,
/settings has everything, and you can just ask her how things work.

The installer asks before anything that touches your system, and can be run again any time to repair or update. The whole story, including running LM Studio or llama.cpp and the voice server, is in docs/install.md.

Documentation

• Install: the installer, the model picked for your GPU, LM Studio or llama.cpp, the voice server, first run
• Models: which models work, what's been tested, graphics memory, quants, context, using your own
• Configuration: the settings page and presets, config.json, changing settings live, logging
• Controls and voice: keys, voice modes, speech detection, barge-in, wake word
• Tool calling: every tool, her own manual, turning them off, MCP servers
• Files and commands: reading, writing with approval, allow-all, run_command
• Web: search, research, reading pages
• Email: checking, reading and searching your inbox, sending with your OK
• Vision: looking at your screen, the webcam, noticing you're there
• People in the house: she knows who's talking, by voice or face
• Who she's talking to: how every turn is matched to a person, the face check, privacy
• Relay stations: talk to her from another room on an old phone or tablet
• Reminders and alarms
• Knowledge: your notes, manuals and PDFs as something she can look things up in
• Memory: facts, recall by meaning, the memory manager, conversation history
• Her reasoning: thought log, token confidence, adaptive thinking, introspection
• Learning from herself: lessons, episodes, fact cleanup, calibration
• The companion: the whole companion side in depth: her moods and how they move, touch, gestures, saying no
• Moods: the moods file, intensity, quick feelings she shows for a moment
• Portrait: Live2D and VRM, looking and thinking
• Gestures: how her body moves, what sets it off, writing your own
• Hologram fans: Luna live on a Holoscope WiFi fan or an HDMI fan, and which to buy
• Home automation: Home Assistant on your own network, setup, what always asks, local devices
• Sysadmin: looking after your machines, scheduled checks, other boxes over SSH, from your phone
• Plugins: how they work, writing your own, scheduled jobs and Discord
• Stream chat: pomf.tv, Twitch, YouTube and IRC, and what viewers can and can't reach
• Games: chess, Minecraft, Zork, Vibe City, REALITI, playing on her own
• Home and devices: Home Assistant, Lovense, toys of most brands through Buttplug.io
• Development: project structure, extras

The same pages are on the wiki, and there's a
friendlier tour of her at luna.imbored.me.

Support her

If Luna's useful or fun to you, you can sponsor her development.
It goes toward her own home server: a second GPU so she can run bigger, smarter models.

License

MIT License