Hermes Live Voice

Realtime speech models for Hermes Agent.

Hermes Agent | Architecture | Plugin | Client Protocol | Live Provider Testing

hermes-live-voice is an open-source Hermes Agent plugin and realtime voice gateway. It lets browsers, mobile apps, desktop clients, or terminal smoke tests talk to Hermes through realtime speech APIs.

Gemini Live and OpenAI Realtime handle the voice loop: speech input, speech output, turn-taking, interruption, and low-latency conversational flow. Hermes remains the brain: memory, tools, skills, terminal/file access, MCP, approvals, and long-running runs stay inside Hermes.

Client app
  -> WebSocket /v1/live
  -> hermes-live gateway
  -> Gemini Live or OpenAI Realtime
  -> gateway tool call: start_hermes_run()
  -> Hermes API Server /v1/runs
  -> Hermes tools, memory, skills, MCP

This repo is Hermes-centered only. It is not tied to any hosted product, app, billing system, or private assistant UX.

How Hermes Talks To You

The gateway gives the realtime model one job: be the live voice interface. It does not hand Gemini or OpenAI your full Hermes toolbelt.

1. Your client streams microphone audio or text to WS /v1/live.
2. hermes-live opens a Gemini Live or OpenAI Realtime session from the server.
3. The realtime model listens and speaks naturally.
4. When the user asks for real work, the model calls start_hermes_run.
5. The gateway starts a Hermes /v1/runs task, streams progress back, forwards approvals, and stops the run if the user interrupts.
6. The realtime model turns the Hermes result into a short spoken response.

That split is the product:

Realtime model = ears, mouth, turn-taking
Hermes         = brain, memory, tools, actions
hermes-live    = secure translator between them

Why It Exists

Hermes already has the hard agent layer. Realtime voice has a different runtime shape: persistent WebSockets, audio frames, provider sessions, fast barge-in, client auth, and gateway-level safety. Keeping that runtime in a plugin-managed gateway lets people add voice to Hermes without forking Hermes core.

Status

v0.1.0 is the first GitHub release line. It is intended for self-hosted Hermes installations and integration work.

Implemented:

• GET /health
• GET /ready
• GET /v1/capabilities
• WS /v1/live
• Gemini Live adapter
• OpenAI Realtime adapter
• Mock provider for local development
• Hermes /v1/runs client
• Hermes run event streaming over SSE
• Run stop/interruption bridge
• Approval response bridge
• Static browser demo
• Hermes plugin registration, discovery helper, status tool, and slash command

Install From GitHub

This project is not published to npm yet. Use the GitHub release/tag.

git clone https://github.com/bielcarpi/hermes-live-voice.git
cd hermes-live-voice
git checkout v0.1.0
npm install
npm run build
node dist/cli.js plugin install --symlink
hermes plugins enable hermes-live

Start the gateway:

HERMES_BASE_URL=http://127.0.0.1:8642 \
HERMES_AGENT_API_SERVER_KEY=... \
HERMES_LIVE_PROVIDER=mock \
npm run dev

Then open the browser demo:

http://127.0.0.1:8788

HERMES_LIVE_PROVIDER=mock is the zero-credential path for checking that the gateway, browser demo, terminal client, and Hermes run bridge are wired correctly. To hear real realtime speech, switch HERMES_LIVE_PROVIDER to gemini or openai and add the matching provider key below.

Or run the Docker example from a clone:

HERMES_AGENT_API_SERVER_KEY=... HERMES_LIVE_AUTH_TOKEN=$(openssl rand -hex 32) \
docker compose -f examples/docker-compose.yml up

Configure

Copy .env.example and set the pieces you need.

cp .env.example .env

Hermes API Server:

HERMES_BASE_URL=http://127.0.0.1:8642
HERMES_AGENT_API_SERVER_KEY=...

Set HERMES_AGENT_API_SERVER_KEY to the same value as Hermes Agent's API_SERVER_KEY. Current Hermes API Server deployments require bearer auth, and Hermes only accepts the long-term memory X-Hermes-Session-Key header from authenticated clients. hermes-live keeps that session key server-side and sends it to Hermes on run creation and follow-up run-scoped calls.

HERMES_API_KEY remains supported as a legacy alias, but new installs should use HERMES_AGENT_API_SERVER_KEY.

Gemini Live:

HERMES_LIVE_PROVIDER=gemini
GEMINI_API_KEY=...
GEMINI_MODEL=gemini-3.1-flash-live-preview

OpenAI Realtime:

HERMES_LIVE_PROVIDER=openai
OPENAI_API_KEY=...
OPENAI_REALTIME_MODEL=gpt-realtime-2
OPENAI_REALTIME_TURN_DETECTION=disabled

For current OpenAI Realtime 1.x behavior:

OPENAI_REALTIME_MODEL=gpt-realtime-1.5

Realtime 2 reasoning effort defaults to low. You can set OPENAI_REALTIME_REASONING_EFFORT to minimal, low, medium, high, or xhigh.

After setting real provider credentials, run an optional live session handshake:

npm run check:live-provider

This wraps the CLI command:

node dist/cli.js provider-smoke

Both commands open and close a Gemini Live or OpenAI Realtime session with the same adapter the gateway uses. They do not require Hermes to be running, send user audio/text, or start a Hermes run.

Local mock provider:

HERMES_LIVE_PROVIDER=mock
HERMES_AGENT_API_SERVER_KEY=...

Network-exposed gateway:

HERMES_LIVE_HOST=0.0.0.0
HERMES_LIVE_AUTH_TOKEN=$(openssl rand -hex 32)
HERMES_LIVE_ALLOW_ORIGIN=https://your-app.example

hermes-live refuses network-accessible binds without a strong HERMES_LIVE_AUTH_TOKEN unless you explicitly set HERMES_LIVE_ALLOW_UNAUTHENTICATED=true for an isolated trusted network.

Run

npm run dev

Then open:

http://127.0.0.1:8788

Useful commands:

npm run dev
node dist/cli.js client "What can Hermes do?"
node dist/cli.js provider-smoke
node dist/cli.js plugin install
node dist/cli.js plugin status
npm run check              # gateway, Hermes API, and provider readiness
npm run verify

The built-in browser demo is enabled by default for local development and disabled by default when NODE_ENV=production.

Client Protocol

Connect to:

ws://127.0.0.1:8788/v1/live

If HERMES_LIVE_AUTH_TOKEN is set, /v1/live, /ready, and /v1/capabilities require either:

Authorization: Bearer <token>

or, for browser WebSocket clients that cannot set upgrade headers:

/v1/live?token=<token>

First message:

{
  "type": "session.start",
  "profileId": "default",
  "userLabel": "alice"
}

Then send audio:

{
  "type": "audio.input",
  "data": "<base64 pcm16>",
  "mimeType": "audio/pcm;rate=24000"
}

Or send text for smoke testing:

{
  "type": "text.input",
  "text": "Summarize my current project state."
}

See docs/client-protocol.md.

Architecture

The TypeScript gateway uses a small ports-and-adapters layout:

domain/                  wire protocol and audio primitives
application/live-gateway gateway session orchestration and ports
adapters/inbound/http    HTTP, WebSocket, and static demo transport
adapters/outbound/hermes Hermes API Server HTTP/SSE client
adapters/outbound/realtime Gemini, OpenAI, and mock realtime providers

LiveGatewaySession depends on application ports, not raw WebSocket, provider SDK, or Hermes HTTP details.

See docs/architecture.md.

Hermes Plugin

The GitHub repo/package name is hermes-live-voice. The installed CLI command and Hermes plugin id are intentionally hermes-live.

The plugin gives Hermes a stable discovery/integration surface, registers a hermes_live_status tool, and adds a /hermes-live slash command for local gateway status. Install it with node dist/cli.js plugin install, then enable it with hermes plugins enable hermes-live.

See docs/plugin.md.

Security

Do not expose Hermes directly to untrusted mobile or browser clients. Expose hermes-live, require gateway auth, restrict origins, and keep provider and Hermes credentials server-side.

See docs/security.md.

Development

npm install
npm run verify

The test suite uses mock providers and fake Hermes clients. Live provider tests require external credentials and are intentionally not part of the default CI gate.

npm run check:gateway builds confidence in the packaged path by starting dist/cli.js serve, opening /v1/live, and driving a fake Hermes run over HTTP/SSE.

npm run check:live-provider and node dist/cli.js provider-smoke are optional and only run when you set HERMES_LIVE_PROVIDER=gemini or HERMES_LIVE_PROVIDER=openai with valid provider credentials.

Use docs/live-provider-testing.md before claiming a real Gemini Live or OpenAI Realtime deployment is ready.

References

• OpenAI Realtime overview: https://developers.openai.com/api/docs/guides/realtime
• OpenAI Realtime API reference: https://developers.openai.com/api/reference/resources/realtime
• OpenAI GPT-Realtime-2 model: https://developers.openai.com/api/docs/models/gpt-realtime-2
• Gemini Live API overview: https://ai.google.dev/gemini-api/docs/live-api
• Gemini Live API reference: https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/models/multimodal-live
• Google Gen AI JavaScript SDK: https://github.com/googleapis/js-genai
• Hermes Agent: https://github.com/NousResearch/hermes-agent