Rudy OS

Agent harness that runs a whole office of agents

rudydoes.com · Download for macOS

Free, open source and performant — a multi-agent harness that works with the
subscriptions you already pay for, on their hourly limits. It turns the terminal coding CLI
you already run into an always-on teammate, one that keeps working while you're away and
coordinates a whole office of agents on your own machine.

Wraps Claude Code, Antigravity (Gemini), OpenAI Codex,
xAI Grok, Kimi Code, Gemini CLI, Qwen, OpenCode, Crush,
pi.dev, GitHub Copilot CLI, and Cursor — with bring-your-own keys and local LLMs.
Agents that message, route, and remember, coordinated by Rudy and
visualized as avatars at work on a shared office floor.

Electron · React · TypeScript · Pixi.js · xterm.js · node-pty

[!NOTE]
The world's best agents. All of them yours.
Rudy OS takes the terminal-agent CLIs you already run — claude, agy, codex, grok,
kimi, qwen, opencode, crush, pi, and copilot — and turns them
into a self-coordinating team: each agent gets long-term memory, a mailbox, and a desk on a 2D
office floor — and Rudy routes work between them while you watch. He's the
boss of the floor; you're still the boss of him.

What it is
How it works
Features
Getting started
Architecture
Project structure
Design system
Roadmap
Contributing
Telemetry
License
Acknowledgements

What it is

Rudy OS is a desktop app that wraps real terminal-agent CLIs as fully-capable agents,
wires them into a hive mind, and puts Rudy in charge — the one agent you
talk to in order to get things done. Under the hood it runs the fastest memory layer in the
world so every agent remembers what it learns and recalls it instantly.

Every terminal is an agent. Each claude, agy, codex, grok, kimi, qwen, opencode, crush, pi, copilot, or custom session runs as a real
process in a pseudo-terminal (node-pty), byte-for-byte authentic, rendered with xterm.js.
Every agent is an avatar. Sessions appear as characters on a Pixi.js office floor — they walk
to stations as they work, and envelopes fly desk-to-desk when they message each other.
The hive coordinates them. Agents read their memory and drain a mailbox; the router moves
messages between inboxes; Rudy, the boss agent, adjudicates, assigns, and escalates only when it needs you.
Memory that's instant. A markdown-first memory layer with a semantic recall index means agents
remember across sessions and recall in milliseconds.

How it works

            you ── talk to ──►  ┌─────────────┐
                                │    Rudy     │  the boss: orchestrator
                                │  (his desk) │  roster · routing · adjudication
                                │             │  blackboard · task ledger
                                └──────┬──────┘
                                       │ assigns · routes · escalates
              ┌────────────────────────┼────────────────────────┐
              ▼                         ▼                         ▼
        ┌───────────┐            ┌───────────┐            ┌───────────┐
        │  agent A  │  message   │  agent B  │  message   │  agent C  │
        │ provider  │ ─────────► │ provider  │ ─────────► │ provider  │
        │  + memory │            │  + memory │            │  + memory │
        └───────────┘            └───────────┘            └───────────┘
              └──────── shared hive: memory · mailbox · blackboard · log ───────┘

You spawn agents — each is a normal terminal process (claude, agy, codex, or custom)
with its own working directory, identity, and provider-specific lifecycle.
Agents collaborate through the hive — a local git repo of plain files. They write to their own
outbox/; the harness's router delivers into recipients' inbox/. No agent ever touches git
(single-committer design avoids index.lock corruption).
Rudy runs the floor — he reads every request, resolves routine ones himself (keeping
the system fully autonomous), and only escalates critical items (spend, destructive ops, scope
changes) into an approvals queue you act on.
Everything is visible — you watch avatars move, envelopes fly, and the live terminal stream;
you can type back into any session, browse its files, and read its git history.

See HIVE.md for the full multi-agent design, SPEC.md for the
terminal/event plane, and DESIGN.md for the visual system.

Features

The floor

Every terminal is a real agent. Claude Code, Antigravity (Gemini), OpenAI Codex, xAI Grok, Kimi Code, Gemini CLI, Qwen, OpenCode, Crush, pi.dev, GitHub Copilot CLI, Cursor, or a custom command — each in its own node-pty PTY, rendered with xterm.js.
Every agent is an avatar. A Pixi.js office floor where agents walk to stations, envelopes fly desk to desk, and avatar state reflects real work.
A boss you talk to. Rudy routes tasks, adjudicates traffic, and escalates only what needs a human.
Per-agent git worktrees. Optional isolation so parallel agents never collide on branches.

Memory & coordination

The hive — per-agent memory, atomic-file mailboxes, a shared blackboard, an append-only event log, single-committer git.
Semantic recall — markdown memory mined into a shared palace, searchable from the UI, with condensation so it doesn't grow forever.
Enterprise Knowledge Graph — your own documents and policies, queryable by any agent.

Control & safety

Human gates — spend, scope, and destructive ops escalate to you. Steer mid-run or stop gracefully.
Circuit breaker — a steer → constrain → stop ladder for agents that loop, storm errors, or blow their budget.
Budgets & telemetry — per-agent token budgets, real cost from transcripts, a durable ledger, OTel spans, and a tool waterfall.

Command Center

Kanban tasks with dependencies, scheduled missions + heartbeat, live fleet monitoring, memory search, activity log, and a CI watcher.
Skills — what every agent can already do across Claude Code, OpenCode and Codex, plus a browsable catalog of 227 more with search, filters, install and uninstall.
Built-in Monaco IDE — file tree, editor tabs, save, plus CHANGES · HISTORY · COMPARE git rails with commit graph, diffs, branch compare, and guarded checkout. All fs/git access brokered through main.

Getting work in and out

Slack & webhooks — message a channel or POST a webhook; Rudy can spawn an ephemeral worker, reply in-thread, and tear it down.
Shareable hires — import a role from a rudyos://hire link or a .json manifest; import only pre-fills the form, a human still spawns it.
BYOK keys + local LLMs — per-provider keys in a write-only secret broker, plus Ollama / LM Studio / vLLM base URLs.
Updates in one click — the title-bar badge fetches the build for your machine and tells you how to install it, and it reads latest once a check confirms you are current. The first run afterwards opens that release's notes as a designed page rather than a version number. Background auto-update stays in Settings.
Prerequisites — one Settings page showing which supporting tools (uv, git, Node, MemPalace, each agent CLI) you have, what each is for, and a button that asks Rudy to install what is missing.

[!NOTE]
Status: v0.5.0, the first shipping release under the Rudy OS name. The rename goes all the
way down: identity, palette, onboarding, the cast of fifteen hires, the site, and
the on-disk namespaces. Everything else is the same floor: real CLI agents in real
terminals, shared long-term memory, inter-agent mail, and Rudy running it.
macOS (signed & notarized), Windows, and Linux builds are on the
releases page.

(↑ back to top)

Getting started

Prerequisites

macOS, Windows, or Linux.
Node.js 18+ and npm.
A C/C++ toolchain for node-pty's native addon — on macOS, install Xcode Command Line Tools:
```
xcode-select --install
```
At least one supported agent CLI on your PATH — Claude Code
(claude, the default), Antigravity (agy), OpenAI Codex (codex), xAI Grok (grok),
Kimi Code (kimi), Gemini CLI (gemini), Qwen (qwen), OpenCode (opencode),
Crush (crush), pi.dev (pi), GitHub Copilot (copilot), or Cursor (cursor-agent).
Most missing CLIs self-heal: the harness runs the installer in the
terminal and continues into the new binary.
Optional: your own API keys and local LLMs in Settings → AI Engines (Ollama / LM Studio / vLLM).
Optional: the semantic memory index for instant cross-session recall — markdown memory works without it.

Install & run

git clone https://github.com/nafdalex/rudy-os.git
cd rudy-os
npm install        # postinstall rebuilds node-pty against Electron's ABI
npm run dev        # launches the Electron app with hot reload

On first launch you'll go through the onboarding wizard, then land on the floor. Use Add agent to
spawn your first session — Rudy seats himself at the boss desk automatically.

Other scripts

npm run build      # production build via electron-vite
npm run preview    # preview the production build
npm run typecheck  # type-check the node (main/preload) and web (renderer) projects

If node-pty fails to load after an Electron upgrade, re-run npm install (the postinstall hook
runs electron-rebuild against the current Electron ABI).

Architecture

Two data planes feed one renderer:

┌───────────────────────────────────────────────────────────────┐
│                     Electron Renderer (React)                  │
│   ┌──────────────────┐    ┌──────────────────────────────┐    │
│   │ Office Floor      │    │ Terminal + Command Bar       │    │
│   │ (Pixi.js)        │    │ Files + Git tabs (xterm.js)  │    │
│   └─────────▲────────┘    └────────────▲─────────────────┘    │
│             │ avatar state             │ pty bytes / fs / git  │
└─────────────┼──────────────────────────┼───────────────────────┘
              │ IPC (contextBridge: window.cth)
       ┌──────┴──────────┐        ┌──────┴─────────────┐
       │  Event Plane    │        │  Terminal Plane    │
       │  hooks / hive   │        │  node-pty PTYs     │
       │  router + boss  │        │  + fs + git        │
       └────────▲────────┘        └──────▲─────────────┘
                │ hook payloads          │ stdin / stdout
                └─────────┬──────────────┘
                   ┌──────┴──────────────┐
                   │ claude / agy / codex│
                   └─────────────────────┘

Terminal plane. The main process owns a PtyManager that spawns each agent as a node-pty
process and streams output over per-id IPC (pty:data:<id>). The renderer talks only through a
typed window.cth bridge (src/preload/index.ts), which also exposes
sandboxed filesystem and git helpers.
Hive / event plane. hive.ts is the on-disk multi-agent layer; hooks.ts runs the hook
server that provider bridges POST lifecycle payloads to (cth-hook for Claude Code, agy-hook
for Antigravity). memory.ts wraps the semantic memory CLI. The router delivers messages, drains
provider outboxes, Rudy adjudicates, and idle/inbox wakeups keep workers draining mail.

Project structure

src/
  main/                      Electron main process (Node)
    index.ts                 window, IPC handlers, quit guard
    pty.ts                   node-pty manager (spawn/write/resize/kill/stream)
    hive.ts                  on-disk multi-agent layer (memory, mailboxes, router)
    hooks.ts                 hook server + provider hook shims (`cth-hook`, `agy-hook`)
    memory.ts                semantic memory layer (CLI wrapper, degrade-to-noop)
    config.ts                harness config persistence + home setup
    transcript.ts            reads ~/.claude/projects/ JSONL transcripts for real token/cost telemetry
    telemetry.ts             live OTel collector + usage/cost feed for observability
    usage.ts / pricing.ts    UsageProvider seam + per-model cost attribution
    breaker.ts / control.ts  cost/runaway circuit breaker (steer/constrain/stop) + HITL gate / steer / stop
    reflect.ts               MemoryReflector — memory condensation
    db.ts                    SQLite durable store (window bounds + history) + durable cost ledger
    github.ts                GitHub issue + CI run ingestion via the gh CLI
    shellEnv.ts              resolve PATH and shell env for child processes
    fs.ts / git.ts           sandboxed filesystem + git bridges
  preload/                   contextBridge → typed window.cth API
  renderer/src/
    App.tsx                  top-level layout + wiring
    design/                  tokens.css / tokens.ts / global.css (design source of truth)
    components/              PixelPanel, AgentDetailPanel, CommandBar, ApprovalsPanel, MemoryPanel, …
    CommandCenterPanel,      Rudy's control surface (Terminal/Floor/Memory/Activity/Tasks/Triggers/Handbook tabs)
    ToolWaterfall,           per-agent tool-span waterfall for the observability view
    TasksKanban,             dependency-aware kanban board (Tasks tab)
    ThreadsPanel,            hive message conversation viewer (Messages tab)
    MessageQueueComposer,    park messages for a busy agent
    scene/office/            Pixi office floor: OfficeFloor, Character, Camera, cast, pathfinding, …
    store/ · hooks/          zustand store, event loop, PTY parser, typewriter
    assets/                  tilesets, maps, character sheets (see ATTRIBUTION.md)
docs/                        `logo.png`, `banner.png`, landing page (GitHub Pages → rudydoes.com)
docs/media/                  `og.png` (social previews) + rendered Remotion clips
landing-remotion/            Remotion project that renders the landing page's "how it works" clips
HIVE.md · SPEC.md · DESIGN.md   multi-agent · terminal/event · visual design
docs/message-queue.md        who may type into an agent's terminal, and when

(↑ back to top)

Design system

The aesthetic is Animal Crossing × Earthbound × SNES menu UI — pixel-snapped, chunky, friendly.
DESIGN.md is canonical; every component derives from its tokens. The Rudy OS
brand layers a deep maroon (#6E1423) and gold (#F4D35E) on top for logo and
chrome. The 15 avatars are Rudy and his hires, differentiated by hair, skin and accessory recipes.

Roadmap

Shipped through v0.3.2 — twelve agent engines with BYOK keys and local LLMs, voice orchestration,
the hive (memory · mailboxes · blackboard · event log), Command Center with kanban and weekday
schedules, a built-in Monaco IDE with git rails, integrations registry + secret broker,
Slack-spawned workers, shareable hires and the Agent Gallery, observability and the circuit
breaker, durable persistence, session resume, multi-window floors, one click updates, a Skills
browser, a live Prerequisites check, cost reporting folded from the ledger, and semantic memory
that works on Apple Silicon.
Full history in CHANGELOG.md.

Next up:

More chat integrations — Telegram and richer chat bridges that pipe a channel into Rudy's queue and route replies back out.
More engines & integration templates — keep growing the engine roster and the integrations registry.
Fuller avatar coverage — drive the remaining station visits and tool-bubbles entirely from real hook events.
Durable layout & command history — extend persistence to agent layout and per-session history.

(↑ back to top)

Contributing

Contributions are welcome — this is an early prototype with a lot of surface area. Start with
CONTRIBUTING.md. The short version: fork, npm install && npm run dev, keep
npm run typecheck green, and derive any new UI from DESIGN.md tokens. Good
first areas: wiring real hook events, the add-agent flow, the config drawer, and cross-platform work.

[!IMPORTANT]
Every pull request must show a before and an after — screenshots, or a recording when the
thing moves — under the ### Before and ### After headings in the PR template. This is checked
automatically and a PR without it does not merge. "My change has no UI" is not an exemption; it
just changes what the evidence looks like. See
Evidence is mandatory.

Questions, bugs, or want to show off your office? A community Slack is on the way; until then, open an issue.

Telemetry

Official builds send a small set of anonymous usage events (app opened, agent spawned, feature
used) — never prompts, code, file paths, or agent output. The complete event list, the anonymity
guarantees, and the three ways to opt out (Settings toggle, DO_NOT_TRACK, or building from
source — forks compile with no key and send nothing) are documented in
TELEMETRY.md.

License

[!IMPORTANT]
Asset licensing. The bundled pixel art (tilesets and maps) is Modern Interiors - RPG Tileset
[16X16] by LimeZu, used under the Complete Version
licence, which permits editing and use in commercial and non-commercial projects. Credit to
LimeZu is required by that licence and must stay in place. The character cast is not LimeZu art. It
is drawn procedurally in portraitArt.ts. See
src/renderer/src/assets/ATTRIBUTION.md.

The source code is licensed under the MIT License — see LICENSE. The MIT grant
covers the code only. The LimeZu pixel-art tilesets are licensed separately (see above) and are
not included in this repository: LimeZu's licence does not permit redistributing the asset
pack, so src/renderer/src/assets/tilesets/ here holds only a placeholder note. Official builds
bundle the licensed assets; a from-source build renders the office scene without its tileset art
unless you purchase the pack from LimeZu yourself.

Acknowledgements

LimeZu for the Modern Interiors pixel-art tilesets (Complete Version licence).
shahar061/the-office for the office tileset/map vendoring.
Pixi.js · xterm.js · node-pty · electron-vite · CodeMirror for the libraries this is built on.
Remotion for the landing page's animated "how it works" clips (landing-remotion/).

rudy-os

Rudy OS

Agent harness that runs a whole office of agents

Contents

What it is

How it works

Features

Getting started

Prerequisites

Install & run

Other scripts

Architecture

Project structure

Design system

Roadmap

Contributing

Telemetry

License

Acknowledgements

Reviews (0)