J.A.R.V.I.S.

Just A Rather Very Intelligent System

A macOS AI assistant with voice control, screen awareness, browser automation, and multi-device support via Apple Watch and iPhone. Talk to it, type to it, or let it watch your screen — JARVIS understands context and takes action.

       ██╗ █████╗ ██████╗ ██╗   ██╗██╗███████╗
       ██║██╔══██╗██╔══██╗██║   ██║██║██╔════╝
       ██║███████║██████╔╝██║   ██║██║███████╗
  ██   ██║██╔══██║██╔══██╗╚██╗ ██╔╝██║╚════██║
  ╚█████╔╝██║  ██║██║  ██║ ╚████╔╝ ██║███████║
   ╚════╝ ╚═╝  ╚═╝╚═╝  ╚═╝  ╚═══╝  ╚═╝╚══════╝

Quick Start

cd ~/Downloads/jarvis
npm install
npm run dev

Say "Jarvis" to activate voice control, or type commands directly.

Features

Voice Assistant

JARVIS has an always-on voice assistant with wake word detection. Say "Jarvis" and it starts listening — no button press needed.

jarvis> voice on
  ✓ Voice assistant started. Say "Jarvis" to activate.

  [voice] Wake word detected, listening...
  [voice] Hello sir, how can I help?

  🗣  "open Safari"
  ✓ Opened Safari

  🗣  "what's my battery"
  ✓ Battery: 60%

  🗣  "good night"
  (runs good night routine — lowers volume, enables DND, locks screen)

Uses Apple's SFSpeechRecognizer for on-device speech recognition and Edge TTS or ElevenLabs for spoken responses. Fully on-device wake word detection, free, no cloud dependency for recognition.

Requires: macOS 13+ with Xcode Command Line Tools (xcode-select --install).

Conversation Engine

JARVIS uses a multi-turn conversation engine that understands context, executes actions mid-conversation, and remembers facts across sessions.

jarvis> turn on dark mode and set volume to 30

  [action] dark mode on
  ✓ Dark mode enabled
  [action] volume 30
  ✓ Volume set to 30%
  Done — dark mode is on and volume is at 30%.

jarvis> what's on my screen right now?

  I can see you have VS Code open with a TypeScript file...

jarvis> remember that my project deadline is March 20th

  ✓ Noted — I'll remember your project deadline is March 20th.

Powered by Claude API with automatic Ollama fallback for offline use. The conversation engine detects [ACTION:] tags in responses and executes them in real time, supports [REMEMBER:] for persistent memory, and maintains full conversation context.

All Commands

App Launcher

jarvis> open Safari
  ✓ Opened Safari

jarvis> close Slack
  ✓ Closed Slack

jarvis> switch to Chrome
  ✓ Switched to Chrome

jarvis> list apps
  ✓ Running applications:
    • Finder
    • Google Chrome
    • Terminal

Browser Control

jarvis> browse youtube.com
  ✓ Navigated to https://youtube.com

jarvis> search "TypeScript tutorials"
  ✓ Searched Google for "TypeScript tutorials"

jarvis> read this page
  ✓ (extracts and displays page content)

jarvis> screenshot
  ✓ Screenshot saved to jarvis-screenshot-1710432000.png

Full browser automation powered by Playwright — navigate, search, click elements, fill forms, read page content, and take screenshots.

Script Runner

jarvis> $ git status
  ✓ On branch main, nothing to commit

jarvis> run ls -la
  ✓ (full directory listing)

Dangerous commands (rm -rf /, sudo rm, mkfs, dd, fork bombs, chmod 777, etc.) are automatically blocked.

System Monitor

jarvis> cpu
  ✓ CPU: Apple M3
    Cores: 8
    Usage: 25.3% user, 21.4% sys, 53.3% idle

jarvis> battery
  ✓ Battery: 60%
    State: charging
    Remaining: 2:34

jarvis> status
  ─── JARVIS SYSTEM REPORT ───
  (full CPU + Memory + Disk + Battery + Network dashboard)

File Operations

jarvis> search package.json
  ✓ Found 5 result(s)

jarvis> open folder ~/Downloads
  ✓ Opened /Users/you/Downloads in Finder

jarvis> delete ~/Desktop/junk.txt
  ✓ Moved to Trash: /Users/you/Desktop/junk.txt

File search uses Spotlight (mdfind) for instant results. Delete moves files to Trash (always recoverable).

System Control

jarvis> volume 50
  ✓ Volume set to 50%

jarvis> dark mode on
  ✓ Dark mode enabled

jarvis> lock
  ✓ Screen locked

Timers & Reminders

jarvis> timer 5 min
  ✓ Timer #1 set for 5m (fires at 3:45:00 PM)

jarvis> remind me in 1 hour to push code
  ✓ Reminder #1 set: "push code" in 1h (at 4:40:00 PM)

jarvis> stopwatch
  ✓ Stopwatch started! Type "stopwatch" again to stop.

When timers fire, you get a macOS notification with sound + a terminal alert.

Time formats: 30s, 5 min, 1h30m, 2.5 hours, 1:30, or just a bare number (assumed minutes).

Process Manager

jarvis> top cpu
  ✓ Top CPU processes:
    1. Google Chrome Helper  — 45.2%
    2. node                  — 12.1%

jarvis> port 3000
  ✓ Port 3000: node (PID 12345)

jarvis> kill node
  ✓ Killed process: node

Clipboard Manager

jarvis> paste
  ✓ Clipboard: (current clipboard contents)

jarvis> clips
  ✓ Clipboard history:
    #1  Hello world
    #2  npm install

jarvis> paste #2
  ✓ Pasted from history: npm install

Window Manager

jarvis> tile Safari left
  ✓ Tiled Safari to left

jarvis> tile Chrome right
  ✓ Tiled Chrome to right

jarvis> Safari and Chrome side by side
  ✓ Arranged Safari and Chrome side by side

jarvis> fullscreen Terminal
  ✓ Maximized Terminal

Media Control (Spotify & Apple Music)

jarvis> play
  ✓ Playing (Spotify)

jarvis> now playing
  ✓ ▶ Bohemian Rhapsody
    by Queen
    on A Night at the Opera

jarvis> next
  ✓ ▶ Don't Stop Me Now

jarvis> play "Daft Punk"
  ✓ Searching Spotify for "Daft Punk"

Auto-detects whether Spotify or Apple Music is running.

Screen Awareness

jarvis> what's on my screen
  ✓ (OCR reads screen content and describes what it sees)

jarvis> read screen
  ✓ (extracts all visible text via OCR)

Uses OCR to read and understand screen content. The conversation engine can inject screen context into conversations for context-aware responses.

Screen Interaction

jarvis> paraphrase this
  (paraphrases selected text using AI)

jarvis> fix grammar
  (fixes grammar in selected text)

jarvis> translate to Spanish
  (translates selected text)

Processes currently selected text with AI for rewriting, grammar fixes, translation, and more.

WhatsApp

jarvis> send whatsapp to John: Hey, running late!
  ✓ Message sent to John

jarvis> read whatsapp
  ✓ Recent messages: ...

Send and read WhatsApp messages through automated browser control.

Weather & News

jarvis> weather
  ✓ Current weather: 72°F, Sunny

jarvis> news
  ✓ Top headlines: ...

Get current weather conditions and top news headlines.

Smart Routines

jarvis> good morning
  ✓ Running morning routine...
    Volume set to 40%
    Here's your weather...
    Here are today's headlines...

jarvis> good night
  ✓ Running night routine...
    Do Not Disturb enabled
    Volume set to 10%
    Screen locked

Voice-triggered routines that chain multiple actions together.

Conversions

jarvis> convert 5 miles to km
  ✓ 5 miles = 8.045 km

jarvis> time in Tokyo
  ✓ Tokyo: 2:30 AM (JST, +9:00)

Unit conversions and timezone lookups — always accurate, no LLM needed.

Site Monitor

jarvis> check if google.com is up
  ✓ google.com is UP (200 OK, 45ms)

jarvis> monitor mysite.com every 5 min
  ✓ Monitoring mysite.com every 5 minutes

Check if websites and services are online, with optional recurring monitoring.

Workflows & Automation

jarvis> create workflow morning: battery && cpu && volume 30
  ✓ Workflow "morning" created with 3 steps

jarvis> workflow morning
  Running workflow "morning" (3 steps)...

jarvis> every 5 min run battery
  ✓ Scheduled #1: "battery" every 5m

jarvis> shortcut "Toggle Dark Mode"
  ✓ Shortcut "Toggle Dark Mode" executed

AI Chat

jarvis> ask what is recursion
  Recursion is a programming technique where a function calls itself...

jarvis> summarize ~/code/main.py
  This file implements a REST API server with 3 endpoints...

jarvis> explain package.json
  This is a Node.js project configuration file...

jarvis> models
  ✓ Available models:
    llama3:latest (4.7 GB)
    mistral:latest (4.1 GB)

Uses Claude API by default for fast, accurate responses. Falls back to Ollama for fully offline, local AI. If neither is available, all other JARVIS features still work normally.

JARVIS Personality

jarvis> hello
  ✓ Good morning, sir. All systems operational. How may I assist you?

jarvis> tell me a joke
  ✓ Why do programmers prefer dark mode? Because light attracts bugs.

jarvis> who are you
  ✓ I'm JARVIS -- Just A Rather Very Intelligent System. I have 45+ modules
    loaded and can manage your apps, files, system, media, browser, and more.

Time-aware greetings, tech jokes, system-aware mood responses, and existential philosophy.

Smart Assist

jarvis> show me heaviest processes
  ✓ (automatically maps to "top cpu" via NLU)

jarvis> what can I do
  ✓ Try something new:
    - cpu
    - open <app>
    - timer <duration>
    - browse <url>
    - ask <question>

jarvis> top commands
  ✓ Your most used commands:
    1. battery (15x)
    2. cpu (12x)
    3. volume 50 (8x)

Research

jarvis> research quantum computing
  ✓ 15 papers analyzed (8 arXiv, 12 Semantic Scholar)
  Report saved to config/research-reports/...

jarvis> deep research transformer architectures
  ✓ Deep research complete: 4 rounds, 28 papers

jarvis> compare React vs Vue
  ✓ Comparison complete with table and recommendation

jarvis> quick research latest iPhone
  ✓ Quick web lookup complete

Intelligence Layer

jarvis> (runs automatically in background)
  Trace recording: every command logged with timing and context
  Route prediction: suggests modules based on usage patterns
  Habit detection: identifies your daily routines
  Memory search: hybrid TF-IDF + Rust vector search

JARVIS learns from your usage. Every command is recorded as a trace with timing, context (time of day, active app, voice mode), and result. The learning engine analyzes these traces to detect habits, predict what you'll do next, and suggest automations.

Rust Sidecar

jarvis> (starts automatically on boot)
  [rust-bridge] Sidecar ready
  Vector search: <1ms memory retrieval
  Fuzzy match: Rust-accelerated Levenshtein
  Trace analytics: single-pass stats over 5000+ traces

A compiled Rust binary (rust-sidecar/) that accelerates performance-critical operations. Falls back gracefully to TypeScript when unavailable.

Multi-Device Support

JARVIS connects to Apple Watch and iPhone via AIM (Advanced Idea Mechanics), a WebSocket relay server.

The menubar app shows JARVIS status at a glance — running state, voice mode, and last command processed.

Command Chaining

Run multiple commands in sequence with && or ;:

jarvis> battery && cpu && disk
  ✓ Battery: 85% ...
  ✓ CPU: Apple M3 ...
  ✓ Disk: 45% used ...

jarvis> open Safari; open Chrome; tile Safari left; tile Chrome right

Variables

Built-in variables expand automatically:

jarvis> $ echo $HOME
  ✓ /Users/you

jarvis> set mydir = ~/Projects
  ✓ Variable $mydir set

jarvis> open folder $mydir
  ✓ Opened /Users/you/Projects in Finder

Command History

Persistent across sessions:

jarvis> history
  Recent commands:
    1. battery
    2. volume 50
    3. open Safari

jarvis> !!
  (repeats last command)

jarvis> history search volume
  Matches: volume 50, volume up, ...

Aliases

Create shortcuts for frequently used commands:

jarvis> alias deploy = run npm run build && npm run deploy
  ✓ Alias created: "deploy" → "run npm run build && npm run deploy"

jarvis> deploy
  (runs the aliased command)

jarvis> aliases
  chrome → open Google Chrome
  safari → open Safari
  stats  → status
  vol    → volume

Startup Commands

Auto-run commands every time JARVIS launches:

jarvis> startup add battery
jarvis> startup add volume
jarvis> startup list
  1. battery
  2. volume

Fuzzy Matching & Typo Tolerance

JARVIS uses Levenshtein distance to handle typos. If your input is within an edit distance of 2 from a known keyword, it still works:

jarvis> baterry
  ✓ Battery: 85% ...

jarvis> neetwork
  ✓ Network: 192.168.1.5, WiFi: MyNetwork

Architecture

User Input → Variable Expansion → Alias Expansion → Pattern Parser → Executor → Module
                                                          ↓                        ↓
                                                    Keyword Fallback    AppleScript / shell / LLM
                                                          ↓
                                                    Fuzzy Match (Rust/Levenshtein)
                                                          ↓
                                                    NLU Mapping (natural language)
                                                          ↓
                                                    Intelligence Layer (traces, prediction, context)
                                                          ↓
                                                    Conversation Engine (Claude / Ollama)

The core parser uses no AI/LLM — it's a seven-phase engine:

1. Variable expansion — $HOME, $DATE, custom $vars
2. Alias expansion — user-defined shortcuts
3. "open" disambiguation — detects if target is app or file path
4. Regex pattern matching — each module declares regex patterns; first match wins (confidence 1.0)
5. Keyword fallback — exact keyword match (confidence 0.6), then Levenshtein fuzzy match (confidence 0.4)
6. NLU mapping — natural language phrases mapped to existing commands (confidence 0.5)
7. Conversation engine — if nothing matches, treats input as natural language and routes to the conversational AI

Project Structure

jarvis/
├── bin/jarvis.ts                  # Entry point
├── rust-sidecar/
│   ├── Cargo.toml
│   └── src/
│       ├── main.rs              # HTTP server (actix-web) — vector search, fuzzy, traces
│       ├── vector.rs            # Cosine similarity vector index
│       ├── embeddings.rs        # TF-IDF text embedder (384-dim)
│       ├── fuzzy.rs             # Levenshtein distance + batch matching
│       └── traces.rs            # Trace analytics + habit detection
├── config/
│   ├── aliases.json               # User command shortcuts
│   ├── startup.json               # Auto-run on boot
│   ├── workflows.json             # Saved workflows
│   └── scheduled-tasks.json       # Recurring tasks
├── src/
│   ├── index.ts                   # REPL loop, wires everything together
│   ├── mac-client.ts              # Thin client for VPS mode
│   ├── core/
│   │   ├── types.ts               # Shared interfaces (45+ module types)
│   │   ├── parser.ts              # 7-phase NL parser + fuzzy matching
│   │   ├── registry.ts            # Module registry
│   │   ├── executor.ts            # Dispatches to modules
│   │   ├── conversation-engine.ts # Multi-turn AI with action execution
│   │   ├── memory.ts              # Persistent memory and conversation context
│   │   ├── capabilities.ts        # System prompt builder for AI
│   │   ├── history.ts             # Persistent command history
│   │   └── context.ts             # Session state + variable expansion
│   ├── intelligence/
│   │   ├── index.ts               # Barrel exports + initialization
│   │   ├── trace-store.ts         # Command trace recording (capped at 5000)
│   │   ├── router-policy.ts       # Route suggestion + next-command prediction
│   │   ├── learning-engine.ts     # Habit detection + failure analysis
│   │   ├── memory-index.ts        # TF-IDF search + hybrid Rust vector search
│   │   ├── context-engine.ts      # Context injection for LLM prompts
│   │   └── energy-tracker.ts      # Operation energy/cost tracking
│   ├── modules/                   # 45+ modules (see table below)
│   ├── utils/
│   │   ├── shell.ts               # Safe child_process wrapper + deny-list
│   │   ├── osascript.ts           # AppleScript helpers
│   │   ├── formatter.ts           # Chalk colored terminal output
│   │   ├── ollama.ts              # Ollama HTTP client (streaming)
│   │   ├── llm.ts                 # Hybrid LLM provider (Claude + Ollama)
│   │   ├── browser-manager.ts     # Playwright browser lifecycle
│   │   ├── voice-output.ts        # TTS via Edge TTS / ElevenLabs
│   │   ├── platform.ts            # OS detection (macOS / Linux)
│   │   ├── mac-proxy.ts           # Remote command proxy for VPS mode
│   │   └── status-reporter.ts     # Status file + AIM broadcast
│   ├── voice/
│   │   ├── voice-assistant.ts     # Wake word + speech recognition + conversation
│   │   └── voice-input.ts         # macOS Speech Recognition via Swift
│   └── watch/
│       ├── aim-bridge.ts          # AIM WebSocket bridge for multi-device
│       └── ws-server.ts           # WebSocket server for Watch connectivity
├── menubar/
│   ├── JarvisMenubar.swift        # Native macOS menubar app
│   └── start-menubar.sh           # Menubar launcher
├── watch/JarvisWatch/             # Xcode project for Watch + iPhone apps
├── package.json
└── tsconfig.json

45+ Modules

Dependencies

Runtime:

• chalk — terminal colors
• ws — WebSocket client for AIM
• playwright — browser automation
• @elevenlabs/elevenlabs-js — voice synthesis (optional)
• dotenv — environment variables
• actix-web — Rust HTTP server (sidecar)

Dev only:

• typescript — compiler
• tsx — run .ts files directly in dev
• @types/node — Node.js type definitions

Everything else is Node.js built-ins (readline, child_process, os, fs) and macOS native tools (osascript, mdfind, pmset, open, pbcopy/pbpaste, lsof, shortcuts).

Requirements

• macOS (tested on macOS 14+ Sonoma, Apple Silicon)
• Node.js 20+
• Xcode Command Line Tools (for voice commands): xcode-select --install
• Ollama (optional, for offline AI): ollama.com — ollama serve then ollama pull llama3
• Claude API key (optional, for cloud AI): set in config/llm-config.json
• Rust (optional, for sidecar): curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Build & Install

# Development (uses tsx, no build step)
npm run dev

# Production build
npm run build
npm start

# Build Rust sidecar (optional — JARVIS works without it)
cd rust-sidecar && cargo build --release

# Install globally (makes 'jarvis' available everywhere)
npm link
jarvis