🧠 J.A.R.V.I.S. OS Agent – The Ultimate Open-Source Autonomous AI Assistant for Windows

Welcome to the Jarvis-OS-Agent by @thekaifansari01. > "We call it an AI Operating System because it autonomously controls your PC, Android Phone, Local Files, and Lifelong Memory — not just your code."

Open-Source AI Agent · Zero Line‑Drift Coding · Lifelong Episodic Memory · Voice‑First Multimodal · Proactive HITL · Android ADB Control · Telegram Remote Execution · Hybrid RAG

Built by an 18‑year‑old solo developer. Commerce background. BCA first year. No team. No funding. Just late nights, coffee, and a burning passion to build the impossible. ☕

📌 Table of Contents

• 🎯 Core Features of J.A.R.V.I.S. AI Agent
• 🔥 J.A.R.V.I.S. vs. Claude Code (The Superior Alternative)
• 🏗️ Technical Architecture of Jarvis OS
• ⚡ Dual‑Engine AI: FastBrain vs 🧠 AgenticBrain
• 🧠 Lifelong Episodic Memory & Hybrid RAG System
• 📱 Android Mobile Control via ADB
• 🛠️ Integrated AI Tool Ecosystem
• 🚀 How to Install Jarvis OS Agent (Quick Setup)
• 📋 Real‑World Autonomous AI Use Cases
• ⚙️ Command Line Interface (CLI) & Agent Configuration
• 🔧 Troubleshooting & FAQ
• 🤝 Contributing to the Open Source Agent
• 📄 License

🎯 Core Features of J.A.R.V.I.S. AI Agent

Jarvis OS Agent is a highly advanced desktop-native AI Operating System built in Python. It bridges sub-second conversational AI with deep, multi-step autonomous engineering execution.

🔥 J.A.R.V.I.S. vs. Claude Code (The Superior Alternative)

For developers seeking an open-source AI coding agent and desktop assistant, the Jarvis OS Agent directly solves the critical pain points of paid enterprise solutions.

🏗️ Technical Architecture of Jarvis OS

Trust through transparency. Here is the complete architecture flow of the Python Autonomous Agent:

flowchart TD
    User[👤 Voice or Text Command] --> Input{Input Type}
    Input -->|Voice| Wake[🎙️ Vosk KWS Wake Word]
    Input -->|Text| Hotkey[⌨️ Ctrl+Shift+J Popup]
    Input -->|Remote| TelegramBot[🤖 Telegram Remote Bot]
    Wake --> STT[⚡ Deepgram Nova-2 STT]
    Hotkey --> Router[🚦 Hybrid Semantic Router]
    TelegramBot -->|Silent Flag| Router
    STT --> Router
    Router -->|Fallback| LocalRouter[🔄 Local Rule-Based Router]
    
    Router -->|Simple / Stateless| FastBrain[⚡ FastBrain<br/>Groq Llama-3.3-70B]
    Router -->|Complex / Stateful| AgenticBrain[🧠 AgenticBrain<br/>Regolo/Gemini/OpenRouter/Custom]
    
    AgenticBrain --> Providers[🔌 Provider Abstraction Layer]
    Providers --> Regolo[Regolo MoE]
    Providers --> Gemini[Gemini Reasoning]
    Providers --> OpenRouter[OpenRouter<br/>Claude 3.7 / o1 / DeepSeek]
    Providers --> Custom[Custom Provider<br/>Any OpenAI‑compatible<br/>endpoint including Ollama]
    
    subgraph Memory[🧠 Memory Ecosystem]
        LTM[(🗄️ Vector Semantic Graph LTM<br/>Weighted Graph + Subgraph + Decay)]
        RAG[(📚 ChromaDB RAG<br/>Hybrid Vector+BM25+RRF)]
        JSONL[📜 JSONL Rolling History<br/>15-Day Context]
        Profile[👤 User Profile & Mood]
    end
    
    AgenticBrain <--> Memory
    FastBrain <--> Memory
    
    subgraph Tools[🛠️ Native Tool Ecosystem]
        Code[💻 Repo-Map / Replace-Block / AST Linter]
        Comms[📨 Gmail / WhatsApp / Telegram / Calendar]
        System[⚙️ OS Control / Apps / Clipboard]
        Search[🌐 Tavily / ArXiv / Scraper]
        Vision[👁️ Vision Multimodal / OCR]
        Image[🎨 Image Gen / Edit]
        Mobile[📱 ADB / Mobile Control]
    end
    
    AgenticBrain --> Tools
    FastBrain --> System
    FastBrain --> Search
    
    subgraph UI[🎨 UI & Visualization]
        AgentPanel[🖥️ ZMQ Agent Panel]
        TypingPopup[📝 Markdown Typing Popup]
        STTPopup[🗣️ STT Status Popup]
        InputPopup[⌨️ Input Popup]
    end
    
    AgenticBrain -->|ZMQ PUB| AgentPanel
    FastBrain -->|typing_status.json| TypingPopup
    STT --> STTPopup
    InputPopup --> Router
    
    subgraph Proactive[🛡️ Proactive HITL Watchdog]
        Listeners[📡 Gmail / WhatsApp / Telegram / Reminders]
        Scout[🛡️ Proactive Scout Agent]
        Consent[🔒 HITL Consent Gate]
    end
    
    Listeners -->|Conditional Start| Scout
    Scout -->|Suggested Action| AgenticBrain
    AgenticBrain -->|Requires Permission| Consent -->|User Confirms| AgenticBrain
    
    subgraph Resilience[⚙️ Resilience Layer]
        Watchdog[🛡️ ServiceWatchdog]
        Failover[🔄 Provider Failover]
        Recovery[🔄 Two-Strike Rule]
    end
    
    Watchdog -.->|Smart Skip| System
    Failover -.-> Providers
    Recovery -.-> AgenticBrain

⚡ Dual‑Engine AI: FastBrain vs 🧠 AgenticBrain

Jarvis utilizes a dual-engine architecture to optimize API costs, speed, and reasoning depth.

🧠 Lifelong Episodic Memory & Hybrid RAG System

JARVIS implements a four‑tier artificial memory system, setting a new standard for open-source AI desktop agents:

1. 📜 Rolling JSONL History (Short‑Term): 15‑day rolling conversation context, auto‑pruned and archived.
2. 🗄️ Weighted Lifetime Episodic Graph (Long‑Term):

• Built on NetworkX with weighted edges and temporal decay (relations older than 6 months lose half their weight).
• Semantic Vector Embeddings: Uses SentenceTransformers (all-MiniLM-L6-v2) for zero-shot semantic matching. Maps conceptual synonyms (e.g., "React" matches "ReactJS") autonomously.
• Batched Array Queries & Preloading: Preloads the embedding engine at startup to process multi-entity graph queries in a single agent step with zero latency.
• Subgraph Retrieval: Extracts complete 2-hop subgraphs around matched nodes to retain secondary contextual facts.

1. 📚 Hybrid RAG (Workspace Documents):

• Smart Chunking: Text is chunked with 1500 character limits and a 200-character overlap to preserve cross-boundary context.
• Hybrid Retrieval: Merges BM25 keyword search and Gemini embedding vector search via Reciprocal Rank Fusion (RRF).
• Recency Boost: Recently modified files receive a 20% score lift, creating a self-updating knowledge base.

1. 👤 User Profile & Mood Tracker: Automatically extracts and tracks user bio, preferences, and mood states into knowledge-graph triplets.

📱 Android Mobile Control via ADB

Control your Android smartphone completely hands-free from your Windows PC using the Jarvis Agent via ADB and Tailscale:

Setup Note: Enable USB debugging on your Android, run adb tcpip 5555, connect your devices via Tailscale, and assign ADB_PHONE_IP in the .env file.

🛠️ Integrated AI Tool Ecosystem

🚀 How to Install Jarvis OS Agent (Quick Setup)

📋 Prerequisites

• Windows 10/11 (Primary supported OS)
• Python 3.10+ (Required for AI models and dependencies)
• Node.js 18+ (Required for the WhatsApp Baileys bridge)

🛠️ Single Block Installation

Copy and paste this exact command block into your PowerShell terminal:

# 1. Clone the Jarvis Repository
git clone [https://github.com/thekaifansari01/Jarvis-OS-Agent.git](https://github.com/thekaifansari01/Jarvis-OS-Agent.git)
cd Jarvis-OS-Agent

# 2. Setup Python Virtual Environment
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -r requirements.txt

# 3. Setup Node.js Dependencies (WhatsApp Integration)
cd tools/Messanger/whatsapp/BaileysServer
npm install
cd ../../../..

# 4. Global CLI Registry Setup (Run ONCE from the root directory)
python SetupRegistry.py

⚠️ CRITICAL: Ensure your terminal is in the root Jarvis-OS-Agent/ directory before executing python SetupRegistry.py.

🌐 Global Command Initialization

Once setup is complete, open a new terminal window and launch your agent globally by typing:

jarvis

You do not need to activate the virtual environment manually again; the jarvis command is now system‑wide.

📋 Real‑World Autonomous AI Use Cases

🐞 Scenario 1: Automated Bug Fixing (Zero Line‑Drift)

1. User: "Run the tests in my Python project and fix any failing ones."
2. Jarvis (AgenticBrain): Executes pytest -v via terminal; identifies 3 failing test cases.
3. Jarvis: Utilizes the view tool to read the specific failing files.
4. Jarvis: Employs replace_block (exact search & replace) to fix the core logic errors safely.
5. Jarvis: Re‑runs pytest -v to verify 100% success.
6. Jarvis (Voice): "Sir, all tests have passed. I have made changes to 3 files. Would you like to review them?"

🚨 Scenario 2: Proactive HITL (Your AI Executive Assistant)

1. Background Listener: The EmailProactive service detects an incoming email: "Meeting shifted to 5 PM."
2. Jarvis (Scout Mode): Evaluates priority and flags it for user consent.
3. Jarvis (Voice Alert): "[Alert] There is an email from Ram saying the meeting has been moved to 5 PM. Should I update the calendar?"
4. User: "Yes, do it."
5. Jarvis: Triggers AgenticBrain → calendar_action to securely update the Google Calendar event.
6. Jarvis: "Done sir, the calendar has been updated."

🧪 Scenario 3: Full‑Stack Application Generation

1. User: "Create a project named 'TaskFlow' on the Desktop. FastAPI backend with SQLite, React frontend with Tailwind, and 10+ unit tests. Run pytest and make all tests pass."
2. Jarvis (Autonomous Execution):

• Scaffolds the folder architecture.
• Generates requirements.txt and package.json.
• Codes FastAPI CRUD endpoints and React UI.
• Writes the test suite.
• Executes npm install and pip install autonomously.
• Runs pytest. If failures occur, utilizes replace_block to iterate and fix until 100% success.
• Invokes complete_task.

🧠 Scenario 4: Weighted Graph Knowledge Recall

1. Day 1: User: "I like BMW." → Graph Database stores [User] --(LIKES)--> [BMW] (Weight=1).
2. Day 60: User: "What is the color of my automobile?" → Semantic vector search identifies automobile maps to BMW, returning the stored relation.
3. Day 100: User: "I really like BMW." → Knowledge Graph Weight increases to 2.
4. Day 400: User: "What do I like?" → Algorithm calculates temporal decay (Day 400 vs Day 100). Adjusted weight prioritizes the most relevant lifelong facts instantly.

⚙️ Command Line Interface (CLI) & Agent Configuration

Agent Session & Memory Management CLI

⚠️ Important: Ensure the main Jarvis application is OFF before running memory, login, or reset commands.

Environment Variables (.env) Configuration

Copy .env.example to .env in the root directory and populate your API credentials. For 100% Local LLM Execution (e.g., Ollama), configure the Custom Provider block:

🔧 Troubleshooting & FAQ

🤝 Contributing to the Open Source Agent

We actively welcome contributions to make the Jarvis OS Agent the best open-source AI desktop assistant! Whether it's patching bugs, integrating new LLM providers, or expanding documentation:

1. Fork the repository.
2. Create a Feature Branch (git checkout -b feature/AdvancedRAGUpdate).
3. Commit your Changes (git commit -m 'Implement AdvancedRAGUpdate').
4. Push to the Branch (git push origin feature/AdvancedRAGUpdate).
5. Open a Pull Request for review.

📄 License

This open-source AI project is distributed under the MIT License. See the LICENSE file for full documentation.

🌟 Show Your Support for the Developer

If the Jarvis OS Agent streamlined your workflow, autonomously fixed your bugs, or inspired your own AI projects:

• ⭐ Star this repository to help it rank and grow the community.
• 🐦 Follow & Tweet your use cases by tagging @thekaifansari01.
• ☕ Buy me a coffee (Link coming soon) – Late nights, compiling code, and building AI operating systems run purely on caffeine.