👁️ Observer AI

The Agent that monitors your screen, so you don't have to.

Using local models, keeping your data 100% private and secure.

What Observer Does Best

When X then Y

• When I'm distracted, call me.
• When my download finishes, SMS me.
• When my render is done, Whatsapp me.
• When this dashboard changes, email me.
• When a person is on camera, telegram me.
• When my dog finishes their food, log it.

How Observer Micro-Agents Work

Sensors →	Models →	Tools
_{Screen • Camera • Mic • Audio}	_{Local LLMs}	_{Messaging • Notifications • Recording • Memory • Code}

Base Micro-Agent Example

Sends an email when the Observer logo is on screen

System Prompt (uses $SCREEN for multimodal screen input)

You are an Observer agent, watch the screen and if you see the Observer logo say OBSERVER, if you don't, say CONTINUE. 
$SCREEN

Code using Email Tool if model identified an Observer logo

if(response.includes("OBSERVER")){
  sendEmail("[email protected]", response, screen); //sends the screen as an attached image
}

Building Your Own Micro-Agent

Creating your own Observer AI consist of three things:

SENSORS - input that your model will have
MODELS - Small LLMs
TOOLS - functions for your model to use

Quick Start

Navigate to the Agent Dashboard and click "Create New Agent"
Fill in the "Configuration" tab with basic details (name, description, model, loop interval)
Give your model a system prompt and Sensors! The current Sensors that exist are:
- Screenshot ($SCREEN) Captures screen as an image for multimodal models.
- Screen OCR ($SCREEN_OCR) Captures screen content as text via OCR.
- CAMERA ($CAMERA) Captures the camera for multimodal models.
- CAMERA OCR ($CAMERA_OCR) Captures the camera text via OCR.
- Agent Memory ($MEMORY or $MEMORY@agent_id) Accesses agents' stored information (defaults to current agent)
- Agent Image Memory ($IMEMORY or $IMEMORY@agent_id) Accesses agents' stored images (defaults to current agent)
- Agent Image Memory OCR ($IMEMORY_OCR) Captures text on an image's memory via OCR.
- Clipboard ($CLIPBOARD) It pastes the clipboard contents
- Microphone* ($MICROPHONE) Captures the microphone and adds a transcription
- Screen Audio* ($SCREEN_AUDIO) Captures the audio transcription of screen sharing a tab.
- All audio* ($ALL_AUDIO) Mixes the microphone and screen audio and provides a complete transcription of both (used for meetings).

* Uses a whisper model with transformers.js

Agent Tools:

getMemory(agentId?)* – Retrieve stored memory
setMemory(agentId?, content)* – Replace stored memory
appendMemory(agentId?, content)* – Add to existing memory
getImageMemory(agentId?)* - Retrieve images stored in memory
setImageMemory(agentId?, images)* - Set images to memory
appendImageMemory(agentId?, images)* - Add images to memory
startAgent(agentId?)* – Starts an agent
stopAgent(agentId?)* – Stops an agent
time() - Gets current time
sleep(ms) - Waits that ammount of miliseconds

* agentId is optional, deaults to agent running code

Notification Tools:

sendDiscord(discord_webhook, message, images?, videos?) - Directly sends a discord message to a server.
sendTelegram(chat_id, message, images?, videos?) Sends a telegram message with the Observer bot. Get the chat_id messaging the bot @observer_notification_bot.
sendEmail(email, message, images?, videos?) - Sends an email. Email must be the signed in email.
sendPushover(user_token, message, images?, title?) - Sends a pushover notification.
call(phone_number, message)* - Makes an automated phone call with text-to-speech message. Needs whitelisting.
sendWhatsapp(phone_number, message, images?, videos?)* - Sends a whatsapp message with the Observer bot. Needs whitelisting.
sendSms(phone_number, message, images?, videos?)* - Sends an SMS to a phone number. Due to A2P policy, blocked for US/Canada. Needs whitelisting.
notify(title, options) – Send browser notification ⚠️IMPORTANT: Some browsers block notifications
sound(name?, volume?) - Plays a notification sound locally. No account needed, works on web and app.

* To whitelist, SMS or call +1 (863)208-5341 or whatsapp +1 (555)783-4727

Video Recording Tools:

startClip() - Starts a recording of any video media and saves it to the recording Tab.
stopClip() - Stops an active recording
markClip(label) - Adds a label to any active recording that will be displayed in the recording Tab.
getVideo() - Returns array of videos on buffer.

App Tools:

ask(question, title="Confirmation") - Pops up a system confirmation dialog
message(message, title="Agent Message") - Pops up a system message
system_notify(body, title="Observer AI") - Sends a system notification
overlay(body) - Pushes a message to the overlay
click('left'|'right') - Triggers a mouse click at the current cursor position accepts either 'left' or 'right', defaults to left.
celebrate() - Triggers a celebration animation in the Observer UI.

Code Tab

The "Code" tab receives the following variables as context before running:

response - The model's response (null when using a decision model)
decision - The decision model's answer (null for text models, see Decision Models)
agentId - The id of the agent running the code
screen - The screen if captured
camera - The camera if captured
imemory - The agent's current image in memory
images - All images sent to the model
prompt - The model's prompt
microphone - Trascription from the microphone in this loop
screenAudio - Transcription from screen audio in this loop
allAudio - Transcription from microphone and screen audio mixed in this loop

JavaScript agents run in the browser sandbox, making them ideal for passive monitoring and notifications:

// Remove Think tags for deepseek model
const cleanedResponse = response.replace(/<think>[\s\S]*?<\/think>/g, '').trim();

// Get time
const time = time();

// Update memory with timestamp
appendMemory(`[${time}] ${cleanedResponse}`);

// Send to Telegram if the model mentions a word
if(response.includes("word")){
  sendTelegram(cleanedResponse, "12345678") // Example chat_id
}

Decision Models

Decision models don't write text, they answer typed questions with probabilities. response is null and the result is in decision instead.

Plain-text prompt – a single yes/no (noul) question, the sensor images are the state:

Is there a person in the room? $CAMERA

decision is one answer:

// { type: 'noul', noul: 0.94, start: 0.12, dropped: false }
if (decision.noul > 0.8) {
  sendTelegram("12345678", "Someone is in the room", camera);
}

noul - Probability of "yes" (0 to 1)
start - Average noul over the run's first 3 iterations
dropped - true once noul has stayed well below start for 3 iterations in a row (useful for "tell me when this finishes")

JSON prompt – several questions at once. decision is an object keyed by question id:

{
  "state": "<image:1>",
  "questions": {
    "person": { "type": "noul", "instructions": "Is there a person?" },
    "pet": { "type": "choice", "instructions": "Which pet is visible?", "criteria": { "dog": "A dog", "cat": "A cat", "none": "No pet" } },
    "mess": { "type": "score", "instructions": "How messy is the room?", "criteria": ["Clean", "A bit messy", "Very messy"] }
  }
}

Add an image sensor ($SCREEN, $CAMERA or $IMEMORY) after the JSON to capture the image. <image:N> marks where each captured image goes in the state; if the state has no markers, the images are placed at the top.

// decision.person → { type: 'noul', noul }
// decision.pet    → { type: 'choice', choice, probabilities, confidence }
// decision.mess   → { type: 'score', score, legend, probabilities, confidence }
if (decision.pet.choice === "dog" && decision.pet.confidence > 0.7) {
  appendMemory(`[${time()}] Dog spotted`);
}

Getting Started with Local Inference for Micro-Agents

There are a few ways to get Observer up and running with local inference. I recommend the Observer App.

Option 1 (Easiest): Pull models using Transformers.js on WebApp

If you go to the ModelHub you can download Gemma 4 e2b and Gemma4 e4b directly on the browser with no install. This is a bit unstable and crashes mobile devices, but no install is required at all!

Option 2 (Easy and Stable): Download the Observer App and use bundled llama.cpp models

Download the Official App:

The Observer App comes bundled with llama.cpp under the hood, so you can run any GGUF model! Be sure to load an mmproj file if using multimodality.

Option 3 (Most stable): Use Desktop App with any OpenAI compatible endpoint (Ollama, llama.cpp, vLLM)

Download Ollama for the best compatibility. Observer can connect directly to any server that provides a v1/chat/completions endpoint.

Set the Custom Model Server URL on the App to any OpenAI compatible endpoint.

NOTE: Your browser app sends the request to localhost:3838 which the ObserverApp proxies to your Custom Model Server URL, this is because of CORS.

Option 4: Full Docker Setup (Deprecated)

For Docker setup instructions, see docker/DOCKER.md.

Setting Up Python (Jupyter Server) (Deprecated)

For Jupyter server setup instructions, see app/JUPYTER.md.

Deploy & Share

Save your agent, test it from the dashboard, and upload to community to share with others!

CLI tool

Cool easter egg, the app comes bundled with the CLI binary. You can use it doing

observe sleep 5 and the bash wrapper will call you when the command exits. Enjoy!

Or built it from source:

cd app && cargo build

Contributing

We welcome contributions from the community! Here's how you can help:

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'feat: add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

This project is licensed under the AGPLv3 License - see the LICENSE file for details.

Built with ❤️ by Roy Medina for the Observer AI Community
Special thanks to the Ollama team for being an awesome backbone to this project!

Observer