OpenCrow

A self-hosted multi-agent AI platform that orchestrates specialized agents across Telegram, WhatsApp, and a web dashboard — with 100+ tools, 16 autonomous data scrapers, vector memory, cron scheduling, and real-time market streaming.

What Can OpenCrow Do?

• Run AI agents on Telegram, WhatsApp, and web — each with its own persona, model, tools, and memory
• Scrape data sources autonomously — HackerNews, Reddit, GitHub, X/Twitter, App Store, Play Store, news, and more
• Remember everything — conversations, facts, and observations are indexed into vector memory and recalled across sessions
• Generate ideas — research agents collect signals, ideation agents synthesize them into product/crypto/AI startup ideas on a schedule
• Monitor markets — real-time crypto derivatives via Binance WebSocket (prices, liquidations, open interest, technical indicators)
• Automate with cron — schedule any agent to run at intervals, one-shot times, or cron expressions
• Self-manage — agents can deploy code, restart processes, manage other agents, and monitor system health
• Scale horizontally — each agent, scraper, and subsystem runs as an isolated process with crash recovery

Architecture

OpenCrow runs as a thin core process that spawns and supervises isolated child processes. Each subsystem (agents, scrapers, web, cron, markets) runs independently with crash-loop detection, exponential backoff, and automatic recovery.

core (orchestrator + internal API)
  └── spawns:
      ├── cron                       (scheduled jobs + agent execution)
      ├── web                        (React dashboard + Hono API)
      ├── agent:default              (Telegram + WhatsApp channels)
      ├── agent:<id>                 (per-agent Telegram/WhatsApp bots)
      ├── scraper:<id>               (one per data source, isolated)
      └── market                     (WebSocket hub + derivatives pipeline)

• Process isolation — Kill a scraper, everything else keeps running. Crash one agent, others are unaffected.
• Dynamic scaling — Agents and scrapers are added/removed via config. The orchestrator reconciles desired vs actual state every 5 seconds.
• Monolith mode — For development, src/gateway.ts runs everything in a single process.

Stack

Agent System

Agents are the core of OpenCrow. Each agent is an AI persona with its own system prompt, model, tools, and channel connections.

How agents work

1. Message arrives via Telegram/WhatsApp/Web
2. Router checks authorization, resolves which agent handles the chat
3. Agent SDK spawns a Claude Code subprocess with the agent's system prompt
4. OpenCrow tools are exposed via an in-process MCP server (no network hop)
5. Agent iterates: thinking → tool use → response, auto-continuing until done
6. SDK session ID is captured and resumed on next message for conversational continuity
7. Observations are extracted from the conversation and indexed into memory

Agent features

• Session resume — Conversations persist across messages via SDK session IDs
• Cross-session memory — Related memories and user preferences are injected into prompts
• Auto-continuation — Agents keep working when they have pending tool use, unlimited iterations
• Sub-agent spawning — Agents can spawn specialized sub-agents for complex tasks
• Activity logging — Real-time progress shown as an editable Telegram message with tool tracking
• Hot-reload — Agent configs stored in DB (config_overrides table), reloaded every 30 seconds
• Agent templates — Predefined templates (chatbot, researcher, coder, etc.) for quick agent creation
• Tool filtering — Per-agent allowlist/denylist controls which tools are available
• Intelligent tool routing — Tools are ranked by category match, recency, success rate, and keyword relevance

Built-in sub-agents

31 specialized agents for orchestrated workflows:

Orchestration workflow

TRIVIAL task   → Agent answers directly
MODERATE task  → Agent states plan, executes
COMPLEX task   → Agent spawns sub-agents:
                 planner → user approval → backend/frontend → reviewer → security-reviewer

Tools (100+)

Every tool is a ToolDefinition with name, JSON Schema, and execute function. Tools are registered per-agent based on capabilities and exposed via MCP.

File Operations

Code Development

Research & Data

Markets & Trading

Memory & Knowledge

Ideas & Signals

Analytics & Monitoring

System & Automation

Data Scrapers

16 autonomous scrapers run as isolated processes, each with its own tick interval and error handling. Scraped data is stored in PostgreSQL and indexed into Qdrant for semantic search.

| Markets | real-time | Crypto derivatives via Binance WebSocket (prices, liquidations, open interest, technical indicators) |

Scraper features

• Browser automation — X/Twitter and News use Playwright for JavaScript-heavy sites

• Anti-detection — Stealth scripts, cookie auth, GraphQL API interception for X/Twitter

• Multi-account — Reddit, Product Hunt support multiple accounts with separate cookies

• Rate limiting — Per-source delays (App Store: 2s between calls)

X/Twitter automation

The X/Twitter scraper is a full automation suite:

• Timeline scraping — Capture tweets from your home timeline via GraphQL interception
• Bookmark sharing — Share bookmarked tweets to configured Telegram chats
• Auto-like — Automatically like tweets matching configured criteria
• Auto-follow — Follow users based on interaction patterns
• Interaction tracking — Track likes, retweets, replies across accounts

Memory & RAG

Hybrid search engine combining vector similarity and full-text search for long-term agent memory.

Search pipeline

1. Query expansion — Semantic expansion with synonyms and related terms
2. Parallel search — Vector (Qdrant) + full-text (PostgreSQL) run concurrently
3. Score merging — 70% vector weight + 30% full-text weight
4. Temporal decay — Per-kind half-life (observations decay faster than documents)
5. MMR deduplication — Maximal marginal relevance to reduce redundancy
6. Channel scoping — Conversation memories stay within their channel

Memory types

Source kinds, each with its own chunk profile and temporal decay: conversations, notes, documents, tweets, articles, products, stories, Reddit posts, GitHub repos, observations, ideas, app reviews, and app rankings.

Observation extraction

After each conversation, observations (facts, preferences, patterns) are automatically extracted and indexed. These are injected into future prompts for cross-session context.

Channels

Plugin-based channel system with a unified adapter interface.

Telegram

• grammY-based with serial polling (no runner — avoids 409 conflicts)
• Per-chat message queuing via sequentialize middleware
• Activity log: editable message showing real-time tool progress
• Smart message chunking for 4096 character limit
• HTML formatting with inline buttons

WhatsApp

• Baileys (Web WhatsApp protocol)
• QR code / pairing code authentication
• Per-number and per-group sender filtering
• Media support (images, documents)

Web Chat

• Hono API route (POST /api/chat)
• Bearer token auth via OPENCROW_WEB_TOKEN

Router

The message router handles authorization, agent selection, and response delivery:

• Authorization — Per-channel allowed senders (userIds for Telegram, numbers for WhatsApp)
• Agent selection — Explicit /agent command > persistent routing rules > default agent
• Commands — /stop (abort), /clear (reset session), /status (health), /agent [id] (switch)
• Concurrency — One active message per chat, prevents overlapping processing
• Input validation — Per-agent max input length enforcement

Cron Scheduler

Agents can be triggered on schedules for autonomous work.

Schedule types

• at — One-time execution at a specific timestamp
• every — Interval-based (e.g., every 6 hours)
• cron — Standard cron expressions via croner

Built-in jobs

• Idea generation — Research agents save signals, ideation agents synthesize them (every 6 hours)
• Watchdog — Health monitoring: processes, errors, cron success, costs, DB, scrapers (every 30 minutes)
• Digests — Scheduled content summaries

Execution

Jobs run agents with full tool access in isolation. Progress is tracked per-run with status, duration, and result summaries stored in the database.

Web Dashboard

React SPA served via Bun HTML imports with Hono API backend. 30+ views covering every subsystem.

Views

| X/Twitter | Timeline, bookmarks, auto-like, auto-follow management |
| Product Hunt | Product browser with account management |

Auth

OPENCROW_WEB_TOKEN environment variable → Bearer token in localStorage → 401 shows login modal.

Quick Start

curl -fsSL https://opencrow.dev/install.sh | bash
opencrow setup

The setup wizard handles everything: Docker containers, database, .env, Telegram/WhatsApp, and service installation.

opencrow doctor     # Check system health
opencrow update     # Pull latest + reinstall + restart
opencrow status     # Show running service status
opencrow start      # Start in foreground (monolith mode)
opencrow version    # Show version info

Manual Setup

Prerequisites

• Bun runtime
• Docker (for PostgreSQL, Qdrant, QuestDB)
• Claude Agent SDK credentials (~/.claude/.credentials.json)

Install

bun install
cp .env.example .env

Environment variables

# Required
OPENCROW_WEB_TOKEN=...                    # Web dashboard auth token
TELEGRAM_BOT_TOKEN=...                    # Telegram bot token from BotFather
OPENROUTER_API_KEY=...                    # For embeddings (text-embedding-3-small)
DATABASE_URL=postgres://opencrow:[email protected]:5432/opencrow

# Optional
QDRANT_URL=http://127.0.0.1:6333         # Vector search (defaults to this)
QUESTDB_ILP_URL=tcp::addr=127.0.0.1:9009 # Time series ingestion
QUESTDB_HTTP_URL=http://127.0.0.1:9000   # Time series queries
OPENCROW_WEB_HOST=127.0.0.1              # Web UI bind address
OPENCROW_WEB_PORT=48080                  # Web UI port

Start services

docker compose up -d    # PostgreSQL + Qdrant + QuestDB

Run

# Development (monolith, single process)
bun run dev

# Production (distributed, process tree)
bun run start

Development

bun run dev           # Watch mode (monolith)
bun run dev:web       # Web UI only
bun test              # Run tests
bun run typecheck     # TypeScript check
bun run tw            # Tailwind CSS watch
bun run tw:build      # Tailwind CSS build

Deployment

systemd

bun run service:install    # Install opencrow.service
bun run service:start      # Start
bun run service:stop       # Stop
bun run service:restart    # Restart
bun run service:status     # Check health

Manual deploy

git push origin master
ssh <server> "cd ~/opencrow && git pull origin master"
sudo systemctl restart opencrow

Project Structure

src/
├── agent/           # Agent SDK integration, MCP bridge, streaming, sessions
├── agents/          # Agent registry and config resolution
├── channels/        # Telegram + WhatsApp plugins, registry, manager
├── config/          # Schema (Zod), loader, env overrides
├── cron/            # Scheduler, executor, job store, delivery poller
├── daemon/          # systemd + launchd service management
├── entries/         # Process entry points (core, agent, cron, scraper, market)
├── health/          # Checkpoint, rollback notifier
├── memory/          # RAG pipeline (indexer, search, embeddings, Qdrant, chunker)
├── process/         # Orchestrator, manifest, bootstrap, supervisor
├── prompts/         # Prompt loader
├── router/          # Message routing, activity logging
├── sources/         # 16 data scrapers (each in own directory)
├── store/           # Database init + migrations
├── tools/           # 100+ tool definitions (registry, types, factories)
├── web/             # Hono routes + React SPA
│   ├── routes/      # API endpoints (/api/*)
│   └── ui/          # React components, views, styles
├── worktree/        # Git worktree management for isolated agent work
├── cli.ts           # CLI entry point
├── gateway.ts       # Monolith mode (all-in-one)
├── index.ts         # Default entry
├── logger.ts        # Pino-based structured logging
└── web-index.ts     # Web-only entry point

prompts/
├── SOUL.md          # Core identity and values
├── TECH.md          # Technical context and conventions
├── WORKFLOW.md      # Decision and execution process
├── ORCHESTRATION.md # Sub-agent coordination rules
└── agents/          # 31 specialized agent prompts

skills/              # Reusable skill documents for agents
bin/                 # Guardian scripts (crash-loop detection + rollback)