Discover, share, and monitor AI coding agents with full observability built in.

If you find Observal useful, please consider giving it a star. It helps others discover the project and keeps development going.

Observal is a self-hosted AI agent registry with built-in observability. Think Docker Hub, but for AI coding agents.

Browse agents created by others, publish your own, and pull complete agent configurations, all defined in a portable YAML format that templates out to Claude Code, Codex CLI, Gemini CLI, and more. Every agent bundles its MCP servers, skills, hooks, prompts, and sandboxes into a single installable package. One command to install, zero manual config.

Every interaction generates traces, spans, and sessions that flow into a telemetry pipeline, giving you full observability, traceability, and real-time metrics for your agents in production. The built-in eval engine (WIP) scores agent sessions so you can measure performance and make your agents better over time.

Supported tools: Claude Code, Codex CLI, Gemini CLI, and Kiro CLI are fully supported. Cursor and VS Code have MCP/rules file support.

See the Changelog for recent updates.

Quick Start

git clone https://github.com/BlazeUp-AI/Observal.git
cd Observal
cp .env.example .env          # edit with your values

cd docker && docker compose up --build -d && cd ..
uv tool install --editable .
observal auth login            # auto-creates admin on fresh server

Already have MCP servers in your IDE? Instrument them in one command:

observal scan                  # auto-detect, register, and instrument everything
observal pull <agent> --ide cursor  # install a complete agent

This detects MCP servers from your IDE config files, registers them with Observal, and wraps them with observal-shim for telemetry without breaking your existing setup. A timestamped backup is created automatically.

The Problem

AI coding agents today are hard to share and impossible to measure. Components (MCP servers, skills, hooks, prompts) are scattered across repos with no standard way to package them together. There's no visibility into what's actually working, and no way to compare one version of an agent against another on real workflows.

Observal solves this by giving you a registry to package and distribute complete agents, and a telemetry pipeline to measure them.

How It Works

Agents in the registry are defined in YAML. Each agent bundles its components (MCP servers, skills, hooks, prompts, sandboxes) into a single configuration. When you run observal pull <agent>, it installs everything and generates the right config files for your tool.

A transparent shim (observal-shim for stdio, observal-proxy for HTTP) sits between your tool and the MCP server. It never modifies traffic, it only observes. Every request/response pair becomes a span, spans group into traces, and traces form sessions. All of this streams into ClickHouse for analysis.

Tool  <-->  observal-shim  <-->  MCP Server / Sandbox
                |
                v (fire-and-forget)
          Observal API  -->  ClickHouse (traces, spans, scores)
                |
                v
          Eval Engine (SLM-as-a-Judge / Deductive Penalty Scoring)  -->  Scorecards

The eval engine runs on collected traces after the fact. It scores agent sessions across five dimensions: Goal Completion, Tool Call Efficiency, Tool Call Failures, Factual Grounding, and Thought Process. Scorecards let you compare agent versions, identify bottlenecks, and track improvements over time.

The Registry

Observal manages 6 component types that agents bundle together:

Component	Description
Agents	Complete configurations that bundle all the components below
MCP Servers	Model Context Protocol servers that expose tools to agents
Skills	Portable instruction packages that agents load on demand
Hooks	Lifecycle callbacks that fire during agent sessions
Prompts	Managed templates with variable substitution
Sandboxes	Docker execution environments for code running and testing

Anyone can publish components to the registry. Admin review controls visibility in the public listing, but your own items are usable immediately without approval. Browse the web UI or CLI to discover agents and components shared by others.

CLI Reference

The CLI is organized into command groups. Run observal --help or observal <group> --help for full details.

Primary Workflows

observal pull <agent> --ide <ide>    # install a complete agent with all dependencies
observal scan [--ide <ide>]          # detect and instrument existing IDE configs
observal use <git-url|path>          # swap IDE configs to a git-hosted profile
observal profile                     # show active profile and backup info

Authentication — observal auth

observal auth login            # auto-creates admin on fresh server, or login with key
observal auth logout           # clear saved credentials
observal auth whoami           # show current user
observal auth status           # check server connectivity and health
observal auth reset-password   # reset a forgotten password (uses server-logged code)

Forgot your password? If you've lost access to an account (e.g. an admin account created before passwords were set up), use the reset flow:

observal auth reset-password --email admin@localhost

This requests a 6-character reset code that gets logged to the server console. Check the server logs (make logs or docker logs <container>) for a line like:

WARNING - PASSWORD RESET CODE for admin@localhost: A7X9B2 (expires in 15 minutes)

Enter the code and your new password to regain access. The same flow is available from the web UI via the "Forgot password?" link on the login page.

For CI/scripts, use environment variables:

export OBSERVAL_SERVER_URL=http://localhost:8000
export OBSERVAL_API_KEY=<your-key>

Component Registry — observal registry <type>

All 5 component types (mcp, skill, hook, prompt, sandbox) support the same core commands:

observal registry <type> submit [<git-url> | --from-file <path>]
observal registry <type> list [--search <term>]
observal registry <type> show <id-or-name>
observal registry <type> install <id-or-name> --ide <ide>
observal registry <type> delete <id-or-name>

Prompts also have observal registry prompt render <id> --var key=value.

Agent Authoring — observal agent

# Browse and manage
observal agent create              # interactive agent creation
observal agent list [--search <term>]
observal agent show <id>
observal agent install <id> --ide <ide>
observal agent delete <id>

# Local YAML workflow
observal agent init                # scaffold observal-agent.yaml
observal agent add <type> <id>     # add component (mcp, skill, hook, prompt, sandbox)
observal agent build               # validate against server (dry-run)
observal agent publish             # submit to registry

Observability — observal ops

observal ops overview              # dashboard stats
observal ops metrics <id> [--type mcp|agent] [--watch]
observal ops top [--type mcp|agent]
observal ops traces [--type <type>] [--mcp <id>] [--agent <id>]
observal ops spans <trace-id>
observal ops rate <id> --stars 5 [--type mcp|agent] [--comment "..."]
observal ops feedback <id> [--type mcp|agent]
observal ops telemetry status
observal ops telemetry test

Admin — observal admin

# Invite team members
observal admin invite              # generate invite code (e.g. OBS-A7X9B2)
observal admin invites             # list all invite codes

# Settings and users
observal admin settings
observal admin set <key> <value>
observal admin users

# Review workflow
observal admin review list
observal admin review show <id>
observal admin review approve <id>
observal admin review reject <id> --reason "..."

# Evaluation engine
observal admin eval run <agent-id> [--trace <trace-id>]
observal admin eval scorecards <agent-id> [--version "1.0.0"]
observal admin eval show <scorecard-id>
observal admin eval compare <agent-id> --a "1.0.0" --b "2.0.0"
observal admin eval aggregate <agent-id> [--window 50]

# Penalty and weight tuning
observal admin penalties
observal admin penalty-set <name> [--amount 10] [--active]
observal admin weights
observal admin weight-set <dimension> <weight>

Configuration — observal config

observal config show               # show current config
observal config set <key> <value>  # set a config value
observal config path               # show config file path
observal config alias <name> <id>  # create @alias for an ID
observal config aliases            # list all aliases

Self-Management & Diagnostics

observal self upgrade              # upgrade CLI to latest version
observal self downgrade            # downgrade to previous version
observal doctor [--ide <ide>] [--fix]  # diagnose IDE settings compatibility

Tech Stack

Component	Technology
Frontend	Next.js 16, React 19, Tailwind CSS 4, shadcn/ui, Recharts
Backend API	Python, FastAPI, Uvicorn
Database	PostgreSQL 16 (primary), ClickHouse (telemetry)
ORM	SQLAlchemy (async) + AsyncPG
CLI	Python, Typer, Rich
Eval Engine	AWS Bedrock / OpenAI-compatible LLMs
Background Jobs	arq + Redis
Real-time	GraphQL subscriptions (Strawberry + WebSocket)
Dependency Management	uv
Telemetry Pipeline	OpenTelemetry Collector
Deployment	Docker Compose (7 services)

Setup & Configuration

For detailed setup, eval engine configuration, environment variables, and troubleshooting, see SETUP.md.

API Endpoints

Auth

Method	Endpoint	Description
`POST`	`/api/v1/auth/bootstrap`	Auto-create admin on fresh server
`POST`	`/api/v1/auth/login`	Login with API key or email+password
`GET`	`/api/v1/auth/whoami`	Current user info
`POST`	`/api/v1/auth/request-reset`	Request password reset (code logged to server console)
`POST`	`/api/v1/auth/reset-password`	Reset password with code + new password
`POST`	`/api/v1/auth/invite`	Create invite code (admin)
`POST`	`/api/v1/auth/redeem`	Redeem invite code → get API key
`GET`	`/api/v1/auth/invites`	List invite codes (admin)

Registry (per type: mcps, agents, skills, hooks, prompts, sandboxes)

All {id} parameters accept either a UUID or a name.

Method	Endpoint	Description
`POST`	`/api/v1/{type}`	Submit / create
`GET`	`/api/v1/{type}`	List approved items
`GET`	`/api/v1/{type}/{id}`	Get details
`POST`	`/api/v1/{type}/{id}/install`	Get IDE config snippet
`DELETE`	`/api/v1/{type}/{id}`	Delete
`GET`	`/api/v1/{type}/{id}/metrics`	Metrics
`POST`	`/api/v1/agents/{id}/pull`	Pull agent (installs all components)

Scan

Method	Endpoint	Description
`POST`	`/api/v1/scan`	Bulk register items from IDE config scan

Review

Method	Endpoint	Description
`GET`	`/api/v1/review`	List pending submissions
`GET`	`/api/v1/review/{id}`	Submission details
`POST`	`/api/v1/review/{id}/approve`	Approve
`POST`	`/api/v1/review/{id}/reject`	Reject

Telemetry

Method	Endpoint	Description
`POST`	`/api/v1/telemetry/ingest`	Batch ingest traces, spans, scores
`POST`	`/api/v1/telemetry/events`	Legacy event ingestion
`GET`	`/api/v1/telemetry/status`	Data flow status

Evaluation

Method	Endpoint	Description
`POST`	`/api/v1/eval/agents/{id}`	Run evaluation
`GET`	`/api/v1/eval/agents/{id}/scorecards`	List scorecards
`GET`	`/api/v1/eval/scorecards/{id}`	Scorecard details
`GET`	`/api/v1/eval/agents/{id}/compare`	Compare versions
`GET`	`/api/v1/eval/agents/{id}/aggregate`	Aggregate scoring stats

Feedback

Method	Endpoint	Description
`POST`	`/api/v1/feedback`	Submit rating
`GET`	`/api/v1/feedback/{type}/{id}`	Get feedback
`GET`	`/api/v1/feedback/summary/{id}`	Rating summary

Admin

Method	Endpoint	Description
`GET`	`/api/v1/admin/settings`	List settings
`PUT`	`/api/v1/admin/settings/{key}`	Set a value
`GET`	`/api/v1/admin/users`	List users
`POST`	`/api/v1/admin/users`	Create user
`PUT`	`/api/v1/admin/users/{id}/role`	Change role
`PUT`	`/api/v1/admin/users/{id}/password`	Reset user password (admin)
`GET`	`/api/v1/admin/penalties`	List penalty catalog
`PUT`	`/api/v1/admin/penalties/{id}`	Modify penalty
`GET`	`/api/v1/admin/weights`	Get dimension weights
`PUT`	`/api/v1/admin/weights`	Set dimension weights

GraphQL

Endpoint	Description
`/api/v1/graphql`	Traces, spans, scores, metrics (query + subscription)

Health

Method	Endpoint	Description
`GET`	`/health`	Health check

Environment Variables

Variable	Required	Default	Description
`DATABASE_URL`	Yes		PostgreSQL connection string (asyncpg)
`CLICKHOUSE_URL`	Yes		ClickHouse connection string
`POSTGRES_USER`	Yes	`postgres`	PostgreSQL user
`POSTGRES_PASSWORD`	Yes	`postgres`	PostgreSQL password
`SECRET_KEY`	Yes		Secret key for API key hashing. Generate with: `python3 -c "import secrets; print(secrets.token_urlsafe(32))"`
`CLICKHOUSE_USER`	No	`default`	ClickHouse user
`CLICKHOUSE_PASSWORD`	No	`clickhouse`	ClickHouse password
`EVAL_MODEL_URL`	No		OpenAI-compatible endpoint for the eval engine
`EVAL_MODEL_API_KEY`	No		API key for the eval model
`EVAL_MODEL_NAME`	No		Model name (e.g. `us.anthropic.claude-3-5-haiku-20241022-v1:0`)
`EVAL_MODEL_PROVIDER`	No		`bedrock`, `openai`, or empty for auto-detect
`AWS_ACCESS_KEY_ID`	No		AWS credentials for Bedrock
`AWS_SECRET_ACCESS_KEY`	No		AWS credentials for Bedrock
`AWS_SESSION_TOKEN`	No		AWS session token (temporary credentials)
`AWS_REGION`	No	`us-east-1`	AWS region for Bedrock

Running Tests

make test      # quick (526 tests)
make test-v    # verbose

All tests mock external services. No Docker needed.

Community

Have a question, idea, or want to share what you've built? Head to GitHub Discussions. Please use Discussions for questions instead of opening issues. Issues are reserved for bug reports and feature requests.

Security

To report a vulnerability, please use GitHub Private Vulnerability Reporting or email [email protected]. Do not open a public issue. See SECURITY.md for full details.

Contributing

See CONTRIBUTING.md for the full guide. The short version:

Fork and clone
make hooks to install pre-commit hooks
Create a feature branch
Make changes, run make lint and make test
Open a PR

See AGENTS.md for internal codebase context useful when working with AI coding agents.

Star History

If you find Observal useful, please star the repo. It helps others discover the project and motivates continued development.

License

Apache License 2.0. See LICENSE.