Skillget

Observal

agent
Security Audit
Fail
Health Pass
  • License — License: Apache-2.0
  • Description — Repository has a description
  • Active repo — Last push 0 days ago
  • Community trust — 250 GitHub stars
Code Fail
  • rm -rf — Recursive force deletion command in demo/run_demo.sh
Permissions Pass
  • Permissions — No dangerous permissions requested
Purpose
This project is an open-source, self-hosted registry and observability platform for AI coding agents. It allows developers to package, share, and monitor AI agents and their components (like MCP servers and hooks) across various coding environments.

Security Assessment
The overall risk is rated as Medium. As an observability platform, the tool inherently accesses and processes telemetry data, which could include sensitive code interactions and session logs. It requires a local Docker setup and database, and explicitly makes network requests to manage agent configurations and reporting. No hardcoded secrets or dangerous permission requests were found in the main application code. However, the automated rule-based scan flagged a `rm -rf` recursive force deletion command inside a demonstration script (`demo/run_demo.sh`). While common in scripts that tear down temporary environments, unchecked recursive deletion commands are an inherent risk and should be reviewed carefully before execution.

Quality Assessment
The project demonstrates strong health and active maintenance. It is licensed under the permissive Apache-2.0 license, which is excellent for open-source adoption. Activity is very high, with the latest repository pushes occurring just today. It also possesses a solid foundation of community trust, currently backed by 250 GitHub stars. Keep in mind that the repository explicitly labels its current status as "alpha," meaning you should anticipate potential breaking changes or bugs as development continues.

Verdict
Use with caution: the active development and transparent licensing make it a promising tool, but its alpha status and the presence of a force deletion command in the demo scripts warrant careful review before deploying in sensitive environments.
SUMMARY

Observal is an observability platform and local registry for MCPs, hooks, skills, graphRAGs and more!

README.md
Observal

Discover, share, and monitor AI coding agents with full observability built in.

License Python Status Stars

If you find Observal useful, please consider giving it a star. It helps others discover the project and keeps development going.

Observal is a self-hosted AI agent registry with built-in observability. Think Docker Hub, but for AI coding agents.

Browse agents created by others, publish your own, and pull complete agent configurations, all defined in a portable YAML format that templates out to Claude Code, Codex CLI, Gemini CLI, and more. Every agent bundles its MCP servers, skills, hooks, prompts, and sandboxes into a single installable package. One command to install, zero manual config.

Every interaction generates traces, spans, and sessions that flow into a telemetry pipeline, giving you full observability, traceability, and real-time metrics for your agents in production. The built-in eval engine (WIP) scores agent sessions so you can measure performance and make your agents better over time.

Supported tools: Claude Code, Codex CLI, Gemini CLI, and Kiro CLI are fully supported. Cursor and VS Code have MCP/rules file support.

See the Changelog for recent updates.

Quick Start

git clone https://github.com/BlazeUp-AI/Observal.git
cd Observal
cp .env.example .env          # edit with your values

cd docker && docker compose up --build -d && cd ..
uv tool install --editable .
observal auth login            # auto-creates admin on fresh server

Already have MCP servers in your IDE? Instrument them in one command:

observal scan                  # auto-detect, register, and instrument everything
observal pull <agent> --ide cursor  # install a complete agent

This detects MCP servers from your IDE config files, registers them with Observal, and wraps them with observal-shim for telemetry without breaking your existing setup. A timestamped backup is created automatically.

The Problem

AI coding agents today are hard to share and impossible to measure. Components (MCP servers, skills, hooks, prompts) are scattered across repos with no standard way to package them together. There's no visibility into what's actually working, and no way to compare one version of an agent against another on real workflows.

Observal solves this by giving you a registry to package and distribute complete agents, and a telemetry pipeline to measure them.

How It Works

Agents in the registry are defined in YAML. Each agent bundles its components (MCP servers, skills, hooks, prompts, sandboxes) into a single configuration. When you run observal pull <agent>, it installs everything and generates the right config files for your tool.

A transparent shim (observal-shim for stdio, observal-proxy for HTTP) sits between your tool and the MCP server. It never modifies traffic, it only observes. Every request/response pair becomes a span, spans group into traces, and traces form sessions. All of this streams into ClickHouse for analysis.

Tool  <-->  observal-shim  <-->  MCP Server / Sandbox
                |
                v (fire-and-forget)
          Observal API  -->  ClickHouse (traces, spans, scores)
                |
                v
          Eval Engine (SLM-as-a-Judge / Deductive Penalty Scoring)  -->  Scorecards

The eval engine runs on collected traces after the fact. It scores agent sessions across five dimensions: Goal Completion, Tool Call Efficiency, Tool Call Failures, Factual Grounding, and Thought Process. Scorecards let you compare agent versions, identify bottlenecks, and track improvements over time.

The Registry

Observal manages 6 component types that agents bundle together:

Component Description
Agents Complete configurations that bundle all the components below
MCP Servers Model Context Protocol servers that expose tools to agents
Skills Portable instruction packages that agents load on demand
Hooks Lifecycle callbacks that fire during agent sessions
Prompts Managed templates with variable substitution
Sandboxes Docker execution environments for code running and testing

Anyone can publish components to the registry. Admin review controls visibility in the public listing, but your own items are usable immediately without approval. Browse the web UI or CLI to discover agents and components shared by others.

CLI Reference

The CLI is organized into command groups. Run observal --help or observal <group> --help for full details.

Primary Workflows

observal pull <agent> --ide <ide>    # install a complete agent with all dependencies
observal scan [--ide <ide>]          # detect and instrument existing IDE configs
observal use <git-url|path>          # swap IDE configs to a git-hosted profile
observal profile                     # show active profile and backup info
Authentication — observal auth 
observal auth login            # auto-creates admin on fresh server, or login with key
observal auth logout           # clear saved credentials
observal auth whoami           # show current user
observal auth status           # check server connectivity and health
observal auth reset-password   # reset a forgotten password (uses server-logged code)

Forgot your password? If you've lost access to an account (e.g. an admin account created before passwords were set up), use the reset flow:

observal auth reset-password --email admin@localhost

This requests a 6-character reset code that gets logged to the server console. Check the server logs (make logs or docker logs <container>) for a line like:

WARNING - PASSWORD RESET CODE for admin@localhost: A7X9B2 (expires in 15 minutes)

Enter the code and your new password to regain access. The same flow is available from the web UI via the "Forgot password?" link on the login page.

For CI/scripts, use environment variables:

export OBSERVAL_SERVER_URL=http://localhost:8000
export OBSERVAL_API_KEY=<your-key>
Component Registry — observal registry <type>

All 5 component types (mcp, skill, hook, prompt, sandbox) support the same core commands:

observal registry <type> submit [<git-url> | --from-file <path>]
observal registry <type> list [--search <term>]
observal registry <type> show <id-or-name>
observal registry <type> install <id-or-name> --ide <ide>
observal registry <type> delete <id-or-name>

Prompts also have observal registry prompt render <id> --var key=value.

Agent Authoring — observal agent 
# Browse and manage
observal agent create              # interactive agent creation
observal agent list [--search <term>]
observal agent show <id>
observal agent install <id> --ide <ide>
observal agent delete <id>

# Local YAML workflow
observal agent init                # scaffold observal-agent.yaml
observal agent add <type> <id>     # add component (mcp, skill, hook, prompt, sandbox)
observal agent build               # validate against server (dry-run)
observal agent publish             # submit to registry
Observability — observal ops 
observal ops overview              # dashboard stats
observal ops metrics <id> [--type mcp|agent] [--watch]
observal ops top [--type mcp|agent]
observal ops traces [--type <type>] [--mcp <id>] [--agent <id>]
observal ops spans <trace-id>
observal ops rate <id> --stars 5 [--type mcp|agent] [--comment "..."]
observal ops feedback <id> [--type mcp|agent]
observal ops telemetry status
observal ops telemetry test
Admin — observal admin 
# Invite team members
observal admin invite              # generate invite code (e.g. OBS-A7X9B2)
observal admin invites             # list all invite codes

# Settings and users
observal admin settings
observal admin set <key> <value>
observal admin users

# Review workflow
observal admin review list
observal admin review show <id>
observal admin review approve <id>
observal admin review reject <id> --reason "..."

# Evaluation engine
observal admin eval run <agent-id> [--trace <trace-id>]
observal admin eval scorecards <agent-id> [--version "1.0.0"]
observal admin eval show <scorecard-id>
observal admin eval compare <agent-id> --a "1.0.0" --b "2.0.0"
observal admin eval aggregate <agent-id> [--window 50]

# Penalty and weight tuning
observal admin penalties
observal admin penalty-set <name> [--amount 10] [--active]
observal admin weights
observal admin weight-set <dimension> <weight>
Configuration — observal config 
observal config show               # show current config
observal config set <key> <value>  # set a config value
observal config path               # show config file path
observal config alias <name> <id>  # create @alias for an ID
observal config aliases            # list all aliases
Self-Management & Diagnostics 
observal self upgrade              # upgrade CLI to latest version
observal self downgrade            # downgrade to previous version
observal doctor [--ide <ide>] [--fix]  # diagnose IDE settings compatibility

Tech Stack

Component Technology
Frontend Next.js 16, React 19, Tailwind CSS 4, shadcn/ui, Recharts
Backend API Python, FastAPI, Uvicorn
Database PostgreSQL 16 (primary), ClickHouse (telemetry)
ORM SQLAlchemy (async) + AsyncPG
CLI Python, Typer, Rich
Eval Engine AWS Bedrock / OpenAI-compatible LLMs
Background Jobs arq + Redis
Real-time GraphQL subscriptions (Strawberry + WebSocket)
Dependency Management uv
Telemetry Pipeline OpenTelemetry Collector
Deployment Docker Compose (7 services)

Setup & Configuration

For detailed setup, eval engine configuration, environment variables, and troubleshooting, see SETUP.md.

API Endpoints

Auth

Method Endpoint Description
POST /api/v1/auth/bootstrap Auto-create admin on fresh server
POST /api/v1/auth/login Login with API key or email+password
GET /api/v1/auth/whoami Current user info
POST /api/v1/auth/request-reset Request password reset (code logged to server console)
POST /api/v1/auth/reset-password Reset password with code + new password
POST /api/v1/auth/invite Create invite code (admin)
POST /api/v1/auth/redeem Redeem invite code → get API key
GET /api/v1/auth/invites List invite codes (admin)

Registry (per type: mcps, agents, skills, hooks, prompts, sandboxes)

All {id} parameters accept either a UUID or a name.

Method Endpoint Description
POST /api/v1/{type} Submit / create
GET /api/v1/{type} List approved items
GET /api/v1/{type}/{id} Get details
POST /api/v1/{type}/{id}/install Get IDE config snippet
DELETE /api/v1/{type}/{id} Delete
GET /api/v1/{type}/{id}/metrics Metrics
POST /api/v1/agents/{id}/pull Pull agent (installs all components)

Scan

Method Endpoint Description
POST /api/v1/scan Bulk register items from IDE config scan

Review

Method Endpoint Description
GET /api/v1/review List pending submissions
GET /api/v1/review/{id} Submission details
POST /api/v1/review/{id}/approve Approve
POST /api/v1/review/{id}/reject Reject

Telemetry

Method Endpoint Description
POST /api/v1/telemetry/ingest Batch ingest traces, spans, scores
POST /api/v1/telemetry/events Legacy event ingestion
GET /api/v1/telemetry/status Data flow status

Evaluation

Method Endpoint Description
POST /api/v1/eval/agents/{id} Run evaluation
GET /api/v1/eval/agents/{id}/scorecards List scorecards
GET /api/v1/eval/scorecards/{id} Scorecard details
GET /api/v1/eval/agents/{id}/compare Compare versions
GET /api/v1/eval/agents/{id}/aggregate Aggregate scoring stats

Feedback

Method Endpoint Description
POST /api/v1/feedback Submit rating
GET /api/v1/feedback/{type}/{id} Get feedback
GET /api/v1/feedback/summary/{id} Rating summary

Admin

Method Endpoint Description
GET /api/v1/admin/settings List settings
PUT /api/v1/admin/settings/{key} Set a value
GET /api/v1/admin/users List users
POST /api/v1/admin/users Create user
PUT /api/v1/admin/users/{id}/role Change role
PUT /api/v1/admin/users/{id}/password Reset user password (admin)
GET /api/v1/admin/penalties List penalty catalog
PUT /api/v1/admin/penalties/{id} Modify penalty
GET /api/v1/admin/weights Get dimension weights
PUT /api/v1/admin/weights Set dimension weights

GraphQL

Endpoint Description
/api/v1/graphql Traces, spans, scores, metrics (query + subscription)

Health

Method Endpoint Description
GET /health Health check
Environment Variables
Variable Required Default Description
DATABASE_URL Yes PostgreSQL connection string (asyncpg)
CLICKHOUSE_URL Yes ClickHouse connection string
POSTGRES_USER Yes postgres PostgreSQL user
POSTGRES_PASSWORD Yes postgres PostgreSQL password
SECRET_KEY Yes Secret key for API key hashing. Generate with: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
CLICKHOUSE_USER No default ClickHouse user
CLICKHOUSE_PASSWORD No clickhouse ClickHouse password
EVAL_MODEL_URL No OpenAI-compatible endpoint for the eval engine
EVAL_MODEL_API_KEY No API key for the eval model
EVAL_MODEL_NAME No Model name (e.g. us.anthropic.claude-3-5-haiku-20241022-v1:0)
EVAL_MODEL_PROVIDER No bedrock, openai, or empty for auto-detect
AWS_ACCESS_KEY_ID No AWS credentials for Bedrock
AWS_SECRET_ACCESS_KEY No AWS credentials for Bedrock
AWS_SESSION_TOKEN No AWS session token (temporary credentials)
AWS_REGION No us-east-1 AWS region for Bedrock

Running Tests

make test      # quick (526 tests)
make test-v    # verbose

All tests mock external services. No Docker needed.

Community

Have a question, idea, or want to share what you've built? Head to GitHub Discussions. Please use Discussions for questions instead of opening issues. Issues are reserved for bug reports and feature requests.

Security

To report a vulnerability, please use GitHub Private Vulnerability Reporting or email [email protected]. Do not open a public issue. See SECURITY.md for full details.

Contributing

See CONTRIBUTING.md for the full guide. The short version:

  1. Fork and clone
  2. make hooks to install pre-commit hooks
  3. Create a feature branch
  4. Make changes, run make lint and make test
  5. Open a PR

See AGENTS.md for internal codebase context useful when working with AI coding agents.

Star History

If you find Observal useful, please star the repo. It helps others discover the project and motivates continued development.

Star History Chart

License

Apache License 2.0. See LICENSE.

Reviews (0)

No results found