VideoDB Record & Replay

Record desktop workflows once. Replay them anywhere.

Record → Compile → Replay

Install · Features · How It Works · Docs · Report Bug

What is Record & Replay?

An MCP server that gives AI agents the ability to watch, learn, and replay human desktop workflows.

• Record — Captures every click, keystroke, and UI element through native accessibility APIs while simultaneously recording screen video to VideoDB for visual reference.
• Compile — An LLM transforms the event log and scene descriptions into reusable SKILL.json and human-readable SKILL.md files.
• Replay — Agents play back skills with variable substitution across macOS, Windows, and Linux.

Demonstrate a task once on screen, and the server produces a self-contained, versioned, agent-executable skill.

Installation

Prerequisites

• Python 3.10+
• uv package manager
• A VideoDB API key (free)

1. Clone and install

git clone https://github.com/video-db/open-record-replay.git
cd open-record-replay
uv sync

2. Set your API key

Create a .env file in the project root:

VIDEODB_API_KEY=sk-your_api_key_here

3. Configure your MCP client

Add to your MCP config (claude_desktop_config.json, VS Code MCP settings, etc.):

{
  "mcpServers": {
    "videodb-record-replay": {
      "command": "uv",
      "args": ["run", "python", "server.py"],
      "cwd": "/path/to/open-record-replay"
    }
  }
}

4. Restart your client

Five tools and two resources will appear. You're ready to record.

macOS — Requires Screen Recording, Microphone, Accessibility, and Input Monitoring permissions. Run the permission helper first:

uv run python scripts/smoke_macos_hook.py --prompt-permissions

If ready_for_event_recording is false, enable the terminal process in System Settings > Privacy & Security > Accessibility and Input Monitoring, then rerun.

Windows — Uses UI Automation. No additional setup required beyond the standard install.

Linux — Uses AT-SPI. Ensure at-spi2-core is installed and your desktop environment has accessibility enabled.

Usage

Recording is human-in-the-loop. The agent starts recording, announces that recording is active, then waits. The human operator performs the UI workflow being captured.

record_skill_tool("my-workflow", lead_in_seconds=5)
    → Agent tells the operator recording is active
    → Operator switches to the target app and performs the workflow
    → Operator returns to the MCP client and says "stop"
    → Agent calls stop_recording_tool(trim_end_seconds=10)
    → Agent calls compile_skill_tool(video_id, "my-workflow")
    → Agent verifies/reports global_skill_md_path for future use

The recorder starts capture immediately, then the compiler ignores events before the effective workflow start. With lead_in_seconds=5, the operator can switch from the MCP client to the target app and should begin the demonstrated workflow after 5 seconds.

Discards events at the tail of the recording. Use when the operator must switch back to the MCP client to say "stop". For example, trim_end_seconds=10 ignores the final 10 seconds so the generated skill does not include the operator returning to the terminal or chat window.

If VideoDB screen capture is unavailable, the system falls back to recording AX events only. Call compile_skill_tool with video_id="" or video_id="none" to compile from events alone.

How It Works

Human performs workflow
        │
        ▼
┌──────────────────────────────────────────────────────────────────┐
│  AX hooks ──► events.jsonl      (deterministic action log)       │
│  Capture SDK ──► video_id       (visual reference)               │
└──────────────────────────────────────────────────────────────────┘
        │
        ▼
┌──────────────────────────────────────────────────────────────────┐
│  Compiler (LLM)                                                  │
│  events.jsonl + matched scene descriptions ──► SKILL.json        │
│  SKILL.json ──► SKILL.md        (agent-readable)                 │
└──────────────────────────────────────────────────────────────────┘
        │
        ▼
┌──────────────────────────────────────────────────────────────────┐
│  Replay                                                          │
│  Read SKILL.json → substitute {{variables}} → native execution  │
│  Self-healing: AX re-lookup → frame delta → semantic search      │
└──────────────────────────────────────────────────────────────────┘

The camera records your screen. The accessibility hooks capture the deterministic truth. Together, they produce skills that are both precise and visually verifiable.

Features

Tools

Resources

Skill Output

Compiled skills land in ~/.mcp-videodb/skills/<name>/:

Every generated SKILL.json includes an execution_strategy — web_browser, desktop_app, hybrid, terminal, file_system, or unknown — so the replaying agent knows which tool path to use. Every SKILL.md includes an execution guidance section and a continuous improvement section.

After SKILL.md is created, compile_skill_tool also installs it into the
agent's global skills directory, ~/.codex/skills/<name>/SKILL.md by default,
and returns global_skill_md_path plus an agent_instruction reminding the
agent to verify or report the global install. Agents should ensure this global
install step has happened so the skill is available in future runs. Set
CODEX_HOME to change the base Codex directory, or set
AGENT_GLOBAL_SKILLS_ROOT to override the global skills directory directly.

Architecture

open-record-replay/
├── server.py                 # FastMCP entry point, tool and resource definitions
├── state.py                  # Shared server state singleton
├── config.py                 # Constants, .env loading
├── registry.py               # Skill CRUD + versioning
│
├── capture/
│   ├── recorder.py           # Records AX events + VideoDB capture simultaneously
│   ├── ax_client.py          # JSONL IPC wrapper for native AX companion
│   ├── capture_client.py     # VideoDB Capture SDK wrapper
│   └── native/
│       ├── ax_hook_win32.py   # Windows: UI Automation + keyboard polling + TCP IPC
│       ├── ax_hook_darwin.py  # macOS: Accessibility API + pynput + pipe IPC
│       └── ax_hook_linux.py   # Linux: AT-SPI + pynput + pipe IPC
│
├── compiler/
│   ├── compiler.py           # LLM compilation: index scenes → match events → prompt → normalize
│   ├── prompts.py            # LLM system prompt for structured skill generation
│   ├── md_generator.py       # Converts SKILL.json to agent-readable SKILL.md
│   ├── tool_manifest.py      # Surface-to-tool mapping for replay guidance
│   └── recommended_tools.json
│
├── schema/
│   └── skill.schema.json     # JSON Schema (draft-07) for SKILL.json validation
│
├── scripts/
│   └── smoke_macos_hook.py   # macOS AX hook permission helper

Troubleshooting

• Verify VIDEODB_API_KEY is set in .env and is valid
• Run request_capture_permissions_tool and approve any permission prompts
• On macOS, check Screen Recording and Accessibility permissions
• Check that no other application is using the accessibility hook

• Ensure the recording has meaningful UI interactions (not just idle time)
• Try events-only compilation (video_id="") if video indexing is slow
• The LLM may need a retry — compilation automatically retries up to 2 times

# Reset permissions and try again
uv run python scripts/smoke_macos_hook.py --prompt-permissions

If ready_for_event_recording is false, manually enable the terminal in System Settings > Privacy & Security > Accessibility and Input Monitoring.

• Ensure the app being recorded has UI Automation support (most modern apps do)

Community & Support

• Docs: docs.videodb.io
• Issues: GitHub Issues
• Discord: Join the VideoDB community
• API Key: console.videodb.io

Made with ❤️ by the VideoDB team