record-and-replay

Agent-agnostic, open-source record & replay skill. Let the user demonstrate a workflow — in the browser or on the desktop — while an AI coding agent records it, then turns the demonstration into a reusable agent skill.

Inspired by OpenAI Codex's macOS-only Record & Replay plugin, rebuilt on open tooling so it works across platforms and across agents: Claude Code, Codex CLI, opencode, and any agent that can run shell commands and read SKILL.md-style skills.

Platform support

How it works

Browser mode (default — richest evidence): launches a headed browser via Playwright (your installed Google Chrome if available, bundled Chromium otherwise) and records every user action into events.jsonl — each with multiple selector candidates (testId → role+name → id → text → css) — plus Playwright trace chunks (DOM snapshots + screenshots, 60 s checkpoints).

Desktop mode: a single-process recorder (pynput) captures global mouse/keyboard events, active-window/title changes and a half-resolution JPEG at each click into desktop-events.jsonl + screens/ — the same philosophy as Codex's native Record & Replay (an event/context stream, not video) at ~0-3% CPU. On macOS every click also records the accessibility description of the element under the cursor (AXButton "Save"), so generated skills target semantics rather than pixels.

The agent then reads the recording (compact markdown summaries via the bundled summarizers), treats it as evidence of intent — not a pixel-by-pixel script — and generates a reusable skill with semantic locators and verification steps.

Install

git clone https://github.com/ugarchance/record-and-replay-skill ~/.agents/skills/record-and-replay
cd ~/.agents/skills/record-and-replay
node setup.mjs      # npm install + (no Chrome? download Chromium) + browser & desktop self-tests
node install.mjs    # link into agent skill dirs (symlink on macOS/Linux, junction on Windows)

install.mjs links the skill into ~/.claude/skills/, ~/.codex/skills/ and ~/.config/opencode/skills/ for every agent that is installed. For any other agent supporting the SKILL.md format, link the folder into its skill directory.

Desktop mode's Python env (.venv-desktop: pynput+mss+pillow, ~50 MB) is created by setup.mjs automatically when uv is installed.

macOS permissions. Desktop capture needs Accessibility/Input Monitoring and Screen Recording. Without them it records nothing at all, so the recorder preflights and reports them instead of failing quietly:

node scripts/desktop-record.mjs permissions

This opens the system dialog and the exact Settings pane, and reports grantTo — the app that actually owns the grant. A TCC grant binds to the .app bundle hosting your agent (e.g. Claude, Terminal, iTerm), never to the Python binary, and that app must be restarted afterwards. start runs the same flow automatically when a permission is missing (--no-permission-prompt opts out).

Usage

Tell your agent "watch me do this and make it a skill" — the skill file guides it. Manually:

# Browser (macOS/Linux; Windows: Start-Process node …)
nohup node scripts/recorder.mjs --minutes 30 --name my-task &
node scripts/summarize.mjs ~/.agents/recordings/my-task

# Desktop
node scripts/desktop-record.mjs start my-task 30
node scripts/desktop-record.mjs pause  my-task   # closes segment desktop-events-partN.jsonl
node scripts/desktop-record.mjs resume my-task
node scripts/desktop-record.mjs stop   my-task
python3 scripts/desktop-summarize.py ~/.agents/recordings/my-task

Recording controls

• Floating control window (bottom-right, separate from the recorded pages — like Codex's native Recording Controls panel): live timer, Pause/Resume, Stop, and Discard (two-tap confirm; deletes all evidence, session ends as discarded). Reopens itself if closed by accident. Pass --badge to use an in-page badge instead.
• While paused, nothing is captured — events and trace both stop.
• Agent/file control: touch <outDir>/PAUSE (delete to resume), touch <outDir>/STOP. Stays in sync with the window buttons.
• Auto-stop at the time limit; closing the browser window also ends the session (avoid Cmd+Q — the last minute of the trace may be lost; everything else survives thanks to 60 s trace checkpoints).
• Desktop pause/resume is segment-based (suspending the capture process could silently kill the macOS event tap) — the summarizer merges segments into one timeline.

Performance defaults (lightweight by design)

• Browser trace defaults to light: full DOM snapshots per action, no continuous screencast (--trace full re-enables screenshots, --trace off disables tracing entirely).
• Desktop capture is a single process with no continuous screen reading and no video — measured at ~0-3% CPU while recording. (Continuous full-res video capture was tried and rejected: the raw frame pipeline can overwhelm a machine.)
• Nothing heavy runs in an input callback. Screenshots (~33 ms warm, ~220 ms cold) and AX queries (up to ~130 ms) run on a worker thread. On macOS the pynput callback executes on the CGEventTap thread, and a slow callback there can get the tap disabled — which silently ends capture mid-recording with no error anywhere.
• Typing is debounced, held keys skip auto-repeat, scroll/selection are sampled, and the event stream is rate-limited — a 30-minute recording stays in the tens of MB, not GBs.
• start returns only when capture is genuinely live (READY handshake), not when the output file appears — otherwise a cold Python start swallowed the first ~10 seconds of the demonstration.
• Stopping is signal-free on every platform (STOP-file watch) — recordings always finalize cleanly, including on Windows.

Benchmarks & self-tests

node scripts/recorder.mjs --self-test              # browser: end-to-end, headless
.venv-desktop/bin/python scripts/desktop-selftest.py   # desktop: end-to-end, no permissions needed
node bench/browser-overhead.mjs                    # in-page recorder cost
.venv-desktop/bin/python bench/desktop-capture.py  # screenshot / AX / startup cost

Both self-tests run as part of node setup.mjs.

Design notes

• Two-layer evidence (browser): events.jsonl (semantic actions, multi-candidate selectors) + trace-NNN.zip (per-action DOM snapshot & screenshot; open with npx playwright show-trace trace-*.zip).
• Persistent browser profile at ~/.agents/recordings/.browser-profile keeps logins across recordings and replays. ⚠ It holds live session cookies — treat it as plaintext credentials; never commit or share it.
• Sensitive data (browser): password/OTP/card fields (English and Turkish field names) arrive pre-masked; paste content is never logged; secret-looking selections/copies are masked; plain-character keydowns aren't logged at all. Masking is heuristic — trace chunks are not masked; treat recordings as sensitive.
• Desktop recordings have NO masking — every keystroke and screenshot is raw. The summarizer warns the agent explicitly.
• Page-JS hardening: the recording binding is nonce-guarded and removed from window; the event stream is rate-limited and capped (no fake-event injection, no disk-fill).
• One active recording per mode (lock files); restarting with the same name archives the previous evidence as .bak instead of appending.
• Replaying: generated skills use semantic Playwright locators (browser) or an accessibility-tree computer-use layer such as cua-driver (desktop) — never raw coordinates unless there is no better target. Desktop recordings carry an AX target per click, so "no better target" is now the exception rather than the rule on macOS.

Files

License

MIT