web-qa

Autonomous web-app QA skill for Claude Code —
explore once, generate specs, run with zero tokens.

Autonomous web-app QA skill for Claude Code: Playwright E2E, visual regression, axe-core accessibility — with test scenarios auto-generated from a git diff or a plain-text task, specs grounded in a real crawled map of your app, and a self-healing loop for failing tests.

One skill, many projects: each project keeps its own .web-qa/ state (scenarios, specs, baselines, reports) inside its repo; the skill itself stays global.

Why

When an AI writes the code, you become the QA. Letting the same AI test its own work is a trap: a model that misread the requirement will happily write a test that passes on the misread — green check, bug intact.

web-qa splits the jobs: the model generates the checks, a deterministic oracle — pixel diff, exit code, playwright test — delivers the verdict. In between: a markdown test plan you review in minutes.

How it works

LLM-driven browser testing is usually done by driving a browser through MCP step by step — powerful, but token-hungry. This skill takes a cheaper route:

1. Explore — a plain Playwright crawler logs in and maps your app once: routes, forms, real button labels, table headers, OpenAPI request schemas → app.context.md (cached in your repo). Routes declared in your source (Next/Nuxt/SvelteKit file routers, router configs) are mined statically and crawled even when no link points at them; an opt-in --interactive pass clicks through runtime-only navigation with all non-GET requests blocked at the network level.
2. Generate — test cases (markdown) are produced from a git diff or a task description, grounded in that map.
3. Automate — each test case becomes a self-contained .spec.ts via a single claude -p call. The prompt embeds the app map, so selectors come from real labels, not guesses. Every spec is validated twice before it's accepted: playwright test --list (does it parse?) and a live locator probe — the generated locators are counted against the running app, and misses come back as a regeneration retry with real-DOM feedback.
4. Run — plain npx playwright test + axe-core injection + pixel-diff visual regression + network assertions (a 5xx fired during a page's navigation fails its test). No LLM in the loop at run time: running your suite costs zero tokens. Visual regressions produce a baseline/current/diff-mask image triple and are accepted per route, not all-or-nothing.
5. Maintain — a failing spec plus its real error output and Playwright's ARIA snapshot of the failure moment goes back to the LLM, which returns a corrected spec. Proposed by default (*.spec.ts.proposed); --apply to overwrite with backup and automatic rollback if the fix doesn't parse.

Plus a deploy gate: web-qa-matrix inventories all tests in a project (scenario TCs + specs), seeds deterministic data via your fixture_cmd, runs everything, reports route coverage and flaky tests, and returns exit 0/1 — drop it in front of your deploy script. It speaks CI natively: --junit exports the matrix as JUnit XML, and inside GitHub Actions the report lands on the run page via $GITHUB_STEP_SUMMARY automatically.

web-qa-matrix --alias my-app && ./deploy.sh

Want to see the output before installing? web-qa-demo runs the full pipeline against a real RealWorld stack — the crawled map, generated scenarios and specs, the deploy-gate matrix, and the real bugs it found in upstream code are all committed there.

Requirements

• Linux/macOS, Python 3.11+, Node.js 18+, uv
• Claude Code CLI on PATH (claude) — used for generation/healing only
• A web app running locally (or reachable) with login credentials

Install

As a Claude Code plugin (from inside Claude Code):

/plugin marketplace add c-c0rtex/web-qa
/plugin install web-qa@c-c0rtex

Then ask your agent to "set up web-qa" — on first use it runs install.sh (uv sync + chromium) for you. Plugin installs keep the project registry in ~/.config/web-qa/projects.json, so it survives plugin updates.

Or as a plain skill:

git clone https://github.com/c-c0rtex/web-qa ~/.claude/skills/web-qa
cd ~/.claude/skills/web-qa
./install.sh          # uv sync + chromium + projects.json from example

Onboard a project:

bin/web-qa-register-project my-app --target-url http://127.0.0.1:3000 --backend-url http://127.0.0.1:8000
# put credentials into projects.json → "auth": {"email": ..., "password": ...}
bin/web-qa-explore --alias my-app          # crawl → <project>/.web-qa/app.context.md

Set up the per-project specs runner (once — see skills/web-qa/SKILL.md, "Per-project specs runner setup").

Talking to your agent

web-qa is a skill: normally you don't type the CLI yourself — you ask your coding agent, and it drives the pipeline. Prompts that work well:

The agent-facing contract (which command for which intent, business rules, config keys) lives in SKILL.md — that's what your agent reads when the skill activates.

Everyday use (CLI)

RBAC and responsive are matrix dimensions: web-qa-matrix --roles admin,viewer --viewports desktop,mobile runs the passive stage per combination (mobile = real device emulation: touch, UA, DPR — via Playwright device descriptors), reports per-role route coverage, and a device viewport also runs specs under mobile emulation. Per-viewport visual baselines are kept apart (@mobile suffix).

All long runs are durable: the matrix report is rewritten after every stage, Playwright progress streams to a log file, generation failures leave *.FAILED markers. If your agent (or you) kill a run, the partial report is already on disk.

Configuration

Machine-local registry projects.json (aliases, URLs, credentials — never committed) + per-project .web-qa/config.json (stack description, auth flow hint for generation, visual masks/excludes, deploy-gate excludes, test-data prefix…). Both documented in SKILL.md, examples in projects.example.json and config.example.json.

Useful environment variables:

• WEBQA_CLAUDE_MODEL=sonnet|opus — model for internal claude -p calls
• WEBQA_GEN_TIMEOUT=600 — seconds per spec generation (default 300)
• WEBQA_WORKERS=4 — Playwright workers (template config)

Token cost, honestly

• Exploration, running tests, the matrix, visual diff, a11y: zero tokens — plain Python + Playwright.
• Visual regression is an algorithmic pixel diff against a committed baseline — no vision model looks at your screenshots. Screenshots are saved as run artifacts for you (or your agent) to inspect.
• Spec generation: one claude -p call per test case (prompt ≈ app map + TC, roughly 10–20k tokens each), cached — regeneration only happens when the TC, the prompt, or the app map changes. Text-only: generation is grounded in the crawled app map, not in screenshots.
• Healing: one call per failing spec.

A 30-test-case project costs roughly one medium Claude conversation to fully automate, then runs for free.

Roadmap

See ROADMAP.md — next: cross-agent support (WEBQA_LLM_CMD for Codex/Gemini/Kimi CLIs) and flake-aware healing. Items marked help wanted are good first contributions.

Attribution

MIT-licensed. If you use this project or build on it, a link back to github.com/c-c0rtex/web-qa is appreciated — it's the only thing asked for.

Generated tests, reports, and app maps produced by this tool are yours, no strings attached.

Third-party

• axe-core (runners/axe.min.js) — © Deque Systems, MPL-2.0, vendored unmodified.
• Playwright — Apache-2.0, installed as a dependency.

The four-stage pipeline architecture was inspired by Playwright's test agents (Planner / Generator / Healer) and TestDino's writing on AI-driven Playwright testing. No code or text was copied — the ideas were reimplemented from scratch.

License

MIT © c-c0rtex