claude-qa-agent

DevOps QA Agent

An autonomous QA agent for DevOps-oriented teams. Give it an issue (GitLab, GitHub, Jira, Azure Boards), a staging URL, or a failing CI run — it writes test cases, executes them in a real browser, tests your APIs, runs accessibility checks, and files bugs to GitLab Issues, all without you lifting a finger.

Built with Claude Code · Playwright + playwright-bdd · works with GitLab, GitHub, Jira, Azure DevOps & Bitbucket · MIT licensed

Quick start: Setup · Modes: Seventeen Modes · See the output: examples/ · Help wanted: Roadmap · Contributing

See It in Action

From one user story ("apply a discount code at checkout"), the agent produces: a test plan + runnable Gherkin/Playwright tests, a developer-ready bug for the failure it found, a session report, an MR impact comment, a traceability matrix and a release go/no-go. All on a fictional shop; see examples/.

What Makes This Different

	Typical QA Agent	This Agent
Bug tracking	One tracker, hard-wired	GitLab, GitHub, Jira or Azure Boards: same workflows, your tracker
Test management	Paid SaaS	Markdown files in your repo + issues in your tracker
Browser tests	UI E2E only	UI E2E + accessibility (axe) built into every scenario
API testing	None	`api it` mode — OpenAPI validation + functional tests
CI diagnosis	None	`ci it`: root cause of a failed GitLab / GitHub Actions / Azure / Bitbucket run
After a bug is filed	Nothing	`verify it` runs the bug's regression test and closes or reopens it
Tests in the pipeline	Manual	`ci setup` writes the CI config for GitLab CI, GitHub Actions, Azure Pipelines or Bitbucket Pipelines
Broken selectors	Re-write by hand	`heal it` repairs renamed/moved elements from Playwright's failure snapshot
Flaky tests	File bogus bugs	`flaky` finds them across runs and quarantines them

Works With

Pick where your issues live and where your code + CI live, separately. Plenty of teams use Jira with GitHub or Bitbucket.

Provider	Issues (`QA_TRACKER`)	MRs/PRs + CI (`QA_CODE_HOST`)	Uses
GitLab (.com or self-hosted)	✅	✅ merge requests, pipelines, CI lint	bundled GitLab MCP server
GitHub (.com or Enterprise)	✅	✅ pull requests, Actions runs	`gh` CLI
Jira (Cloud or Data Center)	✅	—	Atlassian MCP server if connected, otherwise the REST API
Azure DevOps	✅ Boards work items	✅ Repos PRs, Pipelines	`az` CLI + REST
Bitbucket Cloud	— (use Jira)	✅ pull requests, Pipelines	REST API

Every WAY calls one provider-neutral contract (the tracker skill). A provider-<name> skill maps it to that provider, including its refs (#143, SHOP-143, !87), body format (Markdown, Jira wiki, Azure HTML), labels vs tags, and attachments. Adding a provider means adding one skill. See CONTRIBUTING.md.

Output Watermark

Every issue, comment, and saved markdown report this agent produces ends with:

🤖 Generated by DevOps QA Agent — review before acting on this.

Every generated .feature/.steps.ts test file starts with the same attribution as a header comment. This makes AI-generated content clearly identifiable to anyone on the team who opens an issue, a report, or a test file — useful once the agent is shared beyond a single user. The exact wording is defined once in CLAUDE.md's Output Watermark section; sub-agent templates in .claude/agents/ include it directly rather than relying on the caller to add it afterward.

Seventeen Modes

Each mode ("WAY") is a skill. Type a trigger phrase, or /<skill> directly (/qa-agent:<skill> with the plugin). Only the triggered WAY's instructions are loaded.

WAY	Skill	Trigger	What happens
0	`qa-setup`	`qa setup`	Prepare a project: test runner, providers, `.env` values, access check
1	`e2e-session`	`e2e it`	Requirements → browser tests (+a11y) → run → heal locator failures → file bugs → session report
2	`api-session`	`api it`	OpenAPI → API tests → run (schema, perf) → file bugs → API report
3	`quick-bug`	`report it`	One-line bug → issue + a regression test
4	`write-tests`	`write it`	Requirements → test files + `test-plan` issue, no run
5	`diagnose-ci`	`ci it`	Failed pipeline → failure type, root cause, concrete fix
6	`enhance-story`	`analyze us`	Score a story → rewrite with Gherkin ACs, scenarios, DoD → QA analysis comment
7	`benchmark`	`benchmark it`	App/URL/repo → sourced competitor matrix (+ Lighthouse)
8	`gap-stories`	`gap it`	Benchmark → leader → gap user stories
9	`verify-fix`	`verify it #142`	Run the bug's regression test → `verified`/`fix-failed`, close/reopen
10	`ci-test-setup`	`ci setup`	Export the suite + CI config for GitLab, GitHub Actions, Azure or Bitbucket
11	`mr-impact`	`test mr !87`	MR diff → only the affected tests → results + coverage gaps on the MR
12	`release-readiness`	`release check v2.3`	Bugs, QA results, pipeline, untested stories → GO / GO WITH RISKS / NO-GO
13	`flaky-triage`	`flaky`	Last N runs → flaky tests → quarantine + `flaky-test` issues
14	`traceability`	`trace it #148`	AC → tests → latest result → open bugs matrix
15	`load-test`	`load test`	k6 smoke/load/stress/spike/soak with p95 + error-rate thresholds (never prod)
16	`heal-tests`	`heal it`	Repair tests broken by UI renames, with approval
17	`integration-tests`	`integration it` / `e2e api it`	One spec → API integration tests; several specs → E2E API tests across services (async effects polled), no run

Test levels

Level	Scope	Entry point	WAY	Tags / project
API (component)	One endpoint of one service	API	2	`@api` / `api`
API integration	Several endpoints/resources of one service	API	17	`@api @integration` / `api`
UI E2E	A user journey through the whole app	Browser	1, 4	`@e2e` / `e2e`
E2E API	A business flow across several services (async effects, per-service auth)	API	17 (several specs)	`@api @e2e-api` / `api`

The identifiers e2e (Playwright project, tag, issue label) and e2e-session keep their names for compatibility. They mean UI E2E.

Tech Stack

Tool	Role
Claude Code (CLI, VS Code, JetBrains or desktop)	Runs the agent. Needs a Claude plan or API key with Claude Code access. Works with any current Claude model
Playwright + playwright-bdd	Real, runnable tests: Gherkin + compiled step definitions, run with `npx playwright test`
@axe-core/playwright	Accessibility check inside every browser scenario
Your providers' tools	GitLab: bundled GitLab MCP. GitHub: `gh`. Jira: Atlassian MCP or REST. Azure DevOps: `az` + `azure-devops` extension. Bitbucket: REST
k6 (optional)	Load testing, WAY 15 only

All tools except Claude are free and open source.

Project Structure

devops-qa-agent/
├── CLAUDE.md                          # Routing table (17 WAYs → skills, one trigger each) + shared rules + watermark policy
├── README.md                          # This file
├── LICENSE · CONTRIBUTING.md · SECURITY.md · CODE_OF_CONDUCT.md · CHANGELOG.md
├── .claude-plugin/                    # plugin.json (manifest, bundled GitLab MCP, install-time settings) + marketplace.json
├── hooks/hooks.json                   # Plugin SessionStart hook → scripts/session-context.mjs prints CLAUDE.md
├── runner-template/                   # package.json + playwright.config.ts meeting the config contract → copy to qa-artifacts/
├── scripts/
│   ├── validate.py                    # Frontmatter, names, CLAUDE.md references + size, triggers, plugin manifest, watermarks (CI)
│   ├── session-context.mjs            # Plugin hook: loads CLAUDE.md rules into each session
│   └── selftest.mjs                   # Runs test-writer's templates end to end on a demo app (CI)
├── evals/                             # `claude plugin eval` cases: routing, readiness gate, spec boundaries, WAY 17 end to end (CI: weekly / `run-evals` label)
├── .github/                           # CI workflows (validate, evals), issue + PR templates
├── .env.example                       # Copy to .env and fill in your values
├── .mcp.example.json                  # Copy to .mcp.json (GitLab MCP only)
├── .gitignore
├── .vscode/
│   └── mcp.json                       # MCP config for the VS Code extension
├── .claude/
│   ├── settings.example.json          # Copy to settings.json and fill in
│   ├── skills/
│   │   ├── qa-setup/ e2e-session/ … integration-tests/   # WAY 0–17 skills, one folder each (see "Sixteen Modes")
│   │   ├── flaky-triage/flaky.mjs     # Run-history aggregator used by WAY 13
│   │   ├── tracker/                   # Shared: provider-neutral issue / MR-PR / CI operations, labels, failures
│   │   ├── provider-gitlab/ provider-github/ provider-jira/ provider-azure-devops/ provider-bitbucket/
│   │   ├── bug-report/                # Shared: severity × priority + bug body + when not to file
│   │   ├── session-report/            # Shared: WAY 1/2 session summary
│   │   └── user-story-format/         # Shared: the one user-story body format
│   └── agents/                        # 9 sub-agents (isolated context)
│       ├── test-writer.md                 # Requirements → Test#<nn> Gherkin + playwright-bdd (UI/API/visual)
│       ├── test-execution-agent.md        # Runs suites, captures artifacts, classifies outcomes
│       ├── test-healer.md                 # Repairs locator failures from error-context snapshots
│       ├── mr-test-impact.md              # MR diff → affected tests + uncovered changes
│       ├── load-test-writer.md            # k6 scripts + smoke run
│       ├── ci-diagnostician.md            # Fetches + diagnoses failed pipelines
│       ├── user-story-enhancer.md         # Scores + rewrites story issues
│       ├── competitor-benchmark.md        # Sourced competitor benchmark
│       └── benchmark-gap-story-writer.md  # Benchmark gaps → user stories
└── qa-artifacts/                      # Your workspace — gitignored (runner-template copied here)
    ├── steps/                         # Shared step definitions + fixtures.ts (failure capture, API exchange)
    ├── test-cases/                    # Test#<nn>.{md,feature,steps.ts} triples (+ regression/bug-<ref>/)
    ├── visual-baselines/              # @visual screenshots (snapshotPathTemplate)
    ├── runs/<RUN_ID>/                 # results.json, junit.xml (CI), artifacts/ (screenshots, traces, error-context)
    ├── api-responses/                 # Every API request/response pair
    ├── load-tests/                    # k6 scripts + summaries
    ├── sessions/ · ci/ · releases/ · traceability/ · benchmarks/

Setup

Prerequisites

Requirement	Version
Claude Code	2.1+ (terminal, VS Code, JetBrains or desktop)
Node.js	20+
Access to your tracker / code host	GitLab token (`api`), `gh auth login`, Jira API token or Atlassian MCP, `az` + PAT, or a Bitbucket API token
Optional: k6 or Docker	For WAY 15

Option A — Install as a plugin (recommended)

Use it in any project, with updates through the marketplace. In Claude Code, from your app's repository:

/plugin marketplace add GhassenAZZOUZ/claude-qa-agent
/plugin install qa-agent@devops-qa-agent

Pick project scope when asked, so it's enabled only in repos where you do QA (an enabled plugin's skill list is in context every turn). If you use GitLab, Claude Code asks for its URL and token (kept in your system's secure credential store). Other providers leave those empty. Then:

qa setup

WAY 0 copies the test runner into qa-artifacts/, installs Playwright, asks which tracker and code host you use, adds the per-project values (QA_TRACKER, GITHUB_REPO / JIRA_PROJECT_KEY / …, STAGING_URL) to your .env, and checks access. Skills are named /qa-agent:<skill> (e.g. /qa-agent:e2e-session), but trigger phrases like e2e it work the same.

Update later with /plugin marketplace update devops-qa-agent. Change the GitLab settings with /plugin configure qa-agent@devops-qa-agent.

Option B — Clone and work inside the repo

Best for contributing, or for keeping one QA workspace separate from your app.

git clone https://github.com/GhassenAZZOUZ/claude-qa-agent.git
cd claude-qa-agent
cp .env.example .env
cp .mcp.example.json .mcp.json
cp .claude/settings.example.json .claude/settings.json

Set QA_TRACKER / QA_CODE_HOST and your provider's section in .env (and, for GitLab, the token in .mcp.json and settings.json; all are gitignored). Then start Claude Code in the folder and run qa setup.

Verify

qa setup ends with an access check for your providers. Then try a mode that doesn't touch the tracker:

write it "Users can reset their password from the login page via an emailed link" App: https://staging.myapp.com

Usage Examples

WAY 1 — Full UI E2E Session

e2e it --env staging
Feature: User Login
Requirements:
1. Users log in with email + password
2. Invalid credentials show an error message
3. Successful login redirects to /dashboard
4. Locked accounts show a specific locked message

From an issue (GitLab/GitHub #42, Jira SHOP-42, Azure #42):

e2e it --env staging
Issue: #42

WAY 2 — API Testing

From an OpenAPI spec:

api it --env staging
Spec: https://api.staging.myapp.com/openapi.json

Specific endpoints only:

api it --env staging
Endpoints:
- POST /api/users
- GET /api/users/{id}
- DELETE /api/users/{id}

WAY 3 — Quick Bug Report

report it
Auth, Logged in as standard user, Go to /settings > Click Delete Account > Confirm > Observe: 500 error instead of success message

WAY 4 — Write Test Cases

write it
Issue: #55
App: https://staging.myapp.com/upload
Spec: https://api.staging.myapp.com/openapi.json#/paths/~1upload
MR: 88

WAY 5 — Diagnose CI Failure

ci it
Run: https://github.com/your-org/your-app/actions/runs/12345678
(or a GitLab pipeline, Azure Pipelines run, or Bitbucket pipeline URL)

Or:

ci it
Project: your-group/your-project
Run ID: 12345678

WAY 6 — User Story Enhancer

analyze us #143

Or equivalently:

enhance us #143
analyze and update us #143
improve us #143

The agent scores the story out of 100, rewrites the issue body with Gherkin acceptance criteria, test scenarios, and a Definition of Done checklist, then posts the score + QA findings as a separate comment so the body stays clean for developers.

WAY 7 — Competitor Benchmark

benchmark it https://github.com/org/my-app --focus features,perf
benchmark it https://myapp.com --max 8
find competitors "self-hosted test management tool"

Accepts an app URL, a name/description, or a Git repository (GitHub/GitLab URL or local path). For a repo, features are derived from the code as well as the README and tagged code-verified vs readme-claimed. Every cell in the comparison cites a source; anything not found is ❓, never guessed. Private repos are cloned with GITHUB_TOKEN/GITLAB_TOKEN from .env.

WAY 8 — Benchmark Gap → User Stories

gap it
gap it qa-artifacts/benchmarks/finance-2026-10-10.md --scope market --repo https://github.com/org/my-app
gap it --leader PocketSmith --max 5

Reads a WAY 7 report and ranks competitors with a visible weighted score (feature coverage 40, overlap 20, evidence confidence 15, market fit 15, measured quality 10). It picks the leader worth matching, not just the most similar one, and writes one user story per feature the leader has that the target lacks. ❓ cells are never treated as gaps. With a repo given, each gap is grepped first so no story is written for something that already exists. Gaps that contradict a deliberate product choice are flagged decision-needed.

WAY 9 — Verify Fix

verify it #142 --env staging
verify all --close

Runs test-cases/regression/bug-<ref>/ (written by WAY 3). A pass gets a comment and the verified label, and the issue is closed with your OK. A fail gets a comment with the screenshot and the fix-failed label, and the issue is reopened if it was closed.

WAY 10 — CI Test Setup

ci setup ../my-app

Copies the runnable suite from the gitignored qa-artifacts/ into my-app/qa-tests/ (never .auth/, runs/ or node_modules), writes the CI config for your code host (GitLab CI include, GitHub Actions workflow, Azure Pipelines YAML, or a Bitbucket Pipelines step) with a Playwright image pinned to your version, JUnit results and artifacts, validates it (GitLab's lint API, actionlint if installed, otherwise a parse check), and lists the variables and secrets to create. It never commits or pushes.

WAY 11 — MR/PR Test Impact

test mr !87 --env feature/discount-codes

mr-test-impact maps the diff (routes, endpoints, labels, linked issues) to existing tests, then only those run. The MR gets one comment: selected tests with the reason each was picked, the results, heals needed, and the changed areas with no test.

WAY 12 — Release Readiness

release check "v2.3" --min-pass 95

Open bugs by severity, fix-failed bugs, the latest QA results for the milestone's features, the latest pipeline, untested stories, and quarantined tests are combined into GO / GO WITH RISKS / NO-GO, with every signal sourced. It's a recommendation; the team decides.

WAY 13 — Flaky Triage

flaky --window 15

flaky.mjs aggregates the last N results.json. A test that flips pass ↔ fail with the same error each time is proposed for @quarantine plus a flaky-test issue. Quarantined tests keep running but stop producing bugs, and are proposed for unquarantine after 10 straight passes.

WAY 14 — Traceability

trace it #148

Each AC of the story → the tests whose **Covers:** line names it → their latest result → open bugs. Untested must ACs get an offer to write tests.

Tests written from plain text (no story) are traced by folder, against their R-NN requirements:

trace it password-reset-2026-10-13

WAY 15 — Load Test

load test spec=https://api.myapp.com/openapi.json profile=load --env staging

Writes k6 scripts (read-only endpoints by default) and always runs a smoke pass. It asks before running the real profile. Threshold breaches can be filed as performance bugs. --env prod is refused.

WAY 16 — Heal Tests

heal it

Reads Playwright's error-context.md (the page snapshot at the moment of failure) for each locator failure in the latest run, proposes the minimal step/Gherkin change, and applies it with your OK, re-running to confirm. It never changes expected values; an element that's truly gone is a bug, not a heal.

WAY 17 — API Integration & E2E API Tests from Spec

integration it
Spec: https://api.staging.myapp.com/openapi.json
Scope: orders, customers
Issue: #148

Maps the spec into resources (CRUD endpoints, ID fields, dependencies, states), shows you the map to confirm, then writes one @api @integration test per workflow: lifecycle, list consistency, cross-resource chains, referential integrity, state transitions, idempotency, ownership across users. Values flow between calls (stores the response field "id" as "orderId" → GET /orders/{orderId}) and everything created is deleted after the scenario. Nothing runs; file: true also files a test-plan issue. Run them later with api it or npx playwright test --project=api --grep @integration.

E2E API, several services (one spec each):

e2e api it
Specs:
  orders   = https://orders.staging.myapp.com/openapi.json
  payments = https://payments.staging.myapp.com/openapi.json
  stock    = https://stock.staging.myapp.com/openapi.json
Flows: checkout: order → payment → stock reservation

Adds a cross-service map (which ID flows where, sync or async; every link marked (inferred) until you confirm it) and writes @api @e2e-api flows: happy flow, propagation, failure & compensation, cross-service integrity, duplicate delivery. Requests name their service (POST "orders:/v1/orders" → ORDERS_BASE_URL, ORDERS_TOKEN if set). Async effects are polled until a deadline, never awaited with a sleep:

And within 10 seconds GET "stock:/v1/reservations?orderId={orderId}" has field "status" equal to "reserved"

These are slower and depend on every service being up: run them on a schedule, not on each MR (--grep-invert @e2e-api). Queue/DB assertions and contract tests (Pact) are out of scope.

What the Agent Produces

WAY 1 (UI E2E session):

qa-artifacts/test-cases/[feature]-[date]/Test#<nn>.{md,feature,steps.ts} — one Gherkin + playwright-bdd triple per test case, plus an index .md
qa-artifacts/runs/[timestamp]/results.json — execution results, including each scenario's built-in accessibility check
qa-artifacts/runs/[RUN_ID]/artifacts/… — failure screenshot, trace, error-context.md; console/network logs inside results.json
qa-artifacts/sessions/session-[timestamp].md — full session summary
Tracker issues: one test-plan issue + one test-case issue per test case + one bug issue per failure + one qa-session-report

WAY 2 (API session):

qa-artifacts/test-cases/[api-slug]-[date]/Test#<nn>.{md,feature,steps.ts} — API test cases, same triple format
qa-artifacts/api-responses/[endpoint]-[method]-[timestamp].json — full req/resp capture, scrubbed (secret-named fields, JWTs, real-looking emails, secret query params)
qa-artifacts/sessions/api-[timestamp].md — API session summary
Tracker issues: one bug + api issue per failure

WAY 3 (Quick bug):

Tracker issue: one bug issue with full reproduction steps
qa-artifacts/test-cases/regression/bug-<ref>/Test#01.{md,feature,steps.ts} — regression test reproducing it, tagged @regression

WAY 4 (Write test cases):

qa-artifacts/test-cases/[feature]-[date]/Test#<nn>.{md,feature,steps.ts} — UI E2E + API test cases, same triple format
Tracker issue: one test-plan issue
Optional MR discussion comment with test plan summary

WAY 5 (CI diagnosis):

qa-artifacts/ci/ci-[run-id]-[date].md — root cause report
Optional MR discussion comment with the diagnosis

WAY 6 (User story enhancer):

Issue body — replaced with enhanced story (Gherkin ACs, test scenarios, DoD, original preserved in collapsible)
Issue comment — QA analysis report (score/100, grade, strengths, issues found with severity + suggestions)

WAY 7 (Competitor benchmark):

qa-artifacts/benchmarks/[target]-[date].md — fingerprint, competitors, feature matrix, pricing, OSS repo health, differentiators/gaps, sources
qa-artifacts/benchmarks/raw/*.json — raw Lighthouse runs (when perf/a11y focus)
Optional issue labeled benchmark

WAY 8 (Benchmark gap → user stories):

qa-artifacts/benchmarks/stories/[target]-[date]/index.md: leader ranking, backlog, decision-needed, already-implemented
qa-artifacts/benchmarks/stories/[target]-[date]/US-<n>.md: one user story per gap
Optional issues: one user-story + benchmark-gap issue per story, plus a benchmark index issue

WAY 17 (Integration tests):

qa-artifacts/test-cases/[api-slug]-integration-[date]/Test#<nn>.{md,feature,steps.ts} — one workflow per test case, tagged @api @integration, plus an index .md
qa-artifacts/test-cases/[flow-slug]-e2e-api-[date]/… — same, for several specs, tagged @api @e2e-api
Optional test-plan issue and MR/PR comment (file: true)

WAY 9–16: verify report + issue comments/labels (9) · qa-tests/ + CI include in the app repo (10) · MR/PR comment + sessions/change-*.md (11) · releases/<milestone>-<date>.md (12) · @quarantine tags + flaky-test issues + sessions/flaky-*.md (13) · traceability/<ref>-<date>.md (14) · load-tests/<api>-<date>/ scripts + summaries (15) · healed test files + sessions/heal-*.md (16)

Example Bug Filed to Your Tracker

## Bug Report

**Summary:** [Login] No error message shown when password is incorrect

---

## Steps to Reproduce

1. Navigate to https://staging.myapp.com/login
2. Enter email: `[email protected]`
3. Enter password: `WrongPassword123`
4. Click "Sign In"
5. Observe: page reloads with no feedback

## Expected Result
An error message appears: "Invalid email or password"

## Actual Result
Page silently reloads. No error message is displayed. User remains on /login.

## Environment
- URL: https://staging.myapp.com/login
- Browser: Chromium 124
- Date: 2026-06-06T10:32:00Z
- Test Case: Test#03

## Artifacts
- Screenshot: `./qa-artifacts/screenshots/tc3-step4-1234.png`
- Console log: `[no console errors captured]`

## Severity / Priority
Sev-3 (Medium) / P2 — This sprint

Sub-Agents Reference

All 9 sub-agents live in .claude/agents/. They are invoked automatically by the orchestrator — you never call them directly. Work that only applies a rubric or template to data the main session already holds is a skill instead (see Skills Reference below) — no sub-agent cold start, no isolation needed. Tracker and code-host mechanics live in the shared tracker skill and its provider-* skills.

`test-writer`

Used by: WAY 1, WAY 2, WAY 3 (regression test), WAY 4, WAY 17 (integration and E2E API workflows)

The single test-authoring agent for the whole project — first analyzes the requirements into happy path / negative / edge / boundary / security / permission scenarios (mandatory input, ID, upload and numeric checks), then writes both browser and API scenarios, always as a Test#<nn>.{md,feature,steps.ts} triple: Gherkin .feature + a real playwright-bdd step-definition file (page fixture for UI, request fixture for API). Every UI scenario that navigates gets a built-in accessibility step (@axe-core/playwright, WCAG 2.0 A/AA + best-practice) — there's no separate a11y pass. Every API scenario is tagged @api and gets at least 10 mandatory cases per endpoint (happy path, auth failures, missing/invalid fields, IDOR, schema validation, perf threshold, and boundary Outlines built from the spec's schema constraints; +3 for POST/PUT). Replaces the former e2e-test-writer, api-test-writer, and integration-test-writer.

`test-execution-agent`

Used by: WAY 1, WAY 2

The single test-execution agent — runs every .feature/.steps.ts pair test-writer produces via real npx bddgen && npx playwright test, across 2 projects (e2e, api). No live-driven MCP session: it runs compiled, repeatable Playwright suites. Checks the user's config against a documented contract first (a missing project is a setup error), captures screenshot/console/network log on any FAIL/BLOCKED, re-runs only network-failed scenarios once (never assertion failures), and surfaces each scenario's accessibility step result inline with its pass/fail. Replaces the former playwright-navigator, api-executor, and accessibility-checker.

`test-healer`

Used by: WAY 1, 9, 11, 16

Works from Playwright's error-context.md, which holds the failing locator and the page's accessibility snapshot at the moment of failure, so no browser or MCP server is needed. It classifies each locator failure as healable (renamed/moved), ambiguous, or not healable (element missing → likely a product bug), proposes the minimal .feature/step change, and in apply mode edits and re-runs the one scenario. It never touches expected values or @visual baselines.

`mr-test-impact`

Used by: WAY 11

Fetches an MR's changed files and only the diffs it needs, extracts routes, endpoints, UI labels and linked issues, and matches them against Test#<nn> steps and traceability lines. Returns a ranked selection with honest confidence, plus the changed areas no test touches.

`load-test-writer`

Used by: WAY 15

Writes k6 scripts per profile (smoke/load/stress/spike/soak) with per-endpoint tags and p95/error-rate thresholds, read-only by default, with credentials only from __ENV. It always runs smoke, runs heavier profiles only on explicit confirmation, and never runs against prod.

`ci-diagnostician`

Used by: WAY 5

Fetches the CI run, failed jobs (log tail first) and linked MR/PR diff itself, from any supported code host, — so large logs stay in its context, not the main session's — then classifies the failure into one of five types (test-failure / build-failure / infra-failure / config-failure / dependency-failure). Quotes the exact failing log line, cross-references the MR diff, states confidence (HIGH/MEDIUM/LOW), and gives a specific fix recommendation (file + line or command). Marks transient failures as retry-safe.

`user-story-enhancer`

Used by: WAY 6

Scores a user story out of 100 (the enhance-story skill fetches it from any tracker and applies the result) (deductions for missing business value, vague language, absent ACs, no error handling, no NFRs, etc.). Updates the issue body with: enhanced "As a / I want / So that" story, AC summary table with priorities, full Gherkin Given/When/Then scenarios in a collapsible block, test scenarios covering all six types (happy_path, edge_case, negative, boundary, security, performance), open questions that block testing, and a DoD checklist. Posts the QA analysis (score, grade, strengths, issues with severity + concrete suggestions) as a separate comment to keep the body clean for developers. The original story is always preserved verbatim inside a collapsible block.

Skills Reference

Skills live in .claude/skills/ and run inline in the main session.

`bug-report`

Used by: WAY 1 Phase 4, WAY 2 Phase 3, WAY 3 (/bug-report)

Classifies a bug with a fixed Sev-1–4 × P1–3 matrix and hard auto-escalation rules (XSS/IDOR/auth bypass/injection → Sev-1; PII exposure → Sev-2 minimum; payment failure → Sev-1; axe critical/serious → Sev-2), picks labels, parses the WAY 3 shorthand, and formats the developer-ready issue body. Replaces severity-classifier + bug-reporter.

`session-report`

Used by: WAY 1 Phase 5, WAY 2 Phase 4

Builds, saves, and (WAY 1) files the session summary — counts, failed tests, per-page a11y table or API schema/performance flags, bugs filed. Replaces session-reporter.

`tracker` + `provider-*`

Used by: every issue, MR/PR and CI action

tracker defines the operations every WAY uses (issue.create, change.comment, ci.job_log, …), the shared vocabulary and labels, attachment rules and failure handling. provider-gitlab, provider-github, provider-jira, provider-azure-devops and provider-bitbucket map each operation to that provider's tool, ref format, body format and quirks.

`user-story-format`

Used by: user-story-enhancer (WAY 6), benchmark-gap-story-writer (WAY 8)

The single story body definition (story, Gherkin ACs, scenarios, Missing Information, DoD), so WAY 6 can re-score WAY 8's stories and test-writer can trace tests to stable AC IDs.

`requirements-analysis`

Used by: WAY 1, 4, 14, 17 (inline) · test-writer, user-story-enhancer, user-story-format

The single set of rules for analyzing requirements, so stories are scored against the same checks their tests are written from:

Intake: text, issue, enhanced story or spec → one requirements model (requirements, actors, data elements, NFRs, open questions)
Readiness gate: open Missing Information, unassertable outcomes, vague words ("fast", "easy"…), contradictions → questions asked before test-writer starts
Requirement IDs: AC-NN from the story, never renumbered. Plain-text requirements get R-NN, derived one per testable statement, confirmed by you, saved in the test folder's index, so tests without a tracker story are traceable too:
```
**Source:** text
## Requirements
| ID | Requirement | Priority |
|---|---|---|
| R-01 | A user can request a reset link from the login page | must |
| R-03 | The reset link expires after 1 hour | must |
```
Tests cite them like ACs (**Covers:** R-03). trace it <folder> builds the same matrix as for a story
Scenario categories: happy path, negative, edge case, boundary, security, permissions, performance, with their Gherkin tags and when each is required
Checklists per element: input field, ID/resource, upload, numeric, auth/session, concurrency, backend failure

WAY skills

The 17 WAY skills hold each mode's full procedure; see Seventeen Modes above.

Troubleshooting

Problem	Fix
Tracker / code host access fails	Run `qa setup` again: it checks each provider (GitLab MCP connected, `gh auth status`, Jira token or Atlassian MCP, `az` + PAT, Bitbucket token)
GitLab MCP 401 error	Check the token (plugin: `/plugin configure qa-agent@devops-qa-agent`; clone: `.mcp.json`) has the `api` scope
GitHub labels missing	GitHub doesn't create labels on first use; `qa setup` offers to create the label set
`test-execution-agent` fails to launch tests	Run `npx playwright install chromium` and confirm `npx playwright --version` works — it runs the toolchain locally via Bash, not through an MCP server
Agent goes off-task	Ensure `CLAUDE.md` is in the project root; reload VS Code window
Screenshots not saved	Config needs `use: { screenshot: 'only-on-failure', trace: 'retain-on-failure' }`; they land in `runs/<RUN_ID>/artifacts/`
`bddgen`: "Function has 2 arguments, but expected 0-1"	A step takes `testInfo` as a 2nd parameter; use `async ({ page, $testInfo }) => …` instead
`bddgen`: "Multiple definitions matched"	The same step is defined twice; keep generic steps only in `steps/`
Visual tests fail in CI but pass locally	Baselines are OS-specific; run the `qa-update-baselines` job and commit its output
WAY 15 writes scripts but doesn't run	Install k6 or Docker (`grafana/k6`)
GitLab MCP server not connecting	Type `/mcp` in Claude Code panel to verify server status
CI logs truncated	`ci-diagnostician` reads the log tail first and pages back. On GitLab, if pipeline tools are missing, check `GITLAB_TOOLSETS` = `core,issues,merge_requests,pipelines,ci` and `GITLAB_TOOLS=upload_markdown`

Roadmap

Help wanted. Open a "New WAY / feature" issue to discuss first.

More providers. Bitbucket Data Center, Gitea/Forgejo, Linear, YouTrack: each is one provider-<name> skill (see CONTRIBUTING)
Field-testing the provider adapters. GitLab is verified against its API schemas and GitHub's read paths against a live repo; Jira, Azure DevOps and Bitbucket need real-project reports
E2E API, beyond the API surface. Assertions on queues/topics and databases, and consumer-driven contract tests (Pact), for flows WAY 17 can only observe through APIs
Exploratory tester. Explore an app with no requirements and report bugs and untested flows (needs a browser-driving MCP server scoped to that agent)
Claude Code plugin packaging. Install with one command instead of cloning
More trigger languages. French, Spanish, German, Portuguese… trigger phrases in the skill descriptions
Mobile. Appium or Maestro runner alongside Playwright

Contributing

PRs are welcome. Read CONTRIBUTING.md first: skill vs agent, conventions, and the checks CI runs (python scripts/validate.py, node scripts/selftest.mjs, and the evals/ suite weekly). Security issues: see SECURITY.md.

License

MIT. Use freely, attribution appreciated.