OpenQA Skills

Agent Skills that check whether AI-written code still matches the requirement — and turn a PRD into manual test cases and real testdata.

For Cursor, Claude Code, Codex, and OpenClaw. Install with npx skills add (Agent Skills). Local-first; no OpenQA account.

简体中文

Quick Start · What it looks like · How It Works · Blind Evaluation · Limitations · Getting Started · FAQ · Support Matrix

Coding agents make a green pull request cheap. The expensive part is now requirement bugs, rubber-stamp reviews, and test work the model cannot finish alone:

Coding agents write the change. These skills check the requirement, the review, the cases, and the testdata.

Host-by-host scores are not published. Findings are candidates for a human.

What this repository provides

openqa-skills is OpenQA's public, local-first Agent Skills pack. Five skills; they do not share one input:

defect-detection workflow:

• Task creation, repository cloning, branch and diff context collection
• Changed-method analysis with AST rules and optional Java call-graph analysis
• Local JSON providers plus optional HTTP, GitHub, test-case, document, issue, and trace adapters
• Structured finding validation, write-back, ranking, tagging, and HTML reports
• A deterministic known-good/seeded-defect fixture and an automated CLI test suite

Method write-ups live in each skill: defect-detection, code-reviewer, requirements-analyzer, testcase-generation, testdata-generation. Index: docs/HOW_IT_WORKS.md.

How defect-detection works

Static rules catch bugs you can recognise by shape: a swallowed exception, a hardcoded secret, a missing null check. The bugs that survive review are usually different: the code looks fine, the tests are green, and it still does not match the requirement.

• Spec says “discount at 10 items”; the code uses >, so 10 items get no discount.
• A refund pays the requested amount and never caps it at the remaining balance.
• One function updates a counter; the cache that reads the same data is never invalidated.

Those bugs are not in the syntax tree. They sit in the gap between code and intent, and the intent lives in requirements and test cases. A model can compare the two. Left unconstrained, it also invents methods it never read, stamps “looks good” on hundreds of methods, and files so much noise that people stop reading.

So the split is: the model judges meaning; the infrastructure makes that judgement checkable.

Repo + branch + requirements or cases
                  │
                  ▼
     Collect context, analyse the change
                  │
                  ▼
     AST rules + optional call graph
                  │
                  ▼
     Agent review, findings validated
                  │
                  ▼
     Structured findings + HTML report
                  │
                  ▼
              Human review

OpenQA runs the workflow. The host Agent / model does the semantic review. Local providers need no private backend; adapters can attach the same flow to an external platform.

Three controls do the real work:

1. Tiers. Methods tied to a stated requirement or case get deeper analysis; the rest are not treated equally.
2. 23 write-back rules. Rejected if the code was never read, the method name is not in the source, or the batch looks like autopilot output.
3. A close gate. Coverage, report consistency, and evidence depth are checked again before the task can finish.

On the inventory-service blind fixture, seven business-logic defects sit inside ordinary feature work, plus four decoys that look wrong but are correct. One recorded agent run found 7/7 with 0 false positives, and none of the seven came from the 102 Semgrep seed rules. That is one model, one run, one in-house fixture — not a benchmark. Design: How it works. Limits: known limitations.

What the output looks like

These are sample pages (same renderer as a local run, canned findings). Open the HTML if the image is stale.

Defect-detection HTML report — task header, KPIs, and three requirement mismatches. Open the page.

testcase-generation writes Markdown cases. This is that file rendered: steps, expected results, empty Construction column. Open the page.

code-reviewer report: each finding has location, rule, runtime impact, and a fix. Open the page.

requirements-analyzer: one gap / conflict register with an executable check per row. Open the page.

testdata-generation fills {placeholder} with IDs the backend actually returned. Open the page.

Capability map

OpenQA's product direction covers the full AI software engineering quality lifecycle. This repository currently ships defect-detection, code-reviewer, requirements-analyzer, testcase-generation, and testdata-generation. Capabilities marked Available or Partial below are delivered through those workflows unless stated otherwise. The remaining rows describe planned extensions, not features already included here.

Status vocabulary: Available means usable in the current repository; Partial means a working path exists but coverage or integration is incomplete; Planned means not shipped here yet; Product direction means a broader OpenQA platform goal.

[!IMPORTANT]
The current defect-detection workflow is a usable engineering tool, not an experiment-only prototype. It produces candidates for human confirmation and does not replace tests, static analysis, security review, or maintainer judgment.

• OpenQA website
• OpenQA product
• OpenQA Skill Hub

Install

npx skills add openqa-cn/openqa-skills --skill defect-detection
npx skills add openqa-cn/openqa-skills --skill code-reviewer
npx skills add openqa-cn/openqa-skills --skill requirements-analyzer
npx skills add openqa-cn/openqa-skills --skill testcase-generation
npx skills add openqa-cn/openqa-skills --skill testdata-generation

Choose an Agent when prompted. For a user-level Codex installation add --agent codex --global. Each --skill copies one directory.

No OpenQA or npm account is required. See Getting started for runtime requirements, installation scope, and troubleshooting.

Quick start

1. Install the skill you need with the command above.
2. Start a new Coding Agent session.
3. Give the agent the material that skill expects. They are not interchangeable.

Review a branch (defect-detection)

Review REPOSITORY_URL at BRANCH_NAME with defect-detection.
Requirement: checkout amounts must be greater than zero.
For each suspected defect, report the location, trigger, evidence, and fix.

Replace the uppercase placeholders. The workflow collects context, analyzes changed methods, validates findings, and produces a report for human review.

A model-free fixture of the contract (not a detection-accuracy claim):

node examples/checkout-boundary/verify.mjs

Review a local checkout (code-reviewer)

Open the repository in the agent. This skill diffs in place; it does not clone.

Review the current branch with code-reviewer against main.
For each finding give severity, file:line, the rule, the runtime impact, and a fix.

There is no public fixture yet. For method-level requirement defects with write-back gates, use defect-detection instead.

Analyze requirements (requirements-analyzer)

Give documents, not a repo.

Analyze these requirement documents with requirements-analyzer.
Produce one gap/conflict register. P0 items must include verification fields.
Do not invent endpoints or SLAs that are not in the source.

This reviews the PRD. To write a case library from prd/, use testcase-generation.

Write a case library (testcase-generation)

Put PRD / design / specs under prd/ first. code/ is not required for generate.

Generate a manual case library with testcase-generation from the documents under prd/.
Do not invent engineering fields that are not in the source. Mark those TBD.

When the agent stops on a PRD vs design conflict, reply Confirm follow PRD or Item N follow technical design. There is no public fixture yet.

Construct test data (testdata-generation)

Not a git clone. Say what to construct, or point at written cases / OpenAPI:

Create a standard catalog product named Northwind Standard with testdata-generation.
Use the local mock if no enterprise gateway is configured.

For write-back into cases:

Prepare test data for this case file and write the IDs back as preconditions.

Default backend is the bundled mock on http://127.0.0.1:8765. A demo ID is not proof that a real system was written.

Developer verification

Before contributing, run the repository checks:

python3 scripts/check-docs.py
export NODE_OPTIONS=--experimental-strip-types
(cd skills/defect-detection && npm test)
node examples/checkout-boundary/verify.mjs

These checks cover documentation links, CLI and provider behavior, packaging, isolation, write-back validation, and the bundled boundary fixture. They do not prove that every defect will be found.

Scope and limitations

This project helps organize code context and evidence; it does not replace tests, static analysis, security review, or maintainer judgment. It cannot prove the absence of defects or infer business rules that were not provided. Findings are candidates, not automatic merge decisions.

Local providers write data to disk. Repository cloning, document fetching, remote providers, automatic Semgrep/GitNexus installation, and the host Agent/model may use the network. Review FAQ, support matrix, and security policy before using private source.

Roadmap

• Now: harden clean installation, agent compatibility, public fixtures, and developer documentation.
• Next: add specification review, requirement review, and broader analysis skills under the same evidence-and-human-review contract.
• Later: connect cross-repository impact analysis, AI Code Review, CI quality gates, and hosted engineering systems.

A capability is marked available in this repository only when its implementation, example, and limitations are published. Progress is tracked in the public roadmap.

Documentation

Support

• Report reproducible bugs through GitHub Issues
• Ask questions and discuss implementations in GitHub Discussions
• For security vulnerabilities, follow SECURITY.md

When asking for help, include the commit or Skill version, operating system, Agent, command, expected result, and actual result. Remove credentials, private source, and proprietary logs.

Contribute

Useful contributions include minimal public reproductions of false positives or missed defects, known-good controls, new analysis rules, fixtures, and documentation improvements. Remove credentials, private source, and proprietary logs before sharing. Start with Contributing, examples, and Publishing.

Apache-2.0 · GitHub