Verify Regression Tests

skills.sh directory
· 한국어

Your regression test is green. But does it actually catch the bug?

verify-regression-tests is an open Agent Skill that tests the test. It runs one
frozen regression guard against a candidate implementation and a faithful
known-bad control, then traces the RED result to the exact behavior the test
claims to protect.

It is framework-neutral and uses the same SKILL.md in Claude Code and Codex.

Why this exists

A regression test can stay green while the original bug is present because:

• an earlier validation gate rejects the fixture first;
• a mock, cache, fallback, or default bypasses the target branch;
• the test is skipped, filtered out, not awaited, or running a stale artifact;
• another detector satisfies a broad assertion;
• the assertion accepts both the broken and correct behavior.

A passing suite does not reveal those failures. This skill establishes a focused
counterfactual instead:

candidate implementation + frozen target test  -> GREEN
valid known-bad control + frozen target test    -> RED at the intended witness
restored candidate + frozen target test         -> GREEN

The known-bad control must also reproduce the target behavior independently of
the test. A compile error, unrelated exception, unreachable mutation, or
equivalent change is not proof.

Install for Claude Code and Codex

Run this from the project where you want the skill available:

npx skills add aiopshwang/verify-regression-tests \
  --skill verify-regression-tests \
  -a claude-code \
  -a codex

Both hosts can also select the skill implicitly when a request matches its narrow
description.

Use it

Try requests such as:

Use the regression-test verifier on this bug guard. Prove the unchanged test
fails for the original defect, not an earlier validation error.

This retry regression test is suspiciously green. Check whether the cache or
mock bypasses the retry branch, and strengthen only the test if needed.

Verify this existing idempotency regression test. If it already kills a faithful
known-bad control, do not change files; report the evidence.

What it verifies

The skill:

1. defines the trigger, bad behavior, expected contract, decision point, and
   observable witness;
2. confirms the exact target test is discovered and executed;
3. creates one safe, reversible, faithful known-bad control;
4. validates that control independently of the target test;
5. rejects false greens and wrong reds caused by gates, fixtures, mocks,
   fallbacks, artifacts, or broad assertions;
6. restores the candidate exactly and reruns the relevant checks;
7. reports Verified, Not verified, or Inconclusive without extending the
   claim beyond the exercised defect and environment.

How it differs

The skill can use a targeted mutation when it faithfully recreates the defect,
but it does not install mutation tools or optimize a mutation score.

Safety boundaries

Known-bad controls are temporary evidence. The skill requires an isolated copy or
worktree when exact in-place restoration is uncertain and never authorizes a
commit, push, deployment, production service call, shared-data migration, or
external account change.

It preserves unrelated work, keeps the test frozen during candidate/control
comparison, and reports an inconclusive result when a safe control cannot be
constructed.

See SECURITY.md and PRIVACY.md.

Reproducible evaluation assets

The repository includes six dependency-free synthetic fixtures across Node.js,
Python, and Dart:

• wrong upstream gate;
• retry path hidden by cache fallback;
• target detector masked by another detector;
• overly broad exception assertion;
• weak invariant assertion;
• an already-strong idempotency test that should remain unchanged.

All six baselines are intentionally green. Run them with:

python3 evals/check_baselines.py
python3 evals/run_proofs.py

Behavioral results, routing checks, host versions, and evidence limits are in
EVALS.md.

Limits

This skill verifies a named regression guard at the boundary actually exercised.
It does not prove that the full suite is strong, that no other bugs exist, or that
unit evidence applies to integration, device, production, or deployment behavior.

It is designed to work with any repository that has an executable test command,
including Jest, Vitest, pytest, JUnit, Dart, and Flutter projects, but v0.1.0's
published synthetic evaluations cover Node.js, Python, and Dart only.

Open standard

The skill follows the open Agent Skills format documented by
OpenAI,
Claude Code, and the
Agent Skills specification.

Contributing and license

See CONTRIBUTING.md for evidence and privacy requirements.
Licensed under the MIT License.