What this is. A completion gate for coding agents.
It watches the commands your agent already runs, turns their output into evidence, and binds each record to the exact bytes of the files it observed.
When the agent claims it's done, a state gets computed instead of believed.
Edit a file afterwards and its green tests go stale — the way make invalidates an object file.

The thirty-second version

Ask a coding agent to fix a bug and it will tell you the bug is fixed.

It is telling the truth as it understands it. That is the problem. Published measurements put the gap between claiming completion and achieving it in the tens of percent — one study found a frontier model that submitted a patch on every single run and resolved 44% of them. Prompting the model to be more careful does not close that gap, because the model was never lying. It simply has no way to know.

So stop asking it.

Every agent already runs commands. Tests, typechecks, builds, the failing reproduction it wrote three minutes ago. All of that output is flowing past unread on its way to the scrollback. ElevenPowers reads it, files it as evidence, and stamps each record with a fingerprint of the files it saw.

flowchart LR
    A["agent runs<br/>pytest · tsc · build"] --> B["runtime reads<br/>the tool result"]
    B --> C["evidence record<br/>+ fingerprint of<br/>every file observed"]
    C --> D[("ledger")]
    D --> E{"gate computes<br/>a state"}
    E --> V["VERIFIED"]
    E --> U["UNVERIFIED"]
    E --> S["STALE"]
    E --> X["CONTRADICTED"]

    classDef ok fill:#d4f4dd,stroke:#22a06b,color:#0b3d2c
    classDef warn fill:#ffeaa7,stroke:#d98e04,color:#5c3c00
    classDef stale fill:#dfe6ee,stroke:#7b8a9b,color:#1f2d3a
    classDef bad fill:#ffd9d9,stroke:#d64545,color:#5c1111
    class V ok
    class U warn
    class S stale
    class X bad

Then "done" stops being a sentence the model emits and becomes a state you can compute:

UNVERIFIED  bug_fixed
  missing  a test covering the change passes
           run the test that exercises this change, by name or by file
  missing  that test failed before the fix
           run it before applying the fix so the failure is on record
  met      the related test suite passes

And because each record is bound to file content, evidence rots when the code moves under it:

STALE  bug_fixed
  stale    a test covering the change passes
           recorded at tree fb7fb4ae5570c0da, files have changed since
           re-run: python -m pytest tests/test_core.py -q

The closest analogue is not another agent framework. It is make.

A claim moves between them on its own, as the code moves:

stateDiagram-v2
    direction LR
    [*] --> UNVERIFIED: claim opens at the first edit
    UNVERIFIED --> VERIFIED: every obligation met
    VERIFIED --> STALE: a file it observed changes
    STALE --> VERIFIED: re-run the recorded command
    VERIFIED --> CONTRADICTED: latest evidence fails
    CONTRADICTED --> VERIFIED: fixed, and proven again

[!NOTE]
A failure that was later fixed is reproduction evidence, not a contradiction — that's the point of demanding the test failed first. Only the most recent record per identity counts.

First, I went and read the competition

Before writing a line of runtime, I cloned fourteen agent systems, pinned each to a commit, and read them from source. Not the READMEs — the source. Where the loop exits. What is actually enforced in code versus what is merely requested in a prompt. Where each one is genuinely brilliant, and where it quietly gives up.

[!NOTE]
62,608 words of notes. 2,646 lines of runtime. A reading-to-writing ratio of roughly 24:1, which felt indulgent right up until the fourth system turned out to have the same blind spot as the first three.

Every card is in docs/research/cards/ with file:line citations at a recorded commit, and anything inferred rather than read is marked as inferred.

The pattern that made this project exist

Read fourteen of these in a row and the same shape shows up in every one:

[!IMPORTANT]

"Done" means the model said so, or the model stopped.

One of the fourteen binds a single declared command to a tree hash. None computes completion from evidence. None ships tooling for intermittent bugs. Nine of ten enforce their process through text the model is free to ignore — and three admit it in their own repositories.

That is not a knock on any of them. Every one is better engineered than this project, most are years older, and I borrowed from all of them. They were simply solving the turn before this one.

What's different here

Evidence is collected, not demanded

No new ceremony, no protocol for the agent to follow, no sentinel lines to emit. It runs its tests the way it always did; the runtime reads the output. If the agent never cooperates once, the evidence is still there.

Evidence expires

Every record carries a fingerprint of what it observed. Change those files and the record goes STALE, with the exact command to re-run. This is the whole idea, and it is borrowed shamelessly from Test Impact Analysis — which has tracked test-to-source dependencies in industry for years — pointed at agent completion instead of test selection.

Obligations scale with risk, and arrive early

Touching auth/, payments/, migrations, infra or secrets costs more proof than touching a README. And they are announced at the first edit, while the agent can still act on them, rather than sprung at the end as an ambush:

this task will need, before it can be called done:
  a test covering the change passes
    run the test that exercises this change, by name or by file
  the related test suite passes
    run the suite covering the files you changed, not just the one test
  repeated runs show the failure is gone
    ep-repeat 20 -- <the command that reproduced it>

A check that could not have failed is not evidence

Fresh and passing are two facts about a record, and neither is the one that
matters. A test that would have passed before your change proves nothing about
it — and measured elsewhere, 46% of agent validation evidence carries no
bug-discriminating information at all.

How often does it fire here? Measured 2026-09-17: once in 22 runs. Across
two sweeps on the same 16 real tasks — sonnet and opus — exactly one check came
back non-discriminating, and it did not replicate when the other model was given
the identical task. The 46% above is somebody else's corpus and it is cited, not
claimed. On this one the effect is rare, and that is written down rather than
left for a reader to discover:
results/b4-discriminate/findings.md.

So the runtime runs its own checks the other way round: each declared check
again, in a throwaway worktree built from the commit the task started at. If it
already passed there, the report says so.

  ok      the related test suite passes  <- python pytest 1 passed, 0 failed
  could not fail: tests: this check passes without your change, so it is not
                  evidence the change works

It reports. It never refuses — naming a weak check costs nothing, and blocking
on one has to earn its cost first.

The best state you proved is kept, not overwritten

Published measurements put 60–69% of coding-agent failures on runs that
reach and edit the correct functions and then produce a wrong patch. A long
attempt ends at its latest patch, not its best.

When the declared checks are green, the working tree is committed to a private
ref — your branch, index and stash untouched. If a later edit moves off it, the
report offers it back, and refuses to offer it if HEAD has moved underneath,
because restoring then would undo whatever moved it.

"It failed before the fix" is computed, not demanded

The obligation used to need the agent to have run the test red first. Anyone
who wrote the test afterwards — ordinary practice — could never discharge it,
and an earlier failure from an unrelated typo counted as proof.

Now the question is asked of the repository: was this test already failing on
the commit this task started from? If it was and it passes now, that is a
reproduction, whatever order the work happened in. No model call.

cannot_complete is a first-class outcome

A system with no way to say "this should not be done as asked" has quietly reproduced the action bias that causes false completion in the first place.

Intermittent bugs get arithmetic instead of vibes

The one case where a single green run proves nothing. Ask about something flaky and the runtime computes how many clean runs are actually enough, from the failure rate the agent itself measured:

missing  repeated runs show the failure is gone
         ep-repeat 29 -- python -m pytest tests/test_worker.py -q
         so far: still failed 3 of 30

Three failures in thirty is a 10% rate; ruling that out at 95% confidence needs 29 clean runs, because 0.9²⁹ < 0.05. The agent is handed the number rather than left to invent one. The runner stands alone, too:

ep-repeat 50 --jobs 8 -- pytest tests/test_login.py   # is this flaky, and how flaky

None of the fourteen ships anything like it.

The seam is tested, not assumed

A verification layer that silently stops working is worse than none — and this is not hypothetical: three defects in the layer between this runtime and its host each failed by doing nothing, while every unit test stayed green. So ep-doctor feeds the runtime a tool result shaped exactly the way the host shapes one, and checks the answer comes back right:

ok    python 3.13.2
ok    a failing command is recorded as failing
ok    a passing command is recorded as passing
ok    hooks.json subscribes to every event the runtime handles (6 tools recorded)
ok    the ledger directory is writable
ok    nothing unreadable has arrived from the host

Anything it cannot parse lands in .elevenpowers/blindspots.jsonl — so a host that renames a field becomes a diagnosable symptom instead of a tool that quietly went quiet.

It stays inside the task

Agents wander. When an edit lands somewhere the task has neither read nor been asked about, the guard asks — it never denies, because you are the judge:

src/billing/stripe.py is in billing, which this task has not read or edited,
and the request does not mention it. Edit it anyway, or read it first.

Scope is never declared up front, because nobody knows which files a change will touch before making it. It is derived from what the task established: files read, files edited, and the areas the request named.

Install

git clone https://github.com/SatyamSingh-Git/ElevenPowers.git
claude --plugin-dir ElevenPowers/plugin

Nothing to configure. Ask for a change the way you normally would.

python plugin/bin/ep_doctor.py    # is the runtime seeing what the host sends?
python plugin/bin/ep_status.py    # what do I still owe on this task?

[!TIP]
Full instructions, every command and flag, and what to do when something breaks: The Guide. Run ep_doctor.py --host once after installing — the shallower check passed for weeks while silently discarding the flag.

Optionally, in .elevenpowers/config.json — a project with no config behaves exactly as it did before:

{
  "profile": "strict",
  "commands": {"tests": "make test", "typecheck": "npm run typecheck"}
}

[!TIP]
Declaring your commands is the single highest-value line of config. It tells the runtime this project has a suite even when the suite hides behind a Makefile, it replaces a guessed hint with the command you actually use, and it lets the runtime discharge obligations by running them itself instead of interrupting to demand them. That one change took live blocking from 75% of runs to 12%.

What claims cost

Claims are inferred from the request by pattern matching — no model call, no added latency. Ask a question or request a code read and no claim opens at all; the runtime stays entirely out of the way.

Most requests state no claim, because one real prompt in five is four words or fewer and "continue" carries its subject in the conversation rather than in the message. So the claim follows the work: the first edit to a source file opens one. Measured across 3,557 turns of real sessions, obligations attach to 21% of turns that changed nothing — down from 51% under the naive rule.

Where this actually is

Week one. Here is the unflattering version, because the alternative is becoming the thing this project was built to catch.

Every figure below was produced by the command printed next to it.

On the first row. Claude Code writes a transcript of every session including each tool result exactly as the host produced it, so replaying those costs no inference and needs no hand labelling — the host already recorded whether each command failed. The reader this replaced agreed on 0 of 174 failures. It is reported as two rates rather than one because the corpus is 97% successes: a reader that answers "passed" to everything scores 97% accuracy while being wrong about the only thing the gate needs to know.

On the last row. It runs live — 89 free runs through the CLI where the gate fires, refuses the stop and the agent goes back to work, then ninety paid runs of one arm against a pinned corpus and model, and then a hundred more with both arms. Nineteen of those gated runs have since been graded at the moment the gate refused, which nothing here had ever done. The mechanism fired four times and changed no outcome: it blocked a patch that was already correct, one that stayed broken, and one regression that it then failed to catch. Measured cost is 1.3x vanilla; measured benefit is none so far. The fair qualifier is that the case it exists for barely occurs on this corpus, because 80 percent of first proposals are already right.

A literature sweep on 2026-09-15 then explained that null, and re-aimed the plan to
v0.8. Independently measured: 46% of agent validation evidence carries no
bug-discriminating information, and every model saturates the visible test suite. So
PASS and FRESH are two facts about a record and neither is the one that matters — a check
is not evidence until it is shown to discriminate. Separately, the median decisive error
lands at step 7 of 27 steps, which is why a gate at the end changes little. Both fixes were
already written in this repository's own plan and neither had been built. The thesis survives
and is better supported than it has ever been; the placement did not.

And on 2026-09-17 that re-aiming was measured, on the corpus rather than in the
literature — it came back null. Sixteen tasks on opus, against the same sixteen on
sonnet: 0 of 14 checks were non-discriminating, against 1 of 8 before, and the single
earlier case did not reproduce. So the sentence above needs its qualifier: the 46% is
cited, not replicated here, and the claim that agents routinely finish on evidence that
could not have failed is not supported by this project's own runs. The mechanism works —
it caught the one case, and it caught a real regression — but a frequency it has not shown
is not a frequency it may assert. The same sweep found something that outranks the rate:
12.5% of runs bypassed the gate entirely, because the runtime never observed the agent's
edits. Full numbers and caveats in
results/b4-discriminate/findings.md.

The rest came out of auditing the sweep afterwards: 24 of the 100 runs name their own task's fix commit, fetched from GitHub, so the 92 percent is a score for applying a described upstream change with access to that change — not for repairing anything unseen. Whether the output is better is still unanswered, and the next thing to build is not a harder corpus but an information boundary. Two identical plain passes over the same sixteen tasks resolved eleven and fifteen.

That night also contradicted something the harness assumed twice over: the grade turned out to be a function of the grader's Git configuration, and then of what an unrelated agent installed on the machine mid-run. Three claims from the write-up were withdrawn afterwards. 23-spend.md has it, including the part where the verification that should have prevented a bad measurement looked exactly like success.

Answering it honestly needs 31 pairs on which the two arms disagree, against a task suite rebuilt so most of it actually discriminates. This page used to say 252 agent runs; that figure is withdrawn. It came from dividing the required pairs by the within-arm flip rate, and flipping is not disagreement — a baseline that fails every time against a treatment that succeeds every time flips never and disagrees always. Converting pairs into runs needs a measured rate of disagreement, and no run here has produced one.

[!WARNING]
An external audit recorded sixteen observations and named fifteen defects in the runtime and the evaluator — its own appendix says those are “sixteen observations, not sixteen statistically independent findings”, and four more were found here afterwards, so nineteen carry probes. Rather than quietly fixing the embarrassing ones, each became a test that fails on purpose until it doesn't — so tests/test_audit_probes.py is the authority here, not this README. A defect marked xfail(strict=True) that starts passing fails the run, which forces the marker off and makes it impossible to fix something quietly.

All nineteen now pass, eighteen fixed and one narrowed. Four of those probes could not detect the repair of the defect they recorded, and the largest defect found in the process was not in the audit at all: the fingerprint that decides whether evidence is stale could not see a same-length edit, in 220 of 300 attempts. 18-evidence.md has it.

The worst one deserves naming out loud: the task grader ran only the fail-to-pass set, so a patch that broke existing tests scored as resolved — on a measurement whose entire subject is catching exactly that. Fixed 2026-09-12. Every number published before that date deserves the appropriate suspicion.

Not built yet: no workflow engine, no repository index, no memory, no model routing, no subagents. Each is postponed with a written trigger in docs/postponed.md. Invalidation is coarse — any source edit stales everything — and narrowing it to each test's import closure is the documented next step, but only once measurement shows the coarse version is too pessimistic to live with.

Standing on fourteen sets of shoulders

Nothing here was invented from nothing, and pretending otherwise would be both dishonest and a waste of fourteen excellent codebases.

The method was deliberate: read each system from source, separate what it enforces in code from what it merely requests in prose, inventory every strength and every weakness, then map which system's strength covers which other system's weakness. The full mapping is in complementarity-matrix.md, and build-on.md does the same for every component currently being built — naming its prior art, licence and limit before its design.

[!IMPORTANT]
Originality was never the goal. An earlier version of this section said the only honest place to build was wherever all fourteen left a gap. That is a novelty filter, and it was retired: a component starts from the best existing implementation, and what we add is stated in one sentence. If that sentence cannot be written, the component is a reimplementation and the borrow wins.

The same applies to research. Every paper, article, survey, dataset and documentation page that changed a decision here is credited in audit_2026_09_15/sources.md — 167 links, each with what it contributed and how strongly it was verified, including the ones we relied on and then disagreed with.

What came from where:

Licenses, commits, copyright holders and reuse obligations for all of it: docs/research/licenses.md. Where a system's terms forbid reuse, only the idea was taken, and it says so.

And yes — this was built with coding agents, mostly Claude and Codex. A project about whether agent output can be trusted would be a strange place to get coy about that. The evidence layer was pointed at its own construction from day one, and docs/dogfood-log.md records where it caught its own author. The audit above is what happened when someone else's agent was pointed at it too.

Reading further

This is a work in progress, and says so on purpose

Everything above is a snapshot. The thesis has already been demoted once, the objective rewritten once, and a published result withdrawn once — and the questions still open in PLAN.md §7 guarantee more of that is coming. Numbers here will be superseded. Sections will be rewritten. Some of what this README currently argues will turn out to be wrong, and when it does, it gets corrected rather than quietly dropped.

So if you want the real story, don't read this file — read journey/.

It is the complete record, written so that someone who was not here can reconstruct the reasoning, including the parts that were mistaken. Twenty-four chapters, in order:

• Where it started — the original brief, the first plan written from memory, and why that was exactly the wrong way to begin.
• What was read — fourteen systems from source: the method, what each one actually turned out to be, and the findings that overturned the assumptions I walked in with.
• How the design changed — two rewrites and the pivot, from classifying tasks to deriving proof obligations.
• What building it taught — the first working version and the three defects that only surfaced in use; a guard that shipped as dead code, twice; three more defects in the one layer nobody had measured, found by reading 36,000 real commands.
• What measurement destroyed — 75% false blocks and every fix; 89 live runs where the mechanism works and the measurement doesn't; twelve real bugs where the gate changed nothing at all and the obvious fix turned out to be theatre; an external critique that dismantled a conclusion I had just reached.
• Where it is now — an audit and its reproduced defects, a grader blind to the exact regressions it existed to catch, a change of objective, then the evaluator, the evidence layer and the host contract repaired; Phase A closed; a corpus and an instrument that refuse to run unpinned; the first night allowed to cost money, and a corpus rebuilt so it can move.
• Where it's heading — PLAN.md §7, three tracks with explicit prerequisites rather than a chain.

Two files there are worth more than the chapters: decisions.md, every significant decision with its reasoning and whether it still stands — and mistakes.md, every mistake made, kept in full, because the corrections turned out to be the most transferable part of the whole thing.

Every number quoted anywhere in that folder was produced by a command in this repository, and the command is printed next to it.