reproducible-science

Command-line tools that check whether a paper's claims match its artifacts.

1. Three outcomes kept apart: a value that disagrees, a value the artifact is silent on,
   and a check that never ran. A pass/fail report collapses them, and their remedies differ.
2. Deterministic and offline. A check is a digest comparison and a decimal comparison, so
   it runs on every commit rather than once before submission.
3. Facts separated from verdict. The engine computes no verdict; a policy maps outcomes to
   severity, so an unchecked citation is acceptable in a draft and disqualifying in a submission.
4. One contract over all three kinds of evidence — a quotation in a source, a value at a
   location in a machine-readable file, a cell in a table — through one engine and one report.

pip install reproducible-science

See our Claude Code plugin,
which traces results at the time of experimentation.

/plugin marketplace add elliottower/reproducible-science

Recommended: poppler (pdftotext), for reading PDF sources. citations falls back to a
pure-Python reader without it, so pip install is enough; poppler resolves more passages than
either fallback on a measured corpus, and every result records which extractor produced it.

brew install poppler                  # macOS
sudo apt install poppler-utils        # Debian/Ubuntu
pip install "citations[pdf]"          # no system package available

Documentation

reproducible.science

The docs include a notebook that runs the published packages in your browser, with nothing to
install.

Install

pip install reproducible-science brings all four. Each is also its own distribution —
prereg, citations, results-cli — so installing one brings only that one. The code that
checks a manifest's claims is a fifth, claims-cli, with the command
claims: claims verify checks the claims alone and runs nothing the manifest names.

Getting started

repro demo writes a small worked example, runs the sequence below over it for real, and then
breaks it twice — so the first thing you watch the tool do is catch something, and the two
things it catches are not the same kind of failure.

Write down what your paper claims and where each claim comes from. repro verify checks every
one against the file it names and tells you which ones hold, and in the same report checks the
plan, the ledger and the quotations the project keeps, naming any it does not keep as not used.

$ prereg freeze                                    # lock the plan; names a git commit
$ results seal analysis.py data.csv                # hash the inputs, before the run
$ results run output.json --run-id exp_001         # hash the outputs, after it
$ results claim "ICC = 0.42" --run-id exp_001 --location "Table 2"
$ repro verify
repro.yaml

  ok    icc          metric   /icc = 0.42
  MISS  pvalue       metric   p: manuscript prints 0.031, results.json holds 0.051
  GONE  n            metric   /n does not resolve in results.json

  1 mismatch, 1 not_found, 1 verified
  policy publication: FAILED  (2 errors, 0 warnings)
    error   evidence.mismatch          pvalue/metric: p: manuscript prints 0.031, results.js
    error   evidence.not_found         n/metric: /n does not resolve in results.json

/home/me/study

  staleness    not used here: repro.yaml declares no command under `regenerations`
  prereg       passed  (prereg check)
  results      passed  (results verify --files)
  citations    not used here: no claims directory holding a claims file here

2 of 3 checks passed; failed: claims
not used here: staleness, citations

ok holds, MISS disagrees with the artifact, GONE names an address that no longer
resolves. Three further outcomes appear in longer runs: -- for a check that could not run,
none for a claim offering no evidence, and ERR for a source that could not be read.

Quotations are matched against the sources they cite, page numbers included. Reported numbers
are compared against the value stored at an address you gave — never by searching a file for a
number that looks close enough. When a check cannot run at all, it reports that, and the run
does not pass.

Preregistration is optional — you mark a claim confirmatory, exploratory, or not applicable,
and only the first kind needs a plan behind it.

For other programs

repro export record writes one JSON object per checked assertion: the file read and its
digests, the address, the stored and printed values, the comparison and the outcome. Each
record names its version, reproducible.science/claim-check/0.1-experimental, and the format
is experimental until 1.0. Releases in the 0.6 series also read records under
repro.check/0.1-experimental, the name 0.5.6 wrote. The JSON Schema is published at
https://reproducible.science/schemas/claim-check-0.1-experimental.schema.json.
repro export ro-crate writes the same records as an RO-Crate, under the
claim-check profile, and
repro export sarif as a SARIF log for GitHub code scanning. The
check record page says more.

A results file copied in from another repository can name where it came from, as
origin: {repository, commit, path} on its artifact. repro origins compares the pinned
digest with the bytes at that path and commit, read from a local clone or, with --fetch,
from GitHub. repro verify never reads an origin and stays offline.

Contributing

Contributing to get started, Development for the workspace
and release machinery, and docs/SPEC.md for what a decision means. Security
issues: SECURITY.md.

MIT licensed.