Trace File Lineage

Find out which script, notebook, data file, command, or AI agent produced a file —
locally, with evidence, and with honest uncertainty.

Built for Python and notebook work: research code, data analysis, and the piles of
files AI coding agents now generate.

pip install git+https://github.com/uczltw6/trace-file-lineage
lineage demo

lineage demo builds a small project, records a run, and shows you the answer — under
a second, nothing to configure, and it writes only into ./lineage-demo.

Not on PyPI yet. pip install trace-file-lineage will work from the first
release; until then use the Git URL above.

It does two different things

Being precise about this up front, because they give different kinds of answer:

For files you already have — reconstruct the most likely origins and show the
evidence. No setup was needed, and nothing had to be running beforehand. These
answers are ranked guesses with the reasoning attached.

For runs from now on — wrap a command and get verified provenance
automatically. These answers are proof.

lineage demo shows both at once:

[4/4] Asking where figures/trend.svg came from.

  This is proof            @run/run:8aa88ebd
    assurance: verified   evidence: task-boundary-diff
    A command was recorded while it ran, and this file changed during it.

  This is a good guess     analysis/plot.py
    assurance: candidate   evidence: static-callsite at analysis/plot.py:16
    That line writes to this path, but nobody watched it happen.

Guesses are labelled as guesses. A matching filename, a nearby timestamp, a line
of code that mentions a path — none of those are proof, and no amount of them stacked
together ever becomes proof. When the evidence genuinely isn't there, the answer is
insufficient rather than something plausible.

Two things people use it for

1. "Where did this old file come from?"

You have a chart from three weeks ago and no memory of making it.

lineage explain figures/final_panel.png

It reads your code, documents, image metadata, and Git history, then ranks the
candidates and shows the evidence for each. Often the answer is obvious once you see
it — that notebook, reading that CSV. Sometimes the honest answer is that the
trail is gone.

2. "My agent just wrote 150 files"

lineage run --task "Parameter sweep" -- python sweep.py
lineage receipt

Every file that run touched, recorded as proof, grouped rather than dumped as 150
separate mysteries. Works the same for an AI agent's turn as for a script.

Then, before you change an input:

lineage impact data/raw.csv     # what depends on this
lineage stale data/raw.csv      # what is now out of date

The interactive graph

lineage open

An interactive map in your browser: drag, zoom, click a file to focus on just its
neighbourhood and read the evidence behind each link. Solid arrows are proof,
dashed arrows are guesses. One self-contained page — offline, no server, nothing
sent anywhere.

lineage export --format mermaid gives the same graph as a diagram. Here is the demo
project, with the labels written in plain words:

flowchart LR
    n0["recorded run"]
    n1["figures/trend.svg"]
    n0 ==>|"proved it made this"| n1
    n2["analysis/plot.py"]
    n2 -.->|"probably made this"| n1
    n3["data/measurements.csv"]
    n3 -.->|"probably read by"| n2

The tool's own labels are was_generated_by · verified, can_generate · candidate,
and declares_read · candidate.

How sure is it?

Five labels, always shown:

Order of trust: your own confirmation, then recorded runs, then imported provenance,
then declarations, then static code, then content, then names and timestamps.

Where it's strongest

Best at: Python and Jupyter notebooks. Their code is genuinely parsed, so
file reads and writes are understood rather than guessed at — including common
Pandas, NumPy, Matplotlib, PIL, and pathlib patterns.

Good at: Word, PowerPoint, Excel, OpenDocument, EPUB, and PDF (text, structure,
embedded images); PNG/JPEG/TIFF/WebP metadata; Git rename history; and searching
about 50 text and code formats for file references.

Deliberately shallow: JavaScript and TypeScript get a cautious static scan, not
real language understanding. Other languages are searched, not parsed. Spreadsheet
formulas are recorded but not turned into links. Paths built at runtime can't be
resolved.

So it fits research code, data analysis, notebook workflows, Python automation, and
agent-generated artifacts. It is not a general-purpose lineage platform for any
software project, and doesn't claim to be.

Run lineage doctor for what your own machine can read.

Everything stays local

• Nothing is uploaded, ever. No account, no API key, no AI service.
• Your code is never executed. It is read and analysed.
• Passwords, keys, and .env files are skipped automatically.
• Recorded commands have password-looking arguments stripped.
• From an AI agent, only a summary and the changed-file list is kept — never your
  conversations or prompts.

The .file-lineage/ folder holds text extracted from your files, so treat it like the
project itself. It's Git-ignored by default. See SECURITY.md.

Speed

Measured on macOS with Python 3.14, reproducible with tests/benchmark.py:

The first scan reads everything and shows a progress counter; after that only changes
are read. Individual questions answer in milliseconds. node_modules, virtual
environments, caches, and build output are skipped by default, and each scan reports
what it left out.

With an AI coding assistant

Ask Claude Code or Codex "where did this file come from?" and it will use this tool.

claude --plugin-dir .                                     # Claude Code

ln -s "$PWD/skills/trace-file-lineage" \
      "$HOME/.agents/skills/trace-file-lineage"           # Codex

Details: docs/install.md.

Why not Git, DVC, or OpenLineage?

Short version: Git records versions, DVC and OpenLineage record pipelines you
declared in advance, and this records evidence — including for work nobody planned
to track. Full comparison, including when not to use this:
docs/comparison.md.

Integrations — experimental

These exist, are tested against fixtures, and are not part of the core promise:
import and export W3C PROV, read dvc.yaml, read OpenLineage
events, import an external code graph, export to Obsidian.

They have fixture-level coverage only and have not been validated against real
third-party pipelines. Treat them as a starting point rather than a compatibility
guarantee, and ignore all of it if you don't need it. The core — tracing files in
your workspace — does not depend on any of them.

Status

Early release (0.7.0). Tested on Python 3.11–3.14 across macOS, Linux, and
Windows — all twelve combinations green in CI, plus coverage, linting, and real
PDF/OCR fixtures. Commands may still change.

Run against three externally-authored repositories, with the results and the
defect it uncovered written up in
docs/real-world-validation.md — including what
those runs do not establish.

Known limits are written down rather than glossed over:
docs/limitations.md.

Contributing

Issues and pull requests welcome, including from first-timers —
CONTRIBUTING.md. Security reports go privately via
SECURITY.md.

lineage demo                                          # see it work
python -m unittest discover -s tests -p 'test_*.py'   # run the tests

License

MIT. See LICENSE.