paper-fetch — a publisher-aware full-text PDF fetcher

Give it a DOI; it walks a route ladder to get the full-text PDF the cheapest, most
legitimate way first — open access, then official publisher text-mining (TDM) APIs, then
your own institutional library proxy, and finally it just prints your library's resolver
link so you can finish by hand. Every route is one you're already allowed to use; there is
no Sci-Hub fallback here.

It's the download end of a small paper pipeline — the piece that was missing longest,
because it's the one that fights publishers:

# 30 seconds to first PDF (the OA route needs zero keys — just your email in config.yaml)
git clone https://github.com/drpwchen/paper-fetch && cd paper-fetch
pip install -r requirements.txt && cp config.example.yaml config.yaml   # set unpaywall_email
python paper_fetch.py 10.1186/s12984-023-01168-x out.pdf

This project ships no institution's access. You supply your own library's endpoints
(in config.yaml) and your own account (in a local secret store). It automates your
own authenticated session — it is not a paywall bypass and it does not share credentials.
Most readers will point an AI agent at this repo to understand and adapt the method; the
code — and AGENTS.md — are written for exactly that.

The route ladder (the idea)

The whole design is: try the cheapest, most-permitted route first, fall through on failure.

DOI
 ├─ 1. Open Access ─────── Unpaywall + Semantic Scholar openAccessPdf → every oa_location,
 │                          PMC→Europe PMC render, landing-page citation_pdf_url (repositories)
 ├─ 2. Publisher TDM API ─ Elsevier / Wiley / Springer official text-mining endpoints
 │                          (you register for your own key; entirely sanctioned)
 ├─ 3. Institutional proxy your library's off-campus remote-auth + EZproxy/NetScaler proxy
 │                          (your own login; for subscriptions you already have)
 └─ 4. Resolver link ────── print your library's SFX/OpenURL link → finish manually

Layer 1 works out of the box for anyone. Layers 2–3 are where the work is.

Two rules worth stealing:

• Validate %PDF magic bytes, never trust Content-Type. Paywalls and Cloudflare love
  to return 200 text/html that looks like a PDF response but isn't.
• Before blaming a route, check whether you're entitled to that article. A working route
  and a route you have no access through look identical. See
  docs/holdings.md — this is the single most expensive mistake in this
  problem space.

Publisher routes, worked out the hard way

The point of this repo. Each publisher exposes a different shape; these are the ones mapped
and verified against live articles.

10.1016 Elsevier goes through the TDM API in paper_fetch.py, never the proxy.

Adding a new publisher? Reach for the citation_pdf_url route first. Many sites with no
DOI→PDF template still advertise the exact PDF URL in a <meta name="citation_pdf_url"> tag on
the article page (it's what Google Scholar indexes). Resolve the DOI through the proxy, read the
meta, fetch it with the article as Referer. No reverse engineering needed.

And if a route comes back cf_block, try headful navigation before declaring it dead. BMJ was
documented here as a Cloudflare dead end that no stealth browser could clear. That was simply
wrong: the WAF only blocks headless requests, and a real headful navigation passes on the first
try — that's the whole _HEADFUL_META_PREFIXES variant. (Highwire sites like AJNR/JNM also loop
on the generic doi-org resolver → give them an explicit host so the route uses that site's own
/lookup/doi/.)

How this compares to other paper fetchers

Plenty of tools turn a DOI into a PDF. They differ on two axes: where the PDFs come from
and what happens when a route fails.

The two rows nothing else has — entitlement ground truth and log-driven route health — exist
because they answer the question every other tool leaves you guessing on: "is this route
broken, or do I just not have access to this article?"

Install

python -m patchright install chromium      # only needed for the institutional proxy layer

(The clone / pip install / cp config.example.yaml config.yaml steps are in the quickstart
above.) Then store publisher and library credentials in a local DPAPI secret store (Windows) —
never in config.yaml:

powershell -File ~/.secrets/secret.ps1 set ELSEVIER_TDM_KEY
powershell -File ~/.secrets/secret.ps1 set WILEY_TDM_TOKEN
powershell -File ~/.secrets/secret.ps1 set LIB_USER   # your own library account
powershell -File ~/.secrets/secret.ps1 set LIB_PASS

(Any secret store works — the code shells out to ~/.secrets/secret.ps1 get <NAME>; swap in
your own if you're not on Windows DPAPI.)

How to register for the publisher TDM APIs (Elsevier / Wiley / Springer / Unpaywall) →
docs/publisher-tdm-apis.md.

Use

python paper_fetch.py 10.1371/journal.pone.0000000 out.pdf   # OA / TDM — works out of the box
python paper_fetch.py --json 10.1016/xxx out.pdf             # agent mode: JSON envelope on stdout
python holdings.py 10.1097/xxxxx                             # do I even have access to this?
python library_session.py check                              # proxy layer (after you implement login)
python library_session.py fetch 10.1002/xxxxx out.pdf
python library_session.py stats                              # rate / block analysis

examples/example-note.md shows the intended Zotero + Obsidian workflow around it.

Agent mode (--json)

paper_fetch.py --json prints exactly one JSON envelope on stdout (all diagnostics go
to stderr), so an orchestrator can json.loads the last line without scraping logs:

{"schema": 1, "doi": "…", "ok": true, "route": "elsevier", "tried": ["elsevier"],
 "bytes": 117204, "sha256": "…", "path": "out.pdf", "elapsed_s": 1.4}

On failure ok is false, tried lists every route attempted, and resolver_url carries
your library's SFX link for a manual finish. sha256 lets batch callers dedupe.

Exit codes (one table for both scripts)

Codes 4 and 5 mean "try again, one at a time" — never record them as "no full text."
Conflating them is the single easiest way to wrongly conclude a paper is unobtainable.

⚠ Before you batch or parallelize

The proxy layer is strictly serial with a 15 s courtesy delay — publishers block an
entire institution's IP range when they detect systematic downloading, and the browser
profile is an exclusive resource (parallel callers deadlock, then wrongly record papers as
missing). Rate-limit reasoning, batch patterns, and how to call this from LLM agents →
docs/operations.md. Never wrap the script in timeout — it has its
own watchdog.

Adapting it to YOUR library

The proxy layer is deliberately a skeleton: the scaffolding (config, secret store, request
log, rate throttle, stats, host-rewrite, response classifier, the publisher route map,
two-phase orchestration) is all here and the technique is fully documented in the
docstrings — but login() and the LWW/Ovid multi-step flow are left for you to implement
against your own institution, because those are the parts that are specific to one library.

Off-campus access comes in four families (EZproxy, OpenAthens/Shibboleth, VPN, custom portal)
and only login() differs between them — docs/library-setup.md
walks you through identifying your family and finding your endpoints.

Then build your entitlement table → docs/holdings.md. It's what tells
"this route is broken" apart from "you don't have access to this article" — and those two look
exactly alike from the outside.

Not a programmer? That's the intended case. Point an AI coding agent at this repo and ask it
to wire up your library; AGENTS.md is written for it — deployment steps, the
verification smoke test, and the hard lines it must not cross.

Red lines

• For people who already have legitimate subscription access. It automates your own
  authenticated session; it is not a way around a paywall and not a way to share an account.
• Your account, your responsibility. Use your own credentials and follow your library's
  license terms and each publisher's ToS.
• Do not remove the rate limit to bulk-download. Publishers respond to systematic
  downloading by blocking the institution's whole IP range — your colleagues pay for it.
• Never commit config.yaml, *.dpapi, or access_log.jsonl (the .gitignore blocks them).

Full acceptable-use terms, and what is and isn't your responsibility →
DISCLAIMER.md. Short version: no warranty, use your own account, follow
your library's licence and each publisher's ToS, don't redistribute what you download.

MIT licensed. Contributions that add publisher route templates or adapt the proxy layer to
other library systems are welcome. Notable changes: CHANGELOG.md.

☕ Find this useful? Buy me a boba.