lit-review — Literature Fetching & Citation Auditing Skill for LLM Agents

English | 繁體中文

An agent skill that turns Claude (or Codex, or any capable LLM agent) into a rigorous literature assistant. Give it your draft — it finds supporting literature, and audits every citation you already have: does the paper exist, is the bibliography correct, and does it actually support the claim you attached it to?

Built by a grad student finishing a thesis, battle-tested on real thesis chapters. No API keys required to start.

Architecture

(Diagram generated by assets/gen_diagram.py, which emits English and Chinese variants in light and dark — edit text or colors and re-run; the palette is CVD-validated. A Mermaid source lives below for quick in-browser edits.)

flowchart TB
    U["User input<br/>draft / topic / command"] --> R{"Mode router<br/>explicit command wins; otherwise inferred"}

    R -->|find| A["Mode A<br/>Find literature"]
    R -->|check| B["Mode B<br/>Audit citations"]
    R -->|write| C["Mode C<br/>Grounded writing"]
    R -->|"map / gap / matrix / notes / annotate<br/>counter / strength / claims / integrity"| G["Grad toolkit"]

    A --> E
    B --> E
    C --> E
    G --> E

    subgraph E["Shared engine — scripts/lit_api.py (stdlib only)"]
        direction LR
        S2["Semantic Scholar"] -. "fallback on 429" .-> OA["OpenAlex"]
        AX["arXiv"] -. "fallback" .-> S2
        CR["Crossref<br/>bibliographic authority"]
    end

    E --> V["Verification<br/>dual-path identity gate · quality flags · adversarial self-audit<br/>evidence levels (abstract / full text p.X / unknown)"]

    V --> O1["Audit report<br/>+ author fix list"]
    V --> O2["RIS / BibTeX / EndNote XML"]
    V --> O3["Cited article · matrix · note cards"]

Three honesty mechanisms run through every path: bibliographic fields come only from API responses (never from LLM memory), every verdict carries its evidence level, and generated output is audited by a fresh-context skeptic before delivery.

What this is not

If you already found a paper and just want its citation string, use the Google Scholar Button (official, by Google) — it is faster and this tool does not compete there. It also finds full text through your university library, which is the proper licensed route and better than anything this tool can legitimately do.

This tool starts where that ends: with citations you already have — from an AI, a co-author, an old draft, or a copy-paste that went wrong.

Measured, not asserted: Scholar Button results carry no DOI and collapse volume/issue/pages into a truncated string, so they cannot drive bibliographic verification. Conversely, it surfaced a free repository PDF for a paywalled review that Unpaywall reported as closed — fulltext recovers the same URL via a fallback source, but the lesson stands: for getting the paper, use Scholar Button and your library.

Finding you a topic is also out of scope. gap <X and Y> verifies an intersection you have already named: two or three re-worded queries, the nearest neighbours, and an honest "not found ≠ nobody did it". It does not generate candidate directions and will not tell you what to research. If you have no X and Y yet, that is a different job — a separate skill, research-gap-hunter, does it, and lit-review works exactly the same whether or not that skill is installed.

Nor is surveying the techniques. map <topic> charts the literature: which papers are seminal, who the key authors are, where the last three years went. It does not chart the approaches — what methods are in play, what each one buys and costs, which are saturated. That is a technique survey, and research-gap-hunter's landscape does it. Wanting both for the same field is reasonable; they are not substitutes for each other.

What it does

Mode A — Find literature. Extracts claims from your draft, searches Semantic Scholar / OpenAlex / arXiv, snowballs through citation networks to find classics keyword search misses, judges candidates by their abstracts (not titles), and exports an EndNote/Zotero-ready .ris file.

Mode B — Audit citations. For every reference in your draft:

1. Existence — cross-checks Crossref + Semantic Scholar; catches hallucinated references (and knows that not found ≠ fabricated)
2. Retraction — every DOI is checked against Crossref update notices (incl. Retraction Watch data), by default, at zero LLM cost
3. Bibliography — field-by-field diff against Crossref authoritative records (years, venues, author spelling, page numbers); arXiv IDs are cross-checked against their actual titles
4. Claim support — reads the abstract and checks whether the paper supports the specific sentence citing it; flags overreach ("paper says correlation, you wrote causation"), null-result traps, and population mismatches
5. Equations & technical details — when your text attributes a formula to a source, fetches the open-access PDF and compares term by term

Mode C — Grounded writing. Give it a topic; it retrieves first, writes second, and self-audits last — every cited claim traceable to retrieved evidence, expository passages explicitly uncited.

Grad toolkit. Thirteen thesis-lifecycle tools: literature matrix, field map, research-gap detection (for an intersection you name), note cards, citation-integrity check, terminology consistency, committee-question rehearsal, new-literature watch, citation-needed annotation, counter-evidence search, evidence-strength grading, claim–evidence ledger, retraction lookup.

Quick start

Claude Code — as a plugin. The first line registers the marketplace; the second installs from it, so run both, in order:

claude plugin marketplace add Zachariah9420/claude-research-skills

claude plugin install lit-review@zachariah-research-skills

Or clone it as a skill, if you would rather not add a marketplace:

git clone https://github.com/Zachariah9420/lit-review-skill ~/.claude/skills/lit-review

Then just ask: "check the citations in my chapter2.docx" or "find literature for this paragraph: …" — the skill infers the mode from your input.

Or use explicit commands (same words work in Claude Code as /lit-review <cmd> and in Codex as plain chat):

Codex / other agents: add one line to your AGENTS.md:

For literature search and citation auditing, read <clone-path>/SKILL.md and follow its workflow.

The helper script is pure Python 3.8+ standard library — no pip installs, works anywhere.

Full walkthrough: USAGE.md

Optional: API keys (recommended for heavy use)

Create a .env in your project directory:

S2_API_KEY=...        # free from semanticscholar.org/product/api — avoids shared-pool 429s
[email protected]   # joins the Crossref/OpenAlex polite pools

Never commit .env. The script reads it automatically (cwd first, then ~/.env).

Packaging for upload (ChatGPT Skills, or sharing)

A cloned folder is not directly uploadable — zipping it would include the
.git directory, and the top-level folder would be named after the repo rather
than the skill. One command produces a correct package:

python scripts/package_skill.py          # → lit-review.zip

It takes the file list from git ls-files, so the package contains exactly what
a clone contains (nothing gitignored, no .git, no .env), wraps it in a
lit-review/ top-level folder so SKILL.md sits where the platform expects it,
then runs evals/zip_check.py for API keys, personal email addresses and
machine-specific paths. If that check fails the ZIP is deleted rather than
handed to you — a package that leaks is worse than no package.

Verified, not just written

The honesty claims above are enforced by tests, not by good intentions:

python evals/test_regression.py     # 74 frozen cases, no network, seconds
python evals/mutation_check.py      # re-introduces 9 fixed bugs; each must be caught

test_regression.py freezes every defect found so far — each case is tagged with where it came from (TS-* black-box stress test, CX-* independent Codex source review, DR-* design review). It calls the production functions directly with frozen candidate data, so it needs no API access and runs in seconds.

mutation_check.py answers the question a green test suite cannot: does it actually detect anything? It re-introduces each fixed bug one at a time and requires that a specific case fails. This caught two of my own tests that had quietly re-implemented the ranking logic instead of calling it — tests that would have passed forever while the real code rotted.

What has been exercised on real material: a planted-error draft (wrong year, wrong venue, fabricated reference, Chinese-language reference, uncited claim), two real thesis chapters, a labmate's 36-citation reference list (which turned up a misattributed author and a wrong page range), equation checking against an open-access PDF, and a cross-check of the same audit against an independent literature tool — which agreed on every verdict and surfaced one genuine gap (abstract coverage for publishers the free APIs do not index).

What has not been exercised: other people's fields (medicine, social science, humanities have different citation conventions and very different API coverage), author-year (APA) documents, longitudinal use across weeks, and a full audit driven by Codex rather than Claude. Those are the honest frontier.

Design principles

• Honesty over polish. "Not found" is reported as not found; "can't judge" is ❓, never a guess. One wrong "verified" is worse than ten honest "unverified".
• No bibliography from memory. Every DOI, year, and page number comes from an API response — LLM memory of bibliographic data is unreliable by construction.
• Isolation firewalls. Evidence isolation (model memory may raise suspicion, only supplied evidence may convict), role isolation (the writer never audits itself), injection isolation (retrieved abstracts/PDFs are untrusted data — embedded instructions are reported, never followed).
• Quality red flags. Zero-citation papers in low-tier venues get flagged automatically (with an age gate so new papers aren't punished for being new).
• Cost tiers. quick (single agent, labeled "no independent audit") / default (one fresh-context skeptic, ~1.5–2×) / thorough (per-claim adversarial verification, 5–10×) — verification intensity follows the cost of being wrong.
• Resilient. Automatic fallbacks: Semantic Scholar → OpenAlex, arXiv → Semantic Scholar. Rate limiting and 429 backoff built in.

Field applicability (measured, not assumed)

Four fresh sessions ran this tool over medicine, pharmacy, social science and humanities literature with no prior knowledge of the project. The pattern they found is structural: reliability falls off with distance from CS/AI, because the tool's assumptions are CS/AI assumptions — that a work has a DOI, that its title is a unique key, that the author field is populated, that an abstract exists, and that venue-quality data comes back with the search.

Three things are trustworthy in every field tested: a positive retraction result, the paywall verdict from fulltext, and integrity refusing to judge a citation style it does not support. Outside CS/AI, a found verdict deserves one look at the candidate's type and container before you act on it.

Honest limitations

• Default judgments are abstract-based; "gap" findings mean the abstract doesn't show it, not the paper doesn't contain it. For verdicts the abstract can't settle, the skill escalates to open-access full text when available (and labels the evidence level) — but paywalled papers stay honestly marked "cannot judge"
• Chinese-language literature: best-effort via Google Scholar when available, otherwise flagged for manual check (the structured APIs barely cover it)
• Books (especially pre-2010): APIs often only index their reviews — the skill uses reviews as indirect existence evidence and tells you to check the ISBN
• Industry proceedings (IPC/SMTA-style) are a blind spot for every database, including commercial ones — the skill says "not found in any index, verify with the publisher", never "fabricated"
• PubMed/MEDLINE is not searched. Drug package inserts, FDA/EMA approval documents, clinical databases (UpToDate, Micromedex, Lexicomp), pharmacopoeias and ClinicalTrials.gov are all out of range — a not-found there is expected, not a finding
• The tool makes bad citations harder to produce, but the selection and interpretation of literature remains the author's responsibility — it's a guardrail, not a chauffeur

Example

examples/planted_errors_test.md is a test draft with five planted problems (wrong year, wrong venue, a fabricated reference, a Chinese-language reference, an uncited claim) — run the skill on it and see whether everything gets caught.

MIT License. Issues and PRs welcome.