textbook-to-note

Turn your own PDF textbooks into an AI-searchable knowledge base and structured, fully-cited notes — figures included. A local-first pipeline that spends (almost) zero LLM tokens on the heavy lifting and reserves the frontier model for the one thing it's uniquely good at: synthesizing a note you can actually learn from.

繁體中文說明 → README.zh-TW.md

Why I built this

I've loved taking notes since med school, and over the years I've built up thousands of them — but I can no longer keep every one at the same quality by hand. In an age of information overload, trustworthy high-quality sources become the precious thing, and textbooks are among the best; yet my specialty alone has 40+ designated reference books, with a single concept scattered across chapters in many of them. Reading them all cover to cover just isn't realistic. LLMs are great at long context, but you can't dump hundreds of books on one at once — so the real unlock is pairing good search and a database with teaching the AI my own note-making process, so it produces grounded, structured notes I only have to absorb.

Left: one of the templates in templates/. Right: a real note in my vault, written against it — every claim traceable, every section where I expect it.

Why this is hard (and why naive approaches fail)

Feeding a raw PDF to a frontier model seems simple until you hit the real problems:

• Cost & latency — a 600-page book is 1-2M tokens. Re-reading it for every question is untenable.
• Silent data loss — scanned pages and broken font encodings produce garbage that the model quietly skips. You never learn your note is missing half the chapter.
• Scrambled text — most textbooks are two-column. Default PDF text extraction interleaves the columns, and a naive "is this garbled?" check passes the shuffled output because the characters themselves are fine.
• Figures vanish — the anatomy diagram, the classification table, the treatment algorithm: often the most valuable part of a chapter, and text extraction drops all of it.
• Ungrounded output — an AI writing notes from memory hallucinates. Every claim needs to trace back to a source.

This pipeline treats each of those as a distinct engineering problem with a deliberate solution, organized as five stages.

The five stages

1 · Convert — PDF/EPUB → clean markdown, 0 tokens

PyMuPDF text extraction at ~130 ms/page. Two non-obvious pieces:

• Silent-failure detection — a native text layer can lie (CID/Identity-H fonts, PUA codepoints). We score glyph-garble ratio, character density, and font risk to catch pages that "extracted fine" but didn't, and route only those to OCR.
• Column-aware reading order — line-level column clustering reconstructs true reading order on two-column layouts, with an exact-fallback path so single-column pages are byte-identical to the trivial extraction. (T2N_COLUMN_SORT=0 to disable.)
• Table pass, gated — pdfplumber's table detection is the slowest part of conversion. We gate it behind a cheap fitz pre-check (ruling-line signature incl. three-line tables, plus multilingual "Table"/表 keywords), so table-sparse books convert ~3.4× faster without missing tables. (T2N_TABLE_GATE=0 to disable.)
• Cross-page table merge — textbook tables often run past a page break. Opt in with T2N_TABLE_MERGE=1 to stitch a table that ends near a page bottom to a geometrically-matching table at the top of the next page (same column count / x-edges, no intervening heading), deduping a repeated header row and leaving a <!-- table continues from page N --> trace comment. Same exact-fallback discipline: default OFF ⇒ byte-identical output.
• Page-frame pseudo-table rejection — a page's decoration (a content frame plus the rule under the running header) gives pdfplumber enough intersecting edges to "find" a table covering the whole page body, with 1 column and every word on the page dumped into a single cell. That output is worse than a missing table: a real multi-column table arrives with its columns interleaved line by line, carrying its caption and every value, so it reads as clean citable data while the row↔column binding is destroyed. Candidates of one column whose largest cell exceeds 500 characters, or whose bbox alone covers half the page, are dropped and replaced by a <!-- ⚠️ page-frame pseudo-table rejected on page N --> comment — the page's own prose already carries the text. Measured at 9.9% of extracted tables across 128 books; on a hand-read sample of 34 pages from 10 books, 28 were genuine defects and the 6 legitimate boxed lists lost nothing (100% of their tokens are present in the page prose). Default ON — this corrects wrong output; T2N_TABLE_FRAME_REJECT=0 restores the old behaviour.
• Whole-book table-failure warning — table loss is bimodal: a book either extracts tables fine or silently loses every one. If pdfplumber parses 0 pages while fitz opens the file, or a book yields 0 tables despite ≥10 table captions, the conversion report and the markdown itself get a loud warning instead of nothing at all. Detection only — extraction is unchanged. Fires on 22 of 34 zero-table books in our corpus and on none of the 226 that extract tables. (T2N_BOOK_TABLE_CHECK=0 to disable.)

Scanned pages fall through an OCR ladder — Surya → PaddleOCR-VL → local vision model → frontier vision as the true last resort — chosen per-page, never committing a whole book to one method. See docs/ocr-ladder.md, including a hardware-tier model-selection table (no-GPU / Apple Silicon / NVIDIA 8GB / 16GB+) so you pick an engine and ollama model that actually fit your VRAM instead of OOMing mid-book.

2 · Chunk — into semantically searchable units

A table of contents is too coarse (one topic spans many chapters); manual tags don't scale and can't be granular enough. The answer is semantic embedding. But chunking is a design decision, not a fixed window: split too small and you lose context, too large and you dilute meaning. We chunk by heading structure and carry each chunk's parent-section context, so every retrieved unit is a self-contained concept with its provenance attached.

3 · Retrieve — find the right book among dozens

Cross-book search runs on the same engine as its sibling project vault-search — local LanceDB + bge-m3 embeddings, nothing leaves your machine. On top of plain similarity we add source weighting: boost your most-trusted references (exam-designated texts, official society textbooks) and down-weight by edition age, so on any topic the AI reaches for the source you trust first.

4 · Write — a note-writing algorithm, not a prompt

Note quality comes from the workflow, not the model:

• Blind draft first — the AI researches the topic fully from the textbook corpus before looking at any existing note, so an old note's structure and content can't bias the new draft. Merge comes last.
• Template-driven extraction — each topic type has a fixed template. This is load-bearing: it tells the AI exactly what to hunt for in the source, and it means every note has the same predictable shape so you read faster. Sections like a leading Summary and a Management-algorithm block exist specifically to aid comprehension, not just to hold data. The exact templates used daily are included under templates/, in both the original 繁體中文 and an English translation.
• Cite or it didn't happen — every claim carries book + chapter; anything the AI adds from its own knowledge is explicitly flagged as inferred.
• Non-destructive merge — replacing an existing note is gated on your vault being under version control; otherwise it writes a draft beside the original. It will not silently overwrite your hand-written notes.

See workflows/note-writing.md.

5 · Extract figures — the hard one

Every book lays figures out differently, so there's no single crop rule. We use a general geometric-matching method (a caption owns the nearest assignable raster) behind a deterministic QC gate: whitespace-fill, text-bleed, and OCR-long-line checks all run before any AI is allowed to judge the crop — and the gate hard-fails rather than guessing, so a wrong page yields a refusal, not a wrong figure. Everything runs locally, token-frugally.

When a specific book extracts wrong, you fix that book's logic once, and every later extraction from it is correct. This stage has been through many iterations and is still experimental — it doesn't yet handle every book, and improvement PRs are very welcome. See figures/CALIBRATION.md.

Bonus · Pluggable evidence enrichment

The note workflow has optional hook points to enrich a draft from external sources — a clinical-evidence API, a regulations/coverage database, a literature search. They live outside this repo to bound its scope; the workflow doc marks exactly where they slot in so you can wire in your own domain's sources.

The clinical-evidence hook I use daily is published separately as openevidence-tools — an OpenEvidence ask tool paired with a verifier that checks the returned citations. It takes htlin222/openevidence-mcp as its reference and inspiration; it is not a fork. Always run the verify step — cited sources can be wrong.

Designed to be deployed by an AI

You're probably here to have your AI set this up — that's the intended path:

Point Claude Code (or any capable coding agent) at this repo and say:
"Read AGENTS.md and set this up for me."

AGENTS.md is written for the agent: dependency install, configuration, converting a first book, installing the two Claude Code skills, running the note workflow, plus token guardrails so a naive agent doesn't burn a million tokens reading a whole book into context.

Repo layout

converter/    PDF/EPUB → markdown (convert.py — silent-failure + column-sort + table-gate)
figures/      figure extraction + deterministic QC gate (figure_remap.py entrypoint)
skills/       drop-in Claude Code skill definitions (textbook-to-md, figure-remap)
workflows/    the note-writing algorithm (adapt to your own note system)
docs/         architecture, OCR ladder + hardware-tier model table
examples/     a sample output note showing the target format
shared/       env-driven configuration (config.py)

Requirements

• Python 3.10+, pip install -r requirements.txt
• CPU-only is a first-class path for born-digital PDFs (the common case) — no GPU needed
• Optional, for scanned books and figure QC: an NVIDIA GPU or Apple Silicon + Surya OCR, ollama with a small vision model and bge-m3 for embeddings — all local, nothing leaves your machine. See the hardware-tier table in docs/ocr-ladder.md.
• Tested on Windows 11 and macOS; Windows-specific gotchas are handled in code (cp950 subprocess decoding, atomic-ish path ops)

Philosophy

• Local-first, token-frugal — the expensive AI is reserved for synthesis, never mechanical page-by-page reading.
• Deterministic gates over AI vibes — every figure crop and OCR page passes rule-based QC before an AI is allowed to judge it, and thresholds are never tuned just to make a failing case pass.
• Citations or it didn't happen — every claim traces to book + chapter; AI-inferred additions are flagged.

Bring your own books

This tool ships no textbook content. It operates on PDFs you already own — purchased ebooks, institutional-access downloads, open-licensed texts (OpenStax), or scans of your own paper books where your local law permits. Respect your books' licenses.

Related

• vault-search — the local semantic-search engine stage 3 builds on.
• openevidence-tools — the OpenEvidence ask + verify pair that plugs into the Bonus stage.

License

MIT © Po-Wei Chen (drpwchen)

A note on dependency licenses

This project's own code is MIT. Two dependencies carry stronger terms you should be aware of if
you redistribute or offer this as a network service:

• PyMuPDF (fitz), a required dependency, is AGPL-3.0 (or a commercial license from
  Artifex). It is the core PDF reader and is not optional. If you build a closed-source product
  or a hosted service on top of this tool, evaluate that obligation for yourself.
• Docling, the optional table engine (T2N_DOCLING=1), is MIT end-to-end (docling,
  docling-core, docling-ibm-models, docling-parse). It is not installed by default and
  is not in requirements.txt; it runs in its own virtual environment and is reached by
  subprocess, so it is a pluggable tool rather than a linked dependency.

This is disclosure, not legal advice — licenses change, so verify against the versions you
actually install.