Tokenminning

Maximize intelligence per token
Not by minimizing tokens, but by delivering the right context at the right time

Case Studies • Principles • Techniques • Results • Contributing • Articles

🆕 What's New: v0.9.3 — a smaller README (case studies as one table), an Evidence column on every results table, and a runnable Quick Start. Previously: v0.9.2; full changelog.

Why Tokenminning?

Tokenminning is not: minimizing tokens to cut costs, aggressively shortening prompts, removing useful context, or avoiding large context windows.

Tokenminning is: improving information density, eliminating context debt, and designing systems where the right context arrives at the right time.

Quick Start

Tokenminning is a design philosophy; the tools here measure it on your own setup. Three steps, about five minutes:

git clone https://github.com/zwolf25/tokenminning && cd tokenminning

# 1. Audit what loads every session (stdlib Python, read-only, never writes)
python3 tools/lint/claude-md-lint.py

# 2. Read the output: resident size in chars and ~tokens, dangling @imports, duplicated blocks.
# 3. Apply one trim (delete a duplicate or an @import you never use), then re-run step 1.

Then: read the Principles, pick a Case Study, and apply one technique.

Principles

flowchart TD
    Q{"Need this information?"}
    Q -- NO --> N["Don't load it."]
    Q -- YES --> Y["Read the live source."]
    Q -- MAYBE --> G{"Grep first"}
    G -- Found --> F["Read only matches."]
    G -- "Not found" --> A["Ask the human or fail fast."]
    classDef ok fill:#12241F,stroke:#10C99A,color:#E8EAED
    classDef bad fill:#2A1E14,stroke:#F0A04B,color:#E8EAED
    class F,Y ok
    class N,A bad

Default: don't load unless justified.

[!NOTE]
These principles emerged from production Second Brain workflows at ServiceChannel, not theoretical speculation.

Techniques

Real-World Results

Evidence column: measured = counted from real data; estimated = derived with stated assumptions; single run = one observation. The data behind these is private; each case study states its method and caveats, and B-002 in the backlog tracks making them re-checkable.

[!NOTE]
These come from production Second Brain workflows, not synthetic benchmarks, except Case Study 10, which is labeled synthetic.

Tooling Measured Results

Production skill stack figures (from Case Studies 6–8). Token counts for these are author estimates; the case studies do not state a sample size.

Case Studies

One row each; the full problem, fix, result and caveats live in the linked file.

Technique Guides

Practical applications of tokenminning to specific systems:

These are technique guides — not measured case studies. Case studies in examples/ (second-brain-system.md, etc.) show before/after metrics.

How It Works

Tokenminning operates on a simple escalation ladder — the model only sees what it needs, when it needs it:

┌─────────────────────────────────────────────────────────────┐
│  LEVEL 1: LOCAL CACHE                                       │
│  Hot context, recent decisions, active task state           │
│  → Always loaded, ≤ 2K tokens                               │
├─────────────────────────────────────────────────────────────┤
│  LEVEL 2: STUB LOOKUP                                       │
│  Summary + source-path pointers to durable knowledge        │
│  → Loaded on reference, ~500 tokens each                    │
├─────────────────────────────────────────────────────────────┤
│  LEVEL 3: ON-DEMAND READ                                    │
│  Full source content retrieved only when explicitly needed  │
│  → Lazy, precise, unbounded depth                           │
└─────────────────────────────────────────────────────────────┘

The key insight: Most systems preload Level 3. Tokenminning makes Level 1 the default, Level 2 the bridge, Level 3 the exception.

flowchart LR
    subgraph B["Before: 211K words, full-body clones, high token cost"]
        direction TB
        b1["AI session"] --> b2["Read wiki 1"] --> b3["Read wiki 2"] --> b4["... Read wiki 68"] --> b5["Answer"]
    end
    subgraph A["After: 28K words, stub-first, on-demand retrieval"]
        direction TB
        a1["AI session"] --> a2["Read stub"] --> a3{"Relevant?"}
        a3 -- Yes --> a4["Read live source"] --> a5["Answer"]
        a3 -- No --> a6["Skip"] --> a5
    end
    B ~~~ A
    classDef bad fill:#2A1E14,stroke:#F0716A,color:#E8EAED
    classDef ok fill:#12241F,stroke:#10C99A,color:#E8EAED
    class b1,b2,b3,b4,b5 bad
    class a1,a2,a3,a4,a5,a6 ok

Related Projects

Contributing

We welcome real-world examples, counterexamples, benchmarks, and tool-specific patterns.

1. Case studies — Add to examples/ as <kebab-name>.md
2. Techniques — Document patterns in examples/ too
3. Counterexamples — Where tokenminning doesn't apply (valuable!)
4. Tool patterns — Claude Code, Cursor, Codex, Continue, etc.

See CONTRIBUTING.md for details, and BACKLOG.md for planned work.

Star History

Visual Reference

• Context Surface: optimize what crosses the boundary into the model
• Grep Before Read: 402 files filtered to 31 reads
• Knowledge Bloat Cascade: how full clones drift into stale copies
• Second Brain cover: dormant notes versus selective retrieval

License

MIT — see LICENSE for details.

Origin

The term "tokenminning" was introduced in July 2026 as a way to describe an emerging optimization philosophy centered on maximizing intelligence per token.

Inspired by the emerging concept of tokenmaxxing: intentionally using larger context windows and more tokens to improve AI performance. Tokenminning asks the opposite question: what's the minimum context that still produces maximum reasoning?