benchbook

Your AI keeps the wiki. You keep the rules.

ulef1005.github.io/benchbook — docs, demo wiki, and the contract, rendered

TL;DR — benchbook is a personal AI-maintained wiki system. The core idea: you work with an
AI agent on projects, and instead of losing the reasoning behind decisions, the agent files it
into a structured markdown wiki — under a written contract that governs what it may write, what
it must ask you about, and what it can never touch.

The three key problems it solves:

• Reasoning decay — code survives, but why you made a decision doesn't. benchbook captures
  rejected alternatives, the reasoning, and the context alongside the artifact.
• Wiki abandonment — humans stop maintaining wikis because it's tedious. An AI doesn't get
  bored, so maintenance cost drops to near zero.
• Wiki bloat — the counterintuitive failure mode: when maintenance is free, you get too much
  content. A significant chunk of the contract exists to make the agent write less.

You build something with an LLM on a Tuesday. It works. Three months later you open the
folder and have no idea why you chose that library, what the two rejected approaches were,
or which of the four config files is the live one. The code survived. The reasoning didn't.

benchbook is the system I use to fix that: a plain markdown wiki, in git, maintained by an AI
agent under a written contract — a file the model reads at the start of every session that
tells it what it may create, what it must ask about, and what it is never allowed to touch.

It is not tied to any one assistant. Any agentic tool that can read and write files in a repo
can run it.

No app. No database. No embeddings. Just files an agent is disciplined about.

See it work

The fastest way to understand this is to watch it handle three ordinary tasks. Each one is
walked step by step in A Day in the Life:

🔧 You saw a funky ESP32 project on YouTube and want to build it.
The video becomes a source page with the transcript embedded. The build becomes a separate
project page — and nothing gets written until you've approved a plan. Eight months later,
when the display goes blank, the pinned library version and the reason it's pinned are one
click away.

🤖 You want a self-hosted Telegram bot, but don't know which repo to use.
Research four candidates, record why the most popular one lost, then deploy with the seven
boring integration steps pre-filled and unticked so none of them get skipped. The bot token
never touches the wiki. When the framework ships a major version, release tracking tells you.

🚁 You want to learn to fly FPV drones.
Not software at all. A learning path with gates between phases, a price baseline before you
shop, and — the valuable part — every gear option you rejected, with your reasoning, dated.
So when someone enthusiastically recommends the goggles you already ruled out, you know why.

Each example names the command that runs, what you decide versus what the agent does, and
exactly what ends up written down.

The problem it actually solves

Building with an LLM is fast, and that speed is the trap. You produce more decisions per week
than you can remember, and none of them write themselves down. Six months of that leaves you
with a homelab, a pile of scripts, and a strong suspicion that past-you had reasons.

What you want to recover later is rarely the code — it's:

• How was this built (which stack, which wiring, which files are live)
• When was it built, and what has changed since
• Why was it built that way — including the options that were considered and dropped

A README in each project captures the first. Git captures the second. Almost nothing captures
the third, and the third is the one you actually miss.

benchbook captures all three as a side effect of working, because the agent that helps you
build is the same agent that files the record.

Lineage

This is an instantiation of Andrej Karpathy's
LLM Wiki pattern — a
deliberately abstract "idea file" describing three layers (immutable raw sources, an
LLM-owned wiki, a co-evolved schema file) and three operations (ingest, query, lint). It
closes by saying the directory structure, schema conventions, page formats and tooling "will
depend on your domain, your preferences, and your LLM of choice," and invites you to build a
version that fits.

benchbook is what that turned into after six months of daily use. The three layers and the
three operations survived unchanged — they're right. Most of this repo is the specifics the
gist deliberately left open, plus the rules that turned out to be necessary past the scale it
scopes for.

What got added along the way:

The one place six months disagreed with the gist

The gist's argument for why this works is that humans abandon wikis because the maintenance
burden grows faster than the value, and LLMs fix that because they don't get bored and the
cost of maintenance falls to near zero.

That's true, and it creates a second problem the gist doesn't anticipate: when maintenance
costs nothing, you get too much of it. The failure mode in practice was never a neglected
wiki. It was a wiki growing faster than anyone could read — logs filling with narrative that
already existed on a project page, lists mirroring state that then drifted, pages accreting
past the point of usefulness. Boredom was never the enemy. Enthusiasm was.

So a real chunk of the contract exists to make the agent write less, in more specific
places. The numbers behind those rules are in
11 — Keeping It Honest, and they're the part of this repo I'd
read first.

What it is, concretely

Six moving parts, five of them text and one a short script:

The agent plays three roles against that structure — Librarian (files what comes in),
Advisor (answers from what's filed), Project Manager (tracks what's in flight).

Rules that exist because something broke

Running since May 2026, currently 1,597 pages across 9 domains with 20 skills and a
log going back to the first week. The useful output of that isn't the page count — it's the
rules that are in the contract today only because the obvious version failed in practice:

• update was removed as a valid log operation. An audit found 17 of 26 recent entries
  were update, averaging 139 words against a 1–3 line spec, mostly duplicating text already
  written on a project page in the same session. The log was growing ~10× faster than intended.
• The central todo file was deleted. It failed twice: first as a mirror (60 of ~76 copied
  items had silently drifted), then, after being cut back to pointer-only, as a list where
  roughly a third of the remaining entries had a stale or wrong premise. A file whose job is to
  tell you what's open, and which is a third wrong, is worse than no file — you trust it. Open
  work now lives on the page that owns it.
• The "split pages over ~500 words" rule was made measurable, with explicit carve-outs,
  after it turned out 27% of all pages breached it — a rule that flags a quarter of your
  content steers nothing.
• The predicate vocabulary doubled, because its checker had a blind spot. The check that
  enforced it was scoped to one folder and reported clean for months; measured end to end, 185
  of 709 relations sat outside that scope carrying 124 off-vocabulary predicates. A check that
  is right about the region it inspects is not the same thing as a check that is right.

Anyone can write a schema. The interesting part is which parts of it survived contact with
six months of real use, and that is most of what the docs in this repo are about.

Things built with it

Not demos — these run:

• A daily two-host podcast. Calendar, health metrics, weather and news → script → TTS →
  a private feed, generated on demand for the morning commute.
• A used-marketplace buying assistant. Hourly listing pickup, scored against
  project-specific criteria by an LLM, high matches pushed to Telegram with a drafted
  seller message and a price-history baseline.
• A book recommendation pipeline. Captures recommendations from social posts and
  screenshots, collision-checks them against a ~350-book library across four shelves,
  and ranks what's left against a taste profile.
• A self-hosted health data store. Wearable providers into a governed local store,
  feeding a coaching layer with weekly reports and early-warning detection.

Each one has a project page recording its phases, its rejected options, and — in two cases —
an honest post-mortem on the parts that didn't validate.

Quickstart

Requires an agentic tool that can read and write files in a repository — Claude Code,
Codex, Cursor, or similar. No install script, no dependencies.

git clone https://github.com/Ulef1005/benchbook.git my-wiki && cd my-wiki

Then:

1. Read agents-core.md — it's the contract. It's meant to be edited; it's yours now.

2. Look at the demo content, then delete it. wiki/DEMO.md says what each page
   demonstrates and gives you the one-line removal command. Keep the folder shape.

3. Live on the three installed domains for a while. knowledge, home and projects are
   the minimum set that makes the placement rules work. Add a fourth when you reach for it and
   it isn't there.

4. Start your agent in the repo and say "read agents-core.md". It will greet you and
   wait. (CLAUDE.md and AGENTS.md both point at the contract, so most tools pick it up
   without being asked.)

5. Ingest one thing — an article, a video, a decision you already made — with
   /wiki-ingest. Watch where it files it and correct it. That correction is how the
   contract gets tuned to you.

6. Turn on the checks (optional, 30 seconds, worth it):

   python3 scripts/lint.py                                    # full report
   ln -s ../../scripts/pre-commit.sh .git/hooks/pre-commit    # block commits on errors
   

   Pure standard library — no install, no dependencies. Errors block a commit, warnings never
   do, and git commit --no-verify bypasses it when you mean to.

Then read A Day in the Life, which walks three real tasks end
to end — it'll make the rest of the docs land faster than reading them in order.

Documentation

A Day in the Life — the three worked examples from the top of
this page, in full. Start here if you haven't already; the numbered docs below are the
reference material behind it.

Status

Early, but no longer sparse. The system is mature; this repo's packaging of it is catching up.
Docs, contract, skills, demo content, working checks and a live site are all in.

The repo tracks the wiki it came from, so it changes when the rules change — and the rules change
when they break. See 11 — Keeping It Honest for what broke most
recently.

License

MIT.