agent-wiki

A reproducible pipeline that turns a folder of PDFs / EPUBs into a linked, provenance-anchored Markdown knowledge base — Karpathy's "LLM Wiki" idea, fully automated.

Status: Battle-tested on a 196-book corpus (165 ok, $104 spend, 8,249 wiki pages, 95 unit tests green). See docs/usecases/query-exercise-and-brain.md for an end-to-end query walkthrough using real output.

What it does

• YAML frontmatter (id, title, tags, related-page links)
• Provenance anchors like [raw/.../Spark.pdf#p43] — every claim is

clickable back to the source page

• **Explicit Inference: / Uncertain: markers** — agent never silently

blends opinion into facts

• **Cross-page [[wiki-links]]** — the same concept appearing in 8

books becomes one merged page with all 8 books' provenance

Quick start (5 commands)

Prerequisites

Python 3.10+ required (developed on 3.12 / 3.14).

Easiest path — interactive wizard

The wizard checks system deps, copies + opens .env, probes the LLM, takes your books folder (offering to copy + beautify filenames + classify into disciplines), runs the first book as a trial, and lands it into wiki/. Press Ctrl-C anytime — progress persists to tmp/setup_state.json and re-running resumes where you left off.

Manual path

That's it. Each book becomes one git commit; rolling back a bad ingest is one git revert.

How --books-dir works

Point at any folder. Two layouts work: