Karpathy’s LLM Wiki Pattern
A persistent, compounding knowledge base as interlinked markdown files.
The wiki compiles knowledge once and keeps it current. Cross-references are already there. Contradictions have already been flagged. Synthesis reflects everything ingested.
Key Ideas
- Agents summarize, cross-reference, file, and maintain consistency
- Wiki is just a directory of markdown files
- Three layers: raw sources, wiki pages, and schema
- Each page must link to at least 2 other pages
The Transformer
The transformer architecture uses self-attention to process sequences. Unlike RNNs, it can see the entire sequence at once.
Self-attention computes query, key, value matrices for each token, then calculates attention scores based on dot product similarity.
This architecture was introduced in the paper “Attention Is All You Need” (2017).