♟️ llama-modchip

Run 30B Sparse MoE LLMs on a pure CPU at 20–60 tok/s with a 256K context window. Zero GPUs required.

⚡ Quickstart · 📊 Benchmarks · 🧠 How It Works · 🏆 CPU Leaderboard · 📄 Technical Paper · ❓ FAQ

⚡ Quickstart

Clone the repository and run ./modchip chat. On first run, it automatically builds the patched llama.cpp engine (AVX-512 / AVX2 / Apple Metal), downloads Nemotron-3.5-30B-A3B (Q4_0), launches the background server + Web UI, and opens an interactive chat with live tok/s telemetry:

git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat

All ./modchip Commands

💡 Want to estimate your CPU's speed in 2 seconds without downloading anything?

curl -sL https://raw.githubusercontent.com/synchopate/llama-modchip/main/scripts/check_cpu_readiness.py | python3

📊 Benchmarks (Pure CPU — Zero GPUs)

Measured on an AMD Ryzen 9 9955HX (16C/32T Zen 5, 64 MB L3, 64 GB DDR5, 0 GPUs) running NVIDIA-Nemotron-3.5-30B-A3B-Q4_0:

♟️ Why a 1.64 MB Stockfish Chess Engine Beats a 1B Neural Draft Model on CPU

On GPUs (3.35 TB/s HBM3), running a 1B neural draft model takes < 1 ms. On a CPU (71.7 GB/s Dual-Channel DDR5), 1B neural draft models fail catastrophically: streaming 600 MB of 1B draft weights for 5 steps reads 3.0 GB from RAM (40 ms) — slower than running the 30B Sparse MoE target model itself (1.8 GB active $\rightarrow 25\text{ ms}$), while completely flushing the 64 MB L3 cache!

🧠 How It Works

We combine 4 core innovations (backed by 9 low-level C++/kernel patches documented in ARCHITECTURE.md and speculative/ROUTECRACK_TECHNIQUES.md):

1. ♟️ RouteCrack + Stockfish-PVS v2 (1.64 MB Chess-Engine Search in L3 Cache)
   RouteCrack acts as an $O(1)$ candidate move generator (pruning 131,072 vocabulary tokens to 4–8 candidates via an 8-level FNV-64 Hash Cascade, Online Session Trie, and 11,382-entry Macro Codebook). StockfishDrafter (rc_stockfish.h) acts as an orthogonal live-session tactician using Principal Variation Search (PVS) and 4-Way Bucketed Zobrist Transposition Tables (tt5/tt4/tt3), unlocking Singular Horizon Extensions up to M=7 tokens/step (98.74% marginal precision at $D=5..7$).
2. 📦 432 MB for 256K Tokens (Hybrid Mamba-2 + _MM_HINT_NTA Non-Temporal L3 Isolation)
   In Nemotron-3.5-30B-A3B, 46 blocks (23 Mamba-2 SSM + 23 Sparse MoE) maintain $O(1)$ recurrent state while only 6 layers use GQA attention (12 KV heads total = 432 MB at 256K vs 7.1 GB on standard GQA models). Our _MM_HINT_NTA patch streams cold MoE weights and cold KV blocks without evicting hot state from the CPU's victim L3 cache.
3. ⚡ Crushing the 5-Million-Page TLB Wall (2MB HugePages + AVX-512 VNNI Tile Repacking + Core Pinning)
   Collapses 4.9 million 4KB pages into 2MB Transparent HugePages (MADV_HUGEPAGE, expanding per-core L2 STLB coverage to 6.14 GB and eliminating 99.6% of page-table walks), repacks Q4_0 weights into interleaved AVX-512 VNNI / AVX2 SIMD tiles (+19% GEMV), and auto-tunes 1:1 physical core & multi-CCD thread placement (306.8 tok/s prefill).
4. 🔧 "Priiloader" Auto-Profiler & Safety Locks for Any GGUF (scripts/llama_modchip.py)
   Inspects any .gguf header in < 5 ms, writes live runtime controls to /dev/shm/llama_modchip.bin, prunes active MoE experts (6→4 on Nemotron for -33% DRAM traffic; 8→4 on Qwen3-30B-A3B for +38.5% decode with automatic gate renormalization), and engages 3:1 Interleaved SWA Safety Locks (LLAMA_SWA_STRIDE=4) on pure GQA Transformers.

Live Multi-Model Verification Matrix (python3 scripts/llama_modchip.py --audit)

Summary of All 9 Kernel & Engine Optimizations (Full Proofs in ARCHITECTURE.md)

1. Architecture-Aware Sliding Window Attention + 16 Sinks (LLAMA_ATTN_WINDOW=0 vs 4096, LLAMA_SWA_STRIDE) (ARCHITECTURE.md §3.1)
2. Bidirectional L3 Cache Isolation via _MM_HINT_NTA (LLAMA_KV_L3_WINDOW=2048) (ARCHITECTURE.md §3.2)
3. ISA-Adaptive In-Memory Weight Tile Repacking (--repack / q4_0_8x8) (ARCHITECTURE.md §3.3)
4. 2MB Transparent HugePages (MADV_HUGEPAGE Arena + MMap, 6.14 GB L2 STLB coverage) (ARCHITECTURE.md §3.4)
5. Dual-CCD & Multi-CCD Physical Core Pinning (OMP_PROC_BIND, LLAMA_KV_DISABLE_CHUNKING=1) (ARCHITECTURE.md §3.5)
6. Draftless C++ Speculative Engine (RouteCrack + Stockfish-PVS v2, M=7) (ARCHITECTURE.md §3.6 · speculative/ROUTECRACK_TECHNIQUES.md)
7. Entropy-Guided Asymmetric KV Quantization (EAKV + VPI, -ctk q8_0 -ctv q4_0) (ARCHITECTURE.md §3.7)
8. Zero-Prefill Agentic Loops (HARP-Cache + 18.9 ms /dev/shm Slot 0 Restore + SSE Gateway) (ARCHITECTURE.md §3.8)
9. Norm-Preserving Dynamic MoE Top-K Pruning (LLAMA_MOE_TOP_K=4) (ARCHITECTURE.md §3.9)

Manual Build & Custom Corpus Training

# 1. Enable 2MB Transparent HugePages (Linux Host)
echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo defer+madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# 2. Clone llama.cpp, apply Zen 5 / AVX-512 patch & copy RouteCrack + Stockfish-PVS into common/
git clone https://github.com/ggerganov/llama.cpp.git && cd llama.cpp
git apply /path/to/llama-modchip/patches/zen5-nemotron-256k-optimizations.patch
cp /path/to/llama-modchip/speculative/cpp/{rc_predictor.h,rc_trie_v2.h,rc_stockfish.h,ngram-map.cpp} common/

cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_AVX512=ON \
  -DGGML_AVX512_VNNI=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_BF16=ON -DGGML_OPENMP=ON
cmake --build build --config Release -j $(nproc) --target llama-server

Repository Structure

llama-modchip/
├── modchip                                    # Unified CLI (setup | start | chat | bench | web | claude | stop)
├── install.sh                                 # 1-Command Automated Installer (Linux)
├── README.md / ARCHITECTURE.md / FAQ.md       # Overview, Full Mathematical Paper & Engineering FAQ
├── RELEASE_NOTES.md                           # v2.1.0 Release Notes
├── patches/zen5-nemotron-256k-optimizations.patch
├── speculative/
│   ├── ROUTECRACK_TECHNIQUES.md               # Catalog of 96+ RouteCrack Techniques & Stockfish-PVS v2 Ablations
│   └── cpp/                                   # rc_stockfish.h, ngram-map.cpp, rc_predictor.h, rc_trie_v2.h, test_synergy.cpp
├── gateway/sse_gateway.py                     # Agentic SSE Proxy + Embedded Dark-Mode Web Chat & Telemetry HUD (:8787)
├── kv_cache/                                  # HARP-Cache (/dev/shm) & EAKV + VPI Asymmetric Quantization
├── scripts/                                   # chat.py, llama_modchip.py, check_cpu_readiness.py, prewarm_agent_prefix.py
├── apple-silicon/                             # Apple M4 Pro (24 GB) Metal setup & 128K benchmarks
└── fedora/                                    # Fedora 44 / AMD Ryzen 7 8745HS (GEEKOM A8) all-in-one script

🏆 Community CPU Leaderboard

Run ./modchip bench on your machine and ➕ Submit Your CPU Scorecard (1-Click GitHub Issue) to be added below!