Add full README with verified benchmarks and research findings

README.md

@@ -1,3 +1,124 @@
----
-license: mit
----
+---
+library_name: gguf
+tags:
+  - moe
+  - expert-prefetch
+  - madvise
+  - llama-cpp
+  - apple-silicon
+  - cuda
+  - on-device
+license: mit
+pipeline_tag: text-generation
+---
+
+# llama.cpp Expert Sniper — madvise prefetch for MoE inference
+
+~65 lines of C++ that enables MoE models larger than RAM on llama.cpp.
+
+Stock llama.cpp thrashes indefinitely. This build generates tokens.
+
+## Results
+
+| Hardware | RAM | Model | Stock llama.cpp | madvise build |
+|----------|-----|-------|-----------------|---------------|
+| M2 MacBook Air | 8 GB | Qwen3.5-35B-A3B IQ2_M (10.6 GB) | 0 tok/s (thrash) | **0.57 tok/s** |
+| M2 MacBook Air | 8 GB | Same model, no-op callback only | 0 tok/s (thrash) | 0.46 tok/s |
+
+On GPU machines with abundant RAM (A100 251GB, RTX 3090 31GB), stock llama.cpp is faster — the OS page cache handles it. madvise helps specifically when **system RAM < model size** and **layers are on CPU (ngl 0)**.
+
+## How it works
+
+MoE models activate 8 of 128+ experts per token. Consecutive tokens share ~87% of active experts. Stock llama.cpp uses mmap but the OS has no idea which expert pages are hot — it evicts them randomly, causing a page fault storm.
+
+Our patch hooks llama.cpp's eval callback, intercepts every `ggml_mul_mat_id` operation, reads the router's top-k expert selection from `t->src[2]`, and calls `madvise(MADV_WILLNEED)` on each active expert's memory range. This tells the kernel which pages to prefetch before the compute needs them.
+
+Zero allocation. Zero memcpy. Zero mutex. One syscall per expert slice.
+
+## What we learned
+
+**1. madvise beats LRU cache everywhere.** We first built a 460-line LRU cache. It was 2.4x slower than 15 lines of madvise (0.24 vs 0.57 tok/s on 8GB MacBook). The cache stole 5GB from the OS page cache for duplicate data. Don't fight the OS page cache — coach it.
+
+**2. Even a no-op callback prevents thrashing.** Just hooking the eval callback and inspecting tensor pointers (without any madvise) produces 0.46 tok/s where stock produces zero. The callback inadvertently warms mmap pages through pointer inspection.
+
+**3. Device pointer bug.** All prior GPU benchmarks were silently invalid — the callback dereferenced `t->src[2]->data` without checking `ggml_backend_buffer_is_host()`. On GPU layers this was a CUDA device pointer. Fixed.
+
+**4. On abundant RAM, do nothing.** The OS page cache is remarkably good when it has enough room. Any intervention is pure overhead when RAM exceeds model size.
+
+## Build
+
+```bash
+git clone https://github.com/walter-grace/mac-code
+cd mac-code/research/expert-sniper/llama-cpp
+
+# macOS (Metal)
+cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
+cmake --build build -j$(nproc) --target llama-server
+
+# NVIDIA GPU (CUDA)
+cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
+cmake --build build -j$(nproc) --target llama-server
+```
+
+## Usage
+
+```bash
+# Enable madvise prefetch
+./build/bin/llama-server \
+  -m Qwen3.5-35B-A3B-UD-IQ2_M.gguf \
+  -ngl 0 \
+  --expert-cache-size 1 \
+  --port 8201
+
+# Test no-op mode (isolate callback overhead)
+EXPERT_CACHE_NOOP=1 ./build/bin/llama-server \
+  -m model.gguf \
+  -ngl 0 \
+  --expert-cache-size 1 \
+  --port 8201
+```
+
+## Files changed vs stock llama.cpp
+
+**New files (~430 lines):**
+
+| File | Purpose |
+|------|---------|
+| `src/llama-expert-cache-ctx.cpp` | Eval callback, madvise prefetch, tensor identification |
+| `src/llama-expert-cache-ctx.h` | Context struct and declarations |
+| `src/llama-expert-cache.cpp` | LRU cache (deprecated, retained for reference) |
+| `src/llama-expert-cache.h` | Cache class definition |
+
+**Patched files (~30 lines across 5 files):**
+
+| File | Change |
+|------|--------|
+| `src/CMakeLists.txt` | Added new source files to build |
+| `src/llama-context.h` | Added expert cache context member |
+| `common/common.h` | Added `expert_cache_size` parameter |
+| `common/common.cpp` | Cache init + eval callback registration |
+| `common/arg.cpp` | `--expert-cache-size` CLI flag |
+
+## The research journey
+
+```
+460-line LRU cache → 0.24 tok/s (stole RAM from OS page cache)
+15-line madvise    → 0.57 tok/s (coached the OS page cache)
+no-op callback     → 0.46 tok/s (accidental page warming)
+
+The cache was the experiment. madvise was the answer.
+```
+
+## Full three-way benchmark (8 GB MacBook Air)
+
+| Config | tok/s | Mechanism |
+|--------|-------|-----------|
+| Stock llama.cpp | 0 (thrash) | OS blind LRU, no domain knowledge |
+| No-op callback | 0.46 | Accidental page warming from tensor inspection |
+| madvise prefetch | 0.57 | Explicit kernel prefetch hints |
+| LRU cache (5 GB) | 0.24 | Duplicate data in user-space heap |
+
+## Related
+
+- **MLX Expert Sniper** (Apple Silicon, 3.3 tok/s): [huggingface.co/waltgrace/mlx-expert-sniper](https://huggingface.co/waltgrace/mlx-expert-sniper)
+- **Full research + code**: [github.com/walter-grace/mac-code/tree/main/research/expert-sniper](https://github.com/walter-grace/mac-code/tree/main/research/expert-sniper)