|
Download README.md from TheCacheIsALie/Llama-3.1-8B-Instruct-RidgeFill-Projector: direct link, hf CLI and curl.
- Browser
- Download file 8.18 kB
-
https://huggingface.co/TheCacheIsALie/Llama-3.1-8B-Instruct-RidgeFill-Projector/resolve/main/README.md
- Command line
-
hf download hf://TheCacheIsALie/Llama-3.1-8B-Instruct-RidgeFill-Projector/README.md
-
curl -L -o README.md https://huggingface.co/TheCacheIsALie/Llama-3.1-8B-Instruct-RidgeFill-Projector/resolve/main/README.md
8.18 kB
| license: llama3.1 | |
| base_model: meta-llama/Llama-3.1-8B-Instruct | |
| tags: | |
| - ridgefill | |
| - llama.cpp | |
| - gguf | |
| - safetensors | |
| - prefill-acceleration | |
| - llama-3.1 | |
| <p align="center"><img src="https://huggingface.co/TheCacheIsALie/Llama-3.1-8B-Instruct-RidgeFill-Projector/resolve/main/assets/ridgefill-banner.webp" width="100%" alt="RidgeFill"></p> | |
| <p align="center"><b>RidgeFill maps for Llama-3.1-8B-Instruct: the reference study's maps at four split depths.</b></p> | |
| <p align="center"><b>Fitted once in closed form. 39.7 more filled-region RULER points than the identity fill at the same split.</b></p> | |
| <p align="center"> | |
| <a href="https://github.com/Approximate-Science/ridgefill-data">Data and code</a> · | |
| <a href="https://huggingface.co/TheCacheIsALie">All maps</a> · | |
| <a href="https://github.com/Approximate-Science/llama.cpp-ridgefill">llama.cpp fork</a> · | |
| <a href="#citation">Citation</a> | |
| </p> | |
| Built with Llama. | |
| ## What this is | |
| The ridge maps of the RidgeFill paper's reference study for Llama-3.1-8B-Instruct (`unsloth/Llama-3.1-8B-Instruct`, | |
| revision 4699cc75), plus the split-12 maps converted to GGUF for the [llama.cpp fork](https://github.com/Approximate-Science/llama.cpp-ridgefill). The maps are fitted on | |
| Llama 3.1 hidden states and are distributed under the Llama 3.1 Community License (`LICENSE`, `USE_POLICY.md`). | |
| ## Files | |
| | File | Split | Bytes | SHA256 | | |
| |---|---|---|---| | |
| | `safetensors/llama-3.1-8b-instruct-sweep-s8.safetensors` | 8 of 32 | 805,505,392 | `911b01565af665931c1e2cb01dd6fc6a7951f120117e1327281245c51a75b335` | | |
| | `safetensors/llama-3.1-8b-instruct-sweep-s12.safetensors` | 12 of 32 (primary) | 671,254,560 | `89bf425556ca2ab5455b2ab76b686e2a29b1bdf9716bb882cb902f2bd6c18008` | | |
| | `safetensors/llama-3.1-8b-instruct-sweep-s16.safetensors` | 16 of 32 | 537,003,728 | `b076e73dc8c852f7bb27b37bc1b13cbac2bf6e9db769d095cf046b88de80358e` | | |
| | `safetensors/llama-3.1-8b-instruct-sweep-s20.safetensors` | 20 of 32 | 402,752,888 | `086afd8767eaf4d207968ca9a4572151d21f710e02c7734b084c9c9eeeeb1212` | | |
| | `llama-3.1-8b-instruct-s12-bf16.gguf` | 12 of 32, bf16 | 671,419,904 | `83b893b7da619dbf280071f6d5a7e2fb1655eef4b7b5f288a90003d3c612fd34` | | |
| | `llama-3.1-8b-instruct-s12-q8_0.gguf` | 12 of 32, Q8_0 | 356,847,104 | `625cffeb57d7a26659f720bd05931cbaa6cd0ef526f93bb3d77623cb452797a9` | | |
| The safetensors files are the bf16 maps the paper's runs loaded, in the reference implementation (`code/kva` in the | |
| [data repository](https://github.com/Approximate-Science/ridgefill-data)). They were fitted in closed form on 834,541 rows from 71 calibration documents with λ = 0.03. | |
| ## Which split | |
| Llama-3.1-8B has no key normalization, and its fill error sits in the keys. At split 12 (3/8 depth) the fitted fill's | |
| added loss on novel tokens is larger than the identity fill's, +1.006 against +0.505 nats; at split 16 and 20 it is far | |
| smaller, +0.085 and +0.030 against +0.563 and +0.406. For loss, use split 16 or deeper. At split 12, attention-selected | |
| exact rows remove the failure and restore retrieval. The GGUF files in this repository are split 12. | |
| ## Results | |
| From the paper's reference study, with a 4,096-token exact tail. | |
| | Measure | Value | | |
| |---|---| | |
| | Filled-region RULER, RidgeFill − identity fill, split 12 (216 samples, 8K to 64K) | +39.7 [+32.6, +46.5] points | | |
| | Filled-region RULER, RidgeFill + attention-selected rows − exact, split 12 | +0.2 points, lower bound −1.38 (non-inferior at 2 points) | | |
| | Novel-token loss vs exact, split 16 | +0.085 nats (identity fill +0.563) | | |
| | Released SwiftKV checkpoint − RidgeFill, novel-token loss, split 16 | +0.010 [−0.004, +0.025] nats | | |
| ## Use in llama.cpp | |
| ```sh | |
| hf download TheCacheIsALie/Llama-3.1-8B-Instruct-RidgeFill-Projector llama-3.1-8b-instruct-s12-q8_0.gguf --local-dir models | |
| llama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -c 32768 -ngl 99 -fa on \ | |
| --ridgefill models/llama-3.1-8b-instruct-s12-q8_0.gguf --ridgefill-tail 8192 | |
| ``` | |
| Build the fork as described in its [RIDGEFILL.md](https://github.com/Approximate-Science/llama.cpp-ridgefill/blob/main/RIDGEFILL.md). Only the plain fill with an exact | |
| tail applies to this model; the selection modes of the Qwen3.8-27B port do not. | |
| ## Measured in llama.cpp | |
| Earlier measurements of the split-12 GGUF, separate from the paper. One RTX 4090 24 GB, CUDA 12.4, flash attention on, | |
| `-b 2048 -ub 512`, F16 KV cache, fork commit fa8061c4b, wikitext-2 test text. Stock is the same build without | |
| `--ridgefill`. Tail T = 4096 unless stated. | |
| Prompt processing, `llama-bench -p N -n 0 -r 3`, speedup over stock: | |
| | Model | 8K | 16K | 32K | | |
| |---|---|---|---| | |
| | F16 | 1.26× | 1.51× | 1.74× | | |
| | Q4_K_M | 1.26× | 1.52× | 1.73× | | |
| Time to first token through `llama-server`, Q4_K_M, mean of 3 prompts, seconds (stock → RidgeFill): 8K 0.77 → 0.60; | |
| 16K 1.73 → 1.14; 32K 4.30 → 2.47. Generation speed (tg128) is the same with and without RidgeFill. | |
| Tail NLL (mean NLL of the last 4,095 tokens of each window, nats per token, stock → RidgeFill), Q4_K_M: | |
| | Prompt | T = 4096 | T = 8192 | T = 16384 | | |
| |---|---|---|---| | |
| | 8K (35 windows), stock 1.846 | 2.031 | n/a | n/a | | |
| | 16K (17 windows), stock 1.811 | 2.047 (1.52×) | 1.927 (1.27×) | nothing filled | | |
| | 32K (8 windows), stock 1.843 | 2.681 (1.73×) | 2.050 (1.52×) | 1.938 (1.23×) | | |
| `llama-perplexity` (`-c N`; effective tail max(T, N/2)), stock → RidgeFill: | |
| | Model | -c 8192 | -c 16384 | -c 32768 | | |
| |---|---|---|---| | |
| | F16 | 6.1775 → 7.5485 | 5.8588 → 7.0083 | 6.7350 → 7.8851 | | |
| | Q4_K_M | 6.3350 → 7.6229 | 6.0108 → 7.0517 | 6.9105 → 7.9416 | | |
| | F16, Q8_0 maps | 6.1775 → 7.5446 | 5.8588 → 7.0046 | 6.7350 → 7.8886 | | |
| Needle in a haystack (wikitext-2 haystack, synthetic needle, depths 10 to 90 %, 4 needles per depth and length, greedy, | |
| T = 4096), needles found: | |
| | Arm | 8K | 16K | 32K | Needles inside the filled region | | |
| |---|---|---|---|---| | |
| | F16 stock | 20/20 | 20/20 | 20/20 | n/a | | |
| | F16 RidgeFill | 19/20 | 15/20 | 15/20 | 35/44 | | |
| | Q4_K_M stock | 20/20 | 20/20 | 20/20 | n/a | | |
| | Q4_K_M RidgeFill | 18/20 | 17/20 | 14/20 | 34/44 | | |
| ## How RidgeFill works | |
| <p align="center"><img src="https://raw.githubusercontent.com/Approximate-Science/ridgefill-data/v1.1/figures/fig_fill_vs_kva.png" width="85%" alt="Two ways to fill a late layer's cache from a middle layer's state: trained key and value heads, and RidgeFill's closed-form map to the layer's input"></p> | |
| The lower layers run exactly up to a split. For every prompt row except the exact ones, a ridge map per later layer | |
| predicts that layer's input from the split layer's state, and the layer writes its cache from the prediction (b) | |
| instead of running in full: its own key and value projections run unchanged, and on a Gated DeltaNet layer so does | |
| its state update. The maps are fitted once, in closed form, from one pass over calibration text. The last few thousand | |
| prompt tokens, any selected exact rows and every decoding step run the full model. Nothing is trained and no token is | |
| dropped. The design comes from DeepSeek-V4.1-Flash, where the projection is trained into the model; (a) is kishida's | |
| Q3-8B-KVA-Projector, which retrofitted it onto Qwen3-8B with trained key and value heads. | |
| ## Citation | |
| The paper is *RidgeFill: Training-Free Late-Layer Cache Fill for Faster Prefill* by Dylan Johnston and Robert P Smith | |
| (preprint, 2026, [doi:10.5281/zenodo.23890247](https://doi.org/10.5281/zenodo.23890247)). Cite it as: | |
| ```bibtex | |
| @misc{johnston2026ridgefill, | |
| author = {Johnston, Dylan and Smith, Robert P}, | |
| title = {{RidgeFill}: Training-Free Late-Layer Cache Fill for Faster Prefill}, | |
| year = {2026}, | |
| publisher = {Zenodo}, | |
| note = {Preprint}, | |
| doi = {10.5281/zenodo.23890247}, | |
| url = {https://doi.org/10.5281/zenodo.23890247} | |
| } | |
| ``` | |
| To cite the data and code themselves, use the data release's citation | |
| ([Approximate-Science/ridgefill-data](https://github.com/Approximate-Science/ridgefill-data#citation)). | |
| An earlier technical note is archived at [doi:10.5281/zenodo.23179168](https://doi.org/10.5281/zenodo.23179168). The | |
| paper replaces some of its numbers; `SUPERSEDED.md` in the data repository lists which. | |