--- license: llama3.1 base_model: meta-llama/Llama-3.1-8B-Instruct tags: - ridgefill - llama.cpp - gguf - safetensors - prefill-acceleration - llama-3.1 ---

RidgeFill maps for Llama-3.1-8B-Instruct: the reference study's maps at four split depths.
Fitted once in closed form. 39.7 more filled-region RULER points than the identity fill at the same split.
Data and code · All maps · llama.cpp fork · Citation
Built with Llama. ## What this is The ridge maps of the RidgeFill paper's reference study for Llama-3.1-8B-Instruct (`unsloth/Llama-3.1-8B-Instruct`, revision 4699cc75), plus the split-12 maps converted to GGUF for the [llama.cpp fork](https://github.com/Approximate-Science/llama.cpp-ridgefill). The maps are fitted on Llama 3.1 hidden states and are distributed under the Llama 3.1 Community License (`LICENSE`, `USE_POLICY.md`). ## Files | File | Split | Bytes | SHA256 | |---|---|---|---| | `safetensors/llama-3.1-8b-instruct-sweep-s8.safetensors` | 8 of 32 | 805,505,392 | `911b01565af665931c1e2cb01dd6fc6a7951f120117e1327281245c51a75b335` | | `safetensors/llama-3.1-8b-instruct-sweep-s12.safetensors` | 12 of 32 (primary) | 671,254,560 | `89bf425556ca2ab5455b2ab76b686e2a29b1bdf9716bb882cb902f2bd6c18008` | | `safetensors/llama-3.1-8b-instruct-sweep-s16.safetensors` | 16 of 32 | 537,003,728 | `b076e73dc8c852f7bb27b37bc1b13cbac2bf6e9db769d095cf046b88de80358e` | | `safetensors/llama-3.1-8b-instruct-sweep-s20.safetensors` | 20 of 32 | 402,752,888 | `086afd8767eaf4d207968ca9a4572151d21f710e02c7734b084c9c9eeeeb1212` | | `llama-3.1-8b-instruct-s12-bf16.gguf` | 12 of 32, bf16 | 671,419,904 | `83b893b7da619dbf280071f6d5a7e2fb1655eef4b7b5f288a90003d3c612fd34` | | `llama-3.1-8b-instruct-s12-q8_0.gguf` | 12 of 32, Q8_0 | 356,847,104 | `625cffeb57d7a26659f720bd05931cbaa6cd0ef526f93bb3d77623cb452797a9` | The safetensors files are the bf16 maps the paper's runs loaded, in the reference implementation (`code/kva` in the [data repository](https://github.com/Approximate-Science/ridgefill-data)). They were fitted in closed form on 834,541 rows from 71 calibration documents with λ = 0.03. ## Which split Llama-3.1-8B has no key normalization, and its fill error sits in the keys. At split 12 (3/8 depth) the fitted fill's added loss on novel tokens is larger than the identity fill's, +1.006 against +0.505 nats; at split 16 and 20 it is far smaller, +0.085 and +0.030 against +0.563 and +0.406. For loss, use split 16 or deeper. At split 12, attention-selected exact rows remove the failure and restore retrieval. The GGUF files in this repository are split 12. ## Results From the paper's reference study, with a 4,096-token exact tail. | Measure | Value | |---|---| | Filled-region RULER, RidgeFill − identity fill, split 12 (216 samples, 8K to 64K) | +39.7 [+32.6, +46.5] points | | Filled-region RULER, RidgeFill + attention-selected rows − exact, split 12 | +0.2 points, lower bound −1.38 (non-inferior at 2 points) | | Novel-token loss vs exact, split 16 | +0.085 nats (identity fill +0.563) | | Released SwiftKV checkpoint − RidgeFill, novel-token loss, split 16 | +0.010 [−0.004, +0.025] nats | ## Use in llama.cpp ```sh hf download TheCacheIsALie/Llama-3.1-8B-Instruct-RidgeFill-Projector llama-3.1-8b-instruct-s12-q8_0.gguf --local-dir models llama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf -c 32768 -ngl 99 -fa on \ --ridgefill models/llama-3.1-8b-instruct-s12-q8_0.gguf --ridgefill-tail 8192 ``` Build the fork as described in its [RIDGEFILL.md](https://github.com/Approximate-Science/llama.cpp-ridgefill/blob/main/RIDGEFILL.md). Only the plain fill with an exact tail applies to this model; the selection modes of the Qwen3.8-27B port do not. ## Measured in llama.cpp Earlier measurements of the split-12 GGUF, separate from the paper. One RTX 4090 24 GB, CUDA 12.4, flash attention on, `-b 2048 -ub 512`, F16 KV cache, fork commit fa8061c4b, wikitext-2 test text. Stock is the same build without `--ridgefill`. Tail T = 4096 unless stated. Prompt processing, `llama-bench -p N -n 0 -r 3`, speedup over stock: | Model | 8K | 16K | 32K | |---|---|---|---| | F16 | 1.26× | 1.51× | 1.74× | | Q4_K_M | 1.26× | 1.52× | 1.73× | Time to first token through `llama-server`, Q4_K_M, mean of 3 prompts, seconds (stock → RidgeFill): 8K 0.77 → 0.60; 16K 1.73 → 1.14; 32K 4.30 → 2.47. Generation speed (tg128) is the same with and without RidgeFill. Tail NLL (mean NLL of the last 4,095 tokens of each window, nats per token, stock → RidgeFill), Q4_K_M: | Prompt | T = 4096 | T = 8192 | T = 16384 | |---|---|---|---| | 8K (35 windows), stock 1.846 | 2.031 | n/a | n/a | | 16K (17 windows), stock 1.811 | 2.047 (1.52×) | 1.927 (1.27×) | nothing filled | | 32K (8 windows), stock 1.843 | 2.681 (1.73×) | 2.050 (1.52×) | 1.938 (1.23×) | `llama-perplexity` (`-c N`; effective tail max(T, N/2)), stock → RidgeFill: | Model | -c 8192 | -c 16384 | -c 32768 | |---|---|---|---| | F16 | 6.1775 → 7.5485 | 5.8588 → 7.0083 | 6.7350 → 7.8851 | | Q4_K_M | 6.3350 → 7.6229 | 6.0108 → 7.0517 | 6.9105 → 7.9416 | | F16, Q8_0 maps | 6.1775 → 7.5446 | 5.8588 → 7.0046 | 6.7350 → 7.8886 | Needle in a haystack (wikitext-2 haystack, synthetic needle, depths 10 to 90 %, 4 needles per depth and length, greedy, T = 4096), needles found: | Arm | 8K | 16K | 32K | Needles inside the filled region | |---|---|---|---|---| | F16 stock | 20/20 | 20/20 | 20/20 | n/a | | F16 RidgeFill | 19/20 | 15/20 | 15/20 | 35/44 | | Q4_K_M stock | 20/20 | 20/20 | 20/20 | n/a | | Q4_K_M RidgeFill | 18/20 | 17/20 | 14/20 | 34/44 | ## How RidgeFill works