prometheusAIR/Ripple-65B-Preview-GGUF

Ripple-65B-Preview GGUF: quants for 96 GB and 32 GB GPUs, plus the llama.cpp port

Five quants of Ricursive Intelligence's Ripple-65B-Preview, a 65B dense model, made from the BF16 release. This repo also includes the llama.cpp patch that adds the architecture. Three quants are sized for a single 96 GB GPU, two of them at the full 1M-token context. The two IQ3 quants are sized for a 32 GB GPU, with 96K or 32K of context.

We checked the patch against Ricursive's own modeling code before making any quant. Every quant was then measured against the BF16 weights, not against another quant.

These files need a patched llama.cpp. Upstream llama.cpp doesn't support the ricursive architecture yet (no pull request as of 2026-11-04). The patch in patches/ applies to upstream commit 6f1e654. See Build.

Which one do I want?

	Q8_0	Q6_K_XL	Q6_K	IQ3_S	IQ3_XXS
File size	64.11 GiB	52.76 GiB	49.50 GiB	27.58 GiB	25.38 GiB
Mean KL divergence vs BF16	0.0027	0.0036	0.0053	0.066	0.085
Median KL divergence	0.00026	0.00043	0.00082	0.022	0.030
99th-percentile KL divergence	0.026	0.037	0.059	0.77	0.96
Same top-1 token as BF16	98.59%	98.20%	97.74%	89.98%	88.30%
Perplexity vs BF16	+0.28%	+0.02%	−0.30%	+0.90%	+2.52%
Fits on (q8_0 KV)	96 GB, 753,664 ctx	96 GB, 1,048,576 ctx	96 GB, 1,048,576 ctx	32 GB, 32,768 ctx	32 GB, 98,304 ctx
Decode, reference box, short prompt	22.0 tok/s	26.2 tok/s	27.4 tok/s	42.8 tok/s	46.0 tok/s

The IQ3 KL rows were measured with q8_0 KV, as they run on a 32 GB card; the others with f16 KV. On the 6- and 8-bit quants, q8_0 KV moves KL divergence by 0.0002 at most.

For scale: the BF16 weights themselves, run a second time with a different batch size and layer placement, score a mean KL divergence of 0.00045 against the first run, with 99.59% of top-1 tokens the same. Perplexity moves both up and down by small amounts. It isn't a quality measure here; use the KL columns.

• Q8_0: the closest to BF16. Pick it if quality matters most and 736K tokens of context is enough. It also prefills fastest (1,710 tok/s at 32K, against 1,448 for Q6_K_XL), but decodes slowest.
• Q6_K_XL: Q6_K feed-forward weights with Q8_0 attention, embeddings and output head. It is 11.4 GiB smaller than Q8_0 and reaches the full 1M context on a 96 GB card, for about a third more KL divergence. Against plain Q6_K, the Q8_0 attention costs 3.3 GiB and removes about two thirds of the KL gap between Q6_K and Q8_0.
• Q6_K: the smallest of the 96 GB quants, for when memory is tight.
• IQ3_S: for a 32 GB card when 32K of context is enough. It is the closer of the two to BF16.
• IQ3_XXS: for a 32 GB card with 96K of context, at about 30% more KL divergence than IQ3_S.

Our IQ3_S and IQ3_XXS aren't llama.cpp's stock recipes of those names. On this model the stock IQ3_XXS recipe scores a KL divergence of 0.581 and +68% perplexity; see Where the extra bits go.

Recipes

tensor group	share of weights	Q8_0	Q6_K_XL	Q6_K	IQ3_S	IQ3_XXS
Feed-forward gate / up / down (240 tensors)	77.7%	Q8_0	Q6_K	Q6_K	IQ3_S	IQ3_XXS
Attention q / k / v / output / gate (400 tensors)	20.7%	Q8_0	Q8_0	Q6_K	IQ4_XS	IQ4_XS
token_embd, output	1.6%	Q8_0	Q8_0	Q6_K	Q6_K	Q6_K
Norms and per-layer value-residual mix	<0.01%	F32	F32	F32	F32	F32

All quants except Q8_0 use the imatrix; Q8_0 doesn't take one. All were quantized from the BF16 GGUF. The two Q6_K recipes differ only in attention, embeddings and output head, and _XL marks the one with Q8_0 there. Q6_K_XL is our name, not a llama.cpp type, and it isn't Unsloth's UD recipe of the same name.

Where the extra bits go, and why

The feed-forward weights are 77.7% of the model and take the base type. Attention and the embeddings/output head get more bits than that in every quant except Q8_0. Attention is where extra bits buy the most:

• At 6 bits. Moving attention, embeddings and output head from Q6_K to Q8_0 (Q6_K → Q6_K_XL) costs 3.3 GiB and cuts KL divergence from 0.0053 to 0.0036. That is about two thirds of the way to Q8_0, which spends 14.6 GiB.
• At 3 bits the effect is far larger. llama.cpp's stock IQ3_XXS recipe saves space in attention: attn_q and attn_k drop to IQ2_S (2.5 bits per weight), the output gate gets IQ3_XXS, the output projection IQ3_S and attn_v Q4_K, with IQ3_S embeddings and a Q5_K output head. On Ripple that file (23.26 GiB) scores a mean KL divergence of 0.581, 70.7% same top-1 and +68% perplexity. Our IQ3_XXS keeps the same IQ3_XXS feed-forward but puts all five attention projections at IQ4_XS and the embeddings and output head at Q6_K. That is 2.1 GiB more, 1.9 GiB of it in attention, and it cuts KL divergence 7×, to 0.085.

Ricursive's own NVFP4 release points the same way: it quantizes only the feed-forward weights and keeps attention in BF16. Ripple's attention is unusually wide (the query, gate and output projections are 10,240 wide, twice the hidden size), but we haven't tested whether that is the reason, or which of the five projections is the most sensitive. The embeddings and output head are 1.6% of the weights, so keeping them at Q6_K or above costs 0.2-0.4 GiB per step. We didn't measure them on their own.

If you make your own low-bit quants of this model, set the attention types explicitly (see Build). A plain llama-quantize ... IQ3_XXS gives the 0.581 file.

The architecture and the port

Ripple is a thin, deep dense transformer: 80 layers, hidden size 5120, a 40,960-wide SwiGLU feed-forward, 80 query heads of 128 dimensions (twice the hidden size) over 8 KV heads. Every fifth layer is full attention with no positional encoding. The other four use a 1,024-token sliding window with RoPE. Each block has four RMSNorms (before and after attention and feed-forward), per-head QK norms, and a sigmoid gate on the attention output. The SwiGLU inputs are clamped (gate at most 20, up within ±20). The value projection doesn't read the normed hidden state directly. It reads a learned per-layer blend of that state with the scaled token embeddings (a value residual). The embeddings are scaled by √5120, and there is no BOS token.

The patch adds src/models/ricursive.cpp, a converter class, one tensor type (blk.N.attn_value_res, the per-layer blend logit) and one metadata key (ricursive.swiglu_clamp). Everything else is existing llama.cpp: the sliding-window KV cache, the clamped-SwiGLU op (ggml_swiglu_clamp, which already has CUDA kernels), the QK norms and the output gate. The graph also records per-layer hidden states, for DFlash-style drafting (untested; see Not covered).

Checked against Ricursive's code. We ran Ricursive's own HF modeling code in FP32 on the CPU, one layer at a time from the BF16 safetensors. It scored two 2,048-token wikitext-2 sequences at positions 1,024-2,046, all past the sliding window, and wrote the result as a llama.cpp KL-divergence reference. Against it, the BF16 GGUF scores:

backend	mean KL divergence	same top-1 token	perplexity (reference: 8.968)
CPU	0.00022	99.41%	8.973
CUDA build, last 6 layers on the GPU	0.00021	99.61%	8.970

Tokenizer. It matches the HF tokenizer token for token on the whole wikitext-2 test set (289,305 tokens), on our 749,032-token calibration corpus, and on the probe strings from Ricursive's own tokenizer check.

Chat template. The GGUFs embed Ricursive's tools-aware chat_template.jinja from the root of the release repo. That is the one their servers load. The release's tokenizer_config.json carries a plain ChatML template without tool support, and llama.cpp's converter would otherwise pick that one, so the patch makes the converter use the .jinja. llama.cpp renders the embedded template byte-identically to Python Jinja2, including the tools block. Tool calls come back as <tool_call> JSON, and llama.cpp returns them as structured tool_calls.

Build

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 6f1e6543704bc5b080c7a98e7b6f707b31f6ed51
hf download prometheusAIR/Ripple-65B-Preview-GGUF patches/llama.cpp-ricursive.patch --local-dir .
git apply patches/llama.cpp-ricursive.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j --target llama-server llama-cli llama-quantize

We built for CUDA on an sm_120 card (RTX PRO 6000 Blackwell) and ran the CPU backend for the reference check. We haven't tried other backends, or upstream commits other than 6f1e654.

To make the quants yourself (needs the patched tree, PyTorch and transformers):

python convert_hf_to_gguf.py /path/to/Ripple-65B-Preview --outtype bf16 --outfile Ripple-65B-Preview-BF16.gguf
llama-quantize Ripple-65B-Preview-BF16.gguf Ripple-65B-Preview-Q8_0.gguf Q8_0
llama-quantize --imatrix Ripple-65B-Preview-imatrix.gguf \
  Ripple-65B-Preview-BF16.gguf Ripple-65B-Preview-Q6_K.gguf Q6_K
llama-quantize --imatrix Ripple-65B-Preview-imatrix.gguf \
  --tensor-type attn_q=q8_0 --tensor-type attn_k=q8_0 --tensor-type attn_v=q8_0 \
  --tensor-type attn_output=q8_0 --tensor-type attn_gate=q8_0 \
  --token-embedding-type q8_0 --output-tensor-type q8_0 \
  Ripple-65B-Preview-BF16.gguf Ripple-65B-Preview-Q6_K_XL.gguf Q6_K
# 32 GB tier: IQ4_XS attention, Q6_K embeddings/output. For IQ3_S, swap iq3_xxs -> iq3_s and the type to IQ3_S.
llama-quantize --imatrix Ripple-65B-Preview-imatrix.gguf \
  --tensor-type ffn_gate=iq3_xxs --tensor-type ffn_up=iq3_xxs --tensor-type ffn_down=iq3_xxs \
  --tensor-type attn_q=iq4_xs --tensor-type attn_k=iq4_xs --tensor-type attn_v=iq4_xs \
  --tensor-type attn_output=iq4_xs --tensor-type attn_gate=iq4_xs \
  --token-embedding-type q6_k --output-tensor-type q6_k \
  Ripple-65B-Preview-BF16.gguf Ripple-65B-Preview-IQ3_XXS.gguf IQ3_XXS

Reference box and settings

One RTX PRO 6000 Blackwell 96 GB, Ryzen 9 9950X3D, 128 GB DDR5, files on NVMe. Everything runs on the one GPU.

llama-server -m Ripple-65B-Preview-Q6_K_XL.gguf -ngl 999 \
  --ctx-size 1048576 -ctk q8_0 -ctv q8_0 -fa on -b 2048 -ub 1024 -np 1 --jinja

For Q8_0, use the same flags with --ctx-size 753664. At 786,432 it loads with only 941 MiB free, too little headroom for the first requests. Q6_K runs with the same flags as Q6_K_XL.

	Q8_0	Q6_K_XL	Q6_K
Context	753,664	1,048,576	1,048,576
GPU memory after load	95,664 MiB	95,560 MiB	92,342 MiB
Peak during the deepest rung	95,722 MiB (2.1 GiB free, 743,346 tokens)	95,618 MiB (2.2 GiB free, 1,031,887 tokens)	not run

32 GB cards. We don't have one, so we sized this tier on the 96 GB card. For the same build and settings, memory use should be nearly identical on another GPU. The CUDA context's size varies a little between models, and a card that also drives a display has less free. An RTX 5090 has 32,607 MiB and the same memory bandwidth as the reference card, so decode speed should be similar too.

llama-server -m Ripple-65B-Preview-IQ3_XXS.gguf -ngl 999 \
  --ctx-size 98304 -ctk q8_0 -ctv q8_0 -fa on -b 2048 -ub 512 -np 1 --jinja

For IQ3_S, use --ctx-size 32768.

	IQ3_XXS	IQ3_S
Context	98,304	32,768
GPU memory after load	30,270 MiB	30,106 MiB
Peak during the deepest rung	30,306 MiB	30,142 MiB
Left on a 32,607 MiB card at that peak	2.2 GiB	2.4 GiB

Each extra 1K tokens of context costs about 38 MiB. IQ3_XXS at 131,072 tokens leaves only 1.1 GiB on a 32 GB card, too little headroom. -ub 1024 instead of 512 adds 284 MiB at 98,304.

Settings that matter

• -ub 1024 at 1M. At 1M context, -ub 2048 needs an 8.3 GiB compute buffer. With Q6_K_XL that doesn't fit on a 96 GB card. The compute buffer grows with context, so test at your full --ctx-size.
• The KV cache is small for its context. Only the 16 full-attention layers keep a cache that grows with context: 34 KiB per token at q8_0, 64 KiB at f16. The 64 sliding-window layers hold 1,024 tokens each. On the 2,048-token KL test, q8_0 KV changes KL divergence by +0.0001 on Q8_0 and by −0.0002 on Q6_K_XL. Both are within error and below the BF16 rerun floor of 0.00045. The depth ladder below ran with q8_0 KV.
• -np 1. We validated with one slot only.
• Reasoning is the model's choice. Ripple opens a <think> block on its own when a prompt needs it, and often skips it on easy ones. llama.cpp puts it in reasoning_content. As Ricursive's server docs say, there is no thinking switch and no effort levels. After a <think> block, content can start with a newline: llama.cpp keeps the newline after </think>, which Ricursive's own SGLang parser drops. Strip it on the client if it matters.
• Sampling: Ricursive doesn't publish sampling defaults. Our probes ran at temperature 0.
• Long prompts are slow to fill, and decode slows with depth. A 1,031,887-token prompt took 76 minutes to prefill on the reference box. Decode drops from 26.2 tok/s on a short prompt to 6.7 tok/s at 1M (Q6_K_XL), and from 22.0 to 8.2 tok/s at 743K (Q8_0). That is a steeper fall than the KV cache's size explains, and we haven't profiled why.

Validation

KL divergence. Reference = the BF16 GGUF, which matches Ricursive's code (above). Text: wikitext-2 test, 64 sequences of 2,048 tokens, scoring the second half of each (65,472 tokens, all past the sliding window), one sequence per batch. The calibration corpus shares no 64-character window with this text.

Depth ladder. Facts are planted at 10%, 50% and 90% depth in a synthetic log, with four questions per rung: a verbatim needle, a single retrieval, a three-fact multi-hop sum, and a comparison. One prefill per rung; the four questions reuse it. Temperature 0. Prefill is the average over the whole prompt.

config	prompt tokens	questions correct	prefill (whole prompt)	decode
Q6_K_XL, 1M ctx	32,135	4/4	1,448 tok/s	23.9 tok/s
Q6_K_XL, 1M ctx	131,315	4/4	982 tok/s	19.3 tok/s
Q6_K_XL, 1M ctx	1,031,887	4/4	226 tok/s	6.7 tok/s
Q8_0, 736K ctx	32,135	4/4	1,710 tok/s	20.4 tok/s
Q8_0, 736K ctx	131,315	4/4	1,105 tok/s	16.9 tok/s
Q8_0, 736K ctx	743,346	4/4	311 tok/s	8.2 tok/s
IQ3_XXS, 96K ctx	89,242	4/4	1,314 tok/s	32.2 tok/s
IQ3_S, 32K ctx	23,617	4/4	1,693 tok/s	38.3 tok/s

Functional checks: on all five quants, arithmetic, a tool call returned as structured tool_calls, and a tool-result round trip pass. Where the model opened a <think> block (checked on Q6_K_XL and IQ3_XXS), it was split into reasoning_content correctly.

Calibration

The imatrix (imatrix/Ripple-65B-Preview-imatrix.gguf) covers 747,520 tokens: 365 sequences of 2,048 tokens, computed on the Q8_0. The corpus is 2.44M characters:

• general/scientific prose, and an agentic, code and tool-use set
• fineweb-edu (80 documents)
• LLM compression calibration chats (64) and OpenThoughts reasoning traces (64), rendered in Ripple's chat template with the reasoning as <think> blocks
• 16 multi-turn tool dialogues in Ripple's <tool_call> / <tool_response> format
• Ricursive's modeling and chat-format code, and the llama.cpp runtime from the patch

Files

file
Ripple-65B-Preview-Q8_0.gguf	Q8_0, 64.11 GiB
Ripple-65B-Preview-Q6_K_XL.gguf	Q6_K FFN + Q8_0 attention, 52.76 GiB
Ripple-65B-Preview-Q6_K.gguf	Q6_K, 49.50 GiB
Ripple-65B-Preview-IQ3_S.gguf	IQ3_S FFN + IQ4_XS attention, 27.58 GiB
Ripple-65B-Preview-IQ3_XXS.gguf	IQ3_XXS FFN + IQ4_XS attention, 25.38 GiB
imatrix/Ripple-65B-Preview-imatrix.gguf	the imatrix used for every quant except Q8_0
patches/llama.cpp-ricursive.patch	the llama.cpp port, against upstream 6f1e654
SHA256SUMS	sha256sum -c SHA256SUMS --ignore-missing

Each quant is one file. hf download / huggingface_hub fetches them over Xet (hf_xet, the default). With HF_HUB_DISABLE_XET=1 set, the Hub refuses the three files over 50 GB.

Not covered

• Speculative decoding. Ricursive also released a DFlash2 drafter (Ripple-65B-Preview-DFlash2). We haven't converted or tested it.
• The base model (Ripple-65B-Base-Preview) isn't quantized here.
• Multiple slots, other backends, an actual 32 GB card. Untested.

Attribution

• Model: Ricursive Intelligence, Apache License 2.0 (LICENSE).
• The llama.cpp patch is offered under llama.cpp's MIT license.