BoxyFit/Groot-65B-A6B-GyroM

Groot-65B-A6B — Gyro-M (TQ2_T)

120 tokens/s on a single RTX 3090. 20.2 GB total VRAM. Still 57 t/s with a 3 GB hot tier — an 8 GB card slot. Our own re-encode of Groot-65B-A6B's routed experts to trellis TQ2_T (2.25 bpw), encoded from the official FP8 teacher. First publisher of this combination (2026-11-03).

The number that matters is not the download size

The Groot team's own card told us a 72 GB Blackwell "can't fit" this model. That compares the download to your VRAM — they are different numbers. Here is the actual breakdown:

Vocabulary used on this card: expool = the complete set of quantized routed-expert weights — the only component that scales with bpw. Hot tier = the slice of the expool the engine keeps in VRAM. Cold store = the slice that stays in host RAM/NVMe. Never compare the expool to the total file: the file also carries the lookup table and unchanged non-expert tensors.

Component	Size	Lives in
Expool — routed expert weights (TQ2_T, 2.25 bpw)	16.0 GB	GPU (hot tier) + host RAM (cold store)
N-gram lookup table (Q5_0, verbatim from official GGUF)	35.2 GB	host RAM only — never VRAM
Dense weights, embeddings (verbatim from official Q4_K_M)	~1.5 GB	GPU
Total file	54.5 GB	disk

The GPU never sees the 35 GB lookup table. Fully-resident VRAM on a 24 GB card: 20.2 GB (15.3 GiB expert cache + ~5 GB dense/KV/context), with 4 GB to spare.

Fit ladder — every rung measured on our 3090 (capping the expert cache to emulate smaller cards)

Real-card confirmation: the same pack runs fully resident on a Tesla V100-32 (~$500 used): 90 t/s engine decode, 12 GB VRAM spare — measured 2026-11-03 on a CUDA 12 build (CUDA 13 dropped Volta). A $500 2017 datacenter card is a 90 t/s Groot server.

Your GPU	Expool hot tier	Decode t/s	Measured hit rate (coding bench)
24 GB (RTX 3090/4090/A5000)	full 15.3 GiB	68–106 fresh context, 120–124 warm KV	100%, fully resident, 4 GB VRAM spare
16 GB (V100-16, RX 6800, RTX 4060 Ti 16 GB, Radeon VII)	10.1 GiB (65% of expool)	80–95	89–98%
12 GB (RTX 3060 12 GB, RTX 3080 12 GB, Tesla M40 12 GB)	6.0 GiB (39% of expool)	62–95	80–87%, PCIe traffic still ~0.1%
11 GB (RTX 2080 Ti)	~5.2 GiB (32% of expool)	42–48	MEASURED on a real RTX 2080 Ti (2026-11-06): warm 48.0 / cold 42.7 t/s, 776 MiB VRAM spare, sm_75 build. Lands inside the predicted band (0.64x of the 12 GB rung).
8 GB (RTX 3070, RTX 2080 Super, RX 7600)	2.9 GiB (20% of expool)	57–59	~20% hot, cold fetch over PCIe; emulated on 3090. Real-card check done on an 11 GB RTX 2080 Ti at 2.9 GiB hot tier: 36.7–41.7 t/s with 3.0 GiB spare — a weaker card loses only ~0.72x at this tier, not its 0.64x bandwidth ratio: below ~30% hot the PCIe fetch path dominates and the card's own VRAM bandwidth matters less.

There is no speed cliff as the hot tier shrinks, because a cold expert fetch is only ~1.3 MB over PCIe — bandwidth, not residency, is the limit. For workload honesty: on a broad 1.25M-token corpus (IsValorum's independent Groot coverage data, 100% of experts observed), routing is flatter than on focused workloads: a 65% hot tier covers ~84% of activations, 39% covers ~63%. Speeds above already include that streaming cost.

System RAM: 32 GB works. The 35.2 GB lookup table does not need to be resident — measured, after real use only 5.7 GB of it (16%) is ever touched; evicting the whole table from page cache before every request cost zero measurable speed. The cold part lives on NVMe and is read sparsely. A fast NVMe is the real requirement; 16 GB RAM is the practical floor, 32 GB is comfortable. (The 128 GB in some reports is our bench box, not a requirement.)

Same size, different engine — measured contrast

We also benchmarked mradermacher's static Q2_K GGUF (56.3 GB, ~2.6 bpw — nearly the same file size as this pack) on the same RTX 3090 with stock llama.cpp (b11364, --cpu-moe + lookup table pinned to host RAM, because the 21 GB expool + KV does not fit 24 GB):

	this pack (TQ2_T, Strata)	Q2_K (llama.cpp)
decode	120 t/s	3.8–4.5 t/s
prefill	800 t/s	22.6 t/s
fast QA gate	10/10, direct answers	5/10 at short budget, 9/10 given 400 tokens — right answers buried under rambling

Upstream llama.cpp gained an expert cache on 2026-10-07 (#29887, --moe-cache-mib), and its eviction policy moved past plain LRU on 2026-11-06 (#31233, ClassLRU — an 8-class recency+frequency hybrid) — real features in the right direction. The distinction that still matters is reactive vs routing-aware: even a smarter eviction policy decides after the router has already missed; Strata's cache predicts from the router before the miss (measured 85.9% hit) and pre-fills from a learned profile. Third-party comparison on a 16 GB card with a 2-bit Flash-Next pack: llama.cpp + 8 GB expert cache 49.4 t/s vs Strata 85.8 t/s, prefill 206 vs 2,320 t/s (local-llm-lab, 2026-10-16). That gap is the engine, not the bits. For a llama.cpp-native path of this model, mradermacher's repo is the right link — this pack needs the Strata engine above (TQ2_T id 144 is not an upstream type).

Is it still smart?

Quant	Size class	HumanEval pass@1 (0-shot, temp 0, thinking)	MBPP pass@1	Decode on 1×RTX 3090
This pack — Gyro-M TQ2_T (2.25 bpw)	54.5 GB	95.1% (156/164)	93.4% (467/500)	120 t/s
Same pack, one Tesla V100-32 (~$500 used)	54.5 GB	same weights	same weights	90 t/s (fully resident, 12 GB VRAM spare)
Official Q4_K_M (4.8 bpw)	78.8 GB	98.2% (161/164)	not run	68–80 t/s
Community Q2_K, llama.cpp (2.56 bpw)	56.3 GB	fast-gate 5/10 short-budget, 9/10 with room	—	3.8–4.5 t/s

Same model, same harness (boxy-fastcoder-v1, official prompts + asserts, 2026-11-03). The requant at less than half the expert precision costs ~3 points on the coder bench and keeps 120 t/s — while the same-size llama.cpp 2-bit quant rambles and crawls at 4 t/s on plain offload (an 8 GB --moe-cache-mib lifts llama.cpp 2-bit into the 40–50 t/s class; see the engine note above).

Reading the size column honestly: those are whole files (download size), not the streaming number. The expool itself goes ~38 GB (Q4_K_M) → 16.0 GB (TQ2_T) — pool bytes scale linearly with bpw, 4.8/2.25 = 2.1x, exactly as they should. The file shrinks only 1.45x (78.8 → 54.5 GB) because everything outside the experts — lookup table, dense weights, embeddings — is unchanged. Don't divide one column by the other.

• Groot is a reasoning model. Answers arrive only after a thinking trace; stacks with short output budgets or code stop-words truncate it mid-thought and score zero. If your eval said "Groot is dumb", check whether it read reasoning_content and gave the model room to finish — that explains a large share of such reports.
• Honest fine print: Groot's real failure mode is occasional reasoning-loop degeneration (the trace collapses into a repetition spiral and never emits code). 6 of this pack's 8 HumanEval misses are that mode, counted, not hidden.
• Encode quality: 11.35 dB signal-to-error at 2.25 bpw on real expert tensors (healthy for this rate; our production ladder threshold).
• Everything except the routed experts is byte-identical to the official Q4_K_M release — the lookup table, dense weights, and embeddings are copied verbatim.
• Detailed long benches (standard suites, long-context) are running; results will be added to this card and to the boxy.fit leaderboard (row groot-gyro, harness-stamped).

Quickstart (Strata engine)

This GGUF uses the trellis TQ2_T type (id 144) from the Strata/agention kernel family — upstream llama.cpp cannot load it. The filename carries a Q2_0 tag purely so HF's hardware estimator can parse it: Q2_0 is 2.25 bpw, the exact same rate as TQ2_T, so the size estimates are accurate. The tensors themselves are TQ2_T.

# 1. build the serving pack (seconds)
python tools/iq_pack.py --gguf Groot-65B-A6B-GyroM-Q2_0.gguf --compat-bf16 --out pack-groot-gyro
# 2. serve (Strata >= 0.2.8 / engine 0.1.54; CUDA arch 86+)
strata --pack pack-groot-gyro --native Groot-65B-A6B-GyroM-Q2_0.gguf \
       --expert-cache auto --spec 4 --kv k8v4 --max-context 32768

The engine sizes the hot tier to your free VRAM automatically — the 16 GB and 12 GB rows above are what --expert-cache auto produces on those cards.

What this is, exactly

Groot-65B-A6B is a whole-expert-table pruned Flash-Next (256 experts instead of 512, same per-token active path, ~6B active). We halved the stored expool again by re-encoding it with trellis quantization: experts re-encoded from the official FP8 teacher; all other tensors copied byte-for-byte from the official Q4_K_M GGUF. The result: the full 65B expool fits resident on one consumer card, and the model runs faster than it does streamed at 4-bit.

Run it yourself

The harness behind every number on this card is open: harness/fastcoder.py (HumanEval) and fastcoder_mbpp.py (MBPP). pip install datasets, point it at any OpenAI-compatible endpoint (python fastcoder.py <port> <model>), and you get the same prompts, same asserts, same RESULT line. Ran it on a card we don't have? Submit the RESULT line at boxy.fit — accepted claims go on the leaderboard with your credit.

Attribution & license

• Base model: GrootTeam/Groot-65B-A6B-261023 — thank you to the Groot team.
• llama.cpp quants of this model: mradermacher. Independent expert-coverage/imatrix data: IsValorum.
• Architecture and weights lineage: Qwen/Qwen3.8-Flash-Next.
• License: qwen-community-1.0 (text). This requant is a derivative work distributed under the same terms.
• Quantization tooling: Strata engine trellis kernels (Niko1221/Strata). Benchmarks: boxy.fit.