Runtime card (private, pre-flip)

d80a07c verified 7 days ago

2.68 kB

license: mit
library_name: gguf
tags:
  - fraqtl
  - llama-cpp
  - kv-cache-compression
  - cuda
  - inference-runtime

fraQtl Membrane Runtime for llama.cpp (prebuilt, CUDA)

Prebuilt binaries of the fraQtl compression membrane integrated into a llama.cpp fork: the KV cache is stored in calibrated compressed pages and the attention kernel reads them directly at tensor-core speed — no decompress-then-attend stage. At long context, where decode is bandwidth-bound, reading fewer bytes makes generation faster.

Measured (receipts on the model cards): 1.79× faster than q8_0 KV at 128K decode on Qwen3-4B (92% of fp16 speed at ~2.45× less KV memory), 36 vs 28 users before OOM at 32K/user — and on Mistral-Nemo-12B, 32 vs 24 users, second model with zero code changes.

Platform

Linux x86_64 · CUDA 12.4 · SM80 + SM90 (A100 / H100 class). This is the server/workstation lane — no macOS/ARM build.
Built 2026-08-24 (MANIFEST.json in this repo carries per-file sha256 — verify your download against it).

Files

llama-server, llama-cli, llama-fraqtl-niah-parallel (the retrieval-gate harness) and 7 shared libraries (libllama, libllama-common, libggml, -base, -cpu, -cuda, libmtmd). Kernel source is not distributed — same posture as our vLLM runtime wheel; the binaries plus published sidecars are sufficient to run and verify every number on the model cards.

How to run

export FRAQTL_MEMBRANE=1 FRAQTL_MEMBRANE_EXCLUSIVE=1
LD_LIBRARY_PATH=. ./llama-server -m <model>.gguf \
  --fraqtl-kv --fraqtl-eigenbasis <v-sidecar>.bin \
  --fraqtl-kv-protect 32 --fraqtl-k-eigenbasis <k-sidecar>.bin \
  --fraqtl-sink-tokens 0 --fraqtl-residual-window 0

Calibrated sidecars per model: Qwen3-4B-Instruct-2507 · Mistral-Nemo-Instruct-2407

License and credit

This runtime is a fork of llama.cpp (MIT — license included; upstream commit pinned in the receipts). The membrane kernels are fraQtl's; llama.cpp and its contributors are credited at their best — the engine this builds on is excellent.

More from fraQtl

Same membrane, independently implemented in vLLM: nine concurrent ≈128K users on one A100, 134.1 tok/s, 9/9 retrieval — receipt on the Qwen3-4B sidecar card. Calibration-aware Hi-Fi GGUFs (pair well with this runtime): org page.