---
license: mit
library_name: gguf
tags:
  - fraqtl
  - llama-cpp
  - kv-cache-compression
  - cuda
  - inference-runtime
---

# fraQtl Membrane Runtime for llama.cpp (prebuilt, CUDA)

Prebuilt binaries of the fraQtl **compression membrane** integrated into a
llama.cpp fork: the KV cache is stored in calibrated compressed pages and the
attention kernel reads them **directly at tensor-core speed** — no
decompress-then-attend stage. At long context, where decode is
bandwidth-bound, reading fewer bytes makes generation faster.

Measured (receipts on the model cards): **1.79× faster than q8_0 KV at 128K
decode** on Qwen3-4B (92% of fp16 speed at ~2.45× less KV memory), **36 vs 28
users** before OOM at 32K/user — and on Mistral-Nemo-12B, 32 vs 24 users,
second model with zero code changes.

## Platform

- **Linux x86_64 · CUDA 12.4 · SM80 + SM90** (A100 / H100 class). This is
  the server/workstation lane — no macOS/ARM build.
- Built 2026-08-24 (`MANIFEST.json` in this repo carries per-file sha256 —
  verify your download against it).

## Files

`llama-server`, `llama-cli`, `llama-fraqtl-niah-parallel` (the retrieval-gate
harness) and 7 shared libraries (`libllama`, `libllama-common`, `libggml`,
`-base`, `-cpu`, `-cuda`, `libmtmd`). Kernel source is not distributed — same
posture as our [vLLM runtime wheel](https://huggingface.co/fraQtl/fraqtl-sm80-runtime);
the binaries plus published sidecars are sufficient to run and verify every
number on the model cards.

## How to run

```bash
export FRAQTL_MEMBRANE=1 FRAQTL_MEMBRANE_EXCLUSIVE=1
LD_LIBRARY_PATH=. ./llama-server -m <model>.gguf \
  --fraqtl-kv --fraqtl-eigenbasis <v-sidecar>.bin \
  --fraqtl-kv-protect 32 --fraqtl-k-eigenbasis <k-sidecar>.bin \
  --fraqtl-sink-tokens 0 --fraqtl-residual-window 0
```

Calibrated sidecars per model:
[Qwen3-4B-Instruct-2507](https://huggingface.co/fraQtl/qwen3-4b-instruct-2507-kv-sidecars) ·
[Mistral-Nemo-Instruct-2407](https://huggingface.co/fraQtl/mistral-nemo-instruct-2407-kv-sidecars)

## License and credit

This runtime is a fork of [llama.cpp](https://github.com/ggml-org/llama.cpp)
(MIT — license included; upstream commit pinned in the receipts). The
membrane kernels are fraQtl's; llama.cpp and its contributors are credited
at their best — the engine this builds on is excellent.

## More from fraQtl

Same membrane, independently implemented in vLLM: nine concurrent ≈128K
users on one A100, 134.1 tok/s, 9/9 retrieval — receipt on the
[Qwen3-4B sidecar card](https://huggingface.co/fraQtl/qwen3-4b-instruct-2507-kv-sidecars).
Calibration-aware Hi-Fi GGUFs (pair well with this runtime):
[org page](https://huggingface.co/fraQtl).