README.md

metadata

license: other
license_name: fraqtl-verification
license_link: LICENSE
tags:
  - kv-cache
  - kv-cache-compression
  - vllm
  - a100
  - fraqtl
inference: false

fraQtl SM80 runtime — prebuilt verification wheel

Prebuilt binary wheel of the fraqtl vLLM runtime, published so anyone can independently reproduce the fraQtl KV-cache receipts on a rented A100 for ≈$15 — without the kernel or calibration source being distributed.

What's inside

• The fraQtl CUDA kernel as a compiled SM80 (A100) binary (fraqtl/_native/libggml-cuda.so). No .cu/.c/header source ships; the build hard-fails if any leaks in (see wheel_build_receipt.json).
• The vLLM integration layer (attention backend, model adapters, sidecar readers) as Python source — it must stay torch.compile-traceable or the engine provisions a smaller KV pool than the receipts. It is orchestration glue: how eigenbases are computed is not in the wheel in any form (no calibration or sidecar-factory code; remaining runtime modules are Cython-compiled binaries).
• vLLM plugin auto-registration via the vllm.general_plugins entry point — pip install the wheel and vLLM's plugin loader finds the backend in every worker process.

Scope

• Hardware: A100 (SM80) only. The binary will not run on H100, L4, or consumer GPUs.
• vLLM 0.20.2, CUDA 12.4, Python 3.11 — the exact receipt environment. Other versions are unsupported.
• Purpose: verification of the published receipts. It is not a general-purpose release.

Use it

Don't drive this by hand — use the one-command repro script (fraqtl_repro_receipts.py, in this repo and in the sidecar repos):

pip install modal && modal setup
modal secret create huggingface HF_TOKEN=hf_...
modal run fraqtl_repro_receipts.py

It installs this wheel, pulls the calibrated sidecars from fraQtl/mistral-7b-instruct-v0.3-kv-sidecars, runs the three-arm receipt (fraQtl D2 vs fp16 vs fp8-KV) at 8K and 32K with a needle-in-a-haystack gate per arm, and prints the table.

Honesty footnotes

• NIAH-verified, not "lossless."
• vs-fp8 always visible — fp8 KV is the real competitor.
• Batch-1 decode for the published speed rows.

Files

File	What
fraqtl_runtime-*.whl	The prebuilt wheel (binary-only)
wheel_build_receipt.json	Build provenance: sha256, source-leak audit result, module list
fraqtl_repro_receipts.py	The one-command repro script

License

Proprietary; free to install and run for verification and evaluation. No reverse engineering. Sidecars and receipts in the companion repos are Apache 2.0.