| --- |
| license: other |
| license_name: fraqtl-verification |
| license_link: LICENSE |
| tags: |
| - kv-cache |
| - kv-cache-compression |
| - vllm |
| - a100 |
| - fraqtl |
| inference: false |
| --- |
| |
| # fraQtl SM80 runtime — prebuilt verification wheel |
|
|
| Prebuilt binary wheel of the `fraqtl` vLLM runtime, published so anyone can independently reproduce the fraQtl KV-cache receipts on a rented A100 for ≈$15 — **without the kernel or calibration source being distributed**. |
|
|
| ## What's inside |
|
|
| - The fraQtl CUDA kernel as a compiled **SM80 (A100) binary** (`fraqtl/_native/libggml-cuda.so`). No `.cu`/`.c`/header source ships; the build hard-fails if any leaks in (see `wheel_build_receipt.json`). |
| - The vLLM integration layer (attention backend, model adapters, sidecar readers) as Python source — it must stay `torch.compile`-traceable or the engine provisions a smaller KV pool than the receipts. It is orchestration glue: how eigenbases are computed is not in the wheel in any form (no calibration or sidecar-factory code; remaining runtime modules are Cython-compiled binaries). |
| - vLLM plugin auto-registration via the `vllm.general_plugins` entry point — `pip install` the wheel and vLLM's plugin loader finds the backend in every worker process. |
|
|
| ## Scope |
|
|
| - **Hardware: A100 (SM80) only.** The binary will not run on H100, L4, or consumer GPUs. |
| - **vLLM 0.20.2, CUDA 12.4, Python 3.11** — the exact receipt environment. Other versions are unsupported. |
| - Purpose: verification of the published receipts. It is not a general-purpose release. |
|
|
| ## Use it |
|
|
| Don't drive this by hand — use the one-command repro script (`fraqtl_repro_receipts.py`, in this repo and in the sidecar repos): |
|
|
| ```bash |
| pip install modal && modal setup |
| modal secret create huggingface HF_TOKEN=hf_... |
| modal run fraqtl_repro_receipts.py |
| ``` |
|
|
| It installs this wheel, pulls the calibrated sidecars from |
| [`fraQtl/mistral-7b-instruct-v0.3-kv-sidecars`](https://huggingface.co/fraQtl/mistral-7b-instruct-v0.3-kv-sidecars), |
| runs the three-arm receipt (fraQtl D2 vs fp16 vs fp8-KV) at 8K and 32K with a |
| needle-in-a-haystack gate per arm, and prints the table. |
|
|
| ### Honesty footnotes |
|
|
| - **NIAH-verified, not "lossless."** |
| - **vs-fp8 always visible** — fp8 KV is the real competitor. |
| - **Batch-1 decode** for the published speed rows. |
|
|
| ## Files |
|
|
| | File | What | |
| |---|---| |
| | `fraqtl_runtime-*.whl` | The prebuilt wheel (binary-only) | |
| | `wheel_build_receipt.json` | Build provenance: sha256, source-leak audit result, module list | |
| | `fraqtl_repro_receipts.py` | The one-command repro script | |
|
|
| ## License |
|
|
| Proprietary; free to install and run for verification and evaluation. No reverse engineering. Sidecars and receipts in the companion repos are Apache 2.0. |
|
|
