Pier-Jean/Qwen3-4B-LLVQ-2bit

Qwen3-4B — LLVQ 2-bit

Qwen3-4B quantized to 2.16 bits per weight with Leech lattice vector quantization, from an independent Rust implementation of arXiv:2603.11021 (van der Ouderaa, van Baalen, Whatmough, Nagel — Qualcomm AI Research, 2026).

One file, 1.771 GB, opens with no checkpoint, no cache and no network.

⚠️ Read the format section before downloading. This is not GGUF, AWQ or safetensors. It does not load with transformers, llama.cpp, vLLM or TGI. You need the Rust reader linked below. It is published as a reproducible research artifact, not as a drop-in model.

Numbers

Measured, not computed — the file was written, read back, and its 3 633 315 840 weights decode bit for bit to the weights that were evaluated.

Size	1.771 GB against 8.045 GB in FP16 → ×4.54
Rate	2.1595 bits/weight over the quantized projections
WikiText-2 perplexity, ctx 4096	16.9617 (FP32 baseline: 12.2336, ×1.386)

Composition: 252 quantized linear projections (0.981 GB) + 146 tensors the quantizer does not touch, at f16 (0.790 GB — almost all of it the tied embedding, 9.7 % of the model) + config and tokenizer.

Against published 2-bit methods

All figures without fine-tuning, on the same benchmark. Rates are what each method stores.

Method	Wiki ↓	bits/weight
Quip#/E8P12	21.15	2.000
QTIP (3INST)	17.04	2.000
LLVQ, 0 gain bits (paper)	17.05	2.000
This model	16.9617	2.1595
LLVQ, 2 gain bits (paper's best)	15.54	2.000

It lands just under QTIP — at 8 % more bits — and stays 9 % above the paper's best configuration. That is the honest reading; it is not state of the art.

Quantization recipe

Shape–gain over the Leech lattice Λ₂₄, direction capped at shell 12 (47-bit index) plus one gain bit, spherical retraction, per-row scale, input incoherence rotation, tail columns kept exact. Calibration on C4 — out of domain with respect to WikiText-2, matching the paper's protocol — 64 windows of 2048 tokens. 4 h on an M3 Max.

How to run it

git clone https://github.com/pjmalandrino/llvq && cd llvq
hf download Pier-Jean/Qwen3-4B-LLVQ-2bit qwen3-4b-llvq.bin --local-dir .
cargo run --release -p llvq-llm --features metal --bin run -- qwen3-4b-llvq.bin metal 24

Drop --features metal for CPU. Nothing else is required — no Hugging Face cache, no network. That is checked in CI-style with an empty environment:

env -i HOME=/nonexistent PATH=/usr/bin:/bin ./target/release/run qwen3-4b-llvq.bin cpu 14

Limitations

• No inference speedup. The reader decodes weights into memory and then does an ordinary matvec. The gain here is size on disk and in RAM, not latency. A fused dequantize+matvec kernel is unwritten — and does not exist for this regime anywhere, including in the paper, whose own CUDA kernel handles a single shell "for simplicity" and is slower than QTIP.
• The format is not portable. ~250 lines of dependency-free Rust (llvq-artifact) define it; a reader in another language is straightforward but does not exist yet.
• Perplexity only. No MMLU, no commonsense reasoning, no task-specific evaluation. Perplexity can stay flat while individual answers change (arXiv:2607.08734).
• Evaluated on 12 windows, not the full 73. The FP32 baseline lands 1.4 % under the paper's, so this window subset is slightly easier.

License and attribution

Apache 2.0, inherited from Qwen/Qwen3-4B. The LICENSE file in this repository is Qwen's, carried over unchanged.

Modification made to the original work: the 252 linear projection weight tensors of every transformer block have been replaced by Leech lattice codes (index + gain per 24 weights, with a per-row scale) and are reconstructed at load time. All other tensors are the originals, converted to f16. No training, no fine-tuning, no architectural change.

The quantization implementation is at github.com/pjmalandrino/llvq (MIT OR Apache-2.0).