WarpQuant: Dual-Domain LLM Quantization via Hadamard Rotation and Output-Fisher Sensitivity

Harim Choi

HarimxChoi

Follow

I started WarpQuant with two deployment targets in mind: running a recent 27B model within a 16GB packed runtime, and eventually deploying a 4B model on an iPhone 16.

Conventional 4-bit quantization was not small enough for these targets. Moving toward INT3, however, made it increasingly important to decide not only how to quantize weights, but also which quantization errors should be recovered.

WarpQuant addresses these two problems in different coordinate systems:

• Projection weights are quantized in a deterministic signed Hadamard domain.
• Recovery columns are selected in the original domain using Output-Fisher sensitivity.
• Block-GPTQ error feedback is applied with a small recovery budget.
• KV Cache and activations are evaluated separately using K4/V4/R128 and dynamic per-token A8.

The code, evaluation scripts, technical report, and five Hugging Face checkpoints are publicly available.

• Model Collection
• GitHub Repository
• Technical Report

Method

WarpQuant separates the domain used for compression from the domain used for recovery.

$$R=HD,\tilde{W}=Q_3(W{R}^{\mathrm{\top }})R,E=W-\tilde{W}\mathrm{.}R\; =\; HD,\; \backslash qquad\; \backslash widetilde\{W\}\; =\; Q\_3(WR^\backslash top)R,\; \backslash qquad\; E\; =\; W-\backslash widetilde\{W\}.$$

Here, $H$ is a normalized Hadamard matrix and $D$ is a deterministic random sign diagonal. Projection weights are rotated, quantized with a Gaussian Lloyd-Max 3-bit codebook, and reconstructed in the original coordinates.

Block-GPTQ transfers quantization error to later groups during reconstruction [1]. The Hadamard rotation spreads concentrated outliers across the input dimension before low-bit quantization [2].

The next question is which columns should receive the recovery budget.

Selecting columns only by local squared error can preserve large residuals that have little effect on next-token loss. WarpQuant instead combines input activation energy, quantization residual, and diagonal Output-Fisher sensitivity:

$$S_c=\frac{H_{X,cc}{E}_{:,c}^{\mathrm{\top }}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(H_G)E_{:,c}}{16d_{\mathrm{o}\mathrm{u}\mathrm{t}}+32}\mathrm{.}S\_c\; =\; \backslash frac\{\; H\_\{X,cc\}\; E\_\{:,c\}^\{\backslash top\}\; \backslash mathrm\{diag\}(H\_G)\; E\_\{:,c\}\; \}\{\; 16d\_\{\backslash mathrm\{out\}\}+32\; \}.$$

$H_X$ measures input-channel energy, $E$ is the reconstruction residual, and $H_G$ is estimated from squared output gradients collected through next-token NLL backpropagation. Columns are ranked globally by $S_c$ and recovered under a small additional bit budget.

from warpquant import (
    output_fisher_score,
    recover_columns,
    select_weak_columns,
)

scores = output_fisher_score(
    weight,
    base_weight,
    activations,
    output_fisher,
)

columns = select_weak_columns(scores, count=64)
recovered = recover_columns(base_weight, weight, columns)

Qwen3.8-27B results

The main evaluation uses the 26,895,998,464-parameter text backbone as the denominator, excluding vision and MTP parameters.

Commonsense is the macro average of fixed 1,000-example HellaSwag, WinoGrande, and PIQA screens. GSM8K uses the same first 500 examples with 5-shot prompting and flexible answer extraction.

WarpQuant uses 0.0775 fewer bits per text weight than IQ3_S and reduces the packed-equivalent payload by approximately 250 MiB. ARC-Challenge improves from 52.17% to 56.86%, while MMLU and the commonsense average remain close.

Results on Qwen3.5-4B and Llama 3 8B

The same dual-domain procedure was also applied to Qwen3.5-4B and Llama 3 8B.

On Qwen3.5-4B, WarpQuant is approximately 243.6 MiB smaller than IQ3_M while improving WikiText-2 PPL, ARC-Challenge, and MMLU.

On Llama 3 8B, WarpQuant is approximately 40 MiB smaller than IQ3_S and achieves a higher ARC-Challenge score.

KV Cache and activation quantization

Weight compression is only one part of autoregressive inference memory. WarpQuant therefore evaluates two additional memory paths:

• KV Cache: recent 128 tokens remain in BF16; older keys and values use K4/V4/R128 storage inspired by TurboQuant and QJL [4, 5].
• Activation: dynamic per-token INT8 quantization is applied to transient activations.

Released checkpoints

The following checkpoints are included in the WarpQuant Collection:

1. Qwen3.8-27B R16E4H4 Text
2. Qwen3.8-27B R16E4H4 Multimodal
3. Qwen3.5-4B R16E4 Text
4. Qwen3.5-4B R16E4V4 Multimodal
5. Llama 3 8B R16E4H4

The released checkpoints materialize the quantized values in safetensors so that the benchmark results can be reproduced with the existing Hugging Face stack. Packed serialization and fused C++/CUDA/Metal kernels are the next runtime components.

Reproduction

git clone https://github.com/HarimxChoi/WarpQuant
cd WarpQuant

pip install -r requirements.txt

bash benchmarks/run_text_metrics.sh
bash benchmarks/run_commonsense.sh
bash benchmarks/run_gsm8k_500.sh
bash benchmarks/run_kv_activation.sh

The repository contains the exact task definitions, evaluation scripts, payload accounting, and KV/A8 reference implementation used for the tables.

References

[1] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.” ICLR, 2023.

[2] Saleh Ashkboos et al. “QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs.” 2024.

[3] Jinuk Kim et al. “GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance.” ICML, 2025.

[4] Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. “TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate.” 2025.

[5] Amir Zandieh, Majid Daliri, and Insu Han. “QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead.” 2024.

Citation

@misc{choi2026warpquant,
  author = {Harim Choi},
  title = {WarpQuant: Dual-Domain LLM Quantization via Hadamard Rotation and Output-Fisher Sensitivity},
  year = {2026},
  url = {https://github.com/HarimxChoi/WarpQuant}
}