README.md

---
library_name: transformers
base_model: Qwen/Qwen3.5-27B
tags:
  - rotorquant
  - kv-cache-quantization
  - efficient-inference
  - qwen3.5
  - thinking-model
license: apache-2.0
---

# Qwen3.5-27B-RotorQuant -- RotorQuant KV Cache Compression

Qwen3.5-27B with **RotorQuant** KV cache compression applied. RotorQuant uses block-diagonal rotations derived from Clifford algebra to compress KV caches with substantially better speed and efficiency than prior methods. At 3-bit precision, it achieves approximately 10x KV cache compression while maintaining strong output quality.

The base model is [Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B), a 27B parameter hybrid transformer combining gated delta networks with sparse mixture-of-experts. It supports 262K native context with extension to 1M+ tokens and operates in thinking mode by default.

## What is RotorQuant?

[RotorQuant](https://github.com/scrya-com/rotorquant) is a KV cache compression framework that replaces the dense random rotation used in methods like TurboQuant with **block-diagonal rotations** grounded in Clifford algebra. This architectural choice yields major practical advantages:

- **28% faster decode** and **5.3x faster prefill** compared to TurboQuant
- **44x fewer parameters** (128 vs 16,384) for the rotation matrices
- **O(d) complexity** vs O(d log d) for the rotation step
- **Lower perplexity**: 6.91 vs 7.07 (TurboQuant) on standard benchmarks

RotorQuant ships three backend implementations, each offering a different speed/quality tradeoff:

| Backend | Algebra | Best For |
|---------|---------|----------|
| **PlanarQuant** | 2D Givens rotations | Fastest inference -- production deployments |
| **IsoQuant** | 4D quaternion rotations | Balanced speed and quality |
| **RotorQuant** | 3D Clifford rotors | Research and maximum quality |

## Quickstart

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from turboquant import IsoQuantCache

model = AutoModelForCausalLM.from_pretrained(
    "majentik/Qwen3.5-27B-RotorQuant",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("majentik/Qwen3.5-27B-RotorQuant")

# Apply chat template (Qwen3.5 supports thinking mode)
messages = [{"role": "user", "content": "Explain quantum computing"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

# 3-bit IsoQuant cache -- recommended setting (~10x KV compression)
cache = IsoQuantCache(bits=3)
output = model.generate(**inputs, max_new_tokens=2048, past_key_values=cache, use_cache=True)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

### Switching Backends

```python
from turboquant import PlanarQuantCache, IsoQuantCache, RotorQuantCache

# Fastest -- 2D Givens rotations (production)
cache = PlanarQuantCache(bits=3)

# Balanced -- 4D quaternion rotations
cache = IsoQuantCache(bits=3)

# Research -- 3D Clifford rotors (highest quality)
cache = RotorQuantCache(bits=3)
```

## Configuration

| Bit Width | Quality | Compression | Recommended Use |
|-----------|---------|-------------|-----------------|
| **4-bit** | Near-lossless | ~4x KV cache | Quality-sensitive applications |
| **3-bit** | Strong (ppl 6.91) | ~10x KV cache | **Recommended default** -- best quality/compression tradeoff |
| **2-bit** | Moderate degradation | ~16x KV cache | Extreme memory constraints |

The 3-bit setting is recommended as the default. It provides approximately 10x KV cache compression with a perplexity of 6.91, which is lower (better) than TurboQuant's 7.07 at the same bit width.

## Memory Savings

Qwen3.5-27B has substantial KV caches due to its 27B parameter count. RotorQuant's 3-bit mode provides approximately 10x compression, making long-context inference practical on fewer GPUs.

| Context Length | FP16 KV Cache | 4-bit RotorQuant | 3-bit RotorQuant | 2-bit RotorQuant |
|---------------|---------------|-------------------|-------------------|-------------------|
| 8K | ~3.4 GB | ~0.85 GB | ~0.34 GB | ~0.21 GB |
| 32K | ~13.5 GB | ~3.4 GB | ~1.35 GB | ~0.84 GB |
| 128K | ~54 GB | ~13.5 GB | ~5.4 GB | ~3.4 GB |
| 262K (native) | ~110 GB | ~27.5 GB | ~11 GB | ~6.9 GB |

*Estimates based on Qwen3.5-27B KV cache dimensions. Actual savings depend on model configuration and batch size.*

## Performance vs TurboQuant

| Metric | RotorQuant | TurboQuant |
|--------|------------|------------|
| Decode speed | **28% faster** | Baseline |
| Prefill speed | **5.3x faster** | Baseline |
| Rotation parameters | **128** | 16,384 |
| Rotation complexity | **O(d)** | O(d log d) |
| Perplexity (3-bit) | **6.91** | 7.07 |

## Thinking Mode

Qwen3.5-27B generates extended chain-of-thought reasoning before producing its final response. These thinking tokens can consume substantial KV cache memory -- often thousands of tokens of internal reasoning before a single output token is emitted. RotorQuant is especially valuable here because:

- Thinking tokens are generated autoregressively and cached, so KV cache grows rapidly during the reasoning phase.
- At 3-bit with ~10x compression, you can sustain much longer reasoning chains within the same VRAM budget.
- The 5.3x faster prefill directly accelerates the initial prompt processing, which matters for long system prompts and multi-turn conversations.
- The 28% faster decode speeds up the token-by-token generation during both thinking and response phases.

## See Also

- [Base model: Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B)
- [RotorQuant GitHub](https://github.com/scrya-com/rotorquant)
- [TurboQuant paper (arXiv: 2504.19874)](https://arxiv.org/abs/2504.19874)