Why compressed-tensors
Most quantization formats (AWQ, GPTQ, GGUF) target a single inference backend and ship a frozen weight layout that cannot be further composed or modified at load time. compressed-tensors is the format developed by Neural Magic and maintained as a first-class vLLM citizen.
- Native vLLM integration. No format conversion, no plugin shims. vLLM reads compressed-tensors models directly via its built-in
CompressedTensorsWorker. Full PagedAttention, continuous batching, and tensor parallelism work without modification.
- Composable precision. A single checkpoint can carry per-layer or per-group precision assignments. Mixed-precision MoE configurations are expressed in the same file.
- Reproducible calibration metadata. The quantization config, calibration scheme, and per-channel scales are stored inside the checkpoint.
- Forward compatibility. As vLLM adds new kernel support (FP8, INT8, sparse), compressed-tensors models gain that support without re-quantizing.
AWQ and GPTQ remain fine for llama.cpp and older toolchains. If you are deploying on vLLM in production, compressed-tensors is the correct choice.
Quality Standard
W8A16
RTN / AutoRound iters=200
>99.5% MMLU recovery
Ampere+ (A100, A6000, RTX 30xx+)
W4A16
AutoRound iters=200 (SignSGD)
≥99% MMLU recovery
Ampere+ (A100, A6000, RTX 30xx+)
AutoRound at iters=200 runs sign-gradient optimization over a calibration set to minimize weight rounding error. At W4A16, this closes most of the gap between naive round-to-nearest and GPTQ/AWQ, while producing a checkpoint that vLLM can load natively.
Model Catalog
All 16 models in compressed-tensors format, validated for vLLM v0.9.0+.
Qwen3.6-35B-A3B — Mixed-Precision MoE, 1M context
Qwen3.6-27B — Dense Hybrid, 262k context
Qwen3-Omni-30B-A3B — Audio + Vision + Speech
Qwen2.5-Omni-7B — Efficient Omni
Gemma4-E4B-it — Vision-Language Model
Gemma4-E2B-it — Ultra-Efficient VLM
MiniCPM-o-4.5 — Omni Model
Nemotron-3-Nano-30B-A3B — Hybrid SSM/Attention
Quickstart
Requires vLLM v0.9.0+ and an Ampere-class GPU (A100, A6000, RTX 3090/4090, or equivalent).
Install
pip install vllm>=0.9.0
Offline inference
from vllm import LLM, SamplingParams
llm = LLM(
model="88plug/Qwen3.6-35B-A3B-W4A16",
max_model_len=131072,
tensor_parallel_size=1,
)
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, max_tokens=512)
outputs = llm.generate(["Explain W4A16 vs W8A16 tradeoffs."], sampling_params)
print(outputs[0].outputs[0].text)
OpenAI-compatible server
vllm serve 88plug/Qwen3.6-35B-A3B-W4A16 \
--max-model-len 131072 \
--port 8000
Hardware Requirements
| Model Size | W8A16 VRAM | W4A16 VRAM | Recommended |
|---|
| 2B–7B | 8–16 GB | 6–10 GB | Single A6000 / RTX 4090 |
| 27B–35B (dense) | 32–40 GB | 20–28 GB | Single A100 80G or 2× A6000 |
| 30B–35B (MoE, 3B active) | 28–36 GB | 18–24 GB | Single A100 80G or 2× A6000 |