ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ

Qwen3.6-35B-A3B — 2-bit GSQ

Low-bit quantization of Qwen/Qwen3.6-35B-A3B (MoE, 35B total / 3B active) produced with GSQ (Gumbel-Softmax Quantization).

The routed-expert MLP weights are quantized to 2-bit GSQ with an effective storage cost of ≈2.13 bpp, while the attention layers, shared experts, and LM head are quantized to INT8. The checkpoint preserves most of the base model's reasoning, coding, and long-context behaviour while substantially reducing its memory footprint.

• Paper: GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (arXiv:2604.18556)
• Paper page on HF: https://huggingface.co/papers/2604.18556
• Code: https://github.com/IST-DASLab/GSQ
• Collection: https://huggingface.co/collections/ISTA-DASLab/gsq

Evaluation Results

We evaluate the quantized checkpoint against the original Qwen/Qwen3.6-35B-A3B across reasoning, instruction-following, science QA, and math benchmarks.

Benchmark	Base Model	2-bit GSQ
AIME 2025	93.33	93.33
GPQA Diamond	83.84	80.30
IFEval	91.25	87.77
MMLU-Pro	84.87	81.00
GSM8K	96.21	93.93

Quantization details

• Base model: Qwen/Qwen3.6-35B-A3B
• Routed-expert MLP precision: 2-bit GSQ, ≈2.13 bpp effective storage
• Codebook: 2-bit symmetric scalar {-2, -1, 0, +1} × scale
• Group size: 128
• Additional INT8 quantization: self_attn, linear_attn, shared experts, and lm_head
• Format: Humming
• Pipeline: GPTQ initialization → Gumbel-Softmax refinement (Lion optimizer)
• 2-bit quantized: routed-expert MLPs (gate_proj, up_proj, down_proj)
• INT8 quantized: attention (self_attn, linear_attn), shared experts, and LM head
• Kept in BF16: embeddings, layernorms, MoE routing gate, and other non-quantized components

Storage layout (why the HF UI shows I32 + I8 + BF16)

The Hugging Face "Tensor types" widget reports the container dtype of each tensor stored in the safetensors checkpoint, rather than the effective precision of the original model weights.

The routed-expert MLPs use the Humming exact-width 2-bit layout. For every 2-bit expert-MLP Linear with original weight shape [out_features, in_features], the following tensors are stored:

Tensor	Dtype	Shape on disk	Meaning
<layer>.weight	I32	[out_features, in_features × 2 / 32] = [out_features, in_features / 16]	2-bit values bit-packed along the input dimension, LSB-first: 16 weights per INT32 word
<layer>.weight_scale	BF16	[out_features, in_features / 128]	One symmetric scale per group of group_size = 128 weights along the input dimension
self_attn / linear_attn / shared-expert / lm_head weights	I8	implementation-dependent	INT8-quantized weights
Embeddings / norms / MoE routing gate / other unquantized tensors	BF16	unchanged	Copied or retained at BF16 precision

For the 2-bit GSQ expert weights, the effective storage is:

2 bits (packed) + 16 bits / 128 (group scale) ≈ 2.13 bpp

The checkpoint therefore uses a mixed-precision quantization layout: 2-bit GSQ for routed-expert MLP weights, INT8 for attention, shared experts, and the LM head, and BF16 for the remaining unquantized components.

The quantization_config in config.json describes the Humming quantization layout used by the checkpoint.

Note: GSQ training first writes shards in compressed-tensors pack-quantized format, where the 2-bit codebook is padded into a 4-bit INT32 container. The published checkpoint has been re-packed via convert_to_humming.py into exact-width 2-bit Humming storage, hence the 2 / 32 shape factor for the 2-bit expert weights.

Serving with vLLM

Serving this checkpoint requires:

• vLLM 0.27.1
• the vLLM compatibility patch included in this model repository
• an Ampere (SM ≥ 80) or Hopper GPU

Install the required vLLM version:

pip install vllm==0.27.1

Then, from the directory containing the model files, run the included patch_vllm.py script:

python patch_vllm.py

The patch must be executed in the same Python environment in which vLLM 0.27.1 is installed. It patches the installed vLLM package with the changes required to load and serve this checkpoint.

After applying the patch, the model can be served normally:

vllm serve ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ --reasonin-parser qwen3

Important: running the model with an unpatched vLLM installation is not supported. If vLLM is reinstalled or the environment is recreated, run python patch_vllm.py again before serving the model.

Model size / text-only usage: the full checkpoint is approximately 12.6 GB, including the MTP and vision components. These components are optional for ordinary text-only generation: the vision weights are only required for multimodal inputs, while the MTP weights are only required when using MTP/speculative decoding. Removing both the vision and MTP weights reduces the model size to approximately 9.9 GB.

Actual VRAM usage during serving will be higher than the raw model size and depends on KV-cache allocation, context length, batch size, and vLLM runtime overhead.

Citation

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurti{\'c}, Eldar and Kleinegger, Max and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026},
  url    = {https://arxiv.org/abs/2604.18556}
}