---
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/main/LICENSE
base_model: Qwen/Qwen3-Coder-30B-A3B-Instruct
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
tags:
  - qwen3_moe
  - moe
  - code
  - quantized
  - int4
  - w4a16
  - compressed-tensors
  - llm-compressor
  - vllm
---

# Qwen3-Coder-30B-A3B-Instruct-W4A16

**W4A16** (INT4 group-128 weights + FP16 activations) quantization of
[Qwen/Qwen3-Coder-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct).

- **Quantized with** [llm-compressor](https://github.com/vllm-project/llm-compressor) on an
  **NVIDIA H200**, in the
  [compressed-tensors](https://github.com/neuralmagic/compressed-tensors) `pack-quantized`
  format.
- **Designed for inference on 2× NVIDIA A2 (16 GB, Ampere SM 8.6) with
  [vLLM](https://github.com/vllm-project/vllm)** (tensor-parallel across the two cards).

The point of this build is to fit this 30B-A3B MoE onto **small, FP8-less GPUs** like the
A2, where BF16 (~57 GB) and INT8 (~30 GB) don't fit. At 4-bit the checkpoint is ~16 GB
(~8 GB/GPU at TP=2), running via the Marlin INT4 kernel — which, unlike FP8 / W4AFP8, works
on Ampere.

## What's quantized

| Quantized → INT4 (g128, symmetric) | Kept in BF16 |
|---|---|
| all 128 routed experts × 48 layers | token embeddings, `lm_head` |
| attention `q/k/v/o` projections | MoE router gates, all norms |

Only transformer `Linear` weights are quantized; the embedding, output head, router gates,
and norms stay BF16 for quality. It remains a standard `Qwen3MoeForCausalLM` — full GQA
attention, SwiGLU, 128 experts / 8 active — so it uses vLLM's mainstream MoE path.

## Serving with vLLM

```bash
vllm serve systemsofrecord/Qwen3-Coder-30B-A3B-Instruct-W4A16 \
  --tensor-parallel-size 2 \
  --dtype float16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

- **No FP8 required** — runs on Ampere (A2 / A10 / A30 / …) and newer.
- **KV cache** is FP16 (Ampere has no FP8 KV); GQA (4 KV heads) keeps it small.
- Needs a vLLM build with `Qwen3MoeForCausalLM` support (≥ 0.25).

**Verified**: loaded and generated correct code on **2× NVIDIA A2** under vLLM 0.25.1 —
~7.9 GB weights/GPU at TP=2, Marlin wNa16 MoE kernel, CUDA graphs captured cleanly.

## Quantization recipe

- Tool: **llm-compressor** (run on an **NVIDIA H200**).
- Scheme: `W4A16` — weights 4-bit int, `group_size=128`, symmetric; activations unquantized.
- Method: model-free RTN (round-to-nearest) weight quantization.
- Format: `pack-quantized` (INT4 packed into INT32 + group scales).
- Ignore (BF16): `lm_head`, `embed_tokens`, MoE router gates, norms.
- Target: **2× NVIDIA A2** served with **vLLM** (TP=2).

## License & attribution

Apache-2.0, inherited from the base model
[Qwen/Qwen3-Coder-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct).
This repository only redistributes a **quantized** copy of those weights; all model
capabilities and credit belong to the Qwen team.