Qwen

/

Qwen3-Coder-Next

like 341

Follow

Qwen 67.7k

Can anybody run llm-compressor on this model to get a 4-bit quantized version to run on vLLM on memory-restricted hardware?

Thanks in advance.

from llmcompressor import oneshot
from llmcompressor.modifiers.awq import AWQModifier
from datasets import load_dataset
import os
calibration_set = load_dataset("nvidia/Llama-Nemotron-Post-Training-Dataset", split="train")

recipe = [
    AWQModifier(
        ignore=["lm_head"],
        config_groups={
            "group_0": {
                "targets": ["Linear"],
                "weights": {
                    "num_bits": 4,
                    "type": "int",
                    "symmetric": False,
                    "strategy": "group",
                    "group_size": 32,
                }
            }
        }
    ),
]

oneshot(
    model="Qwen/Qwen3-Coder-Next",
    dataset=calibration_set,
    recipe=recipe,
    output_dir="~/Qwen3-Coder-Next-AWQ-4bit",
    max_seq_length=2048,
    num_calibration_samples=1024,
)