metadata
base_model:
- Qwen/Qwen3-8B
Model Details
This example model demonstrates how to use AutoRound’s compatibility to automatically generate a mixed-bit quantization recipe.
For more details, please refer to user guide
Generate the model
auto-round > 0.8.0
from auto_round import AutoRound, AutoScheme
model_name = "Qwen/Qwen3-8B"
avg_bits = 3.0
scheme = AutoScheme(avg_bits=avg_bits, options=("GGUF:Q2_K_S", "GGUF:Q4_K_S"), ignore_scale_zp_bits=True)
layer_config = {"lm_head": "GGUF:Q6_K"}
ar = AutoRound(model=model_name, scheme=scheme, layer_config=layer_config, iters=0)
ar.quantize_and_save()