| --- |
| license: apache-2.0 |
| base_model: |
| - Qwen/Qwen3-32B |
| --- |
| |
| ## Model Details |
|
|
| This model is an mxfp4 quantized version of [Qwen/Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) generated by [intel/auto-round](https://github.com/intel/auto-round) in llm_compressor format. |
| Please follow the license of the original model. |
| |
| ## How to Use |
| |
| The step-by-step README of quantization and evaluation can be found in [Intel Neural Compressor Examples](https://github.com/intel/neural-compressor/blob/master/examples/pytorch/nlp/huggingface_models/language-modeling/quantization/auto_round/qwen/README.md). |
| |
| ## Evaluate Results |
| - Eval with vllm==0.25.1 |
| | Precision | hellaswag | piqa | mmlu | gsm8k | average | |
| | -------------------- | --------- | ------ | ------ | ------ | ------- | |
| | bf16 | 0.6394 | 0.8112 | 0.8078 | 0.9257 | 0.79602 | |
| | mxfp4 RTN | 0.6260 | 0.7922 | 0.7872 | 0.9287 | 0.78353 | |
| |
| ## Ethical Considerations and Limitations |
| |
| The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. |
| Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs. |
| |
| Therefore, before deploying any applications of the model, developers should perform safety testing. |
| |
| ## Caveats and Recommendations |
| |
| Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. |
| |
| Here are a couple of useful links to learn more about Intel's AI software: |
| |
| - [Intel Neural Compressor](https://github.com/intel/neural-compressor) |
| - [AutoRound](https://github.com/intel/auto-round) |
| |
| ## Disclaimer |
| |
| The license on this model does not constitute legal advice. |
| We are not responsible for the actions of third parties who use this model. |
| Please consult an attorney before using this model for commercial purposes. |
| |
