license: apache-2.0
base_model: Qwen/Qwen3-4B
quantized_by: 6block
pipeline_tag: text-generation
tags:
- compressed-tensors
- FP8
- vllm
- quantized
language:
- en
- zh
Qwen3-4B-FP8
FP8 (W8A8 dynamic) quantization of Qwen/Qwen3-4B, produced with llm-compressor in the compressed-tensors format for efficient serving with vLLM and SGLang.
8-bit floating point weights with dynamic activation scales; roughly half the size of bf16, near lossless, and native to Hopper/Blackwell GPUs.
Serving / 部署
vllm serve 6block/Qwen3-4B-FP8
The quantization scheme is declared in config.json and detected automatically — no extra flags are
required. Compatible with any engine that reads compressed-tensors (vLLM, SGLang, …).
config.json 中已声明量化方案，引擎自动识别，无需额外参数。兼容一切支持 compressed-tensors 的推理框架。
Prompt format / 对话格式
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
Details / 说明
| Method / 方法 | FP8 (W8A8 dynamic) |
| Format / 格式 | compressed-tensors (safetensors) |
| Base model / 原始模型 | Qwen/Qwen3-4B |
| Tool / 工具 | llm-compressor |
| License / 许可 | apache-2.0 (inherited from base model / 继承自原始模型) |
English — This checkpoint reproduces the base model's behavior at reduced precision, shrinking memory footprint and increasing throughput while keeping quality close to the original. It is a drop-in replacement for the base model in vLLM/SGLang deployments.
中文 — 本权重是原始模型的低精度量化版本，在保持接近原模型质量的同时，显著降低显存占用、提升吞吐。 在 vLLM/SGLang 部署中可直接替换原始模型使用。
Quantized by 6block. Weights are unmodified apart from quantization; all model behavior and license obligations follow the base model. / 由 6block 量化，除量化外未改动权重，模型行为与许可义务均遵循原始模型。