---
license: mit
base_model: zai-org/GLM-4.7-Flash
tags:
  - fp8
  - quantized
  - glm4
  - moe
library_name: transformers
---

# GLM-4.7-Flash FP8

FP8 quantized version of [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash).

**NOTE**: For optimal generation parameters, unsloth are working on finding the optimal parameters for practical use as of now. 
They are for llama.cpp, but I'm sure they translate to vLLM as well: https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF

## Quantization Details

- **Method**: FP8 E4M3 per-tensor quantization with embedded scales
- **Original size**: ~62GB (BF16)
- **Quantized size**: ~30GB (FP8)
- **Preserved in BF16**: lm_head, embed_tokens, layernorms, router weights



## Performance

Tested on 2x RTX 3090 (24GB each) with vLLM 0.13.0:

| Setting | Value |
|---------|-------|
| Tensor Parallel | 2 |
| Context Length | 8192 |
| VRAM per GPU | 14.7 GB |
| Throughput | **19.4 tokens/sec** |

Note: RTX 3090 lacks native FP8 support, so vLLM uses the Marlin kernel for weight-only FP8 decompression. GPUs with native FP8 (RTX 40xx, Ada Lovelace+) will achieve higher throughput.

## Usage with vLLM

Requires vLLM 0.13.0+ and transformers 5.0+ for `glm4_moe_lite` architecture support.

```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="marksverdhei/GLM-4.7-Flash-fp8",
    tensor_parallel_size=2,
    max_model_len=8192,
    enforce_eager=True,  # Optional: disable CUDA graphs to save VRAM
)

outputs = llm.generate(["Hello, world!"], SamplingParams(max_tokens=100))
print(outputs[0].outputs[0].text)
```

### vLLM Fork Required

Until upstream vLLM adds MLA detection for `glm4_moe_lite`, use our fork:

```bash
pip install git+https://github.com/marksverdhei/vllm.git@fix/glm4-moe-mla-detection
```

Or install from source:
```bash
git clone https://github.com/marksverdhei/vllm.git
cd vllm
git checkout fix/glm4-moe-mla-detection
pip install -e .
```

**Fork**: [marksverdhei/vllm](https://github.com/marksverdhei/vllm/tree/fix/glm4-moe-mla-detection)

## License

MIT (same as base model)