README.md

---
license: mit
base_model: lightseekorg/kimi-k2.5-eagle3
base_model_relation: quantized
library_name: transformers
tags:
- speculative-decoding
- eagle3
- draft-model
- kimi-k2.5
- fp8
- amd-quark
- quantized
- no-lm-head-quantization
---

## Model Overview

**kimi-k2.5-eagle3-fp8** is an FP8-quantized version of [lightseekorg/kimi-k2.5-eagle3](https://huggingface.co/lightseekorg/kimi-k2.5-eagle3), an Eagle3 MTP draft model for accelerating inference of [Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) with speculative decoding.

This checkpoint was quantized with **AMD Quark**. The quantized tensors use **FP8** quantization metadata in the model config. The **LM head is not quantized** and was intentionally excluded from quantization.

## Model Quantization

The checkpoint keeps the original Eagle3 architecture and exports Quark quantization metadata in `config.json`. The `fc` projection and `lm_head` are intentionally **not quantized**.

**Quantization details:**

- **Quantization tool:** AMD Quark
- **Quantization method:** `quark`
- **Quantization scheme:** `ptpc_fp8`
- **FP8 format:** `fp8_e4m3`
- **Weight quantization:** FP8 E4M3, static, per-channel, symmetric, channel axis `0`
- **Input/activation quantization config:** FP8 E4M3, dynamic, per-channel, symmetric, channel axis `1`
- **Export weight format:** `real_quantized`
- **Output tensor quantization:** not enabled
- **KV-cache quantization:** not enabled
- **Excluded from quantization:** `fc`, `lm_head`

### Quantization Command

```bash
cd Quark/examples/torch/language_modeling/llm_ptq/

python3 quantize_quark.py \
  --model_dir lightseekorg/kimi-k2.5-eagle3 \
  --quant_scheme ptpc_fp8 \
  --exclude_layers fc lm_head \
  --output_dir amd/kimi-k2.5-eagle3-fp8 \
  --file2file_quantization
```

No calibration dataset is required for this file-to-file quantization path.

### vLLM Loading Note

When using this FP8 Eagle3 checkpoint as a vLLM draft model, make sure the exported `config.json` records the excluded layers as regex patterns. If Quark exports:

```json
"exclude": [
  "fc",
  "lm_head"
]
```

change it to:

```json
"exclude": [
  "re:.*fc.*",
  "re:.*lm_head.*"
]
```

This keeps `fc` and `lm_head` unquantized while allowing vLLM to correctly load the Quark FP8 Eagle3 draft model.

### Quantized Layers

The following Eagle3 projection weights are stored as `F8_E4M3` with associated `F32` per-channel scale tensors:

- `midlayer.self_attn.q_proj.weight`
- `midlayer.self_attn.k_proj.weight`
- `midlayer.self_attn.v_proj.weight`
- `midlayer.self_attn.o_proj.weight`
- `midlayer.mlp.gate_proj.weight`
- `midlayer.mlp.up_proj.weight`
- `midlayer.mlp.down_proj.weight`

Each quantized weight tensor has a matching `*_weight_scale` tensor stored in FP32.

### Layers Not Quantized

The following tensors are intentionally not stored as FP8:

- `fc.weight`: kept in `F16`
- `lm_head.weight`: kept in `F16`
- `embed_tokens.weight`: kept in `BF16`
- normalization weights: kept in `F16`

### Tensor Dtype Overview

| Tensor dtype | Count | Notes |
| --- | ---: | --- |
| `F8_E4M3` | 7 | Quantized attention and MLP projection weights |
| `F32` | 7 | Per-channel scale tensors for FP8 weights |
| `F16` | 6 | Excluded `fc`, `lm_head`, and normalization weights |
| `BF16` | 1 | Token embedding weight |

## Intended Use

This model is intended to be used as an Eagle3 draft model for speculative decoding with `moonshotai/Kimi-K2.5` as the target model.

Because this is an AMD Quark FP8 checkpoint, make sure your inference runtime supports the quantization format and Eagle3 speculative decoding before deployment. Please validate quality and acceptance length in your own serving stack.

## Citation and Acknowledgements

This model is derived from [lightseekorg/kimi-k2.5-eagle3](https://huggingface.co/lightseekorg/kimi-k2.5-eagle3). Please refer to the source model card for the original training details, benchmarks, and acknowledgements.

## License

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.