---
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags: [vllm_ci]
---

# Model Overview

- **Model Architecture:** Qwen3MoeForCausalLM (tiny, randomly initialized)
  - **Input:** Text
  - **Output:** Text
- **Supported Hardware Microarchitecture:** AMD MI300 / MI350 / MI355 (gfx942 / gfx950), NVIDIA (SM ≥ 7.5)
- **Inference Engine:** [vLLM](https://docs.vllm.ai/en/latest/)
- **Model Optimizer:** [AMD-Quark](https://quark.docs.amd.com/latest/index.html)
  - **Weight quantization:** INT8, per-channel, static, symmetric
  - **Activation quantization:** INT8, per-token, dynamic, symmetric
  - **Quantized layers:** attention (`self_attn`) and the routed MoE experts.
    The MoE router (`mlp.gate`) and `lm_head` are kept in the original precision.

This is a **tiny, randomly-initialized** Qwen3-MoE model quantized to W8A8-INT8,
used purely as **vLLM CI coverage** for the Quark INT8 fused-MoE path
(`QuarkW8A8Int8MoEMethod`), which dispatches through the modular Triton INT8 MoE
kernel. It is not intended to produce meaningful text.

> Note: the router (`mlp.gate`) is intentionally left in bf16. Its weight is
> `[num_experts, hidden]` and `num_experts` (8) is not a multiple of 16, which
> would trip AITER's INT8 linear-kernel shape assertion. Routers are normally
> left in high precision anyway.

# Model Creation

Built and quantized with [AMD-Quark](https://quark.docs.amd.com/latest/index.html):
per-channel INT8 weights (static) + per-token INT8 activations (dynamic),
symmetric, on a tiny `qwen3_moe` model. Attention and routed experts are
quantized; `mlp.gate` and `lm_head` are excluded. `vocab_size` matches the
tokenizer so token ids stay within the embedding table.

# Usage in vLLM

```bash
vllm serve amd/tiny-qwen3-moe-w8a8-int8 --enforce-eager
```

Because the weights are random, outputs are not meaningful — this model is a
structural / smoke-test fixture only.

# License

Apache-2.0. The tiny model is randomly initialized and derives no weights from
any base model.

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.