threcc/qwen3-moe-dummy

Qwen3-MoE Dummy

Dummy Qwen3-MoE model for testing. Not trained — weights are randomly initialized.

Produces garbage output but uses the real Qwen3-MoE architecture (qwen3_moe), so vLLM loads and serves it correctly with expert parallelism enabled.

Purpose

Validate MoE deployment plumbing without downloading a real model (~10 seconds to load vs hours for a real MoE). Use cases:

• Multinode DP+EP (data parallel + expert parallel) deployment validation
• LeaderWorkerSet topology checks
• InferencePool and router-scheduler pod verification
• CI/CD smoke tests for MoE support

Architecture

Adapted from Qwen/Qwen3-235B-A22B with dimensions scaled down to ~20M parameters:

Parameter	Value
Architecture	Qwen3MoeForCausalLM
Total parameters	19.9M
Hidden size	64
Intermediate size	128
MoE intermediate size	128
Experts	8 (top-2 routing)
Attention heads	2
KV heads	1
Head dim	32
Hidden layers	2
Tensor type	BF16
Format	Safetensors

License

Apache-2.0. Weights are randomly initialized — no trained data, no IP. Tokenizer from Qwen/Qwen3-235B-A22B (Apache-2.0).

How this model was created

Generated with transformers.AutoModelForCausalLM.from_config() with random initialization. No training data or pretrained weights were used.