| --- |
| library_name: transformers |
| --- |
| |
| # tiny-mimo-v2-flash |
|
|
| A ~2.34B-parameter tiny random-weight checkpoint of [XiaomiMiMo/MiMo-V2-Flash](https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash), used for internal testing in Hugging Face `transformers` for the native HF implementation. |
|
|
| ## Configuration |
|
|
|
|
|
|
| | Hyperparameter | Value | Original MiMo | |
| |--------|-------|--------------| |
| | `num_hidden_layers` | 5 | 48 | |
| | `layer_types` | `[full, sliding×4]` | matches pattern | |
| | `mlp_layer_types` | `[dense, sparse×4]` | matches pattern (layer 0 dense, rest MoE) | |
| | `hidden_size` | 2048 | 4096 (**ratio 2.0**) | |
| | `intermediate_size` | 8192 | 16384 (**ratio 2.0**) | |
| | `moe_intermediate_size` | 1024 | 2048 (**ratio 2.0**) | |
| | `num_attention_heads` / `num_key_value_heads` | 16 / 1 | 64 / 4 (**ratio 4.0**) | |
| | `head_dim` / `v_head_dim` | 192 / 128 | 192 / 128 | |
| | `n_routed_experts` / `num_experts_per_tok` | 64 / 2 | 256 / 8 (**ratio 4.0**) | |
| | parameters | 2.34B | 300B | |
|
|
|
|
