| language: en | |
| tags: | |
| - causal-lm | |
| - gqa | |
| - qk-norm | |
| - rope | |
| - swiglu | |
| - refresh-gate | |
| - rose-x1 | |
| license: apache-2.0 | |
| # Rose-Mini | |
| Rose X1 SLM (49.4M params). | |
| ## Architecture | |
| | Property | Value | | |
| |---|---| | |
| | Layers | 14 | | |
| | Hidden | 512 | | |
| | Heads | 8 (kv=2) | | |
| | QK Norm | True | | |
| | Refresh Gate | [4, 9] | | |
| | Params | 49.443M | | |
| ## Training | |
| - Tokens: 12,206,861,568 | |
| - Val PPL: 6.14 | |
| - Optimizer: muon_adamw | |