metadata
language: en
tags:
- causal-lm
- gqa
- qk-norm
- rope
- swiglu
- refresh-gate
- rose-x1
license: apache-2.0
Rose-Mini
Rose X1 SLM (49.4M params).
Architecture
| Property | Value |
|---|---|
| Layers | 14 |
| Hidden | 512 |
| Heads | 8 (kv=2) |
| QK Norm | True |
| Refresh Gate | [4, 9] |
| Params | 49.443M |
Training
- Tokens: 12,206,861,568
- Val PPL: 6.14
- Optimizer: muon_adamw