File size: 404 Bytes
c8de259 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | ---
language: en
tags:
- causal-lm
- gqa
- qk-norm
- rope
- swiglu
- refresh-gate
- rose-x1
license: apache-2.0
---
# Rose-Mini
Rose X1 SLM (49.4M params).
## Architecture
| Property | Value |
|---|---|
| Layers | 14 |
| Hidden | 512 |
| Heads | 8 (kv=2) |
| QK Norm | True |
| Refresh Gate | [4, 9] |
| Params | 49.443M |
## Training
- Tokens: 12,206,861,568
- Val PPL: 6.14
- Optimizer: muon_adamw
|