| --- |
| license: mit |
| tags: |
| - scaling-laws |
| - hyperbolic-geometry |
| - language-model |
| - research |
| --- |
| |
| # HyperScale: Euclidean vs Hyperbolic Output Layer Scaling Laws |
|
|
| Scaling law experiments comparing **Euclidean** (standard dot-product) vs **Hyperbolic** (Lorentz model) output layers for Qwen3 language models on OpenWebText. |
|
|
| ## Project |
|
|
| - **Repository**: [github.com/ObliviateRickLin/HyperScale](https://github.com/ObliviateRickLin/HyperScale) |
| - **Base model**: Qwen3 architecture (custom sizes, untied embeddings) |
| - **Dataset**: OpenWebText (8.39B tokens total) |
| - **Optimizer**: NanochatMuon (Muon for 2D transformer matrices + per-group AdamW) |
| - **Training**: DeepSpeed ZeRO-2, bf16 mixed precision, 4x H100 80GB |
|
|
| ## Key Differences |
|
|
| | | Euclidean | Hyperbolic | |
| |---|---|---| |
| | Output layer | Standard linear (dot-product logits) | Lorentz hyperboloid (Minkowski inner product logits) | |
| | lm_head init | zeros | std=0.02 | |
| | embed_tokens init | std=1.0 | std=1.0 | |
| | tie_word_embeddings | false | false | |
| | Logit computation | `hidden @ lm_head.T` (fp32) | `<expmap(h), expmap(w)>_L * scale` (fp32, mean-centered) | |
| | logit_scale | N/A | `d_model / sinh(1)` (learnable) | |
|
|
| ## Known Issues |
|
|
| **c_proj zero init missing**: Neither Euclidean nor Hyperbolic models zero-initialize `self_attn.o_proj` and `mlp.down_proj` (Qwen3 equivalent of Karpathy c_proj). In nanochat/HypGPT reference, these ARE zeroed alongside lm_head. This is a known confound. Euclidean (lm_head=zeros) is more impacted than Hyperbolic (lm_head=std=0.02). |
| |
| ## Results |
| |
| Token budget `t1_N` means `1/N` of the full 8.39B token dataset. **Delta < 0 means hyperbolic is better.** |
| |
| | Size | Params | Tokens | Hyp | Euc | Delta | |
| |---|---|---|---|---|---| |
| | p020m | 20M | 65.6M (t1_128) | 9.9145 | 10.0184 | -0.1039 | |
| | p020m | 20M | 131M (t1_64) | 8.4028 | 8.4610 | -0.0582 | |
| | p020m | 20M | 262M (t1_32) | 7.1117 | 7.4237 | -0.3120 | |
| | p020m | 20M | 524M (t1_16) | 6.0795 | 6.4958 | -0.4163 | |
| | p047m | 47M | 65.6M (t1_128) | 8.5956 | 8.6146 | -0.0190 | |
| | p047m | 47M | 262M (t1_32) | 6.0944 | 6.3668 | -0.2724 | |
| | p047m | 47M | 524M (t1_16) | 5.5383 | 5.7709 | -0.2326 | |
| | p109m | 109M | 131M (t1_64) | 6.1340 | 6.3509 | -0.2169 | |
| | p109m | 109M | 262M (t1_32) | 5.5677 | 5.7453 | -0.1776 | |
| | p109m | 109M | 524M (t1_16) | 5.3841 | 5.5431 | -0.1590 | |
| | p223m | 223M | 131M (t1_64) | 5.8612 | 6.0407 | -0.1795 | |
| | p407m | 407M | 65.6M (t1_128) | 6.8377 | 7.1486 | -0.3109 | |
| | p407m | 407M | 262M (t1_32) | 4.5280 | 5.1510 | -0.6230 | |
| | p407m | 407M | 524M (t1_16) | 4.4119 | 4.5080 | -0.0961 | |
| | p407m | 407M | 1.05B (t1_8) | 3.4614 | 3.9945 | -0.5331 | |
| | p407m | 407M | 2.10B (t1_4) | 3.5738 | 3.6206 | -0.0468 | |
| | p407m | 407M | 4.20B (t1_2) | 3.3230 | 3.3503 | -0.0273 | |
| | p407m | 407M | 8.39B (t1_1) | 3.1236 | -- | -- | |
| | p686m | 686M | 131M (t1_64) | 5.5675 | 5.7105 | -0.1430 | |
| | p686m | 686M | 262M (t1_32) | 4.9169 | 5.0321 | -0.1152 | |
| | p686m | 686M | 524M (t1_16) | 4.2684 | 4.3334 | -0.0650 | |
| | p686m | 686M | 1.05B (t1_8) | 3.7796 | 3.8389 | -0.0593 | |
| | p686m | 686M | 4.20B (t1_2) | 3.3219 | 3.2237 | +0.0982 | |
| | p686m | 686M | 65.6M (t1_128) | 11.9171* | 6.5685 | -- | |
| | p1083m | 1.08B | 65.6M (t1_128) | 6.9223 | 7.9453 | -1.0230 | |
| | p1083m | 1.08B | 262M (t1_32) | 4.4833 | 7.0858* | -- | |
| | p1083m | 1.08B | 524M (t1_16) | 4.1417 | 4.2060 | -0.0643 | |
| | p1083m | 1.08B | 1.05B (t1_8) | 3.3901 | 3.7304 | -0.3403 | |
| | p1083m | 1.08B | 4.20B (t1_2) | 3.1223 | 3.2614 | -0.1391 | |
| | p1083m | 1.08B | 8.39B (t1_1) | 2.9269 | -- | -- | |
| | p1621m | 1.62B | all | NaN | 3.30-6.54 | -- | |
| | p2324m | 2.32B | 262M (t1_32) | 4.6533 | 4.7401 | -0.0868 | |
| | p2324m | 2.32B | 524M (t1_16) | 4.0000 | 4.0594 | -0.0594 | |
| | p2324m | 2.32B | 4.20B (t1_2) | 3.0077 | 3.2524 | -0.2447 | |
| | p2324m | 2.32B | 8.39B (t1_1) | 3.1524 | -- | -- | |
| |
| \* Anomalous values due to training instability/divergence. p1621m hyp diverged entirely (all NaN). |
| |
| ## Summary |
| |
| - **Hyperbolic is consistently better** in most matched comparisons (lower eval loss) |
| - Average improvement: ~5-13% relative reduction in loss |
| - Improvement is larger at medium token budgets (t1_32, t1_8) and diminishes at high token budgets (t1_2, t1_1) |
| - **Caveat**: init difference (lm_head zeros vs std=0.02) confounds the comparison |
| - Hyperbolic models show instability at larger sizes (p686m+ with t1_128, p1621m diverges entirely) |
| |
| ## Repository Structure |
| |
| ``` |
| checkpoints/ |
| qwen3/ # Euclidean models |
| qwen3_p{SIZE}_t1_{N}_.../ |
| attempt{K}_{DATE}/ |
| checkpoint-{STEP}/ # Final checkpoint |
| qwen3_hyp/ # Hyperbolic models |
| qwen3_hyp_p{SIZE}_t1_{N}_.../ |
| attempt{K}_{DATE}/ |
| checkpoint-{STEP}/ |
| |
| results/ # Training logs (trainer_state.json) |
| scaling_law/owt/ |
| qwen3/owt_scaling_v3/ |
| qwen3_hyp/owt_scaling_v3/ |
| ``` |
| |
| ## Experiment Configuration |
| |
| | Parameter | Value | |
| |---|---| |
| | Architecture | Qwen3 (custom sizes) | |
| | Vocab size | 151,936 | |
| | Context length | 1,024 | |
| | Dataset | OpenWebText (8.39B tokens) | |
| | Optimizer | NanochatMuon (Muon + per-group AdamW) | |
| | Muon targets | 2D transformer matrices | |
| | AdamW groups | embed (lr=0.2*scale), lm_head (lr=0.004*scale), misc (lr=0.004*scale) | |
| | LR scaling | (d_model/768)^(-0.5) | |
| | Precision | bf16 mixed precision | |
| | Infrastructure | 4x NVIDIA H100 80GB, DeepSpeed ZeRO-2 | |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{hyperscale2026, |
| title={HyperScale: Scaling Laws for Hyperbolic Output Layers in Language Models}, |
| author={Jinrui Lin}, |
| year={2026}, |
| url={https://github.com/ObliviateRickLin/HyperScale} |
| } |
| ``` |
| |
