28.3 kB

Ctrl+K

v1.5 pivot: 152M (18L x 768d) hero config, ReLU2 FFN, final logit soft-cap. 350M kept as reference.

7a66951 verified about 1 month ago

test_ablate.py

1.36 kB
v2: 363M hero run (Muon hybrid, WSD, Liger, SmolLM 75/15/10 mix) about 1 month ago
test_checkpoint.py

3.52 kB
v2: 363M hero run (Muon hybrid, WSD, Liger, SmolLM 75/15/10 mix) about 1 month ago
test_data.py

3.06 kB
v2: 363M hero run (Muon hybrid, WSD, Liger, SmolLM 75/15/10 mix) about 1 month ago
test_model.py

11.1 kB
v1.5 pivot: 152M (18L x 768d) hero config, ReLU2 FFN, final logit soft-cap. 350M kept as reference. about 1 month ago
test_optim.py

4.69 kB
v2: 363M hero run (Muon hybrid, WSD, Liger, SmolLM 75/15/10 mix) about 1 month ago
test_run.py

1.84 kB
v2: 363M hero run (Muon hybrid, WSD, Liger, SmolLM 75/15/10 mix) about 1 month ago
test_train.py

2.71 kB
v2: 363M hero run (Muon hybrid, WSD, Liger, SmolLM 75/15/10 mix) about 1 month ago