Extrapolation of 760 achievable steps/hour findins to 75k-steps based on what?
Series prerequisite — the horizon fix: every earlier run had max_steps = -1, giving schedulers a 75,210-step (full-epoch) horizon, so no decay ever actually happened inside a 1-hour run. base_train.py now accepts max_steps from config and all runs here pin it to 760 (the achievable steps/hour), so each schedule genuinely completes.
Do you plan to use 760 steps best params for the futer ~75k steps?
If yes, how to know the discovered best params will work as optimal, rather then suboptimal?
cf
We used AdamW for all experiments. In initial experiments, we used Muon, but found that this led to undertraining as we scaled compute.
https://arxiv.org/html/2601.21571
I messed up some settings while learning how to tweak this, but I think I landed on the best setup so far, so I'm building a Llama-77M right now, trained on the Sprocket-n-Say dataset.
If all goes well with this, I hope I'll tweak the next LLM better :)