Optimizer Work
Collection
Nero MLX optimizer experiments trained for 500M tokens at context length 128. • 50 items • Updated
Nero Optimizer Work — SimOW
This repository contains the final MLX checkpoint for the SimOW arm of the Nero Optimizer Work research sweep.
Summary
- Optimizer: simow
- Requested learning rate: the run's configured default/controller schedule
- Training budget: 500,000,000 tokens
- Context length: 128 tokens
- Backend: Apple MLX
- Model family: matched dense-deep decoder, 2,048-token vocabulary, 128-wide residual stream, 6 blocks, 32-dimensional attention heads, 148-wide gated MLP
- Stored model parameters: approximately 999,680
- Final training loss: 5.371432
- Final logged throughput: 418,988 tokens/s
- Tail throughput: 419,229 tokens/s (median of the final logged samples)
- Final tokens seen: 500,000,000
Files
- model.npz — final MLX model weights from checkpoint_000500000000
- state.json — checkpoint state metadata
- run.json — frozen run configuration
- metrics.jsonl — complete training metrics log
- config.json — model and publication metadata
Evaluation
The figures above are training-run measurements. An independent held-out validation artifact was not saved with these runs, so this card does not claim a validation score. Compare checkpoints using the same frozen evaluation pass before drawing quality conclusions.
Intended use and limitations
This is an experimental research checkpoint, not an instruction-tuned or production-ready language model. It is published to make the optimizer comparison reproducible. The raw MLX weights require a compatible local MLX loader and are not a Transformers checkpoint.
Reproducibility
All arms use the same prepared finephrase-balanced-500m-2k-v2 token stream, 128-token context, 32-example batches, and 500M-token target. The full local training log is included in metrics.jsonl.
License
No new model license is asserted by this experimental publication. Review the source-data terms before redistribution or downstream use.
- Downloads last month
- -