| # MoE pretraining run |
|
|
| This repository is updated by a Kaggle notebook that resumes from the last saved state. |
|
|
| ## Status |
| - run_index: `0` |
| - source_cursor: `0` |
| - global_steps: `0` |
| - completed_runs: `0` |
| - finished_window: `False` |
| |
| ## Window policy |
| - rows selected this run: `10000` |
| - raw rows scanned this run: `25000` |
| - after each completed window, the notebook saves the full state and continues from the next raw cursor on the next trigger |
| |
| ## Model shape |
| - layers: `6` |
| - d_model: `512` |
| - heads: `8` |
| - experts: `100` |
| - active experts per token: `12` |
| - expert hidden size: `768` |
|
|
| ## Recent stats |
| - phase: `bootstrap` |
|
|
| ## Notes |
| - tokenizer: custom byte-level BPE |
| - resume: optimizer, scheduler, RNG, config, tokenizer, model weights, and cursor state are all saved |
