# MoE pretraining run

This repository is updated by a Kaggle notebook that resumes from the last saved state.

## Status
- run_index: `0`
- source_cursor: `0`
- global_steps: `0`
- completed_runs: `0`
- finished_window: `False`

## Window policy
- rows selected this run: `10000`
- raw rows scanned this run: `25000`
- after each completed window, the notebook saves the full state and continues from the next raw cursor on the next trigger

## Model shape
- layers: `6`
- d_model: `512`
- heads: `8`
- experts: `100`
- active experts per token: `12`
- expert hidden size: `768`

## Recent stats
- phase: `bootstrap`

## Notes
- tokenizer: custom byte-level BPE
- resume: optimizer, scheduler, RNG, config, tokenizer, model weights, and cursor state are all saved