# MoS-8B-Checkpoints

Checkpoints of the Qwen3-8B MoS DFlash draft models (clean-800K E5 follow-up arms, 8B self-generated corpus `dataclean-8b-20260912`) and the warm-start inputs they were trained from. Project: MoS (mixture of layer-local expert MLPs added to a shared MLP in each DFlash draft layer).

## Layout

```
e5_8b_800k_20260831/arms/<arm>/checkpoints/iter_NNNNNNN/hf/      portable export: config.json, model.safetensors, export_manifest.json (serving weights)
e5_8b_800k_20260831/arms/<arm>/checkpoints/iter_NNNNNNN/         full training checkpoint (DCP model/, optimizer/, lr_scheduler/, rng_rank_*.pt, meta.json) where listed below
inputs/<name>/                                                   warm-start initializations (config.json, model.safetensors)
```

Milestones: 13,325 steps = 1 epoch (E1); 26,650 E2; 39,975 E3; 53,300 E4; 66,625 E5; 6,662 / 19,986 / 33,310 / 46,634 / 59,958 are half-epoch saves.

## Contents (2026-09-30)

| Arm | Recipe | Archived |
|---|---|---|
| `e5-x4rand-mos` | K4 random-init experts, expert LR 6e-4 (rerun from 2026-09-21) | `hf` at 6,662, 26,650, 33,310, 39,975, 46,634, 53,300, 59,958, 66,625. E1 (13,325) and 19,986 were not archived and are lost |
| `e5-x4rand-mos-r2` | continuation of the first x4rand run from its E1 weights, Adam reset (stopped, no reference value) | `hf` at 13,325, 19,986, 26,650 |
| `e5-x8rand-mos` | K8 random-init experts, expert LR 6e-4 | `hf` at 46,634, 53,300, 59,958, 66,625 |
| `e5-x16rand-mos` | K16 random-init experts, expert LR 6e-4, expert parallelism | `hf` at every save 6,662 … 66,625 |
| `e5-x16lr4-mos` | K16 warm-start jitter init, expert LR ×4, expert parallelism (rerun paused at step 18,478) | `hf` at 6,662 and 13,325; full checkpoint (with optimizer) at 13,325 = resume point |
| `e5-x4rand-lr1e3-mos` | ablation: x4rand with expert LR 1e-3, trained to E1 on the 5-epoch schedule | `hf` at 6,662; full checkpoint (with optimizer) at 13,325 |

| Input | Content |
|---|---|
| `inputs/d0-shared-random-experts-8b-x4` | x4rand init: D0 shared MLP/attention + 4 random experts (seed 20260917) |
| `inputs/d0-shared-random-experts-8b-x4-half` | half-width ablation init: the x4 init with each expert cut to its first 6,144 hidden units |
| `inputs/d0-shared-random-experts-8b-x16` | x16rand init: D0 + 16 random experts (seed 20260919) |
| `inputs/d0-fullwidth-init-8b-jitter` | K4 warm-start init (experts = D0 MLP + 1% jitter); also the source from which the random-expert inits are rebuilt |
| `inputs/d0-fullwidth-init-8b-x16-jitter` | K16 warm-start init used by `e5-x16lr4-mos` |

Results, configs and logs: GitHub `MoS` repository, `experiments/aurora/qwen3-8b-800k-e5/bal01/README.md` (§10). Archive records (file counts and byte totals, verified against the local copies before deletion): `experiments/aurora/qwen3-8b-800k-e5/archive/`.