File size: 4,213 Bytes
7481cba | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | # temporal-moe-extras
Supporting artifacts for the Temporal-MoE experiments, 248 files, 39.78 GiB. These are things
that are expensive to regenerate but are not themselves training checkpoints.
## Layout
```
olmoe_adapt/
ckpt_bake_{CE,G,H,Er8,Er64,D}.pt full optimizer and training state behind the router bakes
finetune_ids.pt tokenized finetune corpus token ids
bpb_slice_ids.pt bits-per-byte evaluation slice
cal2_gprime.pt calibration tensors
oseries/, oseries_ce/ paired evaluation slices, val.f16, idx.u8, paired masks
logs/ bake, sweep, calibration and evaluation logs
merged_ce_model/ OLMoE backbone merged with the bake-CE router, loadable with
transformers, includes config and tokenizer
run_captures/<run_name>/
router_log.pt per-run router traces, back the mechanistic interpretability
and residency result tables
delex_capture.pt delexicalization captures
ablations/ 101 result tables as CSV, plus FINDINGS.md and README.md
figures/ 43 figures
tokenizer_tok16k/ the 16k tokenizer, also mirrored in the corpus repository
```
## Notes
`merged_ce_model/` is a derived artifact. It is included rather than regenerated because the merge is
not guaranteed to be bit-reproducible.
`ablations/` holds the result tables the paper draws on. `FINDINGS.md` and `README.md` in that
directory are the written interpretation of those tables.
The base OLMoE model is not mirrored here. It is public at
[`allenai/OLMoE-1B-7B-0924`](https://huggingface.co/allenai/OLMoE-1B-7B-0924).
## Training configuration
Every run directory contains `run.meta` and `train.log`. `run.meta` records the full knob set for
that run in two lines, for example:
```
[run] temporal_fine_g3_1e19 N=185766400 iters=17112 warmup=856 min_lr=0.0003 eval@1711
[run] shape=s19opt H=800 L=14 ffn=4272 moe_ffn=184 grain=3 num_experts=192 shared_int=1100
dense=0 temporal=1 shared_mult=2 topk=18 heads=50 gb=256 mb=8 lr=3e-3 flops=1e19
```
`train.log` is the full Megatron stdout, including the complete argument namespace dumped at startup.
Read these before drawing any conclusion from a checkpoint.
## MANIFEST.csv and the `cited` column
A manifest covering every file in all four repositories lives in the code repository. It has seven
columns: `local_path`, `hf_repo`, `hf_path`, `bytes`, `sha256`, `run_name`, `cited`.
The `cited` column marks whether a run is referenced by a results table in `results/ablations/*.csv`
or by the paper:
- `cited`, the run backs a published number. There are 58 of these.
- `uncited`, the run is infrastructure validation, a smoke test, a throughput probe, or an aborted
run. It is kept for completeness and reproducibility, not because a table depends on it. There are
13 of these.
- empty, the file is not scoped to a single run, for example a batch log or an evaluation output.
Every `sha256` in the manifest was computed on this disk before upload and each file was verified to
exist remotely with a matching byte size.
## Links
- Code: <https://github.com/ncylich/temporal-moe>
- Paper: Temporal-MoE (short paper), see the `paper/` directory in the code repository
- Upstream platform this work forks: [FLAME-MoE](https://github.com/cmu-flame/FLAME-MoE), [arXiv:2505.20225](https://arxiv.org/abs/2505.20225)
## Companion repositories
- [`ncylich/temporal-moe-ckpts`](https://huggingface.co/ncylich/temporal-moe-ckpts), Megatron training checkpoints
- [`ncylich/temporal-moe-router-adapt`](https://huggingface.co/ncylich/temporal-moe-router-adapt), router adaptation safetensors
- [`ncylich/temporal-moe-extras`](https://huggingface.co/ncylich/temporal-moe-extras), captures, merged model, result tables, figures
- [`ncylich/temporal-moe-corpus`](https://huggingface.co/datasets/ncylich/temporal-moe-corpus), tokenized training corpus
## Provenance
Trained with a personal fork of FLAME-MoE. Not affiliated with or endorsed by the FLAME-MoE authors.
|