| # temporal-moe-extras |
|
|
| Supporting artifacts for the Temporal-MoE experiments, 248 files, 39.78 GiB. These are things |
| that are expensive to regenerate but are not themselves training checkpoints. |
|
|
| ## Layout |
|
|
| ``` |
| olmoe_adapt/ |
| ckpt_bake_{CE,G,H,Er8,Er64,D}.pt full optimizer and training state behind the router bakes |
| finetune_ids.pt tokenized finetune corpus token ids |
| bpb_slice_ids.pt bits-per-byte evaluation slice |
| cal2_gprime.pt calibration tensors |
| oseries/, oseries_ce/ paired evaluation slices, val.f16, idx.u8, paired masks |
| logs/ bake, sweep, calibration and evaluation logs |
| merged_ce_model/ OLMoE backbone merged with the bake-CE router, loadable with |
| transformers, includes config and tokenizer |
| run_captures/<run_name>/ |
| router_log.pt per-run router traces, back the mechanistic interpretability |
| and residency result tables |
| delex_capture.pt delexicalization captures |
| ablations/ 101 result tables as CSV, plus FINDINGS.md and README.md |
| figures/ 43 figures |
| tokenizer_tok16k/ the 16k tokenizer, also mirrored in the corpus repository |
| ``` |
|
|
| ## Notes |
|
|
| `merged_ce_model/` is a derived artifact. It is included rather than regenerated because the merge is |
| not guaranteed to be bit-reproducible. |
|
|
| `ablations/` holds the result tables the paper draws on. `FINDINGS.md` and `README.md` in that |
| directory are the written interpretation of those tables. |
|
|
| The base OLMoE model is not mirrored here. It is public at |
| [`allenai/OLMoE-1B-7B-0924`](https://huggingface.co/allenai/OLMoE-1B-7B-0924). |
|
|
| ## Training configuration |
|
|
| Every run directory contains `run.meta` and `train.log`. `run.meta` records the full knob set for |
| that run in two lines, for example: |
|
|
| ``` |
| [run] temporal_fine_g3_1e19 N=185766400 iters=17112 warmup=856 min_lr=0.0003 eval@1711 |
| [run] shape=s19opt H=800 L=14 ffn=4272 moe_ffn=184 grain=3 num_experts=192 shared_int=1100 |
| dense=0 temporal=1 shared_mult=2 topk=18 heads=50 gb=256 mb=8 lr=3e-3 flops=1e19 |
| ``` |
|
|
| `train.log` is the full Megatron stdout, including the complete argument namespace dumped at startup. |
| Read these before drawing any conclusion from a checkpoint. |
|
|
| ## MANIFEST.csv and the `cited` column |
|
|
| A manifest covering every file in all four repositories lives in the code repository. It has seven |
| columns: `local_path`, `hf_repo`, `hf_path`, `bytes`, `sha256`, `run_name`, `cited`. |
|
|
| The `cited` column marks whether a run is referenced by a results table in `results/ablations/*.csv` |
| or by the paper: |
|
|
| - `cited`, the run backs a published number. There are 58 of these. |
| - `uncited`, the run is infrastructure validation, a smoke test, a throughput probe, or an aborted |
| run. It is kept for completeness and reproducibility, not because a table depends on it. There are |
| 13 of these. |
| - empty, the file is not scoped to a single run, for example a batch log or an evaluation output. |
|
|
| Every `sha256` in the manifest was computed on this disk before upload and each file was verified to |
| exist remotely with a matching byte size. |
|
|
| ## Links |
|
|
| - Code: <https://github.com/ncylich/temporal-moe> |
| - Paper: Temporal-MoE (short paper), see the `paper/` directory in the code repository |
| - Upstream platform this work forks: [FLAME-MoE](https://github.com/cmu-flame/FLAME-MoE), [arXiv:2505.20225](https://arxiv.org/abs/2505.20225) |
|
|
| ## Companion repositories |
|
|
| - [`ncylich/temporal-moe-ckpts`](https://huggingface.co/ncylich/temporal-moe-ckpts), Megatron training checkpoints |
| - [`ncylich/temporal-moe-router-adapt`](https://huggingface.co/ncylich/temporal-moe-router-adapt), router adaptation safetensors |
| - [`ncylich/temporal-moe-extras`](https://huggingface.co/ncylich/temporal-moe-extras), captures, merged model, result tables, figures |
| - [`ncylich/temporal-moe-corpus`](https://huggingface.co/datasets/ncylich/temporal-moe-corpus), tokenized training corpus |
|
|
| ## Provenance |
|
|
| Trained with a personal fork of FLAME-MoE. Not affiliated with or endorsed by the FLAME-MoE authors. |
|
|
