TASK — 100M-active MoE, from scratch, physics-sim next-frame prediction, custom minimal vocab

Train a Qwen3-style sparse-MoE LM from scratch on the physics-simulation next-frame-prediction corpus, using a custom minimal tokenizer whose vocab contains only the tokens needed to emit the simulation text (digits, punctuation, structural keywords). Target ~100M active params.

Scaffold

Model/trainer: github.com/AlexWortega/moe-200m-qwen3-100b- (Qwen3-MoE: GQA + partial RoPE
- QK-Norm + RMSNorm, aux-loss-free sigmoid bias router, 1 shared + N routed top-2 SwiGLU experts, tied embed/lm_head, Liger fused-CE, Muon optimizer). MoEModelConfig in model.py.
Sibling 100M config exists: moe-100m-volta-week (good sizing reference).

Data (HF Hub, from the physics-llm project)

AlexWortega/physics-scenarios-raw, AlexWortega/physics-scenarios-packed (~900K scenes, 30 types, 24 train / 6 held-out). Format = the LFM2 serialization (Scene/Gravity/Frame/obj_...).

Key requirement — custom vocab

Vocab = ONLY simulation tokens (tens–low-hundreds). With tied embeddings, shrinking vocab from 151,936 → ~100 frees ~97M embedding params, so the whole ~100M budget goes to the MoE/dynamics (vs the 350M LFM2 whose huge vocab embeddings ate the budget). Drop free-text Scene/Frame descriptions (not needed for physics); keep Type as a categorical token.

Success metric

Pymunk position error as % of scene diagonal (same as LFM2 baseline), via the existing harness at /Users/aleksandrnikolich/Desktop/vae_llm/physics_blog/bench (physics_core.rollout). Baseline to beat: LFM2-350M bf16 — @15f trained 0.38% / held-out 0.93%; orbit 0.75% @80f.

High-impact unknowns -> Clarify

experiment budget (GPU-h / wall-clock) - GPU choice (eva02 A6000 vs eva01 4xV100)
tokenizer/number encoding (char-level vs tiny-BPE) [genuine 2-path fork]