| # TASK — 100M-active MoE, from scratch, physics-sim next-frame prediction, custom minimal vocab |
|
|
| Train a Qwen3-style sparse-MoE LM **from scratch** on the physics-simulation |
| next-frame-prediction corpus, using a **custom minimal tokenizer** whose vocab |
| contains only the tokens needed to emit the simulation text (digits, punctuation, |
| structural keywords). Target ~100M active params. |
|
|
| ## Scaffold |
| - Model/trainer: github.com/AlexWortega/moe-200m-qwen3-100b- (Qwen3-MoE: GQA + partial RoPE |
| + QK-Norm + RMSNorm, aux-loss-free sigmoid bias router, 1 shared + N routed top-2 SwiGLU |
| experts, tied embed/lm_head, Liger fused-CE, Muon optimizer). `MoEModelConfig` in model.py. |
| - Sibling 100M config exists: moe-100m-volta-week (good sizing reference). |
| |
| ## Data (HF Hub, from the physics-llm project) |
| - AlexWortega/physics-scenarios-raw, AlexWortega/physics-scenarios-packed (~900K scenes, |
| 30 types, 24 train / 6 held-out). Format = the LFM2 serialization (Scene/Gravity/Frame/obj_...). |
|
|
| ## Key requirement — custom vocab |
| Vocab = ONLY simulation tokens (tens–low-hundreds). With tied embeddings, shrinking vocab from |
| 151,936 → ~100 frees ~97M embedding params, so the whole ~100M budget goes to the MoE/dynamics |
| (vs the 350M LFM2 whose huge vocab embeddings ate the budget). Drop free-text Scene/Frame |
| descriptions (not needed for physics); keep Type as a categorical token. |
|
|
| ## Success metric |
| Pymunk position error as % of scene diagonal (same as LFM2 baseline), via the existing harness |
| at /Users/aleksandrnikolich/Desktop/vae_llm/physics_blog/bench (physics_core.rollout). Baseline |
| to beat: LFM2-350M bf16 — @15f trained 0.38% / held-out 0.93%; orbit 0.75% @80f. |
| |
| ## High-impact unknowns -> Clarify |
| - experiment budget (GPU-h / wall-clock) - GPU choice (eva02 A6000 vs eva01 4xV100) |
| - tokenizer/number encoding (char-level vs tiny-BPE) [genuine 2-path fork] |
| |
