# TASK — 100M-active MoE, from scratch, physics-sim next-frame prediction, custom minimal vocab

Train a Qwen3-style sparse-MoE LM **from scratch** on the physics-simulation
next-frame-prediction corpus, using a **custom minimal tokenizer** whose vocab
contains only the tokens needed to emit the simulation text (digits, punctuation,
structural keywords). Target ~100M active params.

## Scaffold
- Model/trainer: github.com/AlexWortega/moe-200m-qwen3-100b- (Qwen3-MoE: GQA + partial RoPE
  + QK-Norm + RMSNorm, aux-loss-free sigmoid bias router, 1 shared + N routed top-2 SwiGLU
  experts, tied embed/lm_head, Liger fused-CE, Muon optimizer). `MoEModelConfig` in model.py.
- Sibling 100M config exists: moe-100m-volta-week (good sizing reference).

## Data (HF Hub, from the physics-llm project)
- AlexWortega/physics-scenarios-raw, AlexWortega/physics-scenarios-packed (~900K scenes,
  30 types, 24 train / 6 held-out). Format = the LFM2 serialization (Scene/Gravity/Frame/obj_...).

## Key requirement — custom vocab
Vocab = ONLY simulation tokens (tens–low-hundreds). With tied embeddings, shrinking vocab from
151,936 → ~100 frees ~97M embedding params, so the whole ~100M budget goes to the MoE/dynamics
(vs the 350M LFM2 whose huge vocab embeddings ate the budget). Drop free-text Scene/Frame
descriptions (not needed for physics); keep Type as a categorical token.

## Success metric
Pymunk position error as % of scene diagonal (same as LFM2 baseline), via the existing harness
at /Users/aleksandrnikolich/Desktop/vae_llm/physics_blog/bench (physics_core.rollout). Baseline
to beat: LFM2-350M bf16 — @15f trained 0.38% / held-out 0.93%; orbit 0.75% @80f.

## High-impact unknowns -> Clarify
- experiment budget (GPU-h / wall-clock)  - GPU choice (eva02 A6000 vs eva01 4xV100)
- tokenizer/number encoding (char-level vs tiny-BPE)  [genuine 2-path fork]