| --- |
| license: apache-2.0 |
| tags: |
| - autonomous-driving |
| - alpamayo |
| - waymo |
| - long-tail |
| --- |
| # Alpamayo R1 + Waymo Fine-tuning — 3-recipe ablation |
|
|
| 3 SFT recipes finetuned on Alpamayo R1 base, each going through Stage 1 |
| (VLM finetune, token CE) → Stage 2 (diffusion expert SFT, flow matching). |
| All 3 epochs, 8× H200, DeepSpeed ZeRO-2. |
|
|
| All recipes use the same camera scheme aligned to NVIDIA's 4-cam slots |
| [0, 1, 2, 6] = (cross_left, front_wide, cross_right, front_tele). |
| Waymo data is re-mapped to these slots; front_tele is emulated by |
| center-cropping Waymo FRONT (~30°/50.4°FOV ≈ 0.595 crop ratio). |
| |
| ## Recipes |
| |
| | Recipe | NV component | LT component | Waymo component | |
| |---|---|---|---| |
| | **R-A** | 10k random | 1450 GT (real) | Waymo GT (real, 798 clips) | |
| | **R-B** | 10k random | 1450 Ours-synth | Waymo Ours-synth (Ray-WAN, 800 clips) | |
| | **R-C** | 10k random | 1450 GT (real) | Waymo Ours-synth (Ray-WAN, 800 clips) | |
| |
| ## Final losses (3 epochs each) |
| |
| | Recipe | Stage 1 (token CE) | Stage 2 (flow matching) | |
| |---|---|---| |
| | R-A | 1.277 | 0.385 | |
| | R-B | 1.281 | 0.403 | |
| | R-C | 1.279 | 0.400 | |
| |
| ## Files |
| |
| Per recipe (e.g. `R-A_s2/`): |
| - `model-*-of-*.safetensors`: model weights (~21 GB total) |
| - `model.safetensors.index.json`: shard index |
| - `config.json`: Alpamayo R1 model config |
| - `trainer_state.json`: full training history (loss curve, lr schedule) |
| - `training_args.bin`, `scheduler.pt`: trainer state |
|
|
| Optimizer states are NOT included (only useful for training resume). |
|
|
| ## Loading |
|
|
| ```python |
| from alpamayo_r1.models.alpamayo_r1 import AlpamayoR1 |
| model = AlpamayoR1.from_pretrained("luuuulinnnn/alpamyo_Waymo", subfolder="R-A_s2") |
| ``` |
|
|
| ## Training data sources |
|
|
| - **10k random NV**: subset of nominal NVIDIA PAI dataset (4-cam standard). |
| - **1450 GT**: out-of-distribution train bucket from the 10k+OOD subset. |
| - **1450 Ours-synth**: 603 of the 1450 NV clips replaced by Ray-WAN synthesised views. |
| - **Waymo GT**: 798 Waymo training clips, FRONT_LEFT/FRONT/FRONT_RIGHT |
| resampled to NV slots [0, 1, 2], with FRONT center-cropped to slot 6 (tele). |
| - **Waymo Ours-synth**: 800 clips re-encoded through Ray-WAN-on-Waymo |
| (`waymo_p65v1_arr_185143`), trajectory poses from the original GT clips. |
|
|
| ## v2 ckpts — Waymo regen (Variant A + Qwen3 per-clip caption) |
|
|
| Re-ran the Waymo synth pipeline with two improvements (`waymo_p65v2_va_190308`): |
| - **Variant A**: source-view K = original Waymo intrinsics (target views still PAI K), |
| fixes the camera-pitch artifact that made the v1 synth look like it was tilted down. |
| - **Qwen3-VL-8B-Instruct per-clip caption**: each clip gets its own caption from |
| Qwen3-VL rather than borrowing a PAI prompt that mentions unrelated objects. |
|
|
| Then re-trained R-B and R-C on this new synth (3 epochs each, identical schedule). |
|
|
| | Recipe | Stage 1 (token CE) | Stage 2 (flow matching) | Subfolder | |
| |---|---|---|---| |
| | R-B v2 | 1.282 | 0.402 | `R-B_v2_s2/` | |
| | R-C v2 | 1.280 | 0.398 | `R-C_v2_s2/` | |
|
|
| Loading v2: |
| ```python |
| model = AlpamayoR1.from_pretrained("luuuulinnnn/alpamyo_Waymo", subfolder="R-B_v2_s2") |
| ``` |
|
|
| R-A is unchanged (GT-only — no synth dependency). |
|
|
