| --- |
| tags: |
| - robotics |
| - video-generation |
| - vision-language-action |
| - fruit-picking |
| library_name: pytorch |
| --- |
| |
| # Fruit-picking Flashwam |
|
|
| Public archival bundle for the fruit-picking model trained by Amin. |
|
|
| ## Status |
|
|
| This is a **partial checkpoint: step 12,600 of 18,900 (epoch 20 of 30)**. The job terminated when its shared `/dev/shm` |
| video staging disappeared; it did not finish the planned 30 epochs. The model |
| has not been evaluated on a robot, and the physical meaning of the `-1/+1` |
| gripper polarity still needs confirmation. |
|
|
| Architecture: FlashWAM M1 fused-KV, fixed-RoPE, 1-layer action expert. |
|
|
| Checkpoints follow the same layout as the existing WAM repositories: |
|
|
| - `checkpoints/weights/step_003150.pt` — SHA-256 `2dc3d4ce58a592944975fc6aa2214f3f22cf724b08fa3815588d4d83319e31b0` |
| - `checkpoints/weights/step_006300.pt` — SHA-256 `660268a99a6e298ad5e3948f4784e7d8be6b4e0ad0517d9404b52c1dc7797145` |
| - `checkpoints/weights/step_009450.pt` — SHA-256 `dc0497a96a8c2c7bcf072d00e700302707f260bf6be0c7fa77567520f101ca01` |
| - `checkpoints/weights/step_012600.pt` — SHA-256 `96c9587141286f2bef8f7c38bc31816e5f9ce689f3c66500b5fdf1d6f3b83a06` |
|
|
| ## Conditioning |
|
|
| Exact task text: |
|
|
| > Lift the lid, put it aside, and pick the black plum. |
|
|
| `conditioning/text_embedding.pt` is the exact cached T5 embedding consumed |
| during training. It was generated with the Wan text stack, context length 128, |
| using `Wan-AI/Wan2.1-T2V-1.3B` as the tokenizer model reference. The resolved |
| training config sets `load_text_encoder: false`, so this cached tensor is part |
| of the required inference bundle. |
|
|
| ## Input processing and normalization |
|
|
| - Two 256x256 RGB cameras (`agentview`, then `wrist`). |
| - Each camera is converted to a tensor and resized to 224x224. |
| - Cameras are concatenated horizontally to 224x448. |
| - Horizon: 33 observations; 32 action transitions at 10 Hz. |
| - Original 15-D state was converted to 8-D: |
| `eef_xyz(3) + quat-to-axis-angle(3) + [gripper_width/2, -gripper_width/2]`. |
| - Action is 7-D: delta XYZ, delta rotation XYZ, and gripper. |
| - Delta/padding mask is `[true, true, true, true, true, true, false]`; the |
| gripper channel is absolute rather than delta. |
| - `dataset_stats.json` contains the exact min/max normalization statistics |
| used by this run. |
|
|
| ## Attention masks |
|
|
| The resolved model uses: |
|
|
| - `video_attention_mask_mode: first_frame_causal`: first-frame queries cannot |
| attend to later video frames; later-frame queries can attend to all video |
| tokens. |
| - `action_group_causal_mask_mode: group_diagonal`: each video temporal group |
| attends only to the corresponding action-token group. |
| - Text cross-attention is enabled for the action expert. |
|
|
| The exact implementations and preprocessing classes are included under |
| `training_code/`; the resolved config is `config.yaml`. |
| No license is asserted here for the bundled upstream code; its original terms |
| continue to apply. |
|
|
| ## Base components |
|
|
| This weights-only checkpoint is not standalone. It references |
| `Wan-AI/Wan2.2-TI2V-5B` and requires the matching Wan VAE plus the included |
| FastWAM code/configuration. PyTorch `.pt` files may contain pickled objects; |
| load only in a trusted environment. |
|
|
