| --- |
| tags: |
| - robotics |
| - video-generation |
| - vision-language-action |
| - fruit-picking |
| library_name: pytorch |
| --- |
| |
| # Fruit-picking Fastwam |
|
|
| Public archival bundle for the fruit-picking model trained by Amin. |
|
|
| ## Status |
|
|
| This is a **partial checkpoint: step 12,600 of 18,900 (epoch 20 of 30)**. The job terminated when its shared `/dev/shm` |
| video staging disappeared; it did not finish the planned 30 epochs. The model |
| has not been evaluated on a robot, and the physical meaning of the `-1/+1` |
| gripper polarity still needs confirmation. |
|
|
| Architecture: FastWAM full 30-layer video + 30-layer action MoT. |
|
|
| Checkpoints follow the same layout as the existing WAM repositories: |
|
|
| - `checkpoints/weights/step_003150.pt` — SHA-256 `5fcd3ccfa8d6307a24dedfc9f8aadbec69e698a5a4218ad94f275914474dc9f9` |
| - `checkpoints/weights/step_006300.pt` — SHA-256 `eb3dca41eec86722bcfe5776bb46738417373d17cc11da1e65f4c5b4be34b0ab` |
| - `checkpoints/weights/step_009450.pt` — SHA-256 `f6c54e322ee97171177d68c671254faa1e31045d803f0943e60549a1fd5e13ff` |
| - `checkpoints/weights/step_012600.pt` — SHA-256 `9fc1147a649aeff4139d00fe70adc07f52ff46e7ffc6dc1b379e29d0aa18ed85` |
|
|
| ## Conditioning |
|
|
| Exact task text: |
|
|
| > Lift the lid, put it aside, and pick the black plum. |
|
|
| `conditioning/text_embedding.pt` is the exact cached T5 embedding consumed |
| during training. It was generated with the Wan text stack, context length 128, |
| using `Wan-AI/Wan2.1-T2V-1.3B` as the tokenizer model reference. The resolved |
| training config sets `load_text_encoder: false`, so this cached tensor is part |
| of the required inference bundle. |
|
|
| ## Input processing and normalization |
|
|
| - Two 256x256 RGB cameras (`agentview`, then `wrist`). |
| - Each camera is converted to a tensor and resized to 224x224. |
| - Cameras are concatenated horizontally to 224x448. |
| - Horizon: 33 observations; 32 action transitions at 10 Hz. |
| - Original 15-D state was converted to 8-D: |
| `eef_xyz(3) + quat-to-axis-angle(3) + [gripper_width/2, -gripper_width/2]`. |
| - Action is 7-D: delta XYZ, delta rotation XYZ, and gripper. |
| - Delta/padding mask is `[true, true, true, true, true, true, false]`; the |
| gripper channel is absolute rather than delta. |
| - `dataset_stats.json` contains the exact min/max normalization statistics |
| used by this run. |
|
|
| ## Attention masks |
|
|
| The resolved model uses: |
|
|
| - `video_attention_mask_mode: first_frame_causal`: first-frame queries cannot |
| attend to later video frames; later-frame queries can attend to all video |
| tokens. |
| - `action_group_causal_mask_mode: group_diagonal`: each video temporal group |
| attends only to the corresponding action-token group. |
| - Text cross-attention is enabled for the action expert. |
|
|
| The exact implementations and preprocessing classes are included under |
| `training_code/`; the resolved config is `config.yaml`. |
| No license is asserted here for the bundled upstream code; its original terms |
| continue to apply. |
|
|
| ## Base components |
|
|
| This weights-only checkpoint is not standalone. It references |
| `Wan-AI/Wan2.2-TI2V-5B` and requires the matching Wan VAE plus the included |
| FastWAM code/configuration. PyTorch `.pt` files may contain pickled objects; |
| load only in a trusted environment. |
|
|
