Upload folder using huggingface_hub

Cosmos-Policy-ALOHA-Predict2-2B.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ffd6c761f11a1deb60c87ff185fc93684c24521126d6f635a091cd567a7c79a1
+size 3913017345

README.md

@@ -0,0 +1,128 @@
+# Cosmos-Policy-ALOHA-Predict2-2B
+
+## Model Description
+
+Cosmos-Policy-ALOHA-Predict2-2B is a bimanual robot manipulation policy fine-tuned from the [NVIDIA Cosmos-Predict2-2B-Video2World](https://huggingface.co/nvidia/Cosmos-Predict2-2B-Video2World) video foundation model (checkpoint: `model-480p-16fps.pt`). This model is trained on real-world human teleoperation data collected on the ALOHA 2 robot platform and achieves a 93.6% average completion rate across four challenging bimanual manipulation tasks.
+
+**Paper**: *Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning*
+
+### Key Features
+
+- **Single-stage fine-tuning**: Adapted from pretrained video model with no architectural modifications
+- **Multimodal outputs**: Jointly predicts actions, future states, and values through unified video diffusion
+- **Real-world performance**: 93.6% average score on challenging bimanual manipulation tasks
+
+### Model Architecture
+
+This model uses the same architecture as the base Cosmos-Predict2-2B model (a diffusion transformer with latent video diffusion). Please refer to the [base model card](https://huggingface.co/nvidia/Cosmos-Predict2-2B-Video2World) for detailed architecture specifications.
+
+**Key adaptation**: Actions, proprioceptive states, and values are encoded as latent frames and injected directly into the video model's latent diffusion sequence, enabling the model to generate these modalities alongside predicted future images.
+
+## Model Details
+
+### Inputs
+
+- **Current state images**: 
+  - Top-down third-person camera: Resized to 224x224 RGB
+  - Left wrist-mounted camera: Resized to 224x224 RGB
+  - Right wrist-mounted camera: Resized to 224x224 RGB
+- **Robot proprioception**: 14-dimensional (7 joint angles per arm)
+- **Task description**: Natural language text (e.g., "put candy in ziploc bag")
+
+### Outputs
+
+- **Action chunk**: 50-timestep sequence of 14-dimensional actions (7 per arm: joint positions for 6 joints + 1 gripper)
+- **Future robot proprioception**: 14-dimensional state at timestep t+50
+- **Future state images**: 
+  - Top-down third-person camera prediction at timestep t+50
+  - Left wrist camera prediction at timestep t+50
+  - Right wrist camera prediction at timestep t+50
+- **Future state value**: Expected cumulative reward from future state
+
+**Note on future predictions**: The future state images and value predictions generated by this base policy checkpoint are primarily for visualization and interpretability purposes. For model-based planning with these predictions, please additionally use the separate [Cosmos-Policy-ALOHA-Planning-Model-Predict2-2B](#) checkpoint as the world model and value function. The other checkpoint has been fine-tuned on policy rollout data to refine the world model and value function for more accurate planning.
+
+### Training Details
+
+**Training Data**: [ALOHA-Cosmos-Policy](https://huggingface.co/datasets/nvidia/ALOHA-Cosmos-Policy) dataset
+- 4 bimanual manipulation tasks
+- 185 total real-world human teleoperation demonstrations
+  - put X on plate: 80 demos
+  - fold shirt: 15 demos
+  - put candies in bowl: 45 demos
+  - put candy in ziploc bag: 45 demos
+
+**Training Configuration**:
+- **Base model**: NVIDIA Cosmos-Predict2-2B-Video2World (`model-480p-16fps.pt`)
+- **Training steps**: 50,000 gradient steps
+- **Batch size**: 200 (global)
+- **GPUs**: 8 H100 GPUs
+- **Training time**: ~48 hours
+- **Optimization**: Full model fine-tuning (all weights updated)
+- **Action chunk size**: 50 timesteps (spanning 2 seconds given 25 Hz control frequency)
+- **Execution horizon**: 50 timesteps (full chunk; recommended, though can be varied)
+- **Image resolution**: 224x224 pixels
+- **Control frequency**: 25 Hz (reduced from original 50 Hz)
+
+**Training Objective**: The model is trained with a hybrid log-normal-uniform noise distribution (modified from the base model's log-normal distribution; see paper for details) to improve action prediction accuracy. Training batches are split 50/25/25 for policy, world model, and value function objectives, respectively.
+
+**Inference Settings**:
+- Denoising steps: 10 (note: this can be changed without retraining)
+- Noise level range: σ_min = 4.0, σ_max = 80.0
+- Generation mode: Either parallel (action, future state, and value generated simultaneously) or autoregressive (using this checkpoint as the policy and the separate planning model checkpoint mentioned above as the world model and value function; see paper for more details)
+- Execution: Full 50-timestep action chunk (2 seconds) is executed before requerying the policy
+
+## Performance
+
+### ALOHA Real-World Benchmark Results
+
+| Task | Score |
+|------|-------|
+| put X on plate | 100.0 |
+| fold shirt | 99.5 |
+| put candies in bowl | 89.6 |
+| put candy in ziploc bag | 85.4 |
+| **Average** | **93.6** |
+
+Scores represent average percent completion across 101 trials total (including both in-distribution and out-of-distribution test conditions). The model outperforms baseline policies including Diffusion Policy (33.6), OpenVLA-OFT+ (62.0), π0 (77.9), and π0.5 (88.6).
+
+### Task Characteristics
+
+- **put X on plate**: Language-conditioned object placement (tests language following)
+- **fold shirt**: Multi-step contact-rich manipulation (tests long-horizon planning)
+- **put candies in bowl**: Handling scattered objects (tests multimodal grasp sequences)
+- **put candy in ziploc bag**: High-precision millimeter-tolerance manipulation
+
+## Important Usage Notes
+
+**Hardware Compatibility Warning**: This model was trained on a specific ALOHA 2 robot setup with particular hardware characteristics. Differences between our robot setup and downstream users' hardware setups (including calibration, joint limits, camera positioning, gripper mechanics, etc.) may significantly impact performance. Users must exercise caution during deployment.
+
+**Control Frequency**: This policy must be used with a **25 Hz controller** for satisfactory performance (not the original 50 Hz ALOHA control frequency). The reduced frequency was used during data collection and training.
+
+**Real-World Deployment**: This model operates real robotic hardware. Always ensure that proper safety measures are in place. On the first deployment of this checkpoint, we highly recommend measuring the difference in the current robot state and the next commanded robot state (e.g., difference between current joint angles and predicted actions, which represent target joint angles) and aborting policy execution if the difference is large.
+
+## Notes
+
+- **Real-world data**: This checkpoint is trained on real-world teleoperation data from the ALOHA 2 robot
+- **Bimanual platform**: Designed for dual-arm manipulation with two ViperX 300 S robot arms
+- **Fixed camera setup**: Requires specific camera configuration (top-down + two wrist views)
+- **Task-specific**: Trained on four specific bimanual manipulation tasks
+- **Hardware sensitivity**: Performance may vary with different robot configurations or hardware setups
+
+## Citation
+
+If you use this model, please cite the Cosmos Policy paper by Kim et al.
+<!-- ```bibtex
+# TODO: Add Cosmos Policy BibTeX
+``` -->
+
+## License
+
+Please refer to the [base model license](https://huggingface.co/nvidia/Cosmos-Predict2-2B-Video2World) for licensing information.
+
+## Related Resources
+
+- **Base Model**: [Cosmos-Predict2-2B-Video2World](https://huggingface.co/nvidia/Cosmos-Predict2-2B-Video2World)
+- **Training Dataset**: [ALOHA-Cosmos-Policy](https://huggingface.co/datasets/nvidia/ALOHA-Cosmos-Policy)
+- **Planning Model Checkpoint**: Cosmos-Policy-ALOHA-Planning-Model-Predict2-2B (for model-based planning)
+- **Paper**: *Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning*
+```