|
|
--- |
|
|
license: cc-by-nc-4.0 |
|
|
tags: |
|
|
- computer-vision |
|
|
- 6d-pose-estimation |
|
|
- object-detection |
|
|
- robotics |
|
|
- foundationpose |
|
|
library_name: foundationpose |
|
|
--- |
|
|
|
|
|
# FoundationPose Model Weights |
|
|
|
|
|
Pre-trained weights for [FoundationPose](https://github.com/NVlabs/FoundationPose) 6D object pose estimation model. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Refiner weights:** `2023-10-28-18-33-37/model_best.pth` |
|
|
- **Scorer weights:** `2024-01-11-20-02-45/model_best.pth` |
|
|
- **Source:** [Official FoundationPose release](https://github.com/NVlabs/FoundationPose) |
|
|
- **Paper:** [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects (CVPR 2024)](https://arxiv.org/abs/2312.08344) |
|
|
|
|
|
## Model Architecture |
|
|
|
|
|
FoundationPose is a unified foundation model for 6D object pose estimation and tracking, supporting both: |
|
|
- **Model-based setup**: Using CAD models |
|
|
- **Model-free setup**: Using reference images (16-20 views) |
|
|
|
|
|
## Files |
|
|
|
|
|
``` |
|
|
. |
|
|
├── 2023-10-28-18-33-37/ |
|
|
│ ├── config.yml |
|
|
│ └── model_best.pth (refiner model) |
|
|
└── 2024-01-11-20-02-45/ |
|
|
├── config.yml |
|
|
└── model_best.pth (scorer model) |
|
|
``` |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Download Weights |
|
|
|
|
|
```python |
|
|
from huggingface_hub import snapshot_download |
|
|
|
|
|
# Download all weights |
|
|
weights_path = snapshot_download( |
|
|
repo_id="gpue/foundationpose-weights", |
|
|
local_dir="./weights" |
|
|
) |
|
|
``` |
|
|
|
|
|
### Use with FoundationPose Space |
|
|
|
|
|
This model repository is designed to work with the [gpue/foundationpose](https://huggingface.co/spaces/gpue/foundationpose) Space. |
|
|
|
|
|
Set environment variables: |
|
|
```bash |
|
|
FOUNDATIONPOSE_MODEL_REPO=gpue/foundationpose-weights |
|
|
USE_HF_WEIGHTS=true |
|
|
USE_REAL_MODEL=true |
|
|
``` |
|
|
|
|
|
### Local Usage |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from pathlib import Path |
|
|
|
|
|
# Load refiner |
|
|
refiner_weights = torch.load("weights/2023-10-28-18-33-37/model_best.pth") |
|
|
|
|
|
# Load scorer |
|
|
scorer_weights = torch.load("weights/2024-01-11-20-02-45/model_best.pth") |
|
|
``` |
|
|
|
|
|
## Performance |
|
|
|
|
|
- **Accuracy**: State-of-the-art on BOP benchmark (as of 2024/03) |
|
|
- **Speed**: Real-time capable with GPU acceleration |
|
|
- **Generalization**: Works on novel objects without fine-tuning |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use these weights, please cite: |
|
|
|
|
|
```bibtex |
|
|
@inproceedings{wen2023foundationpose, |
|
|
title={FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects}, |
|
|
author={Wen, Bowen and Yang, Wei and Kautz, Jan and Birchfield, Stan}, |
|
|
booktitle={CVPR}, |
|
|
year={2024} |
|
|
} |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
These weights are from the official FoundationPose release and are subject to NVIDIA's [Source Code License](https://github.com/NVlabs/FoundationPose/blob/main/LICENSE.txt). |
|
|
|
|
|
**Key restrictions:** |
|
|
- Non-commercial use only |
|
|
- No redistribution of derivative works |
|
|
- Academic and research purposes |
|
|
|
|
|
## Related Resources |
|
|
|
|
|
- **Paper**: https://arxiv.org/abs/2312.08344 |
|
|
- **Code**: https://github.com/NVlabs/FoundationPose |
|
|
- **Project Page**: https://nvlabs.github.io/FoundationPose/ |
|
|
- **Inference Space**: https://huggingface.co/spaces/gpue/foundationpose |
|
|
|
|
|
## Model Card |
|
|
|
|
|
**Developed by:** NVIDIA Research (Bowen Wen, Wei Yang, Jan Kautz, Stan Birchfield) |
|
|
|
|
|
**Model type:** Transformer-based 6D pose estimator |
|
|
|
|
|
**Training data:** Large-scale synthetic dataset |
|
|
|
|
|
**Intended use:** 6D object pose estimation and tracking for robotics and AR/VR applications |
|
|
|
|
|
**Out-of-scope:** Commercial deployment (due to license restrictions) |
|
|
|
