| # VINE Model - Quick Start Guide | |
| Get started with VINE video understanding in 2 simple steps! | |
| ## One-Command Setup | |
| ```bash | |
| # Download and run the complete setup script | |
| wget https://huggingface.co/video-fm/vine/resolve/main/setup_vine_complete.sh | |
| bash setup_vine_complete.sh | |
| ``` | |
| **That's it!** This single script: | |
| - ✅ Creates conda environment with Python 3.10 | |
| - ✅ Installs PyTorch with CUDA support | |
| - ✅ Clones all required repositories (laser, sam2, groundingdino, vine_hf) | |
| - ✅ Downloads SAM2 checkpoint (~149 MB) | |
| - ✅ Downloads GroundingDINO checkpoint (~662 MB) | |
| - ✅ Downloads all config files | |
| - ✅ Tests the installation | |
| **Total setup time**: ~10-15 minutes (depending on download speed) | |
| ## What Gets Installed | |
| ``` | |
| your-directory/ | |
| ├── checkpoints/ | |
| │ ├── sam2_hiera_tiny.pt (~149 MB) | |
| │ ├── sam2_hiera_t.yaml | |
| │ ├── groundingdino_swint_ogc.pth (~662 MB) | |
| │ └── GroundingDINO_SwinT_OGC.py | |
| ├── src/ | |
| │ ├── LASER/ (video processing utilities) | |
| │ ├── video-sam2/ (SAM2 segmentation) | |
| │ ├── GroundingDINO/ (object detection) | |
| │ └── vine_hf/ (VINE HuggingFace interface) | |
| └── test_vine.py (test script) | |
| ``` | |
| ## Usage After Setup | |
| ### Activate Environment | |
| ```bash | |
| conda activate vine_demo | |
| ``` | |
| ### Test Installation | |
| ```python | |
| python test_vine.py | |
| ``` | |
| ### Use in Your Code | |
| ```python | |
| from transformers import AutoModel | |
| from vine_hf import VinePipeline | |
| from pathlib import Path | |
| # Load VINE model from HuggingFace | |
| model = AutoModel.from_pretrained('video-fm/vine', trust_remote_code=True) | |
| # Set up checkpoint paths | |
| checkpoint_dir = Path("checkpoints") | |
| # Create pipeline | |
| pipeline = VinePipeline( | |
| model=model, | |
| tokenizer=None, | |
| sam_config_path=str(checkpoint_dir / "sam2_hiera_t.yaml"), | |
| sam_checkpoint_path=str(checkpoint_dir / "sam2_hiera_tiny.pt"), | |
| gd_config_path=str(checkpoint_dir / "GroundingDINO_SwinT_OGC.py"), | |
| gd_checkpoint_path=str(checkpoint_dir / "groundingdino_swint_ogc.pth"), | |
| device="cuda", | |
| trust_remote_code=True | |
| ) | |
| # Process a video | |
| results = pipeline( | |
| "path/to/video.mp4", | |
| categorical_keywords=['person', 'dog', 'ball'], | |
| unary_keywords=['running', 'jumping', 'sitting'], | |
| binary_keywords=['chasing', 'next to', 'holding'], | |
| object_pairs=[(0, 1), (0, 2)], # person-dog, person-ball | |
| return_top_k=5 | |
| ) | |
| # Print results | |
| print(f"Detected {results['summary']['num_objects_detected']} objects") | |
| print(f"Top categories: {results['summary']['top_categories']}") | |
| print(f"Top actions: {results['summary']['top_actions']}") | |
| print(f"Top relations: {results['summary']['top_relations']}") | |
| ``` | |
| ## System Requirements | |
| - **OS**: Linux (tested on Ubuntu) | |
| - **Python**: 3.10+ | |
| - **CUDA**: 11.8+ (for GPU acceleration) | |
| - **GPU**: 8GB+ VRAM recommended (T4, V100, A100, etc.) | |
| - **RAM**: 16GB+ recommended | |
| - **Disk Space**: ~5GB total | |
| - Conda environment: ~3GB | |
| - Checkpoints: ~811MB | |
| - Code repositories: ~1GB | |
| ## Troubleshooting | |
| ### CUDA Not Available | |
| ```bash | |
| # Check CUDA | |
| nvidia-smi | |
| # If not working, install CPU-only version | |
| pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu | |
| ``` | |
| ### Download Failed | |
| ```bash | |
| # Manually download checkpoints | |
| cd checkpoints | |
| # SAM2 | |
| wget https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_tiny.pt | |
| wget https://raw.githubusercontent.com/facebookresearch/sam2/main/sam2/configs/sam2.1/sam2.1_hiera_t.yaml -O sam2_hiera_t.yaml | |
| # GroundingDINO | |
| wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth | |
| wget https://raw.githubusercontent.com/IDEA-Research/GroundingDINO/main/groundingdino/config/GroundingDINO_SwinT_OGC.py | |
| ``` | |
| ### Import Errors | |
| ```bash | |
| # Reinstall packages | |
| conda activate vine_demo | |
| cd src | |
| pip install -e ./LASER | |
| pip install -e ./video-sam2 | |
| pip install -e ./GroundingDINO | |
| pip install -e ./vine_hf | |
| ``` | |
| ## Alternative: Manual Setup | |
| If you prefer to set up manually or the script fails, see [README.md](https://huggingface.co/video-fm/vine) for step-by-step instructions. | |
| ## Next Steps | |
| - **Process your videos**: Use the pipeline with your own videos | |
| - **Customize keywords**: Adjust categorical, unary, and binary keywords | |
| - **Visualize results**: Enable `visualize=True` in config | |
| - **Deploy**: Use in HuggingFace Spaces, FastAPI, or your own app | |
| ## Links | |
| - **Model**: https://huggingface.co/video-fm/vine | |
| - **Setup Script**: https://huggingface.co/video-fm/vine/blob/main/setup_vine_complete.sh | |
| - **Documentation**: https://huggingface.co/video-fm/vine#readme | |
| - **Code**: https://github.com/kevinxuez/LASER | |
| - **Issues**: https://github.com/kevinxuez/LASER/issues | |
| ## Support | |
| - **Setup Issues**: Check the script output for errors | |
| - **Model Issues**: https://huggingface.co/video-fm/vine/discussions | |
| - **Code Issues**: https://github.com/kevinxuez/LASER/issues | |
| --- | |
| **Ready to start?** | |
| ```bash | |
| wget https://huggingface.co/video-fm/vine/resolve/main/setup_vine_complete.sh | |
| bash setup_vine_complete.sh | |
| ``` | |
| 🎉 Happy video understanding with VINE! | |