| # TTV-1B Setup Guide | |
| Complete installation and setup instructions for the TTV-1B text-to-video model. | |
| ## Prerequisites | |
| ### Hardware Requirements | |
| #### Minimum (Inference Only) | |
| - GPU: 8GB VRAM (RTX 3070, RTX 4060 Ti) | |
| - RAM: 16GB | |
| - Storage: 50GB | |
| - OS: Ubuntu 20.04+, Windows 10+, macOS 12+ | |
| #### Recommended (Training) | |
| - GPU: 24GB+ VRAM (RTX 4090, A5000, A100) | |
| - RAM: 64GB | |
| - Storage: 500GB SSD | |
| - OS: Ubuntu 22.04 LTS | |
| #### Production (Full Training) | |
| - GPU: 8× A100 80GB | |
| - RAM: 512GB | |
| - Storage: 2TB NVMe SSD | |
| - Network: High-speed interconnect for multi-GPU | |
| ### Software Requirements | |
| - Python 3.9, 3.10, or 3.11 | |
| - CUDA 11.8+ (for GPU acceleration) | |
| - cuDNN 8.6+ | |
| - Git | |
| ## Installation | |
| ### Step 1: Clone Repository | |
| ```bash | |
| git clone https://github.com/yourusername/ttv-1b.git | |
| cd ttv-1b | |
| ``` | |
| ### Step 2: Create Virtual Environment | |
| ```bash | |
| # Using venv | |
| python3 -m venv venv | |
| source venv/bin/activate # Linux/Mac | |
| # or | |
| venv\Scripts\activate # Windows | |
| # Using conda (alternative) | |
| conda create -n ttv1b python=3.10 | |
| conda activate ttv1b | |
| ``` | |
| ### Step 3: Install PyTorch | |
| Choose the appropriate command for your system from https://pytorch.org/get-started/locally/ | |
| ```bash | |
| # CUDA 11.8 (most common) | |
| pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 | |
| # CUDA 12.1 | |
| pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121 | |
| # CPU only (not recommended) | |
| pip install torch torchvision | |
| ``` | |
| ### Step 4: Install Dependencies | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| ### Step 5: Verify Installation | |
| ```bash | |
| python -c "import torch; print(f'PyTorch {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}')" | |
| ``` | |
| Expected output: | |
| ``` | |
| PyTorch 2.1.0 | |
| CUDA available: True | |
| ``` | |
| ## Quick Start | |
| ### Test the Model | |
| ```bash | |
| # Run evaluation script to verify everything works | |
| python evaluate.py | |
| ``` | |
| This will: | |
| - Create the model | |
| - Count parameters (should be ~1.0B) | |
| - Test forward/backward passes | |
| - Measure inference speed | |
| - Check memory usage | |
| ### Generate Your First Video (After Training) | |
| ```bash | |
| python inference.py \ | |
| --prompt "A beautiful sunset over mountains" \ | |
| --checkpoint checkpoints/checkpoint_best.pt \ | |
| --output my_first_video.mp4 \ | |
| --steps 50 | |
| ``` | |
| ## Preparing Data | |
| ### Data Format | |
| The model expects video-text pairs in the following format: | |
| ``` | |
| data/ | |
| ├── videos/ | |
| │ ├── video_0001.mp4 | |
| │ ├── video_0002.mp4 | |
| │ └── ... | |
| └── annotations.json | |
| ``` | |
| annotations.json: | |
| ```json | |
| { | |
| "video_0001": { | |
| "caption": "A cat playing with a ball of yarn", | |
| "duration": 2.0, | |
| "fps": 8 | |
| }, | |
| "video_0002": { | |
| "caption": "Sunset over the ocean with waves", | |
| "duration": 2.0, | |
| "fps": 8 | |
| } | |
| } | |
| ``` | |
| ### Video Specifications | |
| - Format: MP4, AVI, or MOV | |
| - Resolution: 256×256 (will be resized) | |
| - Frame rate: 8 FPS recommended | |
| - Duration: 2 seconds (16 frames at 8 FPS) | |
| - Codec: H.264 recommended | |
| ### Converting Videos | |
| ```bash | |
| # Using FFmpeg to convert videos | |
| ffmpeg -i input.mp4 -vf "scale=256:256,fps=8" -t 2 -c:v libx264 output.mp4 | |
| ``` | |
| ### Dataset Preparation Script | |
| ```python | |
| import json | |
| from pathlib import Path | |
| def create_annotations(video_dir, output_file): | |
| """Create annotations file from videos""" | |
| video_dir = Path(video_dir) | |
| annotations = {} | |
| for video_path in video_dir.glob("*.mp4"): | |
| video_id = video_path.stem | |
| annotations[video_id] = { | |
| "caption": f"Video {video_id}", # Add actual captions | |
| "duration": 2.0, | |
| "fps": 8 | |
| } | |
| with open(output_file, 'w') as f: | |
| json.dump(annotations, f, indent=2) | |
| # Usage | |
| create_annotations("data/videos", "data/annotations.json") | |
| ``` | |
| ## Training | |
| ### Single GPU Training | |
| ```bash | |
| python train.py | |
| ``` | |
| Configuration in train.py: | |
| ```python | |
| config = { | |
| 'batch_size': 2, | |
| 'gradient_accumulation_steps': 8, # Effective batch size = 16 | |
| 'learning_rate': 1e-4, | |
| 'num_epochs': 100, | |
| 'mixed_precision': True, | |
| } | |
| ``` | |
| ### Multi-GPU Training (Recommended) | |
| ```bash | |
| # Using PyTorch DDP | |
| torchrun --nproc_per_node=8 train.py | |
| # Or using accelerate (better) | |
| accelerate config # First time setup | |
| accelerate launch train.py | |
| ``` | |
| ### Monitoring Training | |
| ```bash | |
| # Install tensorboard | |
| pip install tensorboard | |
| # Run tensorboard | |
| tensorboard --logdir=./checkpoints/logs | |
| ``` | |
| ### Resume from Checkpoint | |
| ```python | |
| # In train.py, add: | |
| trainer.load_checkpoint('checkpoints/checkpoint_step_10000.pt') | |
| trainer.train() | |
| ``` | |
| ## Inference | |
| ### Basic Inference | |
| ```python | |
| from inference import generate_video_from_prompt | |
| video = generate_video_from_prompt( | |
| prompt="A serene lake with mountains", | |
| checkpoint_path="checkpoints/best.pt", | |
| output_path="output.mp4", | |
| num_steps=50, | |
| guidance_scale=7.5, | |
| seed=42 # For reproducibility | |
| ) | |
| ``` | |
| ### Batch Inference | |
| ```python | |
| from inference import batch_generate | |
| prompts = [ | |
| "A cat playing", | |
| "Ocean waves", | |
| "City at night" | |
| ] | |
| batch_generate( | |
| prompts=prompts, | |
| checkpoint_path="checkpoints/best.pt", | |
| output_dir="./outputs", | |
| num_steps=50 | |
| ) | |
| ``` | |
| ### Advanced Options | |
| ```python | |
| # Lower guidance for more creative results | |
| video = generate_video_from_prompt( | |
| prompt="Abstract art in motion", | |
| guidance_scale=5.0, # Lower = more creative | |
| num_steps=100, # More steps = higher quality | |
| ) | |
| # Fast generation (fewer steps) | |
| video = generate_video_from_prompt( | |
| prompt="Quick test", | |
| num_steps=20, # Faster but lower quality | |
| ) | |
| ``` | |
| ## Optimization Tips | |
| ### Memory Optimization | |
| 1. **Reduce Batch Size** | |
| ```python | |
| config['batch_size'] = 1 # Minimum | |
| config['gradient_accumulation_steps'] = 16 # Maintain effective batch size | |
| ``` | |
| 2. **Enable Gradient Checkpointing** | |
| ```python | |
| config['gradient_checkpointing'] = True | |
| ``` | |
| 3. **Use Mixed Precision** | |
| ```python | |
| config['mixed_precision'] = True # Always recommended | |
| ``` | |
| ### Speed Optimization | |
| 1. **Use Torch Compile** (PyTorch 2.0+) | |
| ```python | |
| model = torch.compile(model) | |
| ``` | |
| 2. **Enable cuDNN Benchmarking** | |
| ```python | |
| torch.backends.cudnn.benchmark = True | |
| ``` | |
| 3. **Pin Memory** | |
| ```python | |
| DataLoader(..., pin_memory=True) | |
| ``` | |
| ## Troubleshooting | |
| ### CUDA Out of Memory | |
| ```bash | |
| # Reduce batch size | |
| config['batch_size'] = 1 | |
| # Enable gradient checkpointing | |
| config['gradient_checkpointing'] = True | |
| # Clear cache | |
| torch.cuda.empty_cache() | |
| ``` | |
| ### Slow Training | |
| ```bash | |
| # Check GPU utilization | |
| nvidia-smi | |
| # Increase num_workers | |
| DataLoader(..., num_workers=8) | |
| # Enable mixed precision | |
| config['mixed_precision'] = True | |
| ``` | |
| ### NaN Loss | |
| ```python | |
| # Reduce learning rate | |
| config['learning_rate'] = 5e-5 | |
| # Enable gradient clipping (already included) | |
| torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) | |
| # Check for NaN in data | |
| assert not torch.isnan(videos).any() | |
| ``` | |
| ### Model Not Learning | |
| ```python | |
| # Increase learning rate | |
| config['learning_rate'] = 2e-4 | |
| # Check data quality | |
| # Verify annotations are correct | |
| # Ensure videos are properly normalized | |
| # Reduce regularization | |
| config['weight_decay'] = 0.001 # Lower weight decay | |
| ``` | |
| ## Performance Benchmarks | |
| ### Training Speed (A100 80GB) | |
| | Batch Size | Grad Accum | Eff. Batch | Sec/Batch | Hours/100K steps | | |
| |------------|------------|------------|-----------|------------------| | |
| | 1 | 16 | 16 | 2.5 | 69 | | |
| | 2 | 8 | 16 | 2.5 | 69 | | |
| | 4 | 4 | 16 | 2.7 | 75 | | |
| ### Inference Speed | |
| | GPU | FP16 | Steps | Time/Video | | |
| |-----|------|-------|------------| | |
| | A100 80GB | Yes | 50 | 15s | | |
| | RTX 4090 | Yes | 50 | 25s | | |
| | RTX 3090 | Yes | 50 | 35s | | |
| ### Memory Usage | |
| | Operation | Batch Size | Memory (GB) | | |
| |-----------|------------|-------------| | |
| | Inference | 1 | 6 | | |
| | Training | 1 | 12 | | |
| | Training | 2 | 24 | | |
| | Training | 4 | 48 | | |
| ## Next Steps | |
| 1. **Prepare your dataset** - Collect and annotate videos | |
| 2. **Start training** - Begin with small dataset to verify | |
| 3. **Monitor progress** - Check loss, sample generations | |
| 4. **Fine-tune** - Adjust hyperparameters based on results | |
| 5. **Evaluate** - Test on held-out validation set | |
| 6. **Deploy** - Use for inference on new prompts | |
| ## Getting Help | |
| - GitHub Issues: Report bugs and ask questions | |
| - Documentation: Check README.md and ARCHITECTURE.md | |
| - Examples: See example scripts in the repository | |
| ## Additional Resources | |
| - [PyTorch Documentation](https://pytorch.org/docs/) | |
| - [Diffusion Models Explained](https://lilianweng.github.io/posts/2021-07-11-diffusion-models/) | |
| - [Attention Is All You Need](https://arxiv.org/abs/1706.03762) | |
| - [DiT Paper](https://arxiv.org/abs/2212.09748) | |