| # TTV-1B Model Architecture Specification | |
| ## Model Summary | |
| **Name:** TTV-1B (Text-to-Video 1 Billion) | |
| **Type:** Diffusion Transformer for Text-to-Video Generation | |
| **Total Parameters:** 1,003,147,264 (~1.0 Billion) | |
| ## Architecture Components | |
| ### 1. Text Encoder (50M parameters) | |
| ``` | |
| Input: Text tokens (batch_size, 256) | |
| Architecture: | |
| - Token Embedding: 50,257 vocab → 768 dim | |
| - Position Embedding: 256 positions → 768 dim | |
| - 6 Transformer Layers: | |
| * Multi-head Attention (12 heads) | |
| * Feed-forward (768 → 3072 → 768) | |
| * Layer Normalization | |
| Output: Text features (batch_size, 256, 768) | |
| ``` | |
| ### 2. Text Projection Layer | |
| ``` | |
| Linear: 768 → 1536 dimensions | |
| Purpose: Project text features to model hidden dimension | |
| ``` | |
| ### 3. 3D Patch Embedding | |
| ``` | |
| Input: Video (batch_size, 3, 16, 256, 256) | |
| Patch size: (2, 16, 16) - temporal × height × width | |
| Conv3D: 3 channels → 1536 channels | |
| Output: (batch_size, 128, 1536) where 128 = (16/2) × (256/16) × (256/16) | |
| = 8 × 16 × 16 | |
| ``` | |
| ### 4. Positional Embedding | |
| ``` | |
| Learnable position embeddings for 128 patches | |
| Shape: (1, 128, 1536) | |
| ``` | |
| ### 5. Timestep Embedding | |
| ``` | |
| Sinusoidal timestep encoding → Linear(1536, 6144) → SiLU → Linear(6144, 1536) | |
| Output: Conditioning vector (batch_size, 1536) | |
| ``` | |
| ### 6. DiT Blocks (24 layers, 950M parameters) | |
| Each block contains: | |
| #### a) 3D Spatiotemporal Attention | |
| ``` | |
| - Query, Key, Value projections: Linear(1536, 4608) | |
| - 24 attention heads (64 dimensions each) | |
| - Rotary position embeddings on temporal dimension | |
| - Scaled dot-product attention | |
| - Output projection: Linear(1536, 1536) | |
| ``` | |
| #### b) Feed-Forward Network | |
| ``` | |
| - Linear: 1536 → 6144 (4x expansion) | |
| - GELU activation | |
| - Linear: 6144 → 1536 | |
| ``` | |
| #### c) Adaptive Layer Normalization (AdaLN) | |
| ``` | |
| - Modulation network: SiLU → Linear(1536, 9216) | |
| - Generates 6 modulation parameters: | |
| * scale_msa, shift_msa, gate_msa (for attention) | |
| * scale_mlp, shift_mlp, gate_mlp (for FFN) | |
| ``` | |
| ### 7. Final Layer | |
| ``` | |
| - Adaptive LayerNorm | |
| - Linear: 1536 → 768 (2×16×16×3) | |
| Purpose: Map back to patch space | |
| ``` | |
| ### 8. Unpatchify | |
| ``` | |
| Reshape patches back to video | |
| (batch_size, 128, 768) → (batch_size, 3, 16, 256, 256) | |
| ``` | |
| ## Parameter Breakdown | |
| | Component | Parameters | Percentage | | |
| |-----------|------------|------------| | |
| | Text Encoder | 50,331,648 | 5.0% | | |
| | Text Projection | 1,180,416 | 0.1% | | |
| | Patch Embedding | 589,824 | 0.1% | | |
| | Position Embedding | 196,608 | 0.02% | | |
| | Timestep Embedding | 14,157,312 | 1.4% | | |
| | DiT Blocks (24×) | 927,711,744 | 92.5% | | |
| | Final Layer | 8,979,712 | 0.9% | | |
| | **Total** | **1,003,147,264** | **100%** | | |
| ## Per-Block Parameters (DiT) | |
| Each of 24 DiT blocks contains ~38.7M parameters: | |
| | Sub-component | Parameters | | |
| |---------------|------------| | |
| | Attention QKV | 7,077,888 | | |
| | Attention Proj | 2,362,368 | | |
| | Rotary Embedding | 48 | | |
| | FFN Layer 1 | 9,443,328 | | |
| | FFN Layer 2 | 9,443,328 | | |
| | AdaLN Modulation | 14,155,776 | | |
| | Layer Norms | 0 (no learnable params) | | |
| | **Per Block Total** | **38,654,656** | | |
| ## Data Flow | |
| ``` | |
| 1. Text Input (batch, 256 tokens) | |
| ↓ | |
| 2. Text Encoder (6 transformer layers) | |
| ↓ | |
| 3. Text Features (batch, 256, 768) → Pool → (batch, 768) | |
| ↓ | |
| 4. Project to 1536 dim → (batch, 1536) | |
| ↓ | |
| 5. Add Timestep Embedding → Conditioning (batch, 1536) | |
| ↓ | |
| 6. Video Input (batch, 3, 16, 256, 256) | |
| ↓ | |
| 7. 3D Patch Embed → (batch, 128, 1536) | |
| ↓ | |
| 8. Add Position Embedding | |
| ↓ | |
| 9. 24× DiT Blocks (with conditioning) | |
| ↓ | |
| 10. Final Layer + AdaLN | |
| ↓ | |
| 11. Unpatchify | |
| ↓ | |
| 12. Output: Predicted Noise (batch, 3, 16, 256, 256) | |
| ``` | |
| ## Memory Requirements | |
| ### Model Weights | |
| - FP32: ~4.0 GB | |
| - FP16: ~2.0 GB | |
| - INT8: ~1.0 GB | |
| ### Activations (per sample, 256×256×16) | |
| - Forward pass: ~8 GB (FP16) | |
| - Backward pass: ~16 GB (FP16) | |
| ### Training (batch_size=2, FP16, gradient accumulation=8) | |
| - Model: 2 GB | |
| - Optimizer states (AdamW): 4 GB | |
| - Gradients: 2 GB | |
| - Activations: 16 GB | |
| - **Total: ~24 GB per GPU** | |
| ### Inference (batch_size=1, FP16) | |
| - Model: 2 GB | |
| - Activations: 4 GB | |
| - **Total: ~6 GB** | |
| ## Computational Complexity | |
| ### FLOPs per forward pass (approximate) | |
| - Text Encoder: ~10 GFLOPs | |
| - Patch Embedding: ~5 GFLOPs | |
| - DiT Blocks (24×): ~4,800 GFLOPs | |
| - Unpatchify: ~1 GFLOPs | |
| - **Total: ~4,816 GFLOPs per video** | |
| ### Training Speed Estimates | |
| - Single A100 80GB: ~2-3 seconds per batch (batch_size=2) | |
| - 8× A100 80GB: ~2-3 seconds per batch (batch_size=16) | |
| ### Inference Speed Estimates | |
| - A100 80GB (50 denoising steps): ~15-20 seconds per video | |
| - RTX 4090 (50 denoising steps): ~25-35 seconds per video | |
| ## Diffusion Scheduler | |
| ### DDPM (Denoising Diffusion Probabilistic Model) | |
| - Training steps: 1000 | |
| - Beta schedule: Linear (0.0001 → 0.02) | |
| - Loss: MSE between predicted and actual noise | |
| - Sampling: Iterative denoising from T=999 to T=0 | |
| ### Classifier-Free Guidance | |
| - Unconditional dropout during training: 10% | |
| - Guidance scale at inference: 7.5 (typical) | |
| - Formula: `noise_pred = noise_uncond + guidance_scale × (noise_cond - noise_uncond)` | |
| ## Key Features | |
| 1. **3D Spatiotemporal Attention** | |
| - Full attention across time, height, and width | |
| - Captures motion dynamics and spatial relationships | |
| 2. **Rotary Position Embeddings** | |
| - Applied to temporal dimension | |
| - Better sequence modeling than learned embeddings | |
| 3. **Adaptive Layer Normalization** | |
| - Conditions on text and timestep | |
| - Allows flexible control over generation | |
| 4. **Efficient Design** | |
| - Patch-based processing reduces sequence length | |
| - Mixed precision training support | |
| - Gradient checkpointing compatible | |
| ## Comparison with Other Models | |
| | Model | Parameters | Resolution | Frames | Architecture | | |
| |-------|------------|------------|--------|--------------| | |
| | TTV-1B (ours) | 1.0B | 256×256 | 16 | DiT | | |
| | Stable Diffusion Video | 1.7B | 512×512 | 25 | U-Net | | |
| | Make-A-Video | 9.7B | 256×256 | 16 | U-Net | | |
| | Imagen Video | 11B | 1280×768 | 128 | U-Net Cascade | | |
| ## Optimization Techniques | |
| 1. **Mixed Precision (FP16)** | |
| - Reduces memory by 50% | |
| - Faster computation on modern GPUs | |
| 2. **Gradient Accumulation** | |
| - Enables larger effective batch sizes | |
| - Improves training stability | |
| 3. **Gradient Checkpointing** | |
| - Trades computation for memory | |
| - Enables larger batch sizes | |
| 4. **Flash Attention** | |
| - O(N) memory instead of O(N²) | |
| - Faster attention computation | |
| ## Future Enhancements | |
| 1. **Higher Resolution**: 512×512 or 1024×1024 | |
| 2. **Longer Videos**: 64 or 128 frames | |
| 3. **Better Text Encoding**: CLIP or T5 | |
| 4. **Temporal Super-Resolution**: Increase frame rate | |
| 5. **Motion Control**: Add motion guidance | |
| 6. **Video Editing**: Inpainting, style transfer | |
| 7. **LoRA Fine-tuning**: Efficient adaptation | |
| 8. **Distillation**: Smaller, faster variants | |