README.md

---
title: MIMO - Character Video Synthesis
emoji: 🎭
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 4.7.1
app_file: app.py
pinned: false
license: apache-2.0
python_version: "3.10"
---

# MIMO - Controllable Character Video Synthesis

**🎬 Complete Implementation - Optimized for HuggingFace Spaces**

Transform character images into animated videos with controllable motion and advanced video editing capabilities.

## 🚀 Quick Start

1. **Setup Models**: Click "Setup Models" button (downloads required models)
2. **Load Model**: Click "Load Model" button (initializes MIMO pipeline)
3. **Upload Image**: Character image (person, anime, cartoon, etc.)
4. **Choose Template** (Optional): Select motion template or use reference image only
5. **Generate**: Create animated video

> **Note on Templates**: Video templates are optional. See [TEMPLATES_SETUP.md](TEMPLATES_SETUP.md) for adding custom templates.

## ⚡ Why This Approach?

To prevent HuggingFace Spaces build timeout, we use **progressive loading**:
- **Minimal dependencies** at startup (fast build)
- **Runtime installation** of heavy packages (TensorFlow, OpenCV)
- **Full features** available after one-time setup

## Features

### 🎭 Character Animation Mode
- Simple character animation with motion templates
- Based on `run_animate.py` from original repository
- Fast generation (512x512, 20 steps)

### 🎬 Video Character Editing Mode
- Advanced editing with background preservation
- Human segmentation and occlusion handling
- Based on `run_edit.py` from original repository
- High quality (784x784, 25 steps)

## Available Templates

**Sports:** basketball_gym, nba_dunk, nba_pass, football
**Action:** kungfu_desert, kungfu_match, parkour, BruceLee
**Dance:** dance_indoor, irish_dance
**Synthetic:** syn_basketball, syn_dancing

## Technical Details

- **Models:** Stable Diffusion v1.5 + 3D UNet + Pose Guider
- **GPU:** Auto-detection (T4/A10G/A100) with FP16/FP32
- **Resolution:** 512x512 (Animation), 784x784 (Editing)
- **Processing:** 2-5 minutes depending on template
- **Video I/O:** PyAV (`av` pip package) for frame decoding/encoding

## Credits

**Paper:** [MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling](https://arxiv.org/abs/2409.16160)
**Authors:** Yifang Men, Yuan Yao, Miaomiao Cui, Liefeng Bo (Alibaba Group)
**Conference:** CVPR 2025
**Code:** [GitHub](https://github.com/menyifang/MIMO)