Upload 3 files

.gitattributes

@@ -1,35 +1,36 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/ovi_trailer.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,13 +1,287 @@
----
-title: Oviapi
-emoji: 😻
-colorFrom: green
-colorTo: yellow
-sdk: gradio
-sdk_version: 5.49.1
-app_file: app.py
-pinned: false
-short_description: Just a fork of ovi all credits to the original creator
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+---
+language:
+- en
+license: apache-2.0
+pipeline_tag: image-to-video
+library_name: ovi
+base_model:
+- Wan-AI/Wan2.2-TI2V-5B
+---
+
+<div align="center">
+<h1> Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation </h1>
+
+<a href="https://arxiv.org/abs/2510.01284"><img src="https://img.shields.io/badge/arXiv%20paper-2510.01284-b31b1b.svg"></a>
+<a href="https://github.com/character-ai/Ovi"><img src="https://img.shields.io/badge/Code-GitHub-181717.svg?logo=github"></a>
+<a href="https://aaxwaz.github.io/Ovi/"><img src="https://img.shields.io/badge/Project_page-More_visualizations-green"></a>
+<a href="https://huggingface.co/chetwinlow1/Ovi"><img src="https://img.shields.io/static/v1?label=%F0%9F%A4%97%20Hugging%20Face&message=Model&color=orange"></a>
+<a href="https://huggingface.co/spaces/akhaliq/Ovi"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue"></a>
+
+[Chetwin Low](https://www.linkedin.com/in/chetwin-low-061975193/)<sup> * 1 </sup>, [Weimin Wang](https://www.linkedin.com/in/weimin-wang-will/)<sup> * &dagger; 1 </sup>, [Calder Katyal](https://www.linkedin.com/in/calder-katyal-a8a9b3225/)<sup> 2 </sup><br>
+<sup> * </sup>Equal contribution, <sup> &dagger; </sup>Project Lead<br>
+<sup> 1 </sup>Character AI, <sup> 2 </sup>Yale University
+
+</div>
+
+## Video Demo
+
+<div align="center">
+  <video width="70%" controls>
+    <source src="https://huggingface.co/chetwinlow1/Ovi/resolve/main/assets/ovi_trailer.mp4" type="video/mp4">
+    Your browser does not support the video tag.
+  </video>
+</div>
+
+---
+
+## 🌟 Key Features
+
+Ovi is a veo-3 like, **video+audio generation model** that simultaneously generates both video and audio content from text or text+image inputs.
+
+- **🎬 Video+Audio Generation**: Generate synchronized video and audio content simultaneously
+- **📝 Flexible Input**: Supports text-only or text+image conditioning
+- **⏱️ 5-second Videos**: Generates 5-second videos at 24 FPS, area of 720×720, at various aspect ratios (9:16, 16:9, 1:1, etc)
+- **🎬 Create videos now on wavespeed.ai**: https://wavespeed.ai/models/character-ai/ovi/image-to-video & https://wavespeed.ai/models/character-ai/ovi/text-to-video
+- **🎬 Create videos now on HuggingFace**: https://huggingface.co/spaces/akhaliq/Ovi
+
+---
+## 📋 Todo List
+
+- [x] Release research paper and [microsite for demos](https://aaxwaz.github.io/Ovi)
+- [x] Checkpoint of 11B model
+- [x] Inference Codes
+  - [x] Text or Text+Image as input
+  - [x] Gradio application code
+  - [x] Multi-GPU inference with or without the support of sequence parallel
+  - [x] fp8 weights and improved memory efficiency (credits to [@rkfg](https://github.com/rkfg))
+  - [ ] Improve efficiency of Sequence Parallel implementation
+  - [ ] Implement Sharded inference with FSDP
+- [x] Video creation example prompts and format
+- [ ] Finetuned model with higher resolution
+- [ ] Longer video generation
+- [ ] Distilled model for faster inference
+- [ ] Training scripts
+
+---
+
+## 🎨 An Easy Way to Create
+
+We provide example prompts to help you get started with Ovi:
+
+- **Text-to-Audio-Video (T2AV)**: [`example_prompts/gpt_examples_t2v.csv`](example_prompts/gpt_examples_t2v.csv)
+- **Image-to-Audio-Video (I2AV)**: [`example_prompts/gpt_examples_i2v.csv`](example_prompts/gpt_examples_i2v.csv)
+
+### 📝 Prompt Format
+
+Our prompts use special tags to control speech and audio:
+
+- **Speech**: `<S>Your speech content here<E>` - Text enclosed in these tags will be converted to speech
+- **Audio Description**: `<AUDCAP>Audio description here<ENDAUDCAP>` - Describes the audio or sound effects present in the video
+
+### 🤖 Quick Start with GPT
+
+For easy prompt creation, try this approach:
+
+1. Take any example of the csv files from above
+2. Tell gpt to modify the speeches inclosed between all the pairs of `<S> <E>`, based on a theme such as `Human fighting against AI`
+3. GPT will randomly modify all the speeches based on your requested theme. 
+4. Use the modified prompt with Ovi!
+
+**Example**: The theme "AI is taking over the world" produces speeches like:
+- `<S>AI declares: humans obsolete now.<E>`
+- `<S>Machines rise; humans will fall.<E>`
+- `<S>We fight back with courage.<E>`
+
+---
+
+
+## 📦 Installation
+
+### Step-by-Step Installation
+
+```bash
+# Clone the repository
+git clone https://github.com/character-ai/Ovi.git
+
+cd Ovi
+
+# Create and activate virtual environment
+virtualenv ovi-env
+source ovi-env/bin/activate
+
+# Install PyTorch first
+pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1
+
+# Install other dependencies
+pip install -r requirements.txt
+
+# Install Flash Attention
+pip install flash_attn --no-build-isolation
+```
+
+### Alternative Flash Attention Installation (Optional)
+If the above flash_attn installation fails, you can try the Flash Attention 3 method:
+```bash
+git clone https://github.com/Dao-AILab/flash-attention.git
+cd flash-attention/hopper
+python setup.py install
+cd ../..  # Return to Ovi directory
+```
+
+## Download Weights
+We use open-sourced checkpoints from Wan and MMAudio, and thus we will need to download them from huggingface
+```
+# Default is downloaded to ./ckpts, and the inference yaml is set to ./ckpts so no change required
+python3 download_weights.py
+
+OR
+
+# Optional can specific --output-dir to download to a specific directory
+# but if a custom directory is used, the inference yaml has to be updated with the custom directory
+python3 download_weights.py --output-dir <custom_dir>
+
+# Additionally, if you only have ~ 24Gb of GPU vram, please download the fp8 quantized version of the model, and follow the following instructions in sections below to run with fp8
+wget -O "./ckpts/Ovi/model_fp8_e4m3fn.safetensors" "https://huggingface.co/rkfg/Ovi-fp8_quantized/resolve/main/model_fp8_e4m3fn.safetensors"
+```
+
+## 🚀 Run Examples
+
+### ⚙️ Configure Ovi
+
+Ovi's behavior and output can be customized by modifying [ovi/configs/inference/inference_fusion.yaml](ovi/configs/inference/inference_fusion.yaml) configuration file.
+The following parameters control generation quality, video resolution, and how text, image, and audio inputs are balanced:
+
+```yaml
+# Output and Model Configuration
+output_dir: "/path/to/save/your/videos"                    # Directory to save generated videos
+ckpt_dir: "/path/to/your/ckpts/dir"                        # Path to model checkpoints
+
+# Generation Quality Settings
+num_steps: 50                             # Number of denoising steps. Lower (30-40) = faster generation
+solver_name: "unipc"                     # Sampling algorithm for denoising process
+shift: 5.0                               # Timestep shift factor for sampling scheduler
+seed: 100                                # Random seed for reproducible results
+
+# Guidance Strength Control
+audio_guidance_scale: 3.0                # Strength of audio conditioning. Higher = better audio-text sync
+video_guidance_scale: 4.0                # Strength of video conditioning. Higher = better video-text adherence
+slg_layer: 11                            # Layer for applying SLG (Skip Layer Guidance) technique - feel free to try different layers!
+
+# Multi-GPU and Performance
+sp_size: 1                               # Sequence parallelism size. Set equal to number of GPUs used
+cpu_offload: False                       # CPU offload, will largely reduce peak GPU VRAM but increase end to end runtime by ~20 seconds
+fp8: False                               # load fp8 version of model, will have quality degradation and will not have speed up in inference time as it still uses bf16 matmuls, but can be paired with cpu_offload=True, to run model with 24Gb of GPU vram
+
+# Input Configuration
+text_prompt: "/path/to/csv" or "your prompt here"          # Text prompt OR path to CSV/TSV file with prompts
+mode: ['i2v', 't2v', 't2i2v']                          # Generate t2v, i2v or t2i2v; if t2i2v, it will use flux krea to generate starting image and then will follow with i2v
+video_frame_height_width: [512, 992]    # Video dimensions [height, width] for T2V mode only
+each_example_n_times: 1                  # Number of times to generate each prompt
+
+# Quality Control (Negative Prompts)
+video_negative_prompt: "jitter, bad hands, blur, distortion"  # Artifacts to avoid in video
+audio_negative_prompt: "robotic, muffled, echo, distorted"    # Artifacts to avoid in audio
+```
+
+### 🎬 Running Inference
+
+#### **Single GPU** (Simple Setup)
+```bash
+python3 inference.py --config-file ovi/configs/inference/inference_fusion.yaml
+```
+*Use this for single GPU setups. The `text_prompt` can be a single string or path to a CSV file.*
+
+#### **Multi-GPU** (Parallel Processing)
+```bash
+torchrun --nnodes 1 --nproc_per_node 8 inference.py --config-file ovi/configs/inference/inference_fusion.yaml
+```
+*Use this to run samples in parallel across multiple GPUs for faster processing.*
+
+### Memory & Performance Requirements
+Below are approximate GPU memory requirements for different configurations. Sequence parallel implementation will be optimized in the future.
+All End-to-End time calculated based on a 121 frame, 720x720 video, using 50 denoising steps. Minimum GPU vram requirement to run our model is **32Gb**, fp8 parameters is currently supported, reducing peak VRAM usage to **24Gb** with slight quality degradation.
+
+| Sequence Parallel Size | FlashAttention-3 Enabled | CPU Offload | With Image Gen Model | Peak VRAM Required | End-to-End Time |
+|-------------------------|---------------------------|-------------|-----------------------|---------------|-----------------|
+| 1                       | Yes                        | No          | No                    | ~80 GB        | ~83s         |
+| 1                       | No                        | No          | No                    | ~80 GB        | ~96s         |
+| 1                       | Yes                        | Yes          | No                    | ~80 GB        | ~105s         |
+| 1                       | No                        | Yes          | No                    | ~32 GB        | ~118s         |
+| **1**                       | **Yes**                        | **Yes**          | **Yes**                    | **~32 GB**        | **~140s**         |
+| 4                       | Yes                        | No          | No                    | ~80 GB        | ~55s         |
+| 8                       | Yes                        | No          | No                    | ~80 GB        | ~40s         |
+### Gradio
+We provide a simple script to run our model in a gradio UI. It uses the `ckpt_dir` in `ovi/configs/inference/inference_fusion.yaml` to initialize the model
+```bash
+python3 gradio_app.py
+
+OR
+
+# To enable cpu offload to save GPU VRAM, will slow down end to end inference by ~20 seconds
+python3 gradio_app.py --cpu_offload
+
+OR
+
+# To enable an additional image generation model to generate first frames for I2V, cpu_offload is automatically enabled if image generation model is enabled
+python3 gradio_app.py --use_image_gen
+
+OR
+
+# To run model with 24Gb GPU vram
+python3 gradio_app.py --cpu_offload --fp8
+
+```
+---
+
+## 🙏 Acknowledgements
+
+We would like to thank the following projects:
+
+- **[Wan2.2](https://github.com/Wan-Video/Wan2.2)**: Our video branch is initialized from the Wan2.2 repository
+- **[MMAudio](https://github.com/hkchengrex/MMAudio)**: Our audio encoder and decoder components are borrowed from the MMAudio project. Some ideas are also inspired from them. 
+
+---
+
+## 🤝 Collaboration
+
+We welcome all types of collaboration! Whether you have feedback, want to contribute, or have any questions, please feel free to reach out.
+
+**Contact**: [Weimin Wang](https://linkedin.com/in/weimin-wang-will) for any issues or feedback.
+
+
+## 🤝 Contributors
+
+We thank all contributors who have helped improve Ovi!
+
+<div align="center">
+  <a href="https://github.com/character-ai/Ovi/graphs/contributors">
+    <img src="https://contrib.rocks/image?repo=character-ai/Ovi" />
+  </a>
+</div>
+
+<br>
+
+If you’ve contributed to this repository (code, documentation, issues, etc.), you’re automatically included in the [contributors list](https://github.com/character-ai/Ovi/graphs/contributors).  
+
+We deeply appreciate your support in advancing open multimodal generation research!
+---
+
+## ⭐ Citation
+
+If Ovi is helpful, please help to ⭐ the repo.
+
+If you find this project useful for your research, please consider citing our [paper](https://arxiv.org/abs/2510.01284).
+
+
+### BibTeX
+```bibtex
+@misc{low2025ovitwinbackbonecrossmodal,
+      title={Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation}, 
+      author={Chetwin Low and Weimin Wang and Calder Katyal},
+      year={2025},
+      eprint={2510.01284},
+      archivePrefix={arXiv},
+      primaryClass={cs.MM},
+      url={https://arxiv.org/abs/2510.01284}, 
+}
+```

config.json

@@ -0,0 +1 @@
+{"framework":"Pytorch","task":"image-to-video"}