Upload README.md with huggingface_hub

README.md

@@ -1,3 +1,137 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-to-video
+tags:
+- video generation
+library_name: diffusers
+inference:
+  parameters:
+    num_inference_steps: 10
+---
+# Text2Motion
+
+
+-----
+
+[**Text2Motion: Open and Advanced Large-Scale Video Generative Models**]("") <be>
+
+In this repository, we present **Text2Motion**, a comprehensive and open suite of video foundation models that pushes the boundaries of video generation. **Text2Motion** offers these key features:
+- 👍 **SOTA Performance**: **Text2Motion** consistently outperforms existing open-source models and state-of-the-art commercial solutions across multiple benchmarks.
+- 👍 **Supports Consumer-grade GPUs**: The T2V-1.3B model requires only 8.19 GB VRAM, making it compatible with almost all consumer-grade GPUs. It can generate a 5-second 480P video on an RTX 4090 in about 4 minutes (without optimization techniques like quantization). Its performance is even comparable to some closed-source models.
+- 👍 **Multiple Tasks**: **Text2Motion** excels in Text-to-Video, Image-to-Video, Video Editing, Text-to-Image, and Video-to-Audio, advancing the field of video generation.
+- 👍 **Visual Text Generation**: **Text2Motion** is the first video model capable of generating both Chinese and English text, featuring robust text generation that enhances its practical applications.
+- 👍 **Powerful Video VAE**: **Text2Motion-VAE** delivers exceptional efficiency and performance, encoding and decoding 1080P videos of any length while preserving temporal information, making it an ideal foundation for video and image generation.
+
+This repository features our T2V-14B model, which establishes a new SOTA performance benchmark among both open-source and closed-source models. It demonstrates exceptional capabilities in generating high-quality visuals with significant motion dynamics. It is also the only video model capable of producing both Chinese and English text and supports video generation at both 480P and 720P resolutions.
+
+
+## 🔥 Latest News!!
+
+* Feb 22, 2025: 👋 We've released the inference code and weights of Text2Motion.
+
+
+## 📑 Todo List
+- Text2Motion Text-to-Video
+    - [x] Multi-GPU Inference code of the 14B
+    - [x] Checkpoints of the 14B
+    - [x] Gradio demo
+    - [ ] Diffusers integration
+    - [ ] ComfyUI integration
+- Text2Motion Image-to-Video
+    - [x] Multi-GPU Inference code of the 14B model
+    - [x] Checkpoints of the 14B model
+    - [x] Gradio demo
+    - [ ] Diffusers integration
+    - [ ] ComfyUI integration
+
+
+## Quickstart
+
+#### Installation
+Clone the repo:
+```
+git clone https://huggingface.co/sbapan41/Text2Motion
+cd Text2Motion
+```
+
+Install dependencies:
+```
+# Ensure torch >= 2.4.0
+pip install -r requirements.txt
+```
+
+
+#### Model Download
+
+| Models        |                       Download Link                                           |    Notes                      |
+| --------------|-------------------------------------------------------------------------------|-------------------------------|
+| T2V-14B       |      🤗 [Huggingface](https://huggingface.co/sbapan41/Text2Motion)            | Supports both 480P and 720P   |
+
+
+
+
+Download models using 🤗 huggingface-cli:
+```
+pip install "huggingface_hub[cli]"
+huggingface-cli download sbapan41/Text2Motion --local-dir ./Text2Motion
+```
+#### Run Text-to-Video Generation
+
+This repository supports two Text-to-Video models (14B) and two resolutions (480P and 720P). The parameters and configurations for these models are as follows:
+
+<table>
+    <thead>
+        <tr>
+            <th rowspan="2">Task</th>
+            <th colspan="2">Resolution</th>
+            <th rowspan="2">Model</th>
+        </tr>
+        <tr>
+            <th>480P</th>
+            <th>720P</th>
+        </tr>
+    </thead>
+    <tbody>
+        <tr>
+            <td>t2v-14B</td>
+            <td style="color: green;">✔️</td>
+            <td style="color: green;">✔️</td>
+            <td>Text2Motion-T2V-14B</td>
+        </tr>
+        <tr>
+            <td>t2v-1.3B</td>
+            <td style="color: green;">✔️</td>
+            <td style="color: red;">❌</td>
+            <td>Text2Motion-T2V-1.3B</td>
+        </tr>
+    </tbody>
+</table>
+
+
+##### (1) Without Prompt Extention
+
+To facilitate implementation, we will start with a basic version of the inference process that skips the [prompt extension](#2-using-prompt-extention) step.
+
+- Single-GPU inference
+
+```
+python generate.py  --task 14B --size 1280*720 --ckpt_dir ./Text2Motion --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
+```
+
+If you encounter OOM (Out-of-Memory) issues, you can use the `--offload_model True` and `--t5_cpu` options to reduce GPU memory usage. For example, on an RTX 4090 GPU:
+
+```
+
+- Multi-GPU inference using FSDP + xDiT USP
+
+```
+pip install "xfuser>=0.4.1"
+torchrun --nproc_per_node=8 generate.py --task 14B --size 1280*720 --ckpt_dir ./Text2Motion --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
+```
+
+
+| Model  | Dimension | Input Dimension | Output Dimension | Feedforward Dimension | Frequency Dimension | Number of Heads | Number of Layers |
+|--------|-----------|-----------------|------------------|-----------------------|---------------------|-----------------|------------------|
+| 14B    | 5120      | 16              | 16               | 13824                  | 256                 | 40              | 40              |