| --- |
| title: Video Model Studio |
| emoji: 🎥 |
| colorFrom: gray |
| colorTo: gray |
| sdk: gradio |
| sdk_version: 5.33.1 |
| app_file: app.py |
| pinned: true |
| license: apache-2.0 |
| short_description: All-in-one tool for AI video training |
| --- |
| |
| # 🎥 Video Model Studio (VMS) |
|
|
|  |
|
|
| ## Presentation |
|
|
| ### What is this project? |
|
|
| VMS is a Gradio app that wraps around Finetrainers, to provide a simple UI to train AI video models on Hugging Face. |
|
|
| You can deploy it to a private space, and start long-running training jobs in the background. |
|
|
| ## Funding |
|
|
| VideoModelStudio is 100% open-source project, I develop and maintain it during both my pro and personal time. If you like it, you can tip! If not, have a good day 🫶 |
|
|
| <a href="https://www.buymeacoffee.com/flngr" target="_blank"><img src="https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png" alt="Buy Me A Coffee" style="height: 41px !important;width: 174px !important;box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;-webkit-box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;" ></a> |
|
|
| ## News |
| - 🔥 **2025-06-10**: Fixed a bug with model download, upgrade to 5.33.1 |
| - 🔥 **2025-06-04**: Upgrade to Cuda 12.8, Pytorch 2.6, Gradio 5.32 |
| - 🔥 **2025-03-12**: VMS now officially supports Wan! |
| - 🔥 **2025-03-11**: I have added a tab to preview a model! |
| - 🔥 **2025-03-10**: Various small fixes and improvements |
| - 🔥 **2025-03-09**: I have added a basic CPU/RAM monitor (no GPU yet) |
| - 🔥 **2025-03-02**: Made some fixes to improve Finetrainer reliability when working with big datasets |
| - 🔥 **2025-02-18**: I am working to add better recovery in case of a failed run (this is still in beta) |
| - 🔥 **2025-02-18**: I have added persistence of UI settings. So if you reload Gradio, you won't lose your settings! |
|
|
| ## TODO |
| - Add `Aya-Vision-8B` for frame analysis (currently we use `Qwen2-VL-7B`) |
|
|
| ### See also |
|
|
| #### Internally used project: Finetrainers |
|
|
| VMS uses Finetrainers under the hood: https://github.com/a-r-r-o-w/finetrainers |
|
|
| #### Similar project: diffusion-pipe-ui |
|
|
| I wasn't aware of its existence when I started my project, but there is also this open-source initiative (which is similar in terms of dataset management etc): https://github.com/alisson-anjos/diffusion-pipe-ui |
|
|
| ## Features |
|
|
| ### Run Finetrainers in the background |
|
|
| The main feature of VMS is the ability to run a Finetrainers training session in the background. |
|
|
| You can start your job, close the web browser tab, and come back the next morning to see the result. |
|
|
| ### Automatic scene splitting |
|
|
| VMS uses PySceneDetect to split scenes. |
|
|
| ### Automatic clip captioning |
|
|
| VMS uses `LLaVA-Video-7B-Qwen2` for captioning. You can customize the system prompt if you want to. |
|
|
| ### Download your dataset |
|
|
| Not interested in using VMS for training? That's perfectly fine! |
|
|
| You can use VMS for video splitting and captioning, and export the data for training on another platform eg. on Replicate or Fal. |
|
|
| ## Supported models |
|
|
| VMS uses `Finetrainers` under the hood. In theory any model supported by Finetrainers should work in VMS. |
|
|
| In practice, a PR (pull request) will be necessary to adapt the UI a bit to accomodate for each model specificities. |
|
|
|
|
| ### Wan |
|
|
| I am currently testing Wan LoRA training! |
|
|
| ### LTX-Video |
|
|
| I have tested training a LTX-Video LoRA model using videos (not images), on a single A100 instance. |
| It requires about 18/19 Gb of VRAM, depending on your settings. |
|
|
| ### HunyuanVideo |
|
|
| I have tested training a HunyuanVideo LoRA model using videos (not images),, on a single A100 instance. |
|
|
| It requires about 47~49 Gb of VRAM, depending on your settings. |
|
|
| ### CogVideoX |
|
|
| Do you want support for this one? Let me know in the comments! |
|
|
| ## Limitations |
|
|
| ### No AV1 on A100 |
|
|
| If your dataset contains videos encoded using the AV1 codec, you might not be able to decode them (eg. during scene splitting) if your machine doesn't support hardware decoding. |
|
|
| Nvidia A100 don't support hardware AV1 decoding for instance. |
|
|
| It might be possible to convert them on server-side or use software decoding directly from Python, but I haven't looked into that yet (you can submit a PR if you have an idea). |
|
|
| My recommendation is to make sure your data comes in h264. |
|
|
| You can use FFmpeg to do this, eg: |
|
|
| ```bash |
| ffmpeg -i input_video_in_av1.mp4 -vcodec libx264 -acodec aac output_video_in_h264.mp4 |
| ``` |
|
|
| ### One-user-per-space design |
|
|
| Currently CMS can only support one training job at a time, anybody with access to your Gradio app will be able to upload or delete everything etc. |
|
|
| This means you have to run VMS in a *PRIVATE* HF Space, or locally if you require full privacy. |
|
|
| ## Deployment |
|
|
| VMS is built on top of Finetrainers and Gradio, and designed to run as a Hugging Face Space (but you can deploy it anywhere that has a NVIDIA GPU and supports Docker). |
|
|
| ### Full installation at Hugging Face |
|
|
| Easy peasy: create a Space (make sure to use the `Gradio` type/template), and push the repo. No Docker needed! |
|
|
| That said, please see the "RUN" section for info about environement variables. |
|
|
| ### Dev mode on Hugging Face |
|
|
| I recommend to not use the dev mode for a production usage (ie not use dev mode when training a real model), unless you know what you are doing. |
|
|
| That's because the dev mode can be unstable and cause space restarts. |
|
|
| If you still want to open the dev mode in the space, then open VSCode in local or remote and run: |
|
|
| ``` |
| pip install -r requirements.txt |
| ``` |
|
|
| As this is not automatic, then click on "Restart" in the space dev mode UI widget. |
|
|
| Important: if you see errors like "API not found" etc, it might indicate an issue with the dev mode and Gradio, not an issue with VMS itself. |
|
|
| ### Full installation somewhere else |
|
|
| I haven't tested it, but you can try to provided Dockerfile |
|
|
| ### Prerequisites |
|
|
| About Python: |
|
|
| I haven't tested Python 3.11 or 3.12, but I noticed some incompatibilities with Python 3.13 dependencies failing to install. |
|
|
| So I recommend you to install [pyenv](https://github.com/pyenv/pyenv) to switch between versions of Python. |
|
|
| If you are on macOS, you might already have some versions of Python installed, you can see them by typing: |
|
|
| ```bash |
| % python3.10 --version |
| Python 3.10.16 |
| % python3.11 --version |
| Python 3.11.11 |
| % python3.12 --version |
| Python 3.12.9 |
| % python3.13 --version |
| Python 3.13.2 |
| ``` |
|
|
| Once pyenv is installed you can type: |
|
|
| ```bash |
| pyenv install 3.10.16 |
| ``` |
|
|
| ### Full installation in local |
|
|
| the full installation requires: |
| - Linux |
| - CUDA 12 |
| - Python 3.10 |
|
|
| This is because of flash attention, which is defined in the `requirements.txt` using an URL to download a prebuilt wheel expecting this exact configuration (python bindings for a native library) |
|
|
| ```bash |
| ./setup.sh |
| ``` |
|
|
| ### macOS is NOT supported |
|
|
| Currently macOS is NOT supported due to two separate dependency issues: |
|
|
| - issue with flash attention |
| - issue with decord |
|
|
| However, the paradox is that I actually develop and maintain VMS using macOS! |
|
|
| So don't be surprised by the presence of "degraded_setup.sh" and all, it's just a workaround for me to be able to at least test the web ui on macOS. |
| |
| ## Run |
| |
| ### Running the Gradio app |
| |
| Note: please make sure you properly define the environment variables for `STORAGE_PATH` (eg. `/data/`) and `HF_HOME` (eg. `/data/huggingface/`) |
| |
| ```bash |
| python3.10 app.py |
| ``` |
| |
| ### Running locally |
| |
| See above remarks about the environment variable. |
| |
| By default `run.sh` will store stuff in `.data/` (located inside the current working directory): |
| |
| ```bash |
| ./run.sh |
| ``` |
| |
| ### Environment Variables |
| |
| - `STORAGE_PATH`: Specifies the base storage path (default: '.data') |
| - `HF_API_TOKEN`: Your Hugging Face API token for accessing models and publishing |
| - `USE_LARGE_DATASET`: Set to "true" or "1" to enable large dataset mode, which: |
| - Hides the caption list in the caption tab |
| - Disables preview and editing of individual captions |
| - Disables the dataset download button |
| - Use this when working with large datasets that would be too slow to display in the UI |
| - `PRELOAD_CAPTIONING_MODEL`: Preloads the captioning model at startup |
| - `ASK_USER_TO_DUPLICATE_SPACE`: Prompts users to duplicate the space |
|
|
