| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - omnivae |
| - audio-video |
| - vae |
| - t2av |
| - diffusion |
| - synchronization |
| - multimodal-generation |
| --- |
| |
| # OmniVAE Release Assets |
|
|
| This Hugging Face repository contains the release checkpoints, text encoders, |
| evaluation data, and metric weights for the |
| [OmniVAE GitHub repository](https://github.com/OpenMOSS/OmniVAE). |
|
|
| ## Paper |
|
|
| **OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation** |
|
|
| Paper: [https://arxiv.org/abs/2607.23855](https://arxiv.org/abs/2607.23855) |
|
|
| The public generation weights in this release are T2AV-only. VAE weights are |
| included because they are required by the T2AV pipeline. |
|
|
| The source code is intentionally kept out of this repository. Use this bundle |
| with the GitHub code by setting `OMNIVAE_RELEASE_ROOT` to the local download |
| path. |
|
|
| ## Download |
|
|
| ```bash |
| pip install -U huggingface_hub |
| huggingface-cli download OpenMOSS-Team/OmniVAE \ |
| --repo-type model \ |
| --local-dir /path/to/omnivae_release \ |
| --local-dir-use-symlinks False |
| |
| git clone https://github.com/OpenMOSS/OmniVAE.git |
| cd OmniVAE |
| export OMNIVAE_RELEASE_ROOT=/path/to/omnivae_release |
| export OPEN_SOURCE_ROOT="${OMNIVAE_RELEASE_ROOT}" |
| ``` |
|
|
| Equivalently: |
|
|
| ```bash |
| source scripts/setup_release_root.sh /path/to/omnivae_release |
| ``` |
|
|
| ## Layout |
|
|
| ```text |
| omnivae_release/ |
| manifest.json |
| models/ |
| text_encoder/ frozen Qwen3.5 text encoder used by T2AV inference |
| vae/ OmniVAE checkpoints |
| dit/t2av/ T2AV generation checkpoints |
| eval/ |
| data/t2av/ validation manifests and small reference data |
| models/t2av/ metric weights for T2AV evaluation |
| ``` |
|
|
| All config and metadata paths are relative to the repository root, so the code |
| can load this directory from any local path. |
|
|
| ## Text Encoders |
|
|
| The release includes the frozen Qwen3.5 text encoder directory used by the |
| validated T2AV configs: |
|
|
| - `models/text_encoder/Qwen3.5-0.8B-Base` |
|
|
| It is a third-party dependency weight, not an additional OmniVAE checkpoint. |
| Keeping it preserves the exact inference/evaluation settings used for the |
| released results. |
|
|
| ## Checkpoints |
|
|
| ### VAE |
|
|
| - `models/vae/video_only/recon` |
| - `models/vae/video_only/recon_distill` |
| - `models/vae/audio_only/recon` |
| - `models/vae/audio_only/recon_distill` |
| - `models/vae/audio_video/recon_avclip` |
| - `models/vae/audio_video/recon_distill_avclip` |
| - `models/vae/audio_only/recon_avclip_ft_decoder` |
| - `models/vae/audio_only/recon_distill_avclip_ft_decoder` |
|
|
| For T2AV AVCLIP experiment families, the audio branch uses the |
| `*_ft_decoder` checkpoint by default. |
|
|
| ### Generation DiT |
|
|
| The public bundle contains T2AV generation packages only: |
|
|
| ```text |
| t2av: t2av_recon, t2av_recon_distill, t2av_recon_avclip, t2av_recon_distill_avclip |
| ``` |
|
|
| Each package is inference-only and excludes optimizer, scheduler, dataloader, |
| random state, and trainer state snapshots. |
|
|
| ## Evaluation Assets |
|
|
| - `eval/data/t2av/versebench_minimal`: VerseBench validation manifest used by |
| the release smoke and comparison scripts. |
| - `eval/models/t2av`: T2AV metric weights for the bundled `my_eval` pipeline. |
|
|
| ### rFVD I3D Weight |
|
|
| The rFVD I3D TorchScript weight is not included in this Hugging Face bundle. |
| When it is not already present in the installed OmniVAE package, the rFVD |
| loader attempts to download it from the upstream Dropbox URL automatically. |
| For offline or reproducible evaluation, download the weight separately and |
| provide its path explicitly: |
|
|
| ```bash |
| export I3D_TORCHSCRIPT_PT=/path/to/i3d_torchscript.pt |
| bash vae/scripts/eval/run_video_eval_rfvd_only.sh |
| ``` |
|
|
| ## Quick Validation |
|
|
| ```bash |
| cd /path/to/OmniVAE |
| export OMNIVAE_RELEASE_ROOT=/path/to/omnivae_release |
| |
| # T2AV release validation on VerseBench set3-large. |
| bash scripts/release_launchers/run_t2av_release_compare_distributed.sh |
| ``` |
|
|
| See the GitHub repository `README.md` and `docs/` for full installation, |
| training, inference, and evaluation commands. |
|
|
| ## License |
|
|
| The OmniVAE release assets are provided under the Apache-2.0 license. Some |
| metric packages and third-party components retain their original licenses; see |
| the corresponding source-code subdirectories in the GitHub repository. |
|
|
