| --- |
| license: other |
| license_name: ltx-2 |
| license_link: https://github.com/Lightricks/LTX-2/blob/main/LICENSE |
| tags: |
| - video |
| - vae |
| - decoder |
| - ltx |
| - ltx-2.3 |
| - pruned |
| - diffusers |
| - image-to-video |
| - text-to-video |
| library_name: diffusers |
| base_model: diffusers/LTX-2.3-Diffusers |
| pipeline_tag: text-to-video |
| --- |
| <!-- header start --> |
| <!-- 200823 --> |
| <div style="width: auto; margin-left: auto; margin-right: auto"> |
| <a href="https://docs.pruna.ai/en/stable/index.html" target="_blank" rel="noopener noreferrer"> |
| <img src="https://imgur.com/rVAgqMY.png" alt="PrunaAI" style="width: 100%; min-width: 400px; display: block; margin: auto;"> |
| </a> |
| </div> |
| <!-- header end --> |
| |
| [](https://github.com/PrunaAI/pruna) |
| [](https://twitter.com/PrunaAI) |
| [](https://www.linkedin.com/company/pruna-ai) |
| [](https://discord.com/invite/JFQmtFKCjd) |
| [](https://dashboard.pruna.ai/login?utm_source=huggingface&utm_medium=org_card&utm_campaign=hf_traffic) |
|
|
| <div align="center"> |
|
|
| <h1 style="color: #9334E9;">⚡ PrunaVAED</h1> |
|
|
| <h2>A drop-in replacement decoder for LTX-2.3</h2> |
|
|
| <h3> |
| <span style="color: #9334E9;">1.7× faster</span> |
| · |
| <span style="color: #9334E9;">~50% lower peak VRAM</span> |
| · |
| Near-original visual quality |
| </h3> |
|
|
| </div> |
|
|
| **PrunaVAED directly replaces the video VAE decoder in |
| [`diffusers/LTX-2.3-Diffusers`](https://huggingface.co/diffusers/LTX-2.3-Diffusers). |
| The encoder and latent format remain unchanged, making it a drop-in upgrade |
| for faster, more memory-efficient LTX-2.3 decoding.** |
|
|
| ## Examples |
|
|
| Both columns decode the same LTX-2.3 latent from the distilled pipeline. |
|
|
| <table> |
| <tr> |
| <th align="center" width="50%">LTX-2.3 decoder</th> |
| <th align="center" width="50%">⚡ PrunaVAED</th> |
| </tr> |
| <tr> |
| <td width="50%"> |
| <video controls autoplay muted loop playsinline |
| style="width:100%; display:block;" |
| src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/04MhZs7YN08-Scene-0048.mp4"> |
| </video> |
| </td> |
| <td width="50%"> |
| <video controls autoplay muted loop playsinline |
| style="width:100%; display:block;" |
| src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/04MhZs7YN08-Scene-0048.mp4"> |
| </video> |
| </td> |
| </tr> |
| <tr> |
| <td width="50%"> |
| <video controls autoplay muted loop playsinline |
| style="width:100%; display:block;" |
| src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/FAfMSWi0FSA-Scene-0200.mp4"> |
| </video> |
| </td> |
| <td width="50%"> |
| <video controls autoplay muted loop playsinline |
| style="width:100%; display:block;" |
| src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/FAfMSWi0FSA-Scene-0200.mp4"> |
| </video> |
| </td> |
| </tr> |
| </table> |
| |
| <details> |
| <summary><strong>More side-by-side comparisons</strong></summary> |
|
|
| <br> |
|
|
| <table> |
| <tr> |
| <th align="center" width="50%">LTX-2.3 decoder</th> |
| <th align="center" width="50%">⚡ PrunaVAED</th> |
| </tr> |
| <tr> |
| <td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/8YSha7iU2ZI-Scene-1025.mp4"></video></td> |
| <td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/8YSha7iU2ZI-Scene-1025.mp4"></video></td> |
| </tr> |
| <tr> |
| <td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/DZt-RtpcJZQ-Scene-0043.mp4"></video></td> |
| <td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/DZt-RtpcJZQ-Scene-0043.mp4"></video></td> |
| </tr> |
| <tr> |
| <td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/9GDE5-rOOfk-Scene-0010.mp4"></video></td> |
| <td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/9GDE5-rOOfk-Scene-0010.mp4"></video></td> |
| </tr> |
| <tr> |
| <td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/7y57zc-GvBA-Scene-0009.mp4"></video></td> |
| <td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/7y57zc-GvBA-Scene-0009.mp4"></video></td> |
| </tr> |
| </table> |
| </details> |
| |
|
|
| ## Benchmark |
|
|
| Metrics compare videos decoded from the **same latents** by the LTX-2.3 VAED and PrunaVAED. Two set of latents were |
| generated with the full |
| [`ti2vid_two_stages`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py) |
| pipeline and the |
| [`distilled`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/distilled.py) |
| pipeline (recommended). |
|
|
| <table class="w-full table-fixed"> |
| <tr><th>Setting</th><th>Value</th></tr> |
| <tr><td>Model</td><td>PrunaVAED</td></tr> |
| <tr><td>Precision</td><td><code>bfloat16</code></td></tr> |
| <tr><td>Batch size</td><td>1</td></tr> |
| <tr><td>Device</td><td>NVIDIA H100 80GB</td></tr> |
| <tr><td>Decode</td><td>Full (no tiling / rolling / <code>torch.compile</code>)</td></tr> |
| </table> |
|
|
| <table class="w-full table-fixed"> |
| <tr> |
| <th>Clip length</th><th>LTX-2.3 VAED</th><th>PrunaVAED</th><th>Speedup</th> |
| </tr> |
| <tr> |
| <td>5 s @ 720p</td><td align="right">841.3 ms</td><td align="right">500.2 ms</td><td align="right"><strong>1.68×</strong></td> |
| </tr> |
| <tr> |
| <td>10 s @ 720p</td><td align="right">1670.7 ms</td><td align="right">998.0 ms</td><td align="right"><strong>1.67×</strong></td> |
| </tr> |
| <tr> |
| <td>5 s @ 1080p</td><td align="right">1959.3 ms</td><td align="right">1152.1 ms</td><td align="right"><strong>1.7×</strong></td> |
| </tr> |
| <tr> |
| <td>10 s @ 1080p</td><td align="right">4903 ms*</td><td align="right">2356 ms</td><td align="right"><strong>2.08×</strong></td> |
| </tr> |
| </table> |
| |
| \* Tiling required on H100; 14533ms otherwise |
|
|
| <table class="w-full table-fixed"> |
| <tr> |
| <th>Latent set</th><th>Clips</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>SSIM ↑</th><th>Δ-frame PSNR ↑</th> |
| </tr> |
| <tr> |
| <td>Distilled two-stage · 720p</td><td align="right">200</td><td align="right"><strong>39.23</strong></td><td align="right"><strong>0.0087</strong></td><td align="right"><strong>0.9811</strong></td><td align="right"><strong>38.32</strong></td> |
| </tr> |
| <tr> |
| <td>TI2Vid two-stage · 720p</td><td align="right">200</td><td align="right"><strong>40.34</strong></td><td align="right"><strong>0.0094</strong></td><td align="right"><strong>0.9823</strong></td><td align="right"><strong>40.06</strong></td> |
| </tr> |
| <tr> |
| <td>Distilled two-stage · 1080p</td><td align="right">200</td><td align="right"><strong>41.06</strong></td><td align="right"><strong>0.0052</strong></td><td align="right"><strong>0.9876</strong></td><td align="right"><strong>40.43</strong></td> |
| </tr> |
| </table> |
| |
| **Δ-frame PSNR** is PSNR on consecutive-frame |
| differences (temporal consistency). |
| |
| ## Quickstart |
|
|
| End-to-end smoke demo: generate a short video with the diffusers LTX-2.3 |
| **distilled** two-stage pipeline, then decode the same latent with the stock |
| LTX-2.3 VAE and with PrunaVAED. Writes two mp4s and prints decode time (ms). |
| Needs a **CUDA GPU**. |
|
|
| ```bash |
| # 1. Install the Hugging Face CLI |
| pip install hf |
| |
| # 2. Download this repo |
| hf download PrunaAI/PrunaVAED --local-dir PrunaVAED |
| cd PrunaVAED |
| |
| # 3. Install dependencies |
| pip install -r requirements-demo.txt |
| |
| # 4. Run the demo (~1080p, ~5 s @ 24 fps) |
| python demo/demo_distilled_decode.py |
| ``` |
|
|
| Outputs land in `outputs/demo_distilled/` (`ltx23.mp4`, `prunavaed.mp4`). |
| Edit `PROMPT` / resolution at the top of `demo/demo_distilled_decode.py` if needed. |
|
|
|
|
| ## Architecture |
|
|
| PrunaVAED is bitwise identical to LTX 2.3 VAED from `conv_in` to `up_blocks.0`. Pruning starts |
| at `up_blocks.1`. |
|
|
| | Stage | Channel reduction | |
| |---|---:| |
| | `up_blocks.0` | unchanged | |
| | `up_blocks.1` | 25% | |
| | `up_blocks.2` | 50% | |
| | `up_blocks.3` | 50% | |
|
|
| | | Teacher (LTX-2.3) | PrunaVAED | |
| |---|---:|---:| |
| | Encoder params | 318.9 M | 318.9 M (unchanged) | |
| | Decoder params | 407.2 M | 345.0 M (−15%) | |
| | **Total VAE params** | **726.1 M** | **663.9 M** | |
|
|
| `prunavaed/patch_diffusers.py` provides the required compatibility shim for |
| the pinned diffusers version. |
|
|
| ## Limitations |
|
|
| - Evaluated on 4-second clips at 24 fps. |
| - Benchmarked on one H100 80GB with bfloat16 and batch size 1. |
| - Speed and VRAM vary with hardware, resolution, batch size, and software. |
| - Evaluated only on two differents pipeline of LTX 2.3 ([`ti2vid_two_stages`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py) |
| pipeline and the |
| [`distilled`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/distilled.py)) |
| - The weights can be adapted to the custom [`LTX-2`](https://github.com/Lightricks/LTX-2) library without retraining, but no implementation is provided for this yet. |
|
|
| ## What this is not |
|
|
| - Not a full T2V model. |
| - Not a replacement for the LTX denoiser/generator. |
| - Not compatible with arbitrary VAE latents. |
| - Not bit-exact with the original decoder. |
|
|
| ## License |
|
|
| PrunaVAED is a derivative of LTX-2.3 and is distributed under the |
| **[LTX-2 Community License Agreement](LICENSE)**. Review its use restrictions |
| and commercial terms before using or redistributing the model. |
|
|
| Helper code adapted from Hugging Face diffusers retains its Apache-2.0 |
| attribution; see [`NOTICE`](NOTICE). |
|
|
| ## What's next? |
|
|
| - **Use PrunaVAED to speed up LTX-2.3 decoding.** |
| - Compress your own models with [Pruna](https://github.com/PrunaAI/pruna) and give us a ⭐️ for more efficiency! |
| - Want to use our optimized video model right away? Check |
| [P-Video](https://www.pruna.ai/p-video) and |
| [P-Video documentation](https://docs.pruna.ai/en/stable/docs_pruna_endpoints/performance_models/p-video.html). |
|
|
| <style> |
| .model-button { |
| display: inline-flex; |
| flex-direction: row; |
| justify-content: center; |
| align-items: center; |
| gap: 8px; |
|
|
| padding: 8px 20px; |
| border: none; |
| border-radius: 8px; |
|
|
| background: #9334e9; |
| color: #ffffff; |
|
|
| font-size: 14px; |
| font-weight: 400; |
| line-height: 1; |
| text-decoration: none; |
| white-space: nowrap; |
| cursor: pointer; |
|
|
| box-sizing: border-box; |
| overflow: visible; |
| opacity: 1; |
| } |
| </style> |
|
|
| <a href="https://dashboard.pruna.ai/login?utm_source=huggingface&utm_medium=model_card&utm_campaign=hf_traffic" class="model-button">Try our models</a> |
