File size: 16,886 Bytes
e9932d5 9e752d2 0d01c91 9e752d2 0d01c91 e9932d5 4baacd7 e9932d5 7f05a4f e9932d5 7f05a4f e9932d5 7f05a4f 7ed6358 e9932d5 7f05a4f e9932d5 7f05a4f 7ed6358 e9932d5 7f05a4f e9932d5 7f05a4f 7ed6358 e9932d5 7f05a4f e9932d5 7f05a4f 7ed6358 e9932d5 7f05a4f e9932d5 7f05a4f e9932d5 7f05a4f e9932d5 7f05a4f e9932d5 7f05a4f e9932d5 9626294 e9932d5 9aa003c 287f858 9aa003c 287f858 9aa003c 287f858 e730d5b 287f858 e730d5b 287f858 e730d5b 287f858 e9932d5 7b2d9de e9932d5 7ed6358 e9932d5 d64f260 e9932d5 3c342fd e9932d5 7ed6358 7b2d9de e9932d5 7ed6358 df69250 e9932d5 9aa003c e9932d5 9aa003c e9932d5 9aa003c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 | ---
license: other
license_name: ltx-2
license_link: https://github.com/Lightricks/LTX-2/blob/main/LICENSE
tags:
- video
- vae
- decoder
- ltx
- ltx-2.3
- pruned
- diffusers
- image-to-video
- text-to-video
library_name: diffusers
base_model: diffusers/LTX-2.3-Diffusers
pipeline_tag: text-to-video
---
<!-- header start -->
<!-- 200823 -->
<div style="width: auto; margin-left: auto; margin-right: auto">
<a href="https://docs.pruna.ai/en/stable/index.html" target="_blank" rel="noopener noreferrer">
<img src="https://imgur.com/rVAgqMY.png" alt="PrunaAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
</a>
</div>
<!-- header end -->
[](https://github.com/PrunaAI/pruna)
[](https://twitter.com/PrunaAI)
[](https://www.linkedin.com/company/pruna-ai)
[](https://discord.com/invite/JFQmtFKCjd)
[](https://dashboard.pruna.ai/login?utm_source=huggingface&utm_medium=org_card&utm_campaign=hf_traffic)
<div align="center">
<h1 style="color: #9334E9;">⚡ PrunaVAED</h1>
<h2>A drop-in replacement decoder for LTX-2.3</h2>
<h3>
<span style="color: #9334E9;">1.7× faster</span>
·
<span style="color: #9334E9;">~50% lower peak VRAM</span>
·
Near-original visual quality
</h3>
</div>
**PrunaVAED directly replaces the video VAE decoder in
[`diffusers/LTX-2.3-Diffusers`](https://huggingface.co/diffusers/LTX-2.3-Diffusers).
The encoder and latent format remain unchanged, making it a drop-in upgrade
for faster, more memory-efficient LTX-2.3 decoding.**
## Examples
Both columns decode the same LTX-2.3 latent from the distilled pipeline.
<table>
<tr>
<th align="center" width="50%">LTX-2.3 decoder</th>
<th align="center" width="50%">⚡ PrunaVAED</th>
</tr>
<tr>
<td width="50%">
<video controls autoplay muted loop playsinline
style="width:100%; display:block;"
src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/04MhZs7YN08-Scene-0048.mp4">
</video>
</td>
<td width="50%">
<video controls autoplay muted loop playsinline
style="width:100%; display:block;"
src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/04MhZs7YN08-Scene-0048.mp4">
</video>
</td>
</tr>
<tr>
<td width="50%">
<video controls autoplay muted loop playsinline
style="width:100%; display:block;"
src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/FAfMSWi0FSA-Scene-0200.mp4">
</video>
</td>
<td width="50%">
<video controls autoplay muted loop playsinline
style="width:100%; display:block;"
src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/FAfMSWi0FSA-Scene-0200.mp4">
</video>
</td>
</tr>
</table>
<details>
<summary><strong>More side-by-side comparisons</strong></summary>
<br>
<table>
<tr>
<th align="center" width="50%">LTX-2.3 decoder</th>
<th align="center" width="50%">⚡ PrunaVAED</th>
</tr>
<tr>
<td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/8YSha7iU2ZI-Scene-1025.mp4"></video></td>
<td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/8YSha7iU2ZI-Scene-1025.mp4"></video></td>
</tr>
<tr>
<td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/DZt-RtpcJZQ-Scene-0043.mp4"></video></td>
<td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/DZt-RtpcJZQ-Scene-0043.mp4"></video></td>
</tr>
<tr>
<td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/9GDE5-rOOfk-Scene-0010.mp4"></video></td>
<td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/9GDE5-rOOfk-Scene-0010.mp4"></video></td>
</tr>
<tr>
<td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/Ltx23VAED/7y57zc-GvBA-Scene-0009.mp4"></video></td>
<td width="50%"><video controls autoplay muted loop playsinline style="width:100%; display:block;" src="https://huggingface.co/PrunaAI/PrunaVAED/resolve/main/example/PrunaVAED/7y57zc-GvBA-Scene-0009.mp4"></video></td>
</tr>
</table>
</details>
## Benchmark
Metrics compare videos decoded from the **same latents** by the LTX-2.3 VAED and PrunaVAED. Two set of latents were
generated with the full
[`ti2vid_two_stages`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py)
pipeline and the
[`distilled`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/distilled.py)
pipeline (recommended).
<table class="w-full table-fixed">
<tr><th>Setting</th><th>Value</th></tr>
<tr><td>Model</td><td>PrunaVAED</td></tr>
<tr><td>Precision</td><td><code>bfloat16</code></td></tr>
<tr><td>Batch size</td><td>1</td></tr>
<tr><td>Device</td><td>NVIDIA H100 80GB</td></tr>
<tr><td>Decode</td><td>Full (no tiling / rolling / <code>torch.compile</code>)</td></tr>
</table>
<table class="w-full table-fixed">
<tr>
<th>Clip length</th><th>LTX-2.3 VAED</th><th>PrunaVAED</th><th>Speedup</th>
</tr>
<tr>
<td>5 s @ 720p</td><td align="right">841.3 ms</td><td align="right">500.2 ms</td><td align="right"><strong>1.68×</strong></td>
</tr>
<tr>
<td>10 s @ 720p</td><td align="right">1670.7 ms</td><td align="right">998.0 ms</td><td align="right"><strong>1.67×</strong></td>
</tr>
<tr>
<td>5 s @ 1080p</td><td align="right">1959.3 ms</td><td align="right">1152.1 ms</td><td align="right"><strong>1.7×</strong></td>
</tr>
<tr>
<td>10 s @ 1080p</td><td align="right">4903 ms*</td><td align="right">2356 ms</td><td align="right"><strong>2.08×</strong></td>
</tr>
</table>
\* Tiling required on H100; 14533ms otherwise
<table class="w-full table-fixed">
<tr>
<th>Latent set</th><th>Clips</th><th>PSNR ↑</th><th>LPIPS ↓</th><th>SSIM ↑</th><th>Δ-frame PSNR ↑</th>
</tr>
<tr>
<td>Distilled two-stage · 720p</td><td align="right">200</td><td align="right"><strong>39.23</strong></td><td align="right"><strong>0.0087</strong></td><td align="right"><strong>0.9811</strong></td><td align="right"><strong>38.32</strong></td>
</tr>
<tr>
<td>TI2Vid two-stage · 720p</td><td align="right">200</td><td align="right"><strong>40.34</strong></td><td align="right"><strong>0.0094</strong></td><td align="right"><strong>0.9823</strong></td><td align="right"><strong>40.06</strong></td>
</tr>
<tr>
<td>Distilled two-stage · 1080p</td><td align="right">200</td><td align="right"><strong>41.06</strong></td><td align="right"><strong>0.0052</strong></td><td align="right"><strong>0.9876</strong></td><td align="right"><strong>40.43</strong></td>
</tr>
</table>
**Δ-frame PSNR** is PSNR on consecutive-frame
differences (temporal consistency).
## Quickstart
End-to-end smoke demo: generate a short video with the diffusers LTX-2.3
**distilled** two-stage pipeline, then decode the same latent with the stock
LTX-2.3 VAE and with PrunaVAED. Writes two mp4s and prints decode time (ms).
Needs a **CUDA GPU**.
```bash
# 1. Install the Hugging Face CLI
pip install hf
# 2. Download this repo
hf download PrunaAI/PrunaVAED --local-dir PrunaVAED
cd PrunaVAED
# 3. Install dependencies
pip install -r requirements-demo.txt
# 4. Run the demo (~1080p, ~5 s @ 24 fps)
python demo/demo_distilled_decode.py
```
Outputs land in `outputs/demo_distilled/` (`ltx23.mp4`, `prunavaed.mp4`).
Edit `PROMPT` / resolution at the top of `demo/demo_distilled_decode.py` if needed.
## Architecture
PrunaVAED is bitwise identical to LTX 2.3 VAED from `conv_in` to `up_blocks.0`. Pruning starts
at `up_blocks.1`.
| Stage | Channel reduction |
|---|---:|
| `up_blocks.0` | unchanged |
| `up_blocks.1` | 25% |
| `up_blocks.2` | 50% |
| `up_blocks.3` | 50% |
| | Teacher (LTX-2.3) | PrunaVAED |
|---|---:|---:|
| Encoder params | 318.9 M | 318.9 M (unchanged) |
| Decoder params | 407.2 M | 345.0 M (−15%) |
| **Total VAE params** | **726.1 M** | **663.9 M** |
`prunavaed/patch_diffusers.py` provides the required compatibility shim for
the pinned diffusers version.
## Limitations
- Evaluated on 4-second clips at 24 fps.
- Benchmarked on one H100 80GB with bfloat16 and batch size 1.
- Speed and VRAM vary with hardware, resolution, batch size, and software.
- Evaluated only on two differents pipeline of LTX 2.3 ([`ti2vid_two_stages`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py)
pipeline and the
[`distilled`](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/distilled.py))
- The weights can be adapted to the custom [`LTX-2`](https://github.com/Lightricks/LTX-2) library without retraining, but no implementation is provided for this yet.
## What this is not
- Not a full T2V model.
- Not a replacement for the LTX denoiser/generator.
- Not compatible with arbitrary VAE latents.
- Not bit-exact with the original decoder.
## License
PrunaVAED is a derivative of LTX-2.3 and is distributed under the
**[LTX-2 Community License Agreement](LICENSE)**. Review its use restrictions
and commercial terms before using or redistributing the model.
Helper code adapted from Hugging Face diffusers retains its Apache-2.0
attribution; see [`NOTICE`](NOTICE).
## What's next?
- **Use PrunaVAED to speed up LTX-2.3 decoding.**
- Compress your own models with [Pruna](https://github.com/PrunaAI/pruna) and give us a ⭐️ for more efficiency!
- Want to use our optimized video model right away? Check
[P-Video](https://www.pruna.ai/p-video) and
[P-Video documentation](https://docs.pruna.ai/en/stable/docs_pruna_endpoints/performance_models/p-video.html).
<style>
.model-button {
display: inline-flex;
flex-direction: row;
justify-content: center;
align-items: center;
gap: 8px;
padding: 8px 20px;
border: none;
border-radius: 8px;
background: #9334e9;
color: #ffffff;
font-size: 14px;
font-weight: 400;
line-height: 1;
text-decoration: none;
white-space: nowrap;
cursor: pointer;
box-sizing: border-box;
overflow: visible;
opacity: 1;
}
</style>
<a href="https://dashboard.pruna.ai/login?utm_source=huggingface&utm_medium=model_card&utm_campaign=hf_traffic" class="model-button">Try our models</a> |