Document self-contained loading and the guided modes
Browse files
README.md
CHANGED
|
@@ -83,17 +83,23 @@ V100 — the encoder fits one consumer card for the first time.
|
|
| 83 |
### Standalone
|
| 84 |
|
| 85 |
```python
|
|
|
|
| 86 |
from ltx_packed_codec import load_packed_model
|
| 87 |
from transformers import AutoTokenizer
|
| 88 |
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
|
|
|
|
|
|
|
|
|
| 92 |
```
|
| 93 |
|
| 94 |
-
`encoder-hf` is
|
| 95 |
-
|
| 96 |
-
|
|
|
|
|
|
|
| 97 |
|
| 98 |
### ComfyUI
|
| 99 |
|
|
@@ -113,6 +119,26 @@ CONDITIONING.
|
|
| 113 |
> nodes themselves have not been loaded in a live ComfyUI, and that is a
|
| 114 |
> different claim.
|
| 115 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
## How it was built
|
| 117 |
|
| 118 |
nvfp4 4.5 bpw (E2M1, group 16 with an fp8-e4m3 scale) on 320 projections;
|
|
|
|
| 83 |
### Standalone
|
| 84 |
|
| 85 |
```python
|
| 86 |
+
from huggingface_hub import hf_hub_download, snapshot_download
|
| 87 |
from ltx_packed_codec import load_packed_model
|
| 88 |
from transformers import AutoTokenizer
|
| 89 |
|
| 90 |
+
repo = "topabaem/LTX-2.5-TextEncoder-nvfp4-portable"
|
| 91 |
+
packed = hf_hub_download(repo, "A3.packed.safetensors")
|
| 92 |
+
encoder_dir = snapshot_download(repo, allow_patterns=["encoder-hf/*"]) + "/encoder-hf"
|
| 93 |
+
|
| 94 |
+
model = load_packed_model(encoder_dir, packed, resident=True)
|
| 95 |
+
tokenizer = AutoTokenizer.from_pretrained(encoder_dir)
|
| 96 |
```
|
| 97 |
|
| 98 |
+
`encoder-hf/` here is config and tokenizer only, 31 MB. **The 26 GB original is
|
| 99 |
+
not needed**: `load_packed_model` builds the skeleton with
|
| 100 |
+
`AutoModel.from_config` and feeds every parameter from the packed file, so
|
| 101 |
+
nothing reads the Lightricks weights. The asset blobs that ship inside the
|
| 102 |
+
original safetensors are already in the packed file as raw BF16.
|
| 103 |
|
| 104 |
### ComfyUI
|
| 105 |
|
|
|
|
| 119 |
> nodes themselves have not been loaded in a live ComfyUI, and that is a
|
| 120 |
> different claim.
|
| 121 |
|
| 122 |
+
## Image- and video-to-video
|
| 123 |
+
|
| 124 |
+
Both work, and both needed a fix ComfyUI does not ship. `LTXVAddGuide` is the
|
| 125 |
+
only producer of guided LTX latents and it cannot take an LTX-2.5 one: it calls
|
| 126 |
+
`torch.cat` on what is a `NestedTensor` for this model. The `ValueError` in that
|
| 127 |
+
function saying AV guides are unsupported never fires — `NestedTensor.shape`
|
| 128 |
+
proxies to the video half, whose channel count is exactly the 128 it checks for
|
| 129 |
+
— so the real failure is a `TypeError`, and the message is stale.
|
| 130 |
+
|
| 131 |
+
Everything below the node already supports AV guides: the model routes
|
| 132 |
+
`keyframe_idxs` to the video branch, the sampler pads a video-only denoise mask
|
| 133 |
+
with ones for audio, and `model_base` splits the packed mask apart again. So
|
| 134 |
+
`ltx_av_guide.py` unwraps the pair, runs the stock node on the video half and
|
| 135 |
+
re-wraps; the guide arithmetic stays the vendor's.
|
| 136 |
+
|
| 137 |
+
Measured on a 16 GB V100 at 512x320, 25 frames: t2v 98.8 s / 5.84 GiB, i2v
|
| 138 |
+
62.7 s / 6.80 GiB, v2v 86.5 s / 5.84 GiB. An i2v first frame lands **relL2
|
| 139 |
+
0.0766** from its guide image against **0.7061** for the same seed and prompt
|
| 140 |
+
without the guide, so the guide is honoured rather than merely accepted.
|
| 141 |
+
|
| 142 |
## How it was built
|
| 143 |
|
| 144 |
nvfp4 4.5 bpw (E2M1, group 16 with an fp8-e4m3 scale) on 320 projections;
|