25-second renders, with and without temporal position interpolation
Browse files- .gitattributes +4 -0
- samples/long/README.md +66 -0
- samples/long/ab.png +3 -0
- samples/long/crop_ab.png +3 -0
- samples/long/idol25-fit.mp4 +3 -0
- samples/long/idol25-nofit.mp4 +3 -0
.gitattributes
CHANGED
|
@@ -118,3 +118,7 @@ samples/sharp/03-road.mp4 filter=lfs diff=lfs merge=lfs -text
|
|
| 118 |
samples/sharp/04-smoke.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 119 |
samples/sharp/05-idol-15s.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 120 |
samples/sharp/sheet.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
samples/sharp/04-smoke.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 119 |
samples/sharp/05-idol-15s.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 120 |
samples/sharp/sheet.png filter=lfs diff=lfs merge=lfs -text
|
| 121 |
+
samples/long/ab.png filter=lfs diff=lfs merge=lfs -text
|
| 122 |
+
samples/long/crop_ab.png filter=lfs diff=lfs merge=lfs -text
|
| 123 |
+
samples/long/idol25-fit.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 124 |
+
samples/long/idol25-nofit.mp4 filter=lfs diff=lfs merge=lfs -text
|
samples/long/README.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Past 20 seconds: what the temporal RoPE does, and what to do about it
|
| 2 |
+
|
| 3 |
+
## There is a hard number, and it is in the config
|
| 4 |
+
|
| 5 |
+
`LTXAVModel` carries `positional_embedding_max_pos=[20, 2048, 2048]`, and
|
| 6 |
+
`get_fractional_positions` divides each coordinate by its entry. The temporal
|
| 7 |
+
one is **seconds, not frames** — both `_prepare_positional_embeddings`
|
| 8 |
+
(`model.py:841`) and the AV cross-attention path (`av_model.py:865`) multiply
|
| 9 |
+
the temporal coordinate by `1 / frame_rate` first.
|
| 10 |
+
|
| 11 |
+
So the temporal rotary embedding was built for **20 seconds**. Past that, the
|
| 12 |
+
fraction exceeds 1.0 and the frequencies extrapolate off the end of anything the
|
| 13 |
+
model saw in training. This is the failure
|
| 14 |
+
[Train Short, Inference Long (arXiv 2602.14027)](https://arxiv.org/html/2602.14027v1)
|
| 15 |
+
attributes long-horizon degradation to.
|
| 16 |
+
|
| 17 |
+
`fit_temporal_positions` raises `max_pos[0]` to the clip's own duration, mapping
|
| 18 |
+
it back onto [0, 1] — position interpolation, the same trick used to extend a
|
| 19 |
+
language model's context, and training-free. It is safe to patch at runtime
|
| 20 |
+
because `generate_freq_grid_np`'s cache is keyed on theta, dimension count and
|
| 21 |
+
device, never on `max_pos`.
|
| 22 |
+
|
| 23 |
+
## Measured at 25 seconds
|
| 24 |
+
|
| 25 |
+
601 frames, 1024x640, two-pass, same prompt and seed, on the Space's H200.
|
| 26 |
+
|
| 27 |
+
| | segment sharpness (5 equal parts) | last/first |
|
| 28 |
+
|---|---|---:|
|
| 29 |
+
| untreated | 0.00086 0.00130 0.00143 0.00139 0.00096 | 1.12 |
|
| 30 |
+
| **interpolated** | **0.00247 0.00302 0.00286 0.00318 0.00198** | 0.80 |
|
| 31 |
+
|
| 32 |
+
The interpolated render is **2.2-2.9x sharper in every segment**.
|
| 33 |
+
`crop_ab.png` at 1:1: hair strands, eyelashes and background props are resolved
|
| 34 |
+
on the treated side and waxy on the untreated one.
|
| 35 |
+
|
| 36 |
+
## What this did not find
|
| 37 |
+
|
| 38 |
+
**No progressive collapse.** The premise being tested was that quality decays as
|
| 39 |
+
the clip runs on. It does not, in either version — the last-fifth to first-fifth
|
| 40 |
+
sharpness ratio is 1.12 untreated and 0.80 treated, and neither drifts in
|
| 41 |
+
character, wardrobe or room across 25 seconds. Most of the long-video literature
|
| 42 |
+
found in this search — compounding error, history forgetting, attribute drift —
|
| 43 |
+
describes **autoregressive or sliding-window** generation, where each window
|
| 44 |
+
conditions on the last. LTX-2.5 as driven here samples the whole clip at once,
|
| 45 |
+
so that class of failure does not apply and searching for it would have been
|
| 46 |
+
looking in the wrong place.
|
| 47 |
+
|
| 48 |
+
What extrapolation costs here is **detail across the whole clip**, not stability
|
| 49 |
+
at the end.
|
| 50 |
+
|
| 51 |
+
## Limits
|
| 52 |
+
|
| 53 |
+
* The two renders are different trajectories, not the same trajectory rendered
|
| 54 |
+
twice, so the sharpness gap is what changed and not a controlled isolation of
|
| 55 |
+
cause. Same prompt, seed, size, sampler and refine strength; only `max_pos[0]`
|
| 56 |
+
differs.
|
| 57 |
+
* One duration (25 s), one prompt, one seed. 30 s, 40 s and beyond were not
|
| 58 |
+
tried, and the point where interpolation itself starts to hurt — it compresses
|
| 59 |
+
temporal detail by construction — was not located.
|
| 60 |
+
* Below 20 s this changes nothing and is not applied.
|
| 61 |
+
* Laplacian variance tracks contrast as well as sharpness. It is used here
|
| 62 |
+
between two renders of the same scene, which is the only place it is fair.
|
| 63 |
+
|
| 64 |
+
## Files
|
| 65 |
+
|
| 66 |
+
`idol25-fit.mp4`, `idol25-nofit.mp4`, `ab.png` (timeline), `crop_ab.png` (1:1).
|
samples/long/ab.png
ADDED
|
Git LFS Details
|
samples/long/crop_ab.png
ADDED
|
Git LFS Details
|
samples/long/idol25-fit.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:eb69e83909f4ef5b5308d13c4b431b001579d206b4dae902996d8931e605a18c
|
| 3 |
+
size 16806661
|
samples/long/idol25-nofit.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a31a7b8539f709ef18cac15d518e90dc3de29cc3a8f908bc88c6cb380b550b3a
|
| 3 |
+
size 16186038
|