Add a 15-second long-form sample, with its BF16 control
Browse files- .gitattributes +4 -0
- samples/idol/README.md +57 -0
- samples/idol/idol-10s-4bit.mp4 +3 -0
- samples/idol/idol-10s-768x512.mp4 +3 -0
- samples/idol/idol-10s-bf16.mp4 +3 -0
- samples/idol/idol-15s.mp4 +3 -0
- samples/idol/prompt.txt +1 -0
.gitattributes
CHANGED
|
@@ -104,3 +104,7 @@ samples/compare-det-03.mp4 filter=lfs diff=lfs merge=lfs -text
|
|
| 104 |
samples/compare-det-04.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 105 |
samples/comparison-sheet.png filter=lfs diff=lfs merge=lfs -text
|
| 106 |
banner.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
samples/compare-det-04.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 105 |
samples/comparison-sheet.png filter=lfs diff=lfs merge=lfs -text
|
| 106 |
banner.png filter=lfs diff=lfs merge=lfs -text
|
| 107 |
+
samples/idol/idol-10s-4bit.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 108 |
+
samples/idol/idol-10s-768x512.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 109 |
+
samples/idol/idol-10s-bf16.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 110 |
+
samples/idol/idol-15s.mp4 filter=lfs diff=lfs merge=lfs -text
|
samples/idol/README.md
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Long-form sample — an anime idol, 15 seconds
|
| 2 |
+
|
| 3 |
+
`idol-15s.mp4` is the one to watch: **14.7 s, 353 frames, 512x320, 24 fps**,
|
| 4 |
+
video and audio generated together, rendered on a **16 GB Tesla V100** from this
|
| 5 |
+
4-bit encoder. `prompt.txt` is the prompt, unedited.
|
| 6 |
+
|
| 7 |
+
It follows a lot of what was asked. Blonde hair, large blue eyes, star hair
|
| 8 |
+
ornament, the white and light-blue frilly dress with its navy ribbon, the bright
|
| 9 |
+
pastel bedroom with white furniture, blue curtains and a mirrored dresser. The
|
| 10 |
+
choreography lands in order too — it opens on a hand over the lens, peeking
|
| 11 |
+
through, then the presenting gesture, the shy eye-rub, the point at the camera.
|
| 12 |
+
|
| 13 |
+
## What it does not do
|
| 14 |
+
|
| 15 |
+
**It is not speaking Japanese, and probably not speaking at all.** The prompt
|
| 16 |
+
asked for it. There is audio — 14.7 s, 71% of its energy in the 300-3400 Hz band
|
| 17 |
+
a voice occupies — but its envelope peaks at **0.20 Hz**, and articulated speech
|
| 18 |
+
modulates at 3-8 Hz, once per syllable. The structure is closer to music or a
|
| 19 |
+
sustained vocalisation than to sentences. Judge it by ear; the measurement only
|
| 20 |
+
says do not expect dialogue.
|
| 21 |
+
|
| 22 |
+
**Style is not reliably controllable.** The same prompt, same encoder, same
|
| 23 |
+
seed, rendered at a different length or resolution, comes out photoreal instead
|
| 24 |
+
of animated:
|
| 25 |
+
|
| 26 |
+
| file | length | size | result |
|
| 27 |
+
|---|---|---|---|
|
| 28 |
+
| `idol-15s.mp4` | 14.7 s | 512x320 | **animated, on-prompt** |
|
| 29 |
+
| `idol-10s-4bit.mp4` | 9.7 s | 512x320 | photoreal, right scene |
|
| 30 |
+
| `idol-10s-768x512.mp4` | 9.7 s | 768x512 | photoreal, wrong scene — a dark stage with an audience |
|
| 31 |
+
|
| 32 |
+
Resolution and length change the latent shape, so they change the trajectory:
|
| 33 |
+
these are different samples, not the same sample rendered better or worse.
|
| 34 |
+
|
| 35 |
+
## That is the model, not the compression
|
| 36 |
+
|
| 37 |
+
`idol-10s-bf16.mp4` is the same prompt through the **BF16 original**, same DiT,
|
| 38 |
+
same seed, same size. It is photoreal too, and it tracks the 4-bit render
|
| 39 |
+
gesture for gesture at the same timestamps. So "2D anime animation" is being
|
| 40 |
+
ignored by LTX-2.5 itself at that length, and the encoder is not what dropped
|
| 41 |
+
it.
|
| 42 |
+
|
| 43 |
+
Frame-to-frame difference between the two, over all 233 frames: **MAE 0.0563**.
|
| 44 |
+
That is higher than the 0.0261 mean across the five benchmark prompts in
|
| 45 |
+
`../README.md` — a 317-token prompt with ten sequenced actions gives compression
|
| 46 |
+
more to lose than "a blacksmith hammers steel".
|
| 47 |
+
|
| 48 |
+
## Cost
|
| 49 |
+
|
| 50 |
+
| clip | wall clock | peak VRAM |
|
| 51 |
+
|---|---:|---:|
|
| 52 |
+
| 15 s, 512x320 | 123 s | 6.26 GiB |
|
| 53 |
+
| 10 s, 512x320 | 165 s | 6.10 GiB |
|
| 54 |
+
| 10 s, 768x512 | 178 s | 6.53 GiB |
|
| 55 |
+
|
| 56 |
+
All on one 16 GB V100 with `--fp16-unet --novram`. Fifteen seconds costs no more
|
| 57 |
+
memory than ten; the length ceiling here was not reached.
|
samples/idol/idol-10s-4bit.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:dda2a5186d6443e97c8d4e925c50d2f61dd8a082804c0673cc2e5c698e78140c
|
| 3 |
+
size 2390698
|
samples/idol/idol-10s-768x512.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4f243b8db28705ae7db177e87f9a6532e32ce073215bab6fead2d94e1db54db7
|
| 3 |
+
size 3687439
|
samples/idol/idol-10s-bf16.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e8581400cd17dd76aad723cfb055f2e7dd17c9895e82e79ae7636a291c6b9046
|
| 3 |
+
size 2404505
|
samples/idol/idol-15s.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2e31f2d27c8b53f59560329c22d744360fbad1ce2cf33f0f53745f0ceae5ef0b
|
| 3 |
+
size 4201457
|
samples/idol/prompt.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
A cute anime girl with shoulder-length blonde hair, large sparkling blue eyes, golden star earrings, fair skin and rosy cheeks, wearing a white and light blue frilly idol dress with sheer puffy sleeves, navy ribbon with gold star brooch, light blue bustier top with white dots, and gold buckle belt. She is in a bright pastel blue girl's bedroom with soft natural lighting from the window, white furniture, blue curtains, and a dresser with mirror. Dynamic, energetic, and extremely expressive performance directed at the camera: starts with her hand covering the camera lens, peeking with a surprised open mouth; pulls her hand away and raises both arms high in a joyful celebration pose, big smile; lowers arms and clasps hands near her chest in a cute pose; opens both palms upward in a presenting gesture with an open-mouthed surprised expression; looks down shyly with half-closed eyes while fidgeting with her hands; covers her face with both hands in a shy laughing motion; rubs her eyes cutely; points her index finger strongly at the camera with a confident expression; changes to a peace sign while smiling; ends by extending one arm outward and smiling at the viewer. Smooth fluid 2D anime animation, high quality, detailed clothing frills and hair movement, soft pastel blue and white color palette, bright cheerful atmosphere, shallow depth of field, medium close-up shots, camera slightly follows her movements, cinematic lighting, highly expressive facial animations, idol-like performance. Audio: a young woman speaking cheerfully in Japanese, bright and energetic idol voice.
|