Instructions to use Viggle/Viggle-Animate with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use Viggle/Viggle-Animate with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("Viggle/Viggle-Animate", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
docs: head-to-head vs Wan2.2-Animate, with clips
Browse files- .gitattributes +10 -0
- README.md +86 -2
- examples/media/swap-airliner.mp4 +3 -0
- examples/media/swap-anime-duo.mp4 +3 -0
- examples/media/swap-claymation.mp4 +3 -0
- examples/media/swap-corgi.mp4 +3 -0
- examples/media/swap-panda.mp4 +3 -0
- examples/media/swap-penguin.mp4 +3 -0
- examples/media/swap-photoreal-duo.mp4 +3 -0
- examples/media/swap-robot.mp4 +3 -0
- examples/media/swap-wushu-animals.mp4 +3 -0
- examples/media/swap-wushu-robots.mp4 +3 -0
.gitattributes
CHANGED
|
@@ -42,3 +42,13 @@ examples/media/compare-horns.mp4 filter=lfs diff=lfs merge=lfs -text
|
|
| 42 |
examples/media/compare-costume.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 43 |
examples/media/compare-prop.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 44 |
examples/media/compare-style.mp4 filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
examples/media/compare-costume.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 43 |
examples/media/compare-prop.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 44 |
examples/media/compare-style.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 45 |
+
examples/media/swap-airliner.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 46 |
+
examples/media/swap-anime-duo.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 47 |
+
examples/media/swap-claymation.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 48 |
+
examples/media/swap-corgi.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 49 |
+
examples/media/swap-panda.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 50 |
+
examples/media/swap-penguin.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 51 |
+
examples/media/swap-photoreal-duo.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 52 |
+
examples/media/swap-robot.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 53 |
+
examples/media/swap-wushu-animals.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 54 |
+
examples/media/swap-wushu-robots.mp4 filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -101,6 +101,77 @@ These three are selected. Across the 24 clips we ran head to head, Wan came out
|
|
| 101 |
tight talking-head shot, where its explicit face channel tracks expression that we render flat. That
|
| 102 |
is the same weakness listed under Limitations below. Roughly six were a wash.
|
| 103 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
## What this is
|
| 105 |
|
| 106 |
Two parts, both derived from [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3)'s
|
|
@@ -169,9 +240,22 @@ in **40β60 s**. That quantization path is not shipped here.
|
|
| 169 |
- **More steps is not better.** `--steps 4` is the operating point, not a degraded `--steps 12`. It
|
| 170 |
renders sharper than its own teacher β Laplacian variance 16β50 % higher β which reads as detail
|
| 171 |
sometimes and over-sharpening other times, and worsens with more steps.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 172 |
- **No facial-landmark channel**, so tight close-up dialogue is the weakest case β Wan drives the
|
| 173 |
-
face from explicit landmarks and is visibly more expressive there.
|
| 174 |
-
|
|
|
|
|
|
|
|
|
|
| 175 |
- The driving clip's audio is dropped at input; the model emits its own track, and the fixed prompt
|
| 176 |
asks for silence.
|
| 177 |
|
|
|
|
| 101 |
tight talking-head shot, where its explicit face channel tracks expression that we render flat. That
|
| 102 |
is the same weakness listed under Limitations below. Roughly six were a wash.
|
| 103 |
|
| 104 |
+
## What you can paint in
|
| 105 |
+
|
| 106 |
+
The model is never told what the new character is. It sees one edited frame and has to carry it
|
| 107 |
+
through five seconds of motion it did not choreograph β so the question is how far from a person
|
| 108 |
+
that character can get. Three panels each: **painted reference Β· driving video Β· this model.** All
|
| 109 |
+
ten are single attempts at both stages: one paint, one render, `--steps 4 --seed 42`, no best-of-N.
|
| 110 |
+
|
| 111 |
+
### Animals
|
| 112 |
+
|
| 113 |
+
<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-corgi.mp4"></video>
|
| 114 |
+
|
| 115 |
+
A corgi in the driver's own tank top and trousers. The ears track head rotation independently β
|
| 116 |
+
nothing in the driving clip has ears to copy.
|
| 117 |
+
|
| 118 |
+
<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-panda.mp4"></video>
|
| 119 |
+
|
| 120 |
+
A giant panda in a denim jacket and ripped jeans. Clothing deforms onto fur without arguing about
|
| 121 |
+
it, and the eye patches stay symmetric through the hands-over-face beat.
|
| 122 |
+
|
| 123 |
+
<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-penguin.mp4"></video>
|
| 124 |
+
|
| 125 |
+
An emperor penguin in a red floral suit. The paint put the flippers where the arms were, and the
|
| 126 |
+
render animates them *as* arms β they sweep on the arms' trajectories while keeping a flipper's shape.
|
| 127 |
+
|
| 128 |
+
### Not humanoid
|
| 129 |
+
|
| 130 |
+
<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-airliner.mp4"></video>
|
| 131 |
+
|
| 132 |
+
The hardest case here. A man standing with his arms out becomes an airliner whose wings are his arms
|
| 133 |
+
and whose landing gear are his legs, and that binding survives the whole take. The geometry is done
|
| 134 |
+
in the paint β the wings placed exactly where the arms were β and the model only has to keep it.
|
| 135 |
+
|
| 136 |
+
<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-robot.mp4"></video>
|
| 137 |
+
|
| 138 |
+
Specular metal has to relight itself as the body turns rather than just deform. The cable joints at
|
| 139 |
+
the elbows and knees stay put instead of migrating.
|
| 140 |
+
|
| 141 |
+
### Stylised
|
| 142 |
+
|
| 143 |
+
<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-claymation.mp4"></video>
|
| 144 |
+
|
| 145 |
+
A stop-motion plasticine character in front of a kitchen that stays photographic. The style boundary
|
| 146 |
+
holds for the whole clip β fingerprints and seams on the figure, real appliances behind it.
|
| 147 |
+
|
| 148 |
+
### Two characters, one edit
|
| 149 |
+
|
| 150 |
+
The prompt has to bind each new character to a specific person β "the girl on the left", "the athlete
|
| 151 |
+
on the right" β or the image editor swaps one and leaves the other. All four of these held.
|
| 152 |
+
|
| 153 |
+
<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-anime-duo.mp4"></video>
|
| 154 |
+
|
| 155 |
+
Two guitarists become a cat and a corgi, drawn in the clip's own cel-shaded style, so the swap has to
|
| 156 |
+
survive as line art rather than as texture. The fretting hands stay on the necks.
|
| 157 |
+
|
| 158 |
+
<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-photoreal-duo.mp4"></video>
|
| 159 |
+
|
| 160 |
+
The same clip with photoreal fur in front of a hand-drawn night sky. Neither side pulls the other
|
| 161 |
+
across: the background stays anime and the characters stay photographic.
|
| 162 |
+
|
| 163 |
+
<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-wushu-animals.mp4"></video>
|
| 164 |
+
|
| 165 |
+
A panda and a tiger through a synchronised *duilian* routine, in a wide arena shot where each figure
|
| 166 |
+
is a few dozen pixels tall β landscape framing and two subjects, both outside the training
|
| 167 |
+
distribution. They stay distinct from each other and neither swaps back.
|
| 168 |
+
|
| 169 |
+
<video controls muted loop playsinline width="100%" src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/swap-wushu-robots.mp4"></video>
|
| 170 |
+
|
| 171 |
+
Same clip, two robots that differ only in metal β gold on the left, chrome on the right. They stay
|
| 172 |
+
told apart through the jumps and stance changes, which is what would fail if the pair were being
|
| 173 |
+
treated as one region.
|
| 174 |
+
|
| 175 |
## What this is
|
| 176 |
|
| 177 |
Two parts, both derived from [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3)'s
|
|
|
|
| 240 |
- **More steps is not better.** `--steps 4` is the operating point, not a degraded `--steps 12`. It
|
| 241 |
renders sharper than its own teacher β Laplacian variance 16β50 % higher β which reads as detail
|
| 242 |
sometimes and over-sharpening other times, and worsens with more steps.
|
| 243 |
+
- **The reference defines only the views it contains.** A character painted from behind has no
|
| 244 |
+
front, and when the driving clip rotates to reveal one the model fills it from its own prior
|
| 245 |
+
rather than from your character. On a figure skater who spins to face camera, a swapped-in cat is
|
| 246 |
+
a cat while her back is turned, a generic woman at the moment she faces us, and a cat again when
|
| 247 |
+
she turns away. Prefer driving clips that keep one side to camera, or paint a frame that shows the
|
| 248 |
+
view the clip will eventually reach.
|
| 249 |
+
- **A silhouette far from the driver's survives as colour, not as geometry.** A LEGO minifigure on
|
| 250 |
+
that same skating clip kept its palette and its yellow claw hands, but the moulded plastic body
|
| 251 |
+
reverted to human anatomy. The airliner above works because the paint bound its wings to the arms
|
| 252 |
+
β geometry the *driving pose* can still explain.
|
| 253 |
- **No facial-landmark channel**, so tight close-up dialogue is the weakest case β Wan drives the
|
| 254 |
+
face from explicit landmarks and is visibly more expressive there.
|
| 255 |
+
- **Out of distribution is not the same as broken.** Training was one subject filling a portrait
|
| 256 |
+
frame. The last four gallery clips are two subjects in landscape framing, including a wide arena
|
| 257 |
+
shot, and they hold β but that is untuned territory with no evaluation behind it, and crowds
|
| 258 |
+
remain untested.
|
| 259 |
- The driving clip's audio is dropped at input; the model emits its own track, and the fixed prompt
|
| 260 |
asks for silence.
|
| 261 |
|
examples/media/swap-airliner.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5a7a31fd225e258633e208f13c18ff8523517834a9ea2c67b97ed060972becf3
|
| 3 |
+
size 397277
|
examples/media/swap-anime-duo.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:205700e55841a5fd22f517c6eb08c447068d0417a8e902bc91c863dede743d86
|
| 3 |
+
size 108986
|
examples/media/swap-claymation.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:dd08b83d8a57e77408203710cf55eccdf8d1ee136c22123cc42ed8ac14434528
|
| 3 |
+
size 286267
|
examples/media/swap-corgi.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5954aeab0360ac8c1c5c62b5d35935520ac0feffdb8d11b8c55a6ff84f9d9239
|
| 3 |
+
size 272402
|
examples/media/swap-panda.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bec174f093bf553b39d1f022b6fedd46b9616a6ac30fa10a939753cf5e4a5150
|
| 3 |
+
size 390485
|
examples/media/swap-penguin.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ea2ccf8ce410331d194403903e2d1e836079b2d8fbafebf113265e52aafb3cd0
|
| 3 |
+
size 346127
|
examples/media/swap-photoreal-duo.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a2d7c7fb36c5b2779176484b0a2fcb1735578ea3cad2fa0a2c0b057218881554
|
| 3 |
+
size 108311
|
examples/media/swap-robot.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e8147fb8f85eca14e372ecd26f6d7b4556c649f5c980e0dd514a582ac8619077
|
| 3 |
+
size 194541
|
examples/media/swap-wushu-animals.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1ed178d0c88053df75803d904bef26c83c81da78c33d575ec8f5897701b622a7
|
| 3 |
+
size 204001
|
examples/media/swap-wushu-robots.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:63cc5dfe297ca472b7c54a9ef32aaa575f84e6cc1088a42d01236e40888785b2
|
| 3 |
+
size 201465
|