Spaces:
Running
Running
File size: 10,994 Bytes
503ee13 d867717 efe4cbd d867717 efe4cbd d867717 efe4cbd d867717 503ee13 d867717 efe4cbd d867717 efe4cbd d867717 efe4cbd d867717 efe4cbd d867717 503ee13 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 | <!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Wan2.2-S2V — audio-driven video generation</title>
<meta name="description" content="Reference for Wan2.2-S2V, the speech-to-video checkpoint from Alibaba's Tongyi Lab that animates a reference portrait from an audio track." />
<link rel="canonical" href="https://wavespeed.ai/models/wavespeed-ai/wan-2.2/speech-to-video?utm_source=huggingface&utm_medium=space&utm_campaign=wan2_2_s2v" />
<meta property="og:type" content="website" />
<meta property="og:title" content="Wan2.2-S2V — audio-driven video generation" />
<meta property="og:description" content="Reference for Wan2.2-S2V, the speech-to-video checkpoint from Alibaba's Tongyi Lab that animates a reference portrait from an audio track." />
<meta property="og:url" content="https://wavespeed.ai/models/wavespeed-ai/wan-2.2/speech-to-video" />
<meta name="twitter:card" content="summary_large_image" />
<link rel="stylesheet" href="style.css" />
</head>
<body>
<header class="site-header">
<div class="wrap">
<a class="brand" href="https://wavespeed.ai?utm_source=huggingface&utm_medium=space&utm_campaign=wan2_2_s2v" target="_blank" rel="noopener">
<span class="brand-mark" aria-hidden="true"></span>
<span>WaveSpeed AI</span>
</a>
<nav class="header-nav">
<a class="jump opt" href="#overview">Overview</a>
<a class="jump opt" href="#inputs">Inputs</a>
<a class="jump" href="#run">Run it</a>
<a class="jump" href="#resources">Resources</a>
<a href="https://wavespeed.ai/models/wavespeed-ai/wan-2.2/speech-to-video?utm_source=huggingface&utm_medium=space&utm_campaign=wan2_2_s2v" target="_blank" rel="noopener">wavespeed.ai ↗</a>
</nav>
</div>
</header>
<main class="wrap">
<div class="hero">
<p class="eyebrow">Alibaba · Tongyi Lab</p>
<h1>Wan2.2-S2V</h1>
<p class="lede">The speech-to-video member of the Wan 2.2 family. Given one reference image and an audio track, it generates a video whose lip movement, head motion and gesture follow the audio.</p>
<ul class="meta">
<li><b>Developer</b> Alibaba Tongyi Lab</li>
<li><b>Task</b> audio + image → video</li>
<li><b>Params</b> 14B</li>
<li><b>License</b> Apache-2.0</li>
</ul>
</div>
<section id="overview">
<h2>Overview</h2>
<p class="section-note">S2V extends the Wan 2.2 video backbone with audio conditioning. Rather than driving only the mouth region, the audio signal conditions the whole frame, so posture and gesture move with the speech as well.</p>
<div class="grid">
<div class="card">
<h3>Audio-conditioned motion</h3>
<p>Lip shape, head pose and upper-body movement are generated jointly from the audio track instead of being composited from a separate talking-head module.</p>
</div>
<div class="card">
<h3>Identity from one image</h3>
<p>A single reference frame supplies appearance and identity; no per-subject fine-tuning or reference video is required.</p>
</div>
<div class="card">
<h3>Speech and song</h3>
<p>The checkpoint is trained on both spoken and sung audio, so it is not restricted to conversational delivery.</p>
</div>
<div class="card">
<h3>Long-form generation</h3>
<p>Clips are produced in chunks with motion carried across boundaries, which keeps identity stable past the length of a single window.</p>
</div>
</div>
</section>
<section id="inputs">
<h2>Inputs</h2>
<div class="table-scroll">
<table>
<thead><tr><th>Field</th><th>Type</th><th>Notes</th></tr></thead>
<tbody>
<tr><td><code>image</code></td><td>URL</td><td>Reference frame. A clear, front-facing subject works best.</td></tr>
<tr><td><code>audio</code></td><td>URL</td><td>Driving audio track. Determines the clip length.</td></tr>
<tr><td><code>prompt</code></td><td>string</td><td>Optional description of scene, framing and camera behaviour.</td></tr>
<tr><td><code>seed</code></td><td>int</td><td>Optional. Fix it to make a run reproducible.</td></tr>
</tbody>
</table>
</div>
</section>
<section id="run">
<h2>Run it</h2>
<p class="section-note">Upload local files first with <code>POST /media/upload/binary</code> and pass the returned URLs, or reference any publicly reachable URL directly.</p>
<div class="code">
<div class="code-tabs" role="tablist">
<button type="button" role="tab" aria-selected="true" data-panel="run-0">cURL</button>
<button type="button" role="tab" aria-selected="false" data-panel="run-1">Python</button>
<button type="button" role="tab" aria-selected="false" data-panel="run-2">JavaScript</button>
</div>
<pre id="run-0" role="tabpanel"><code># 1. submit the job
curl -X POST "https://api.wavespeed.ai/api/v3/wavespeed-ai/wan-2.2/speech-to-video" \
-H "Authorization: Bearer $WAVESPEED_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"image": "https://example.com/portrait.jpg",
"audio": "https://example.com/speech.mp3",
"prompt": "A woman speaking to camera in a sunlit studio, medium shot",
"enable_sync_mode": false
}'
# -> {"code": 200, "data": {"id": "<request-id>", "status": "created", ...}}
# 2. poll until status is "completed"
curl "https://api.wavespeed.ai/api/v3/predictions/<request-id>/result" \
-H "Authorization: Bearer $WAVESPEED_API_KEY"
# -> {"code": 200, "data": {"status": "completed", "outputs": ["https://..."]}}</code></pre>
<pre id="run-1" role="tabpanel" hidden><code>import os, time, requests
API = "https://api.wavespeed.ai/api/v3"
KEY = os.environ["WAVESPEED_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}"}
# submit
res = requests.post(
f"{API}/wavespeed-ai/wan-2.2/speech-to-video",
headers={**HEADERS, "Content-Type": "application/json"},
json={
"image": "https://example.com/portrait.jpg",
"audio": "https://example.com/speech.mp3",
"prompt": "A woman speaking to camera in a sunlit studio, medium shot",
"enable_sync_mode": false
},
timeout=30,
)
res.raise_for_status()
request_id = res.json()["data"]["id"]
# poll
while True:
data = requests.get(
f"{API}/predictions/{request_id}/result",
headers=HEADERS,
timeout=30,
).json()["data"]
if data["status"] == "completed":
print(data["outputs"][0])
break
if data["status"] == "failed":
raise RuntimeError(data.get("error", "generation failed"))
time.sleep(1.5)</code></pre>
<pre id="run-2" role="tabpanel" hidden><code>const API = "https://api.wavespeed.ai/api/v3";
const KEY = process.env.WAVESPEED_API_KEY;
const headers = { Authorization: `Bearer ${KEY}` };
// submit
const submit = await fetch(`${API}/wavespeed-ai/wan-2.2/speech-to-video`, {
method: "POST",
headers: { ...headers, "Content-Type": "application/json" },
body: JSON.stringify({
"image": "https://example.com/portrait.jpg",
"audio": "https://example.com/speech.mp3",
"prompt": "A woman speaking to camera in a sunlit studio, medium shot",
"enable_sync_mode": false
}),
});
const { data: { id } } = await submit.json();
// poll
for (;;) {
const res = await fetch(`${API}/predictions/${id}/result`, { headers });
const { data } = await res.json();
if (data.status === "completed") {
console.log(data.outputs[0]);
break;
}
if (data.status === "failed") throw new Error(data.error ?? "generation failed");
await new Promise((r) => setTimeout(r, 1500));
}</code></pre>
</div>
<div class="callout"><p>Requests are asynchronous: <code>POST</code> returns a request id, then you poll <code>/predictions/<id>/result</code> until <code>status</code> is <code>completed</code>. Set <code>enable_sync_mode: true</code> to have the call block and return outputs directly.</p><p>API keys are created in the <a href="https://wavespeed.ai/dashboard?utm_source=huggingface&utm_medium=space&utm_campaign=wan2_2_s2v" target="_blank" rel="noopener">WaveSpeed dashboard</a>.</p></div>
<div class="btn-row">
<a class="btn" href="https://wavespeed.ai/models/wavespeed-ai/wan-2.2/speech-to-video?utm_source=huggingface&utm_medium=space&utm_campaign=wan2_2_s2v" target="_blank" rel="noopener">Open on WaveSpeed</a>
<a class="btn secondary" href="https://wavespeed.ai/docs?utm_source=huggingface&utm_medium=space&utm_campaign=wan2_2_s2v" target="_blank" rel="noopener">API reference</a>
</div>
</section>
<section id="resources">
<h2>Resources</h2>
<ul class="links">
<li><a href="https://huggingface.co/Wan-AI/Wan2.2-S2V-14B" target="_blank" rel="noopener"><span>Wan2.2-S2V-14B weights</span><span class="host">huggingface.co</span></a></li>
<li><a href="https://github.com/Wan-Video/Wan2.2" target="_blank" rel="noopener"><span>Wan2.2 on GitHub</span><span class="host">github.com</span></a></li>
<li><a href="https://wavespeed.ai/collections/wan-2-2?utm_source=huggingface&utm_medium=space&utm_campaign=wan2_2_s2v" target="_blank" rel="noopener"><span>Wan 2.2 collection</span><span class="host">wavespeed.ai</span></a></li>
<li><a href="https://wavespeed.ai/docs?utm_source=huggingface&utm_medium=space&utm_campaign=wan2_2_s2v" target="_blank" rel="noopener"><span>WaveSpeed API docs</span><span class="host">wavespeed.ai</span></a></li>
</ul>
</section>
</main>
<footer class="site-footer">
<div class="wrap">
<p>This page is a model reference maintained by WaveSpeed AI. The model itself is developed and released by its respective authors; trademarks belong to them. WaveSpeed AI provides hosted inference for it.</p>
<div class="footer-links">
<a href="https://wavespeed.ai?utm_source=huggingface&utm_medium=space&utm_campaign=wan2_2_s2v" target="_blank" rel="noopener">WaveSpeed AI</a>
<a href="https://wavespeed.ai/docs?utm_source=huggingface&utm_medium=space&utm_campaign=wan2_2_s2v" target="_blank" rel="noopener">Docs</a>
<a href="https://huggingface.co/wavespeed" target="_blank" rel="noopener">Hugging Face</a>
</div>
</div>
</footer>
<script src="tabs.js"></script>
</body>
</html>
|