README.md

---
license: apache-2.0
base_model: Wan-AI/Wan2.1-T2V-1.3B
pipeline_tag: text-to-video
library_name: livewan
tags:
  - text-to-video
  - streaming
  - real-time
  - video-generation
  - distillation
  - dmd
  - wan2.1
  - wanstreamer
---

# LiveWan: streaming text 2 video, 3000 steps

> **Unofficial community project.** Not affiliated with, endorsed by, or produced
> by Alibaba Group or the Wan-Video team. Built on their Apache-2.0
> [Wan2.1](https://github.com/Wan-Video/Wan2.1) release.

A 1.3 B student distilled from a Wan2.1-14B teacher that generates video
**continuously** rather than as a fixed clip: 750 ms of 640x368 video per block,
generated in 428 ms on one A100 80GB, extended block by block for as long as you
let it run. The text conditioning can be changed mid-stream without clearing the
K/V cache, so the scene continues instead of cutting.

**Code, setup and documentation: [JonathanColetti/LiveWan](https://github.com/JonathanColetti/LiveWan).** This repository holds
weights and data only. It is not usable on its own.

## Not audio

The model is text-to-video and was never trained to articulate speech. There is no
audio, no lip sync and no speech anywhere in this project. Text steers **the scene**.
A face that appears to talk is generating plausible mouth motion, not saying your
sentence.

## Files

**To run the model — 6.0 GB.** This is what `setup.sh` fetches.

| path | size | what |
|---|---|---|
| `checkpoints/t14b_b64/milestones/step002250_noema.pt` | 5.3 GB | **the model** — 825 tensors, bf16, no EMA copy |
| `checkpoints/t14b_b64/history.json` | 100 KB | the full training history of the run |
| `data/prompts.pt` | 388 MB | umt5-xxl embeddings for 96 prompts, indices 0–95 |
| `out/world_p{0,44,60,82}.pt` | 4 × 59 MB | the four cached evaluation worlds — skip base-model generation entirely |
| `samples/` | 16 MB | reference clips and analysis filmstrips from those worlds |

**To continue the run** Not needed for inference. do not
download these unless you intend to train.

| path | size | what |
|---|---|---|
| `checkpoints/t14b_b64/latest.pt` | 10.6 GB | step 3000, **carries its EMA copy**. The comparison arm, and the weights a resume starts from |
| `checkpoints/t14b_b64/opt_rank00{0..7}.pt` | 8 × 2.2 GB | FSDP-sharded AdamW state at step 3000, one shard per rank |

Nothing here duplicates a public download. The Wan2.1 base model, its VAE and the
umt5-xxl encoder are not mirrored in this repository. `setup.sh` pulls
[Wan-AI/Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) (17 GB)
from its own repo, and the 14B teacher is only needed if you are distilling.

The layout mirrors the code repository, so `setup.sh` downloads straight into place.
To take only part of it:

```bash
hf download JonathanColetti/LiveWan --local-dir . \
  --exclude "checkpoints/t14b_b64/latest.pt" "checkpoints/t14b_b64/opt_rank*"
```

## Use

```bash
git clone https://github.com/JonathanColetti/LiveWan && cd LiveWan
pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
./setup.sh                       # pulls this repo, checksum-verified

python scripts/demo.py \
  --weights checkpoints/t14b_b64/milestones/step002250_noema.pt \
  --world-cache out/world_p60.pt --prompt-idx 60 \
  --latent-norm 1.0 --block 3 --steps 2 --window 6 --units 60 --fps 16 \
  --out out/demo_w60
```

Needs ~10 GiB of VRAM. `scripts/live_demo.py` serves the same model as a steerable
browser stream.

**`--prompt-idx` must match the world**: `world_p60.pt` goes with `--prompt-idx 60`.
Indices run 0–95; anything above fails.


## Training

3000 iterations of SF-DMD distillation from a Wan2.1-T2V-14B teacher into a
Wan2.1-T2V-1.3B student, effective batch 64 (8×H200, accum 8, FSDP), **41.6 hours**
at 66.5 s/it, zero interventions. Losses do not decrease in this trainer and should
not (the critic is retrained every step, so the generator holds position against a
strengthening opponent).

## Continuing the run

The optimizer shards make extending the run a **resume rather than a restart**.
Put `latest.pt`, all eight `opt_rank*.pt` and `history.json` in the same run
directory and:

```bash
ITERS=4000 ./run_b64_resume.sh
```

Four things decide whether this works:

- **Exactly 8 ranks.** These are per-rank FSDP shards, not a consolidated state
  dict. World size 8 is what wrote them and world size 8 is what can read them;
  fewer ranks means *larger* shards, not smaller, so there is no repacking
  shortcut. `run_b64_resume.sh` checks for all eight up front and refuses rather
  than half-starting.
- **You resume from step 3000, not from the released step 2250.** Optimizer state
  and the EMA copy exist only for step 3000 — 2250 was archived `_noema` and its
  EMA is gone. The released checkpoint is the better *model*; `latest.pt` is the
  only *resumable* one.
- **`--resume` overwrites the checkpoint it resumed from.** It reads
  `OUT/latest.pt` and writes that same path at the first save.
  `run_b64_resume.sh` preserves it as `step003000.pt` first — if you drive the
  trainer directly instead, copy it aside yourself.
- **~30 GiB must be free transiently at each save**, because the shards are
  written as `.tmp` and renamed, so old and new coexist. Prune before the save,
  not after; a post-save janitor cannot help, since the save is what runs out of
  space.

The resume path is verified rather than assumed: killing a run at step 12 brought
AdamW's internal counters back as 16/18 rather than reset, and the real 750→3000
resume held `loss_gen`, `loss_critic`, `dmd_grad_norm` and `gn_gen` within one
standard deviation of the 60-point pre-crash reference. A subtly broken restore
shows up as a shifted mean or changed variance; neither appeared.

## Verified

- all 825 tensors load, **zero non-finite values**, weight norms 0.018–115.4
- three correctness gates pass with their deliberately broken controls failing, both
  before and after training
- 24 streaming cells across every arm and world met the real-time budget
- same box, same seed → **byte-identical** output

## Limits

- **Quality holds for roughly a minute.** By 160 s sharpness has fallen to ~73% of
  the world's.
- **Maximum 1024 latent frames** (4.3 minutes at block 3), set by `WanModel.freqs`.
- **The pixel check has not been done.** Checkpoint selection rests on proxy metrics,
  and in this project nine automatic measurements have pointed the wrong way — every
  one caught by looking at pixels at 1:1.
- **World 82 is unexplained**: every trained checkpoint drifts on it while the
  untrained baseline sits at −0.001.
- **The batch-64 premise is unsettled** — confounded with having simply trained
  longer.



## Licence

Apache-2.0. Derived from [Wan2.1](https://github.com/Wan-Video/Wan2.1)
(Apache-2.0, Alibaba Group).