Update README.md
Browse files
README.md
CHANGED
|
@@ -13,11 +13,32 @@ datasets:
|
|
| 13 |
pipeline_tag: unconditional-image-generation
|
| 14 |
---
|
| 15 |
|
| 16 |
-
# Dancing Stick Figures —
|
| 17 |
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
| file | arch | res | steps | recipe | oracle tvr / lie / clean (floor) |
|
| 23 |
|---|---|---|---|---|---|
|
|
@@ -28,7 +49,9 @@ optimizer state stripped. Load with the trainers' `--init` (warm-start) or `--re
|
|
| 28 |
| `unet_img128.pt` | UNet, min-SNR-5 | 128² | 20k | batch 128 | .226 / .073 / .22 (.203 / .047 / .23) |
|
| 29 |
| `dit_img128_p4.pt` | DiT-FM patch 4 | 128² | 40k | batch 128 | .251 / .065 / .23 (.209 / .048 / .21) |
|
| 30 |
|
| 31 |
-
|
|
|
|
|
|
|
| 32 |
8-frame windows (floor); FVD with a frame-repeated I3D (real-vs-real ≈ 110–120 in this setup):
|
| 33 |
|
| 34 |
| file | init | steps | tvr↓ | lie↓ | head-jitter↓ | angle-jerk↓ | FVD↓ |
|
|
@@ -39,18 +62,26 @@ optimizer state stripped. Load with the trainers' `--init` (warm-start) or `--re
|
|
| 39 |
|
| 40 |
Read: warm-starting from the image model gets there **~2.5× sooner** (loss at 4k ≈ scratch at 10k) but ends in the same place — a
|
| 41 |
convergence-speed win, not a quality win. Per-frame anatomy is within ~0.03 of real frames; temporal jitter ~1.2× real; FVD ~80–100
|
| 42 |
-
above the real-vs-real floor. Sample
|
|
|
|
|
|
|
| 43 |
|
| 44 |
-
|
|
|
|
|
|
|
| 45 |
frames per chunk, 10 fps), initialised from `unet_img64.pt`, 60k steps on one H100 (batch 16, ~5.5 h). Rolls out to any length:
|
| 46 |
`python scripts/rollout.py --ckpt unet_ar64.pt --seconds 5`. Oracle on 32 five-second rollouts vs 32 real 5 s clips —
|
| 47 |
tvr .149 / .128, lie .121 / .103, head-jitter .69 / .57, angle-jerk .19 / .14 (model / real). `unet_ar64_rollout.gif` = 8 samples × 5.6 s.
|
| 48 |
`unet_ar64_interim.pt` = the same run at 30k.
|
| 49 |
|
|
|
|
|
|
|
| 50 |
DiT track (interim, stopped at 44k/42k of 61k): `dit_vid64_from_img_interim.pt` (init from `dit_img64_p2.pt`, patch 2, shift 2, 10 % image
|
| 51 |
batches, 20 % I2V) and `dit_vid64_scratch_interim.pt`; class-conditional image models: `unet_img64_cond.pt`, `dit_img64_p2_cond.pt` (5 prompt groups, CFG).
|
| 52 |
|
| 53 |
-
|
|
|
|
|
|
|
| 54 |
```bash
|
| 55 |
python scripts/compare.py --ckpt runs/img64/ckpt.pt --cache data/cache --ref unet_img64 # image models
|
| 56 |
python -m eval.run_ckpt --run runs/vid64 --cache data/cache --n 64 # video: temporal oracle + FVD
|
|
|
|
| 13 |
pipeline_tag: unconditional-image-generation
|
| 14 |
---
|
| 15 |
|
| 16 |
+
# Dancing Stick Figures — reference baselines and checkpoints
|
| 17 |
|
| 18 |
+
**Companion baselines for [Dancing Stick Figures](https://huggingface.co/datasets/sprited/dancing-stick-figures), a small teaching dataset for learning video generation from scratch.** The dataset is the main project; this repository contains reference checkpoints, sample outputs, and evaluation results.
|
| 19 |
+
|
| 20 |
+
<p align="center">
|
| 21 |
+
<img src="unet_ar64_rollout.gif" alt="Eight generated stick-figure dances rolled out for 5.6 seconds" width="640">
|
| 22 |
+
</p>
|
| 23 |
+
|
| 24 |
+
<p align="center"><em>Eight unconditional samples from the 64×64 autoregressive reference baseline, rolled out for 5.6 seconds.</em></p>
|
| 25 |
+
|
| 26 |
+
**Start here:** [Dataset](https://huggingface.co/datasets/sprited/dancing-stick-figures) · [Colab](https://colab.research.google.com/github/sprited-ai/dancing-stick-figures/blob/main/notebooks/dancing_stick_figures_colab.ipynb) · [Code](https://github.com/sprited-ai/dancing-stick-figures)
|
| 27 |
+
|
| 28 |
+
## What this repository is for
|
| 29 |
+
|
| 30 |
+
These checkpoints are reference points, not the dataset itself. They show that the data can support a complete learning path: train a small image generator, warm-start an 8-frame video model, roll it out autoregressively, and compare it with real clips.
|
| 31 |
+
|
| 32 |
+
- **Dataset:** 1,430 six-second clips / 514,800 frames
|
| 33 |
+
- **Small configuration:** 64×64, 0.85 GB
|
| 34 |
+
- **Labels:** exact 2D + 3D pose for 27 joints, depth, normals, part segmentation, camera parameters, and raw motion
|
| 35 |
+
- **Licenses:** CC0 data and MIT code
|
| 36 |
+
- **Reference architectures:** factorised UNet and DiT flow matching
|
| 37 |
+
- **Reproducible path:** download → cache → train → roll out → evaluate
|
| 38 |
+
|
| 39 |
+
All checkpoints are RGBA and include EMA weights with optimizer state removed. Use a checkpoint with the trainers' `--init` or `--resume`, or evaluate it with `eval/score_images.py`.
|
| 40 |
+
|
| 41 |
+
## Image checkpoints
|
| 42 |
|
| 43 |
| file | arch | res | steps | recipe | oracle tvr / lie / clean (floor) |
|
| 44 |
|---|---|---|---|---|---|
|
|
|
|
| 49 |
| `unet_img128.pt` | UNet, min-SNR-5 | 128² | 20k | batch 128 | .226 / .073 / .22 (.203 / .047 / .23) |
|
| 50 |
| `dit_img128_p4.pt` | DiT-FM patch 4 | 128² | 40k | batch 128 | .251 / .065 / .23 (.209 / .048 / .21) |
|
| 51 |
|
| 52 |
+
## Short-video checkpoints
|
| 53 |
+
|
| 54 |
+
**64² × 8 frames @ 20 fps, UNet 46 M, batch 16, unconditional.** Oracle on 64 samples × 2 seeds vs 64 real
|
| 55 |
8-frame windows (floor); FVD with a frame-repeated I3D (real-vs-real ≈ 110–120 in this setup):
|
| 56 |
|
| 57 |
| file | init | steps | tvr↓ | lie↓ | head-jitter↓ | angle-jerk↓ | FVD↓ |
|
|
|
|
| 62 |
|
| 63 |
Read: warm-starting from the image model gets there **~2.5× sooner** (loss at 4k ≈ scratch at 10k) but ends in the same place — a
|
| 64 |
convergence-speed win, not a quality win. Per-frame anatomy is within ~0.03 of real frames; temporal jitter ~1.2× real; FVD ~80–100
|
| 65 |
+
above the real-vs-real floor. Sample outputs:
|
| 66 |
+
|
| 67 |
+
<table><tr><td align="center"><img src="unet_vid64_scratch_samples.gif" width="360"><br><em>trained from scratch</em></td><td align="center"><img src="unet_vid64_from_img_samples.gif" width="360"><br><em>warm-started from the image checkpoint</em></td></tr></table>
|
| 68 |
|
| 69 |
+
## Long autoregressive rollout
|
| 70 |
+
|
| 71 |
+
**`unet_ar64.pt`**: same UNet, `--ar_ctx 8 --frames 8 --stride 2` (8 context + 8 new
|
| 72 |
frames per chunk, 10 fps), initialised from `unet_img64.pt`, 60k steps on one H100 (batch 16, ~5.5 h). Rolls out to any length:
|
| 73 |
`python scripts/rollout.py --ckpt unet_ar64.pt --seconds 5`. Oracle on 32 five-second rollouts vs 32 real 5 s clips —
|
| 74 |
tvr .149 / .128, lie .121 / .103, head-jitter .69 / .57, angle-jerk .19 / .14 (model / real). `unet_ar64_rollout.gif` = 8 samples × 5.6 s.
|
| 75 |
`unet_ar64_interim.pt` = the same run at 30k.
|
| 76 |
|
| 77 |
+
## Additional checkpoints
|
| 78 |
+
|
| 79 |
DiT track (interim, stopped at 44k/42k of 61k): `dit_vid64_from_img_interim.pt` (init from `dit_img64_p2.pt`, patch 2, shift 2, 10 % image
|
| 80 |
batches, 20 % I2V) and `dit_vid64_scratch_interim.pt`; class-conditional image models: `unet_img64_cond.pt`, `dit_img64_p2_cond.pt` (5 prompt groups, CFG).
|
| 81 |
|
| 82 |
+
## Compare your run
|
| 83 |
+
|
| 84 |
+
Compare your model against these references in one command:
|
| 85 |
```bash
|
| 86 |
python scripts/compare.py --ckpt runs/img64/ckpt.pt --cache data/cache --ref unet_img64 # image models
|
| 87 |
python -m eval.run_ckpt --run runs/vid64 --cache data/cache --n 64 # video: temporal oracle + FVD
|