sprited commited on
Commit
3b5f47a
·
verified ·
1 Parent(s): 3b5f45c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +39 -8
README.md CHANGED
@@ -13,11 +13,32 @@ datasets:
13
  pipeline_tag: unconditional-image-generation
14
  ---
15
 
16
- # Dancing Stick Figures — baseline checkpoints (v0.1)
17
 
18
- Reference models trained on [sprited/dancing-stick-figures](https://huggingface.co/datasets/sprited/dancing-stick-figures)
19
- with the code in <https://github.com/sprited-ai/dancing-stick-figures>. All unconditional, RGBA, EMA weights included,
20
- optimizer state stripped. Load with the trainers' `--init` (warm-start) or `--resume`, or with `eval/score_images.py`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
 
22
  | file | arch | res | steps | recipe | oracle tvr / lie / clean (floor) |
23
  |---|---|---|---|---|---|
@@ -28,7 +49,9 @@ optimizer state stripped. Load with the trainers' `--init` (warm-start) or `--re
28
  | `unet_img128.pt` | UNet, min-SNR-5 | 128² | 20k | batch 128 | .226 / .073 / .22 (.203 / .047 / .23) |
29
  | `dit_img128_p4.pt` | DiT-FM patch 4 | 128² | 40k | batch 128 | .251 / .065 / .23 (.209 / .048 / .21) |
30
 
31
- **Video (64² × 8 frames @ 20 fps, UNet 46 M, batch 16, unconditional) — finished runs.** Oracle on 64 samples × 2 seeds vs 64 real
 
 
32
  8-frame windows (floor); FVD with a frame-repeated I3D (real-vs-real ≈ 110–120 in this setup):
33
 
34
  | file | init | steps | tvr↓ | lie↓ | head-jitter↓ | angle-jerk↓ | FVD↓ |
@@ -39,18 +62,26 @@ optimizer state stripped. Load with the trainers' `--init` (warm-start) or `--re
39
 
40
  Read: warm-starting from the image model gets there **~2.5× sooner** (loss at 4k ≈ scratch at 10k) but ends in the same place — a
41
  convergence-speed win, not a quality win. Per-frame anatomy is within ~0.03 of real frames; temporal jitter ~1.2× real; FVD ~80–100
42
- above the real-vs-real floor. Sample GIFs: `unet_vid64_scratch_samples.gif`, `unet_vid64_from_img_samples.gif`.
 
 
43
 
44
- **Autoregressive video (the route's step 4) — `unet_ar64.pt`**: same UNet, `--ar_ctx 8 --frames 8 --stride 2` (8 context + 8 new
 
 
45
  frames per chunk, 10 fps), initialised from `unet_img64.pt`, 60k steps on one H100 (batch 16, ~5.5 h). Rolls out to any length:
46
  `python scripts/rollout.py --ckpt unet_ar64.pt --seconds 5`. Oracle on 32 five-second rollouts vs 32 real 5 s clips —
47
  tvr .149 / .128, lie .121 / .103, head-jitter .69 / .57, angle-jerk .19 / .14 (model / real). `unet_ar64_rollout.gif` = 8 samples × 5.6 s.
48
  `unet_ar64_interim.pt` = the same run at 30k.
49
 
 
 
50
  DiT track (interim, stopped at 44k/42k of 61k): `dit_vid64_from_img_interim.pt` (init from `dit_img64_p2.pt`, patch 2, shift 2, 10 % image
51
  batches, 20 % I2V) and `dit_vid64_scratch_interim.pt`; class-conditional image models: `unet_img64_cond.pt`, `dit_img64_p2_cond.pt` (5 prompt groups, CFG).
52
 
53
- **Compare your model against these in one command:**
 
 
54
  ```bash
55
  python scripts/compare.py --ckpt runs/img64/ckpt.pt --cache data/cache --ref unet_img64 # image models
56
  python -m eval.run_ckpt --run runs/vid64 --cache data/cache --n 64 # video: temporal oracle + FVD
 
13
  pipeline_tag: unconditional-image-generation
14
  ---
15
 
16
+ # Dancing Stick Figures — reference baselines and checkpoints
17
 
18
+ **Companion baselines for [Dancing Stick Figures](https://huggingface.co/datasets/sprited/dancing-stick-figures), a small teaching dataset for learning video generation from scratch.** The dataset is the main project; this repository contains reference checkpoints, sample outputs, and evaluation results.
19
+
20
+ <p align="center">
21
+ <img src="unet_ar64_rollout.gif" alt="Eight generated stick-figure dances rolled out for 5.6 seconds" width="640">
22
+ </p>
23
+
24
+ <p align="center"><em>Eight unconditional samples from the 64×64 autoregressive reference baseline, rolled out for 5.6 seconds.</em></p>
25
+
26
+ **Start here:** [Dataset](https://huggingface.co/datasets/sprited/dancing-stick-figures) · [Colab](https://colab.research.google.com/github/sprited-ai/dancing-stick-figures/blob/main/notebooks/dancing_stick_figures_colab.ipynb) · [Code](https://github.com/sprited-ai/dancing-stick-figures)
27
+
28
+ ## What this repository is for
29
+
30
+ These checkpoints are reference points, not the dataset itself. They show that the data can support a complete learning path: train a small image generator, warm-start an 8-frame video model, roll it out autoregressively, and compare it with real clips.
31
+
32
+ - **Dataset:** 1,430 six-second clips / 514,800 frames
33
+ - **Small configuration:** 64×64, 0.85 GB
34
+ - **Labels:** exact 2D + 3D pose for 27 joints, depth, normals, part segmentation, camera parameters, and raw motion
35
+ - **Licenses:** CC0 data and MIT code
36
+ - **Reference architectures:** factorised UNet and DiT flow matching
37
+ - **Reproducible path:** download → cache → train → roll out → evaluate
38
+
39
+ All checkpoints are RGBA and include EMA weights with optimizer state removed. Use a checkpoint with the trainers' `--init` or `--resume`, or evaluate it with `eval/score_images.py`.
40
+
41
+ ## Image checkpoints
42
 
43
  | file | arch | res | steps | recipe | oracle tvr / lie / clean (floor) |
44
  |---|---|---|---|---|---|
 
49
  | `unet_img128.pt` | UNet, min-SNR-5 | 128² | 20k | batch 128 | .226 / .073 / .22 (.203 / .047 / .23) |
50
  | `dit_img128_p4.pt` | DiT-FM patch 4 | 128² | 40k | batch 128 | .251 / .065 / .23 (.209 / .048 / .21) |
51
 
52
+ ## Short-video checkpoints
53
+
54
+ **64² × 8 frames @ 20 fps, UNet 46 M, batch 16, unconditional.** Oracle on 64 samples × 2 seeds vs 64 real
55
  8-frame windows (floor); FVD with a frame-repeated I3D (real-vs-real ≈ 110–120 in this setup):
56
 
57
  | file | init | steps | tvr↓ | lie↓ | head-jitter↓ | angle-jerk↓ | FVD↓ |
 
62
 
63
  Read: warm-starting from the image model gets there **~2.5× sooner** (loss at 4k ≈ scratch at 10k) but ends in the same place — a
64
  convergence-speed win, not a quality win. Per-frame anatomy is within ~0.03 of real frames; temporal jitter ~1.2× real; FVD ~80–100
65
+ above the real-vs-real floor. Sample outputs:
66
+
67
+ <table><tr><td align="center"><img src="unet_vid64_scratch_samples.gif" width="360"><br><em>trained from scratch</em></td><td align="center"><img src="unet_vid64_from_img_samples.gif" width="360"><br><em>warm-started from the image checkpoint</em></td></tr></table>
68
 
69
+ ## Long autoregressive rollout
70
+
71
+ **`unet_ar64.pt`**: same UNet, `--ar_ctx 8 --frames 8 --stride 2` (8 context + 8 new
72
  frames per chunk, 10 fps), initialised from `unet_img64.pt`, 60k steps on one H100 (batch 16, ~5.5 h). Rolls out to any length:
73
  `python scripts/rollout.py --ckpt unet_ar64.pt --seconds 5`. Oracle on 32 five-second rollouts vs 32 real 5 s clips —
74
  tvr .149 / .128, lie .121 / .103, head-jitter .69 / .57, angle-jerk .19 / .14 (model / real). `unet_ar64_rollout.gif` = 8 samples × 5.6 s.
75
  `unet_ar64_interim.pt` = the same run at 30k.
76
 
77
+ ## Additional checkpoints
78
+
79
  DiT track (interim, stopped at 44k/42k of 61k): `dit_vid64_from_img_interim.pt` (init from `dit_img64_p2.pt`, patch 2, shift 2, 10 % image
80
  batches, 20 % I2V) and `dit_vid64_scratch_interim.pt`; class-conditional image models: `unet_img64_cond.pt`, `dit_img64_p2_cond.pt` (5 prompt groups, CFG).
81
 
82
+ ## Compare your run
83
+
84
+ Compare your model against these references in one command:
85
  ```bash
86
  python scripts/compare.py --ckpt runs/img64/ckpt.pt --cache data/cache --ref unet_img64 # image models
87
  python -m eval.run_ckpt --run runs/vid64 --cache data/cache --n 64 # video: temporal oracle + FVD