File size: 5,797 Bytes
de96f78 d9577be de96f78 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | ---
license: mit
library_name: pytorch
pipeline_tag: text-to-video
tags:
- text-to-video
- diffusion
- rectified-flow
- trained-from-scratch
- toy-model
---
# Movetoy
A tiny text-to-video model trained **from random weights** on one RTX 4070 by Claude and Codex (GPT).
It reads *any* text and turns it into a 2-second, 16-frame, 32×32 clip of coloured shapes doing things.
It is deliberately small and often wrong, but it really does respond to what you write.
`movetoy.pt` (51,029,040 parameters). Code to load it is in `tiny_video/`.
## Load it yourself
```bash
pip install torch numpy imageio[ffmpeg] huggingface_hub
```
```python
import torch, imageio
from huggingface_hub import hf_hub_download, snapshot_download
import sys; sys.path.insert(0, snapshot_download("Kompaktion/Movetoy", allow_patterns=["tiny_video/*"]))
from tiny_video.model import load_checkpoint, sample, tokenizer_for, to_uint8
model, _ = load_checkpoint(hf_hub_download("Kompaktion/Movetoy", "movetoy.pt"), "cpu")
tokens = tokenizer_for(model.cfg)(["a red circle and blue circle merge into a purple circle"])
video = sample(model, tokens, steps=24, cfg_scale=3.0, generator=torch.Generator().manual_seed(0))
frames = to_uint8(video)[0].numpy().repeat(8, 1).repeat(8, 2) # 16 frames, 32x32 -> 256x256
imageio.mimwrite("movetoy.mp4", list(frames), fps=8, macro_block_size=1)
```
Runs on CPU (about 0.8 s per sampling step with 2 threads) or any GPU. Try it in the browser, or call the free API (`gradio_client` / HTTP, no key, up to 15 s videos): [Movetoy Space](https://huggingface.co/spaces/Kompaktion/Movetoy).
## What it was trained on
A procedural world (`tiny_video/world.py`) generated on the fly in RAM. Nothing was downloaded and no pretrained
weights were used:
- **12 colours** (with synonyms like crimson, violet, grey), **9 shapes** (circle/ball/dot, square/box, triangle, star,
ring/donut, cross, heart, diamond, hexagon), **5 sizes**, **7 backgrounds** (black, white, gray, night, sky, sunset, grass).
- **16 single-object events:** moves in 8 directions (with many verbs), falls, rises, jumps, bounces, zigzags, circles,
grows, shrinks, spins, stays still, fades in/out, blinks, changes colour, teleports.
- **7 two-object events:** **merge with colour mixing** (red+blue→purple, red+yellow→orange, blue+yellow→green, …), split,
collide, chase, orbit, swap places, and two independent actions.
- Many caption templates, random articles, word order, case and punctuation.
## Architecture
Pixel-space video diffusion transformer with rectified flow: 4×4 patches, 8 blocks (dim 384, 6 heads) with
joint space-time self-attention, 3D rotary positions (t, y, x), cross-attention to the caption, adaLN-Zero
conditioning, and classifier-free guidance. The text encoder is a from-scratch 4-layer transformer over
`[BOS] bytes [SEP] hash(word)…`, where `hash(word) = crc32(casefold(word)) mod 16384`.
## Training (≈112 min on the 4070, 23,874 optimizer steps, batch 16)
| phase | steps | time | what changed and why |
|---|---|---|---|
| 1 | 7,874 | 36 min | raw-byte text only, from random init. The model learned generic scenes but **ignored the caption**: with right, wrong and no caption, the error was the same (0.0284 / 0.0284 / 0.0282). Stopped early. |
| 2 | 10,000 | 48 min | added hashed-word tokens (byte rows and visual weights carried over); training biased toward high noise; unmentioned background = black; 8% of clips were your exact merge example. The caption started to matter, but the model over-learned merging (every two-object scene became a merge). |
| 3 | 6,000 | 28 min | rebalanced: merges 30%→13% of clips, featured example 8%→2%; lr 1e-4. Exported (raw weights). |
The right caption's advantage over a shuffled one, measured at high noise (t = 0.9 / 0.98), went from ~0 in
phase 1 to **56% / 66%** by the end. `verification.json` reconstructs the whole lineage from the seeded random init
(phase 1 → 2 → 3 hashes) and shows flow loss on 16 fresh clips falling from **1.568** (random weights) to **0.0237**.
## How well it works
Fixed suite `tiny_video/world_eval.py`: 10 prompts × 4 seeds, 64 sampler steps, guidance 3. The pixel checks are
initial/final object counts, colours, final shape, and the action itself. As sanity checks, rendered references
pass 100%, and blank, noise, reversed, and static controls pass 0%.
| prompt | all checks pass |
|---|---|
| a red circle and blue circle merge into a purple circle (**your example**) | **3/4** |
| Two balls, one red and one blue, come together and become one purple ball. | 2/4 |
| A blue circle fuses with a red circle to form a single violet circle. | 2/4 |
| a yellow circle and red circle merge into an orange circle | 1/4 |
| A purple ball floats upwards. | 1/4 |
| A small blue square grows bigger. | 0/4 (it does grow in 4/4, but into a blob, not a square) |
| A red circle changes color and becomes blue. | 0/4 |
| A little green ball glides from left to right. | 0/4 |
| A red circle and a blue circle collide and bounce apart. | 0/4 |
| A purple circle splits into a red circle and a blue circle. | 0/4 |
| **overall** | **9/40 (22.5%)** |
**Plain words:** merging and colour mixing work well, colours are almost always right, and backgrounds follow the
text. Size changes and simple motions are hit-or-miss (direction is often wrong), and collisions/splits don't work.
Shapes are usually blobby at 32×32. Everything is clearly neural-network output: no frames are drawn by the
renderer at generation time, and there's no interpolation.
Fixed-seed demo (`outputs/demo-movetoy/`, one seed per prompt): 40% of event checks, including the exact merge prompt.
|