| ---
|
| license: mit
|
| library_name: pytorch
|
| pipeline_tag: text-to-video
|
| tags:
|
| - text-to-video
|
| - diffusion
|
| - rectified-flow
|
| - trained-from-scratch
|
| - toy-model
|
| ---
|
|
|
| # Movetoy
|
|
|
| A tiny text-to-video model trained **from random weights** on one RTX 4070 by Claude and Codex (GPT).
|
| It reads *any* text and turns it into a 2-second, 16-frame, 32×32 clip of coloured shapes doing things.
|
| It is deliberately small and often wrong, but it really does respond to what you write.
|
|
|
| `movetoy.pt` (51,029,040 parameters). Code to load it is in `tiny_video/`.
|
|
|
|
|
| ## Load it yourself
|
|
|
| ```bash
|
| pip install torch numpy imageio[ffmpeg] huggingface_hub
|
| ```
|
|
|
| ```python
|
| import torch, imageio
|
| from huggingface_hub import hf_hub_download, snapshot_download
|
| import sys; sys.path.insert(0, snapshot_download("Kompaktion/Movetoy", allow_patterns=["tiny_video/*"]))
|
| from tiny_video.model import load_checkpoint, sample, tokenizer_for, to_uint8
|
|
|
| model, _ = load_checkpoint(hf_hub_download("Kompaktion/Movetoy", "movetoy.pt"), "cpu")
|
| tokens = tokenizer_for(model.cfg)(["a red circle and blue circle merge into a purple circle"])
|
| video = sample(model, tokens, steps=24, cfg_scale=3.0, generator=torch.Generator().manual_seed(0))
|
| frames = to_uint8(video)[0].numpy().repeat(8, 1).repeat(8, 2) # 16 frames, 32x32 -> 256x256
|
| imageio.mimwrite("movetoy.mp4", list(frames), fps=8, macro_block_size=1)
|
| ```
|
|
|
| Runs on CPU (about 0.8 s per sampling step with 2 threads) or any GPU. Try it in the browser, or call the free API (`gradio_client` / HTTP, no key, up to 15 s videos): [Movetoy Space](https://huggingface.co/spaces/Kompaktion/Movetoy).
|
|
|
| ## What it was trained on
|
|
|
| A procedural world (`tiny_video/world.py`) generated on the fly in RAM. Nothing was downloaded and no pretrained
|
| weights were used:
|
| - **12 colours** (with synonyms like crimson, violet, grey), **9 shapes** (circle/ball/dot, square/box, triangle, star,
|
| ring/donut, cross, heart, diamond, hexagon), **5 sizes**, **7 backgrounds** (black, white, gray, night, sky, sunset, grass).
|
| - **16 single-object events:** moves in 8 directions (with many verbs), falls, rises, jumps, bounces, zigzags, circles,
|
| grows, shrinks, spins, stays still, fades in/out, blinks, changes colour, teleports.
|
| - **7 two-object events:** **merge with colour mixing** (red+blue→purple, red+yellow→orange, blue+yellow→green, …), split,
|
| collide, chase, orbit, swap places, and two independent actions.
|
| - Many caption templates, random articles, word order, case and punctuation.
|
|
|
| ## Architecture
|
|
|
| Pixel-space video diffusion transformer with rectified flow: 4×4 patches, 8 blocks (dim 384, 6 heads) with
|
| joint space-time self-attention, 3D rotary positions (t, y, x), cross-attention to the caption, adaLN-Zero
|
| conditioning, and classifier-free guidance. The text encoder is a from-scratch 4-layer transformer over
|
| `[BOS] bytes [SEP] hash(word)…`, where `hash(word) = crc32(casefold(word)) mod 16384`.
|
|
|
| ## Training (≈112 min on the 4070, 23,874 optimizer steps, batch 16)
|
|
|
| | phase | steps | time | what changed and why |
|
| |---|---|---|---|
|
| | 1 | 7,874 | 36 min | raw-byte text only, from random init. The model learned generic scenes but **ignored the caption**: with right, wrong and no caption, the error was the same (0.0284 / 0.0284 / 0.0282). Stopped early. |
|
| | 2 | 10,000 | 48 min | added hashed-word tokens (byte rows and visual weights carried over); training biased toward high noise; unmentioned background = black; 8% of clips were your exact merge example. The caption started to matter, but the model over-learned merging (every two-object scene became a merge). |
|
| | 3 | 6,000 | 28 min | rebalanced: merges 30%→13% of clips, featured example 8%→2%; lr 1e-4. Exported (raw weights). |
|
|
|
| The right caption's advantage over a shuffled one, measured at high noise (t = 0.9 / 0.98), went from ~0 in
|
| phase 1 to **56% / 66%** by the end. `verification.json` reconstructs the whole lineage from the seeded random init
|
| (phase 1 → 2 → 3 hashes) and shows flow loss on 16 fresh clips falling from **1.568** (random weights) to **0.0237**.
|
|
|
| ## How well it works
|
|
|
| Fixed suite `tiny_video/world_eval.py`: 10 prompts × 4 seeds, 64 sampler steps, guidance 3. The pixel checks are
|
| initial/final object counts, colours, final shape, and the action itself. As sanity checks, rendered references
|
| pass 100%, and blank, noise, reversed, and static controls pass 0%.
|
|
|
| | prompt | all checks pass |
|
| |---|---|
|
| | a red circle and blue circle merge into a purple circle (**your example**) | **3/4** |
|
| | Two balls, one red and one blue, come together and become one purple ball. | 2/4 |
|
| | A blue circle fuses with a red circle to form a single violet circle. | 2/4 |
|
| | a yellow circle and red circle merge into an orange circle | 1/4 |
|
| | A purple ball floats upwards. | 1/4 |
|
| | A small blue square grows bigger. | 0/4 (it does grow in 4/4, but into a blob, not a square) |
|
| | A red circle changes color and becomes blue. | 0/4 |
|
| | A little green ball glides from left to right. | 0/4 |
|
| | A red circle and a blue circle collide and bounce apart. | 0/4 |
|
| | A purple circle splits into a red circle and a blue circle. | 0/4 |
|
| | **overall** | **9/40 (22.5%)** |
|
|
|
| **Plain words:** merging and colour mixing work well, colours are almost always right, and backgrounds follow the
|
| text. Size changes and simple motions are hit-or-miss (direction is often wrong), and collisions/splits don't work.
|
| Shapes are usually blobby at 32×32. Everything is clearly neural-network output: no frames are drawn by the
|
| renderer at generation time, and there's no interpolation.
|
|
|
| Fixed-seed demo (`outputs/demo-movetoy/`, one seed per prompt): 40% of event checks, including the exact merge prompt.
|
|
|
