File size: 4,915 Bytes
f9e16d2 d07628f 14295c9 d07628f 14295c9 f9e16d2 d07628f 14295c9 d07628f 14295c9 d07628f 14295c9 d07628f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | ---
license: cc-by-sa-4.0
tags:
- diffusion
- rectified-flow
- flow-matching
- pixel-art
- emoji
- sprites
- image-generation
- text-to-image
- clip
- pytorch
library_name: pytorch
pipeline_tag: text-to-image
---
# sprite-gpt
Small pixel-space diffusion models for emoji and game sprites, trained with
rectified flow (flow matching) on a single RTX 5090. Code:
[github.com/gmmeyer/sprite-diffusion](https://github.com/gmmeyer/sprite-diffusion)
| file | model | params | training |
|---|---|---|---|
| `sprite-gpt-text64.pt` | **text-conditioned 64px** (CLIP ViT-B/32 + cross-attention) | 61.9M | 120k steps @ bs128 on 24,134 emoji+sprites |
| `sprite-gpt-cond64.pt` | class-conditional 64px, 31 classes | 55.8M | 95k steps @ bs128 on 24,134 emoji+sprites |
| `sprite-gpt-emoji32.pt` | unconditional 32px emoji | 29.1M | 60k steps @ bs256 on 8,504 emoji |
The text model composes attributes not seen together in training — prompts
below are "a purple dragon emoji", "an angry ghost sprite", "a blue cat face
emoji", "a green skull sprite", "ice dungeon wall tile", "a red sword item",
"a crying robot face emoji", "a yellow snake monster" (8 seeds each):
Both are ADM-style U-Nets (scale-shift GroupNorm conditioning, self-attention
at 16x16 and 8x8) trained with the rectified-flow objective
(`x_t = (1-t)·x0 + t·ε`, predict `v = ε − x0`, logit-normal timestep
sampling). Weights are fp32 EMA (decay 0.9995). Sampling: 50-step Euler ODE;
the conditional model uses classifier-free guidance (null class = index 31,
condition dropout 10% at train time; guidance 3 works well).
## Usage
Text-to-image (easiest via the repo CLI):
```sh
git clone https://github.com/gmmeyer/sprite-diffusion && cd sprite-diffusion && uv sync
uv run python -m spritegpt.sample --ckpt sprite-gpt-text64.pt \
--prompt "a blue ghost sprite" --guidance 5 --out ghost.png
```
(`sample.py` accepts these exported files as well as training checkpoints.)
Class-conditional model, by hand:
```python
# git clone https://github.com/gmmeyer/sprite-diffusion && cd sprite-diffusion && uv sync
import torch
from spritegpt.unet import UNet
from spritegpt.flow import sample
from spritegpt.utils import save_image_grid
ck = torch.load("sprite-gpt-cond64.pt", map_location="cuda", weights_only=False)
cfg = ck["config"]
model = UNet(
img_size=cfg["img_size"],
base=cfg["base"],
ch_mult=tuple(int(c) for c in cfg["ch_mult"].split(",")),
num_res=cfg["num_res"],
attn_res=tuple(int(r) for r in cfg["attn_res"].split(",")),
num_classes=cfg["num_classes"],
).cuda()
model.load_state_dict(ck["ema_state_dict"])
model.eval()
y = torch.full((16,), 24, device="cuda", dtype=torch.long) # sprite/dcss-mon
with torch.autocast("cuda", dtype=torch.bfloat16):
imgs = sample(model, 16, 64, steps=50, y=y, guidance=3.0,
num_classes=cfg["num_classes"], device="cuda")
save_image_grid(imgs.float(), "monsters.png", nrow=4)
```
## Classes (conditional model)
Class ids are the alphabetical index into:
`emoji/activities, emoji/animals-nature, emoji/component, emoji/extras-openmoji,
emoji/extras-unicode, emoji/flags, emoji/food-drink, emoji/objects,
emoji/people-body, emoji/smileys-emotion, emoji/symbols, emoji/travel-places,
emoji/unknown, sprite/dawnlike, sprite/dcss-dungeon, sprite/dcss-effect,
sprite/dcss-item, sprite/dcss-misc, sprite/dcss-mon, sprite/dcss-player,
sprite/kenney-1bit, sprite/kenney-caves, sprite/kenney-characters,
sprite/kenney-micro-roguelike, sprite/kenney-modern-city,
sprite/kenney-pixel-platformer, sprite/kenney-pixel-shmup, sprite/kenney-rpg,
sprite/kenney-tiny-battle, sprite/kenney-tiny-dungeon, sprite/kenney-tiny-town`
## Training data and attribution
Images composited onto white backgrounds at 32/64px:
- [OpenMoji](https://openmoji.org) — CC-BY-SA 4.0 (the reason this model card
is CC-BY-SA)
- [Twemoji](https://github.com/jdecked/twemoji) — CC-BY 4.0, © Twitter/X and
contributors
- [Dungeon Crawl Stone Soup tiles](https://github.com/crawl/crawl) — public domain
- [Kenney asset packs](https://kenney.nl) — CC0
- [DawnLike tileset](https://opengameart.org/content/dawnlike-16x16-universal-rogue-like-tileset-v181)
— CC-BY 4.0, credit DragonDePlatino and DawnBringer
## Limitations
- The unconditional 32px emoji model partially memorizes its 8.5k-image
training set (~1/3 of samples are near-duplicates of training images).
- Rare classes (a few hundred images) produce less diverse samples.
- `sprite/kenney-characters` is trained on paper-doll layers (bodies,
clothing, hats as separate sprites), so it generates those parts rather
than assembled characters, and retains some flat-green artifacts from a
data-contamination issue fixed partway through training (see repo history).
- Outputs are white-background RGB; no alpha channel.
|