README.md

---
license: cc-by-sa-4.0
tags:
- diffusion
- rectified-flow
- flow-matching
- pixel-art
- emoji
- sprites
- image-generation
- text-to-image
- clip
- pytorch
library_name: pytorch
pipeline_tag: text-to-image
---

# sprite-gpt

Small pixel-space diffusion models for emoji and game sprites, trained with
rectified flow (flow matching) on a single RTX 5090. Code:
[github.com/gmmeyer/sprite-diffusion](https://github.com/gmmeyer/sprite-diffusion)

| file | model | params | training |
|---|---|---|---|
| `sprite-gpt-text64.pt` | **text-conditioned 64px** (CLIP ViT-B/32 + cross-attention) | 61.9M | 120k steps @ bs128 on 24,134 emoji+sprites |
| `sprite-gpt-cond64.pt` | class-conditional 64px, 31 classes | 55.8M | 95k steps @ bs128 on 24,134 emoji+sprites |
| `sprite-gpt-emoji32.pt` | unconditional 32px emoji | 29.1M | 60k steps @ bs256 on 8,504 emoji |

The text model composes attributes not seen together in training — prompts
below are "a purple dragon emoji", "an angry ghost sprite", "a blue cat face
emoji", "a green skull sprite", "ice dungeon wall tile", "a red sword item",
"a crying robot face emoji", "a yellow snake monster" (8 seeds each):

![compositional prompts](compositional.png)

Both are ADM-style U-Nets (scale-shift GroupNorm conditioning, self-attention
at 16x16 and 8x8) trained with the rectified-flow objective
(`x_t = (1-t)·x0 + t·ε`, predict `v = ε − x0`, logit-normal timestep
sampling). Weights are fp32 EMA (decay 0.9995). Sampling: 50-step Euler ODE;
the conditional model uses classifier-free guidance (null class = index 31,
condition dropout 10% at train time; guidance 3 works well).

![conditional samples](samples_grid.png)

## Usage

Text-to-image (easiest via the repo CLI):

```sh
git clone https://github.com/gmmeyer/sprite-diffusion && cd sprite-diffusion && uv sync
uv run python -m spritegpt.sample --ckpt sprite-gpt-text64.pt \
    --prompt "a blue ghost sprite" --guidance 5 --out ghost.png
```

(`sample.py` accepts these exported files as well as training checkpoints.)

Class-conditional model, by hand:

```python
# git clone https://github.com/gmmeyer/sprite-diffusion && cd sprite-diffusion && uv sync
import torch
from spritegpt.unet import UNet
from spritegpt.flow import sample
from spritegpt.utils import save_image_grid

ck = torch.load("sprite-gpt-cond64.pt", map_location="cuda", weights_only=False)
cfg = ck["config"]
model = UNet(
    img_size=cfg["img_size"],
    base=cfg["base"],
    ch_mult=tuple(int(c) for c in cfg["ch_mult"].split(",")),
    num_res=cfg["num_res"],
    attn_res=tuple(int(r) for r in cfg["attn_res"].split(",")),
    num_classes=cfg["num_classes"],
).cuda()
model.load_state_dict(ck["ema_state_dict"])
model.eval()

y = torch.full((16,), 24, device="cuda", dtype=torch.long)  # sprite/dcss-mon
with torch.autocast("cuda", dtype=torch.bfloat16):
    imgs = sample(model, 16, 64, steps=50, y=y, guidance=3.0,
                  num_classes=cfg["num_classes"], device="cuda")
save_image_grid(imgs.float(), "monsters.png", nrow=4)
```

## Classes (conditional model)

Class ids are the alphabetical index into:

`emoji/activities, emoji/animals-nature, emoji/component, emoji/extras-openmoji,
emoji/extras-unicode, emoji/flags, emoji/food-drink, emoji/objects,
emoji/people-body, emoji/smileys-emotion, emoji/symbols, emoji/travel-places,
emoji/unknown, sprite/dawnlike, sprite/dcss-dungeon, sprite/dcss-effect,
sprite/dcss-item, sprite/dcss-misc, sprite/dcss-mon, sprite/dcss-player,
sprite/kenney-1bit, sprite/kenney-caves, sprite/kenney-characters,
sprite/kenney-micro-roguelike, sprite/kenney-modern-city,
sprite/kenney-pixel-platformer, sprite/kenney-pixel-shmup, sprite/kenney-rpg,
sprite/kenney-tiny-battle, sprite/kenney-tiny-dungeon, sprite/kenney-tiny-town`

## Training data and attribution

Images composited onto white backgrounds at 32/64px:

- [OpenMoji](https://openmoji.org) — CC-BY-SA 4.0 (the reason this model card
  is CC-BY-SA)
- [Twemoji](https://github.com/jdecked/twemoji) — CC-BY 4.0, © Twitter/X and
  contributors
- [Dungeon Crawl Stone Soup tiles](https://github.com/crawl/crawl) — public domain
- [Kenney asset packs](https://kenney.nl) — CC0
- [DawnLike tileset](https://opengameart.org/content/dawnlike-16x16-universal-rogue-like-tileset-v181)
  — CC-BY 4.0, credit DragonDePlatino and DawnBringer

## Limitations

- The unconditional 32px emoji model partially memorizes its 8.5k-image
  training set (~1/3 of samples are near-duplicates of training images).
- Rare classes (a few hundred images) produce less diverse samples.
- `sprite/kenney-characters` is trained on paper-doll layers (bodies,
  clothing, hats as separate sprites), so it generates those parts rather
  than assembled characters, and retains some flat-green artifacts from a
  data-contamination issue fixed partway through training (see repo history).
- Outputs are white-background RGB; no alpha channel.