π Trout: a diffusion language model for 16Γ16 pixel art
Trout is a 47.6M-parameter absorbing-state masked diffusion model that treats each 16Γ16 sprite as a 256-token "sentence" over a learned 512-color palette (token 0 = transparent). A bidirectional hourglass transformer starts from a fully masked canvas and unmasks all 256 pixels in parallel in about 16 steps, conditioned on a frozen CLIP text encoder. No VAE, no latent space, no autoregression.
Trout was built and trained from scratch (architecture after
Tessera-small by Jagrat Patel
(TheJagStudio), whose open model card and demo space document the design) on a
2-vCPU CPU-only sandbox. All numbers below are real and reproducible from
metrics.jsonl.
Model
| Parameters | 47,597,568 (Tessera-small: 47.6M) |
| Architecture | joint text+pixel bidirectional transformer, hourglass 2 + 8 + 2 blocks (2Γ2 patch-merged core), width 512, 8 heads, QK-RMSNorm, 2D axial RoPE (ΞΈ=100), Conv-SwiGLU, adaLN-single, 4 registers, self-conditioning, learned pos-emb |
| Text encoder | frozen openai/clip-vit-base-patch32 text tower, 24 tokens (not included, MIT) |
| Vocabulary | 512 palette tokens (frequency-weighted OKLab k-means, mean ΞE 0.0099, snapped to dataset colors) + <MASK> |
| Objective | absorbing-state masked diffusion, cosine mask schedule, stratified t, blob masks (25%), perceptual OKLab label smoothing (Ξ΅=0.1, Ο=0.05), CFG dropout (10%), hflip aug |
| Sampler | parallel confidence-ordered unmasking with CFG, Gumbel-annealed choice (T=2), spatial NMS (3Γ3, 0.9), self-conditioning feedback; inpainting supported |
| Optimizer | Muon (batched Newton-Schulz, RMS-matched 0.2Β·βmax(m,n)) on hidden matrices + AdamW elsewhere, Warmup-Stable-Decay, EMA |
Training β and the honest compute gap
Tessera-small was trained for 82,342 steps Γ 512 sprites on an RTX PRO 6000 (β5.5 h). Trout was trained from scratch on 2 vCPUs / 4 GB RAM for 2,400 steps Γ 8 sprites (~19k sprite-views of a 60k-sprite subset of the same dataset) in CPU fp32. The recipe (schedule, smoothing, self-conditioning, CFG dropout, Muon+AdamW, WSD, EMA) is followed faithfully; only the compute budget is smaller by ~4 orders of magnitude, so Trout's metrics are correspondingly weaker than Tessera's β this is a genuine budget-limited from-scratch run, not a distilled or fine-tuned copy.
Metrics (exported weights)
| Metric | Trout (this run) | Tessera-small (reference) |
|---|---|---|
| Validation NLL | 2.176 nats/pixel (perplexity 8.81 of 512) | 0.859 nats/pixel (2.36) |
| Validation accuracy | 54.6% of masked pixels | 78.8% |
| CLIP score (ViT-B/32) | 0.23 generated vs 0.26 real sprites |
The full per-step log is in metrics.jsonl.
Usage
from huggingface_hub import hf_hub_download
from trout import load_model_from_checkpoint, FrozenTextEncoder, sample, tokens_to_images, make_grid
ckpt = hf_hub_download("FrameXlabs/Trout", "trout-fp16.pt")
model, palette, meta = load_model_from_checkpoint(ckpt, device="cpu")
enc = FrozenTextEncoder(meta["text_encoder"], model.cfg.text_len, "cpu")
states, pooled = enc.encode(["golden crown with blue gems"] * 4)
toks = sample(model, states, pooled, steps=16, cfg_scale=3.0,
generator=torch.Generator().manual_seed(0))
images = tokens_to_images(toks, palette)
make_grid(images, ncols=4).save("crowns.png") # or try the live demo Space
Files
| File | Contents |
|---|---|
trout-fp16.pt |
exported weights (fp16; raw/EMA chosen by validation NLL), model config, palette, text-encoder id, step |
trout.py |
single-file from-scratch implementation (model, palette, sampler, pipeline helpers) |
model_config.json, train_config.json |
architecture + training hyperparameters |
metrics.jsonl |
full training and validation log |
showcase.png |
sample grid (16 prompts Γ 4 samples, steps=16, cfg=3.0) |
Sources & credits (the knowledge this model is built on)
- Tessera-small by Jagrat Patel (TheJagStudio) β the architecture this from-scratch reimplementation follows; its open model card and demo space were the design reference.
- Pixel Art Atlas 1.4M by Angais (CC0) β training data.
- Sahoo et al., Simple and Effective Masked Diffusion Language Models, NeurIPS 2024 (arXiv:2406.07524).
- Shi et al., Simplified and Generalized Masked Diffusion for Discrete Data, NeurIPS 2024 (arXiv:2406.04329).
- Chang et al., MaskGIT: Masked Generative Image Transformer, CVPR 2022 (arXiv:2202.04200).
- Blecher et al., Nougat hourglass transformer (arXiv:2308.13418).
- Peebles & Xie, DiT adaLN (arXiv:2212.09748); PixArt-Ξ± adaLN-single.
- Darcet et al., Vision Transformers Need Registers, ICLR 2024.
- Chen, Analog Bits self-conditioning (arXiv:2208.04202).
- Jordan et al., Muon (arXiv:2502.16982); Moonshot, Muon is Scalable (arXiv:2508.02535).
- Ottosson, OKLab perceptual color space (2020).
License
Apache-2.0 (Trout weights and code; trained on CC0 data; CLIP text tower keeps its own MIT license and is not redistributed here). Architecture credit to Tessera-small as noted above.