Trout / README.md
FrameXlabs's picture
Trout: from-scratch 47.6M masked-diffusion pixel-art model
c5dd0d7 verified
|
Raw History Blame Contribute Delete
5.5 kB
metadata
license: apache-2.0
library_name: pytorch
pipeline_tag: text-to-image
datasets:
  - Angais/pixel-art-atlas-1.4m
tags:
  - pixel-art
  - text-to-image
  - masked-diffusion
  - discrete-diffusion
  - diffusion-lm

🐟 Trout: a diffusion language model for 16×16 pixel art

Trout is a 47.6M-parameter absorbing-state masked diffusion model that treats each 16×16 sprite as a 256-token "sentence" over a learned 512-color palette (token 0 = transparent). A bidirectional hourglass transformer starts from a fully masked canvas and unmasks all 256 pixels in parallel in about 16 steps, conditioned on a frozen CLIP text encoder. No VAE, no latent space, no autoregression.

Trout was built and trained from scratch (architecture after Tessera-small by Jagrat Patel (TheJagStudio), whose open model card and demo space document the design) on a 2-vCPU CPU-only sandbox. All numbers below are real and reproducible from metrics.jsonl.

Model

Parameters 47,597,568 (Tessera-small: 47.6M)
Architecture joint text+pixel bidirectional transformer, hourglass 2 + 8 + 2 blocks (2×2 patch-merged core), width 512, 8 heads, QK-RMSNorm, 2D axial RoPE (θ=100), Conv-SwiGLU, adaLN-single, 4 registers, self-conditioning, learned pos-emb
Text encoder frozen openai/clip-vit-base-patch32 text tower, 24 tokens (not included, MIT)
Vocabulary 512 palette tokens (frequency-weighted OKLab k-means, mean ΔE 0.0099, snapped to dataset colors) + <MASK>
Objective absorbing-state masked diffusion, cosine mask schedule, stratified t, blob masks (25%), perceptual OKLab label smoothing (ε=0.1, τ=0.05), CFG dropout (10%), hflip aug
Sampler parallel confidence-ordered unmasking with CFG, Gumbel-annealed choice (T=2), spatial NMS (3×3, 0.9), self-conditioning feedback; inpainting supported
Optimizer Muon (batched Newton-Schulz, RMS-matched 0.2·√max(m,n)) on hidden matrices + AdamW elsewhere, Warmup-Stable-Decay, EMA

Training — and the honest compute gap

Tessera-small was trained for 82,342 steps × 512 sprites on an RTX PRO 6000 (≈5.5 h). Trout was trained from scratch on 2 vCPUs / 4 GB RAM for 2,400 steps × 8 sprites (~19k sprite-views of a 60k-sprite subset of the same dataset) in CPU fp32. The recipe (schedule, smoothing, self-conditioning, CFG dropout, Muon+AdamW, WSD, EMA) is followed faithfully; only the compute budget is smaller by ~4 orders of magnitude, so Trout's metrics are correspondingly weaker than Tessera's — this is a genuine budget-limited from-scratch run, not a distilled or fine-tuned copy.

Metrics (exported weights)

Metric Trout (this run) Tessera-small (reference)
Validation NLL 2.176 nats/pixel (perplexity 8.81 of 512) 0.859 nats/pixel (2.36)
Validation accuracy 54.6% of masked pixels 78.8%
CLIP score (ViT-B/32) 0.23 generated vs 0.26 real sprites

The full per-step log is in metrics.jsonl.

Usage

from huggingface_hub import hf_hub_download
from trout import load_model_from_checkpoint, FrozenTextEncoder, sample, tokens_to_images, make_grid

ckpt = hf_hub_download("FrameXlabs/Trout", "trout-fp16.pt")
model, palette, meta = load_model_from_checkpoint(ckpt, device="cpu")
enc = FrozenTextEncoder(meta["text_encoder"], model.cfg.text_len, "cpu")

states, pooled = enc.encode(["golden crown with blue gems"] * 4)
toks = sample(model, states, pooled, steps=16, cfg_scale=3.0,
              generator=torch.Generator().manual_seed(0))
images = tokens_to_images(toks, palette)
make_grid(images, ncols=4).save("crowns.png")   # or try the live demo Space

Files

File Contents
trout-fp16.pt exported weights (fp16; raw/EMA chosen by validation NLL), model config, palette, text-encoder id, step
trout.py single-file from-scratch implementation (model, palette, sampler, pipeline helpers)
model_config.json, train_config.json architecture + training hyperparameters
metrics.jsonl full training and validation log
showcase.png sample grid (16 prompts × 4 samples, steps=16, cfg=3.0)

Sources & credits (the knowledge this model is built on)

  • Tessera-small by Jagrat Patel (TheJagStudio) — the architecture this from-scratch reimplementation follows; its open model card and demo space were the design reference.
  • Pixel Art Atlas 1.4M by Angais (CC0) — training data.
  • Sahoo et al., Simple and Effective Masked Diffusion Language Models, NeurIPS 2024 (arXiv:2406.07524).
  • Shi et al., Simplified and Generalized Masked Diffusion for Discrete Data, NeurIPS 2024 (arXiv:2406.04329).
  • Chang et al., MaskGIT: Masked Generative Image Transformer, CVPR 2022 (arXiv:2202.04200).
  • Blecher et al., Nougat hourglass transformer (arXiv:2308.13418).
  • Peebles & Xie, DiT adaLN (arXiv:2212.09748); PixArt-α adaLN-single.
  • Darcet et al., Vision Transformers Need Registers, ICLR 2024.
  • Chen, Analog Bits self-conditioning (arXiv:2208.04202).
  • Jordan et al., Muon (arXiv:2502.16982); Moonshot, Muon is Scalable (arXiv:2508.02535).
  • Ottosson, OKLab perceptual color space (2020).

License

Apache-2.0 (Trout weights and code; trained on CC0 data; CLIP text tower keeps its own MIT license and is not redistributed here). Architecture credit to Tessera-small as noted above.