--- license: apache-2.0 library_name: pytorch pipeline_tag: text-to-image datasets: - Angais/pixel-art-atlas-1.4m tags: - pixel-art - text-to-image - masked-diffusion - discrete-diffusion - diffusion-lm --- # 🐟 Trout: a diffusion language model for 16×16 pixel art **Trout** is a 47.6M-parameter absorbing-state **masked diffusion** model that treats each 16×16 sprite as a 256-token "sentence" over a learned **512-color palette** (token 0 = transparent). A bidirectional hourglass transformer starts from a fully masked canvas and unmasks all 256 pixels **in parallel in about 16 steps**, conditioned on a frozen CLIP text encoder. No VAE, no latent space, no autoregression. Trout was **built and trained from scratch** (architecture after [Tessera-small](https://huggingface.co/thejagstudio/tessera-small) by Jagrat Patel (TheJagStudio), whose open model card and demo space document the design) on a 2-vCPU CPU-only sandbox. All numbers below are real and reproducible from `metrics.jsonl`. ## Model | | | |---|---| | Parameters | 47,597,568 (Tessera-small: 47.6M) | | Architecture | joint text+pixel bidirectional transformer, hourglass 2 + 8 + 2 blocks (2×2 patch-merged core), width 512, 8 heads, QK-RMSNorm, 2D axial RoPE (θ=100), Conv-SwiGLU, adaLN-single, 4 registers, self-conditioning, learned pos-emb | | Text encoder | frozen `openai/clip-vit-base-patch32` text tower, 24 tokens (not included, MIT) | | Vocabulary | 512 palette tokens (frequency-weighted OKLab k-means, mean ΔE 0.0099, snapped to dataset colors) + `` | | Objective | absorbing-state masked diffusion, cosine mask schedule, stratified t, blob masks (25%), perceptual OKLab label smoothing (ε=0.1, τ=0.05), CFG dropout (10%), hflip aug | | Sampler | parallel confidence-ordered unmasking with CFG, Gumbel-annealed choice (T=2), spatial NMS (3×3, 0.9), self-conditioning feedback; inpainting supported | | Optimizer | Muon (batched Newton-Schulz, RMS-matched 0.2·√max(m,n)) on hidden matrices + AdamW elsewhere, Warmup-Stable-Decay, EMA | ## Training — and the honest compute gap Tessera-small was trained for **82,342 steps × 512 sprites** on an RTX PRO 6000 (≈5.5 h). Trout was trained **from scratch on 2 vCPUs / 4 GB RAM** for **2,400 steps × 8 sprites** (~19k sprite-views of a 60k-sprite subset of the same dataset) in CPU fp32. The recipe (schedule, smoothing, self-conditioning, CFG dropout, Muon+AdamW, WSD, EMA) is followed faithfully; only the compute budget is smaller by ~4 orders of magnitude, so Trout's metrics are correspondingly weaker than Tessera's — this is a genuine budget-limited from-scratch run, not a distilled or fine-tuned copy. ## Metrics (exported weights) | Metric | Trout (this run) | Tessera-small (reference) | |---|---|---| | Validation NLL | 2.176 nats/pixel (perplexity 8.81 of 512) | 0.859 nats/pixel (2.36) | | Validation accuracy | 54.6% of masked pixels | 78.8% | | CLIP score (ViT-B/32) | 0.23 generated vs 0.26 real sprites | The full per-step log is in `metrics.jsonl`. ## Usage ```python from huggingface_hub import hf_hub_download from trout import load_model_from_checkpoint, FrozenTextEncoder, sample, tokens_to_images, make_grid ckpt = hf_hub_download("FrameXlabs/Trout", "trout-fp16.pt") model, palette, meta = load_model_from_checkpoint(ckpt, device="cpu") enc = FrozenTextEncoder(meta["text_encoder"], model.cfg.text_len, "cpu") states, pooled = enc.encode(["golden crown with blue gems"] * 4) toks = sample(model, states, pooled, steps=16, cfg_scale=3.0, generator=torch.Generator().manual_seed(0)) images = tokens_to_images(toks, palette) make_grid(images, ncols=4).save("crowns.png") # or try the live demo Space ``` ## Files | File | Contents | |---|---| | `trout-fp16.pt` | exported weights (fp16; raw/EMA chosen by validation NLL), model config, palette, text-encoder id, step | | `trout.py` | single-file from-scratch implementation (model, palette, sampler, pipeline helpers) | | `model_config.json`, `train_config.json` | architecture + training hyperparameters | | `metrics.jsonl` | full training and validation log | | `showcase.png` | sample grid (16 prompts × 4 samples, steps=16, cfg=3.0) | ## Sources & credits (the knowledge this model is built on) - **Tessera-small** by Jagrat Patel ([TheJagStudio](https://huggingface.co/thejagstudio)) — the architecture this from-scratch reimplementation follows; its open model card and demo space were the design reference. - **Pixel Art Atlas 1.4M** by [Angais](https://huggingface.co/Angais) (CC0) — training data. - Sahoo et al., *Simple and Effective Masked Diffusion Language Models*, NeurIPS 2024 (arXiv:2406.07524). - Shi et al., *Simplified and Generalized Masked Diffusion for Discrete Data*, NeurIPS 2024 (arXiv:2406.04329). - Chang et al., *MaskGIT: Masked Generative Image Transformer*, CVPR 2022 (arXiv:2202.04200). - Blecher et al., *Nougat* hourglass transformer (arXiv:2308.13418). - Peebles & Xie, *DiT* adaLN (arXiv:2212.09748); PixArt-α adaLN-single. - Darcet et al., *Vision Transformers Need Registers*, ICLR 2024. - Chen, *Analog Bits* self-conditioning (arXiv:2208.04202). - Jordan et al., *Muon* (arXiv:2502.16982); Moonshot, *Muon is Scalable* (arXiv:2508.02535). - Ottosson, *OKLab* perceptual color space (2020). ## License Apache-2.0 (Trout weights and code; trained on CC0 data; CLIP text tower keeps its own MIT license and is not redistributed here). Architecture credit to Tessera-small as noted above.