|
Download README.md from FrameXlabs/Trout: direct link, hf CLI and curl.
- Browser
- Download file 5.5 kB
-
https://huggingface.co/FrameXlabs/Trout/resolve/main/README.md
- Command line
-
hf download hf://FrameXlabs/Trout/README.md
-
curl -L -o README.md https://huggingface.co/FrameXlabs/Trout/resolve/main/README.md
5.5 kB
| license: apache-2.0 | |
| library_name: pytorch | |
| pipeline_tag: text-to-image | |
| datasets: | |
| - Angais/pixel-art-atlas-1.4m | |
| tags: | |
| - pixel-art | |
| - text-to-image | |
| - masked-diffusion | |
| - discrete-diffusion | |
| - diffusion-lm | |
| # 🐟 Trout: a diffusion language model for 16×16 pixel art | |
| **Trout** is a 47.6M-parameter absorbing-state **masked diffusion** model that treats each | |
| 16×16 sprite as a 256-token "sentence" over a learned **512-color palette** | |
| (token 0 = transparent). A bidirectional hourglass transformer starts from a | |
| fully masked canvas and unmasks all 256 pixels **in parallel in about 16 steps**, | |
| conditioned on a frozen CLIP text encoder. No VAE, no latent space, no autoregression. | |
| Trout was **built and trained from scratch** (architecture after | |
| [Tessera-small](https://huggingface.co/thejagstudio/tessera-small) by Jagrat Patel | |
| (TheJagStudio), whose open model card and demo space document the design) on a | |
| 2-vCPU CPU-only sandbox. All numbers below are real and reproducible from | |
| `metrics.jsonl`. | |
| ## Model | |
| | | | | |
| |---|---| | |
| | Parameters | 47,597,568 (Tessera-small: 47.6M) | | |
| | Architecture | joint text+pixel bidirectional transformer, hourglass 2 + 8 + 2 blocks (2×2 patch-merged core), width 512, 8 heads, QK-RMSNorm, 2D axial RoPE (θ=100), Conv-SwiGLU, adaLN-single, 4 registers, self-conditioning, learned pos-emb | | |
| | Text encoder | frozen `openai/clip-vit-base-patch32` text tower, 24 tokens (not included, MIT) | | |
| | Vocabulary | 512 palette tokens (frequency-weighted OKLab k-means, mean ΔE 0.0099, snapped to dataset colors) + `<MASK>` | | |
| | Objective | absorbing-state masked diffusion, cosine mask schedule, stratified t, blob masks (25%), perceptual OKLab label smoothing (ε=0.1, τ=0.05), CFG dropout (10%), hflip aug | | |
| | Sampler | parallel confidence-ordered unmasking with CFG, Gumbel-annealed choice (T=2), spatial NMS (3×3, 0.9), self-conditioning feedback; inpainting supported | | |
| | Optimizer | Muon (batched Newton-Schulz, RMS-matched 0.2·√max(m,n)) on hidden matrices + AdamW elsewhere, Warmup-Stable-Decay, EMA | | |
| ## Training — and the honest compute gap | |
| Tessera-small was trained for **82,342 steps × 512 sprites** on an RTX PRO 6000 | |
| (≈5.5 h). Trout was trained **from scratch on 2 vCPUs / 4 GB RAM** for | |
| **2,400 steps × 8 sprites** (~19k sprite-views of a 60k-sprite | |
| subset of the same dataset) in CPU fp32. The recipe (schedule, smoothing, | |
| self-conditioning, CFG dropout, Muon+AdamW, WSD, EMA) is followed faithfully; | |
| only the compute budget is smaller by ~4 orders of magnitude, so Trout's | |
| metrics are correspondingly weaker than Tessera's — this is a genuine | |
| budget-limited from-scratch run, not a distilled or fine-tuned copy. | |
| ## Metrics (exported weights) | |
| | Metric | Trout (this run) | Tessera-small (reference) | | |
| |---|---|---| | |
| | Validation NLL | 2.176 nats/pixel (perplexity 8.81 of 512) | 0.859 nats/pixel (2.36) | | |
| | Validation accuracy | 54.6% of masked pixels | 78.8% | | |
| | CLIP score (ViT-B/32) | 0.23 generated vs 0.26 real sprites | | |
| The full per-step log is in `metrics.jsonl`. | |
| ## Usage | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| from trout import load_model_from_checkpoint, FrozenTextEncoder, sample, tokens_to_images, make_grid | |
| ckpt = hf_hub_download("FrameXlabs/Trout", "trout-fp16.pt") | |
| model, palette, meta = load_model_from_checkpoint(ckpt, device="cpu") | |
| enc = FrozenTextEncoder(meta["text_encoder"], model.cfg.text_len, "cpu") | |
| states, pooled = enc.encode(["golden crown with blue gems"] * 4) | |
| toks = sample(model, states, pooled, steps=16, cfg_scale=3.0, | |
| generator=torch.Generator().manual_seed(0)) | |
| images = tokens_to_images(toks, palette) | |
| make_grid(images, ncols=4).save("crowns.png") # or try the live demo Space | |
| ``` | |
| ## Files | |
| | File | Contents | | |
| |---|---| | |
| | `trout-fp16.pt` | exported weights (fp16; raw/EMA chosen by validation NLL), model config, palette, text-encoder id, step | | |
| | `trout.py` | single-file from-scratch implementation (model, palette, sampler, pipeline helpers) | | |
| | `model_config.json`, `train_config.json` | architecture + training hyperparameters | | |
| | `metrics.jsonl` | full training and validation log | | |
| | `showcase.png` | sample grid (16 prompts × 4 samples, steps=16, cfg=3.0) | | |
| ## Sources & credits (the knowledge this model is built on) | |
| - **Tessera-small** by Jagrat Patel ([TheJagStudio](https://huggingface.co/thejagstudio)) — | |
| the architecture this from-scratch reimplementation follows; its open model card and demo | |
| space were the design reference. | |
| - **Pixel Art Atlas 1.4M** by [Angais](https://huggingface.co/Angais) (CC0) — training data. | |
| - Sahoo et al., *Simple and Effective Masked Diffusion Language Models*, NeurIPS 2024 (arXiv:2406.07524). | |
| - Shi et al., *Simplified and Generalized Masked Diffusion for Discrete Data*, NeurIPS 2024 (arXiv:2406.04329). | |
| - Chang et al., *MaskGIT: Masked Generative Image Transformer*, CVPR 2022 (arXiv:2202.04200). | |
| - Blecher et al., *Nougat* hourglass transformer (arXiv:2308.13418). | |
| - Peebles & Xie, *DiT* adaLN (arXiv:2212.09748); PixArt-α adaLN-single. | |
| - Darcet et al., *Vision Transformers Need Registers*, ICLR 2024. | |
| - Chen, *Analog Bits* self-conditioning (arXiv:2208.04202). | |
| - Jordan et al., *Muon* (arXiv:2502.16982); Moonshot, *Muon is Scalable* (arXiv:2508.02535). | |
| - Ottosson, *OKLab* perceptual color space (2020). | |
| ## License | |
| Apache-2.0 (Trout weights and code; trained on CC0 data; CLIP text tower keeps | |
| its own MIT license and is not redistributed here). Architecture credit to | |
| Tessera-small as noted above. | |