FrameXlabs commited on
Commit
c5dd0d7
·
verified ·
1 Parent(s): 8a5d945

Trout: from-scratch 47.6M masked-diffusion pixel-art model

Browse files
Files changed (9) hide show
  1. LICENSE +22 -0
  2. README.md +109 -0
  3. clip_score.json +6 -0
  4. metrics.jsonl +250 -0
  5. model_config.json +18 -0
  6. showcase.png +0 -0
  7. train_config.json +55 -0
  8. trout-fp16.pt +3 -0
  9. trout.py +1017 -0
LICENSE ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ Licensed under the Apache License, Version 2.0 (the "License");
6
+ you may not use this file except in compliance with the License.
7
+ You may obtain a copy of the License at
8
+
9
+ http://www.apache.org/licenses/LICENSE-2.0
10
+
11
+ Unless required by applicable law or agreed to in writing, software
12
+ distributed under the License is distributed on an "AS IS" BASIS,
13
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ See the License for the specific language governing permissions and
15
+ limitations under the License.
16
+
17
+ ----
18
+
19
+ Trout was trained from scratch on the Pixel Art Atlas 1.4M dataset
20
+ (Angais, CC0 1.0). The CLIP text encoder used for conditioning
21
+ (openai/clip-vit-base-patch32) keeps its own MIT license and is not
22
+ redistributed with these weights.
README.md ADDED
@@ -0,0 +1,109 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: pytorch
4
+ pipeline_tag: text-to-image
5
+ datasets:
6
+ - Angais/pixel-art-atlas-1.4m
7
+ tags:
8
+ - pixel-art
9
+ - text-to-image
10
+ - masked-diffusion
11
+ - discrete-diffusion
12
+ - diffusion-lm
13
+ ---
14
+
15
+ # 🐟 Trout: a diffusion language model for 16×16 pixel art
16
+
17
+ **Trout** is a 47.6M-parameter absorbing-state **masked diffusion** model that treats each
18
+ 16×16 sprite as a 256-token "sentence" over a learned **512-color palette**
19
+ (token 0 = transparent). A bidirectional hourglass transformer starts from a
20
+ fully masked canvas and unmasks all 256 pixels **in parallel in about 16 steps**,
21
+ conditioned on a frozen CLIP text encoder. No VAE, no latent space, no autoregression.
22
+
23
+ Trout was **built and trained from scratch** (architecture after
24
+ [Tessera-small](https://huggingface.co/thejagstudio/tessera-small) by Jagrat Patel
25
+ (TheJagStudio), whose open model card and demo space document the design) on a
26
+ 2-vCPU CPU-only sandbox. All numbers below are real and reproducible from
27
+ `metrics.jsonl`.
28
+
29
+ ## Model
30
+
31
+ | | |
32
+ |---|---|
33
+ | Parameters | 47,597,568 (Tessera-small: 47.6M) |
34
+ | Architecture | joint text+pixel bidirectional transformer, hourglass 2 + 8 + 2 blocks (2×2 patch-merged core), width 512, 8 heads, QK-RMSNorm, 2D axial RoPE (θ=100), Conv-SwiGLU, adaLN-single, 4 registers, self-conditioning, learned pos-emb |
35
+ | Text encoder | frozen `openai/clip-vit-base-patch32` text tower, 24 tokens (not included, MIT) |
36
+ | Vocabulary | 512 palette tokens (frequency-weighted OKLab k-means, mean ΔE 0.0099, snapped to dataset colors) + `<MASK>` |
37
+ | Objective | absorbing-state masked diffusion, cosine mask schedule, stratified t, blob masks (25%), perceptual OKLab label smoothing (ε=0.1, τ=0.05), CFG dropout (10%), hflip aug |
38
+ | Sampler | parallel confidence-ordered unmasking with CFG, Gumbel-annealed choice (T=2), spatial NMS (3×3, 0.9), self-conditioning feedback; inpainting supported |
39
+ | Optimizer | Muon (batched Newton-Schulz, RMS-matched 0.2·√max(m,n)) on hidden matrices + AdamW elsewhere, Warmup-Stable-Decay, EMA |
40
+
41
+ ## Training — and the honest compute gap
42
+
43
+ Tessera-small was trained for **82,342 steps × 512 sprites** on an RTX PRO 6000
44
+ (≈5.5 h). Trout was trained **from scratch on 2 vCPUs / 4 GB RAM** for
45
+ **2,400 steps × 8 sprites** (~19k sprite-views of a 60k-sprite
46
+ subset of the same dataset) in CPU fp32. The recipe (schedule, smoothing,
47
+ self-conditioning, CFG dropout, Muon+AdamW, WSD, EMA) is followed faithfully;
48
+ only the compute budget is smaller by ~4 orders of magnitude, so Trout's
49
+ metrics are correspondingly weaker than Tessera's — this is a genuine
50
+ budget-limited from-scratch run, not a distilled or fine-tuned copy.
51
+
52
+ ## Metrics (exported weights)
53
+
54
+ | Metric | Trout (this run) | Tessera-small (reference) |
55
+ |---|---|---|
56
+ | Validation NLL | 2.176 nats/pixel (perplexity 8.81 of 512) | 0.859 nats/pixel (2.36) |
57
+ | Validation accuracy | 54.6% of masked pixels | 78.8% |
58
+ | CLIP score (ViT-B/32) | 0.23 generated vs 0.26 real sprites |
59
+
60
+ The full per-step log is in `metrics.jsonl`.
61
+
62
+ ## Usage
63
+
64
+ ```python
65
+ from huggingface_hub import hf_hub_download
66
+ from trout import load_model_from_checkpoint, FrozenTextEncoder, sample, tokens_to_images, make_grid
67
+
68
+ ckpt = hf_hub_download("FrameXlabs/Trout", "trout-fp16.pt")
69
+ model, palette, meta = load_model_from_checkpoint(ckpt, device="cpu")
70
+ enc = FrozenTextEncoder(meta["text_encoder"], model.cfg.text_len, "cpu")
71
+
72
+ states, pooled = enc.encode(["golden crown with blue gems"] * 4)
73
+ toks = sample(model, states, pooled, steps=16, cfg_scale=3.0,
74
+ generator=torch.Generator().manual_seed(0))
75
+ images = tokens_to_images(toks, palette)
76
+ make_grid(images, ncols=4).save("crowns.png") # or try the live demo Space
77
+ ```
78
+
79
+ ## Files
80
+
81
+ | File | Contents |
82
+ |---|---|
83
+ | `trout-fp16.pt` | exported weights (fp16; raw/EMA chosen by validation NLL), model config, palette, text-encoder id, step |
84
+ | `trout.py` | single-file from-scratch implementation (model, palette, sampler, pipeline helpers) |
85
+ | `model_config.json`, `train_config.json` | architecture + training hyperparameters |
86
+ | `metrics.jsonl` | full training and validation log |
87
+ | `showcase.png` | sample grid (16 prompts × 4 samples, steps=16, cfg=3.0) |
88
+
89
+ ## Sources & credits (the knowledge this model is built on)
90
+
91
+ - **Tessera-small** by Jagrat Patel ([TheJagStudio](https://huggingface.co/thejagstudio)) —
92
+ the architecture this from-scratch reimplementation follows; its open model card and demo
93
+ space were the design reference.
94
+ - **Pixel Art Atlas 1.4M** by [Angais](https://huggingface.co/Angais) (CC0) — training data.
95
+ - Sahoo et al., *Simple and Effective Masked Diffusion Language Models*, NeurIPS 2024 (arXiv:2406.07524).
96
+ - Shi et al., *Simplified and Generalized Masked Diffusion for Discrete Data*, NeurIPS 2024 (arXiv:2406.04329).
97
+ - Chang et al., *MaskGIT: Masked Generative Image Transformer*, CVPR 2022 (arXiv:2202.04200).
98
+ - Blecher et al., *Nougat* hourglass transformer (arXiv:2308.13418).
99
+ - Peebles & Xie, *DiT* adaLN (arXiv:2212.09748); PixArt-α adaLN-single.
100
+ - Darcet et al., *Vision Transformers Need Registers*, ICLR 2024.
101
+ - Chen, *Analog Bits* self-conditioning (arXiv:2208.04202).
102
+ - Jordan et al., *Muon* (arXiv:2502.16982); Moonshot, *Muon is Scalable* (arXiv:2508.02535).
103
+ - Ottosson, *OKLab* perceptual color space (2020).
104
+
105
+ ## License
106
+
107
+ Apache-2.0 (Trout weights and code; trained on CC0 data; CLIP text tower keeps
108
+ its own MIT license and is not redistributed here). Architecture credit to
109
+ Tessera-small as noted above.
clip_score.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "clip_score_generated": 0.2284971207845956,
3
+ "clip_score_real": 0.25555067602545023,
4
+ "n": 64,
5
+ "note": "CLIP ViT-B/32 cos-sim, 16-step cfg=3.0 samples vs real val sprites"
6
+ }
metrics.jsonl ADDED
@@ -0,0 +1,250 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"step": 10, "loss": 6.200986862182617, "nll": 6.200520038604736, "acc": 0.3830431401729584, "lr": 2.7499999999999998e-05, "sec_per_step": 6.398, "gnorm": 6.468178749084473}
2
+ {"step": 20, "loss": 6.085262298583984, "nll": 6.084070682525635, "acc": 0.34061792492866516, "lr": 5.25e-05, "sec_per_step": 6.11, "gnorm": 6.487669944763184}
3
+ {"step": 30, "loss": 5.7887115478515625, "nll": 5.784521579742432, "acc": 0.34792807698249817, "lr": 7.75e-05, "sec_per_step": 6.32, "gnorm": 8.065339088439941}
4
+ {"step": 40, "loss": 5.292694091796875, "nll": 5.2869391441345215, "acc": 0.3832145631313324, "lr": 0.0001025, "sec_per_step": 6.314, "gnorm": 10.66641902923584}
5
+ {"step": 50, "loss": 4.211287021636963, "nll": 4.189800262451172, "acc": 0.41723355650901794, "lr": 0.0001275, "sec_per_step": 6.067, "gnorm": 13.290396690368652}
6
+ {"step": 60, "loss": 4.039217472076416, "nll": 4.010549545288086, "acc": 0.3768569231033325, "lr": 0.0001525, "sec_per_step": 5.947, "gnorm": 4.629364013671875}
7
+ {"step": 70, "loss": 4.080512046813965, "nll": 4.0558061599731445, "acc": 0.3692913353443146, "lr": 0.00017750000000000003, "sec_per_step": 5.796, "gnorm": 4.4744062423706055}
8
+ {"step": 80, "loss": 3.756932497024536, "nll": 3.736483097076416, "acc": 0.422374427318573, "lr": 0.00020250000000000002, "sec_per_step": 5.894, "gnorm": 3.615117311477661}
9
+ {"step": 90, "loss": 4.03838586807251, "nll": 4.009660243988037, "acc": 0.35103243589401245, "lr": 0.0002275, "sec_per_step": 5.949, "gnorm": 9.545635223388672}
10
+ {"step": 100, "loss": 3.926302671432495, "nll": 3.896104335784912, "acc": 0.35153844952583313, "lr": 0.00025, "sec_per_step": 5.966, "gnorm": 5.883094787597656}
11
+ {"step": 110, "loss": 3.689901828765869, "nll": 3.681730031967163, "acc": 0.44033873081207275, "lr": 0.00025, "sec_per_step": 6.052, "gnorm": 5.032026767730713}
12
+ {"step": 120, "loss": 2.9494550228118896, "nll": 2.9360105991363525, "acc": 0.5653495192527771, "lr": 0.00025, "sec_per_step": 6.461, "gnorm": 3.8694350719451904}
13
+ {"step": 130, "loss": 3.870502233505249, "nll": 3.849687337875366, "acc": 0.37245696783065796, "lr": 0.00025, "sec_per_step": 6.388, "gnorm": 3.324218273162842}
14
+ {"step": 140, "loss": 3.858419418334961, "nll": 3.8451130390167236, "acc": 0.3631415367126465, "lr": 0.00025, "sec_per_step": 5.897, "gnorm": 2.8710551261901855}
15
+ {"step": 150, "loss": 3.7835071086883545, "nll": 3.757303237915039, "acc": 0.3560490012168884, "lr": 0.00025, "sec_per_step": 6.176, "gnorm": 3.1728737354278564}
16
+ {"step": 160, "loss": 3.879119873046875, "nll": 3.8416779041290283, "acc": 0.3188854455947876, "lr": 0.00025, "sec_per_step": 6.097, "gnorm": 6.751031398773193}
17
+ {"step": 170, "loss": 3.833686351776123, "nll": 3.8111066818237305, "acc": 0.3591117858886719, "lr": 0.00025, "sec_per_step": 6.261, "gnorm": 3.194800615310669}
18
+ {"step": 180, "loss": 3.9325289726257324, "nll": 3.9077517986297607, "acc": 0.32421875, "lr": 0.00025, "sec_per_step": 6.429, "gnorm": 5.764603614807129}
19
+ {"step": 190, "loss": 3.7423737049102783, "nll": 3.7159183025360107, "acc": 0.3640483319759369, "lr": 0.00025, "sec_per_step": 5.951, "gnorm": 3.4916608333587646}
20
+ {"step": 200, "loss": 3.4003562927246094, "nll": 3.387075662612915, "acc": 0.44687265157699585, "lr": 0.00025, "sec_per_step": 6.264, "gnorm": 3.4254374504089355}
21
+ {"step": 210, "loss": 3.4553167819976807, "nll": 3.4268157482147217, "acc": 0.3961038887500763, "lr": 0.00025, "sec_per_step": 6.379, "gnorm": 3.3052334785461426}
22
+ {"step": 220, "loss": 3.253338098526001, "nll": 3.226210594177246, "acc": 0.4332015812397003, "lr": 0.00025, "sec_per_step": 5.931, "gnorm": 4.595219135284424}
23
+ {"step": 230, "loss": 3.795321226119995, "nll": 3.771483898162842, "acc": 0.3651289939880371, "lr": 0.00025, "sec_per_step": 6.087, "gnorm": 5.327313423156738}
24
+ {"step": 240, "loss": 3.2495949268341064, "nll": 3.2198522090911865, "acc": 0.4190329909324646, "lr": 0.00025, "sec_per_step": 6.114, "gnorm": 5.86907958984375}
25
+ {"step": 250, "loss": 3.48093581199646, "nll": 3.4433395862579346, "acc": 0.38876405358314514, "lr": 0.00025, "sec_per_step": 6.339, "gnorm": 3.7719953060150146}
26
+ {"step": 250, "ema_val_nll": 5.711481094360352, "ema_val_acc": 0.399124912917614, "raw_val_nll": 3.4989233016967773, "raw_val_acc": 0.396411269903183}
27
+ {"step": 260, "loss": 3.0223655700683594, "nll": 2.9900503158569336, "acc": 0.4696296155452728, "lr": 0.00025, "sec_per_step": 6.07, "gnorm": 3.129960298538208}
28
+ {"step": 270, "loss": 3.310396432876587, "nll": 3.2859034538269043, "acc": 0.4375951290130615, "lr": 0.00025, "sec_per_step": 5.982, "gnorm": 6.224829196929932}
29
+ {"step": 280, "loss": 3.3965539932250977, "nll": 3.3703815937042236, "acc": 0.444947212934494, "lr": 0.00025, "sec_per_step": 6.051, "gnorm": 5.335007667541504}
30
+ {"step": 290, "loss": 3.239400863647461, "nll": 3.199361801147461, "acc": 0.4470762312412262, "lr": 0.00025, "sec_per_step": 5.781, "gnorm": 5.158711910247803}
31
+ {"step": 300, "loss": 2.912628173828125, "nll": 2.868650197982788, "acc": 0.489393949508667, "lr": 0.00025, "sec_per_step": 6.085, "gnorm": 3.7020480632781982}
32
+ {"step": 310, "loss": 3.418224573135376, "nll": 3.376781463623047, "acc": 0.38988327980041504, "lr": 0.00025, "sec_per_step": 5.879, "gnorm": 4.393625736236572}
33
+ {"step": 320, "loss": 3.3609254360198975, "nll": 3.3164608478546143, "acc": 0.38576778769493103, "lr": 0.00025, "sec_per_step": 5.719, "gnorm": 4.6255059242248535}
34
+ {"step": 330, "loss": 2.9795143604278564, "nll": 2.921900749206543, "acc": 0.47132351994514465, "lr": 0.00025, "sec_per_step": 6.111, "gnorm": 3.5180745124816895}
35
+ {"step": 340, "loss": 3.5651557445526123, "nll": 3.5129222869873047, "acc": 0.3803863227367401, "lr": 0.00025, "sec_per_step": 6.429, "gnorm": 6.238819122314453}
36
+ {"step": 350, "loss": 3.0168468952178955, "nll": 2.974374771118164, "acc": 0.46493300795555115, "lr": 0.00025, "sec_per_step": 6.299, "gnorm": 5.946975231170654}
37
+ {"step": 360, "loss": 2.665656566619873, "nll": 2.6218910217285156, "acc": 0.5297216176986694, "lr": 0.00025, "sec_per_step": 6.026, "gnorm": 4.707192420959473}
38
+ {"step": 370, "loss": 3.0414562225341797, "nll": 2.9841203689575195, "acc": 0.43898695707321167, "lr": 0.00025, "sec_per_step": 6.234, "gnorm": 5.299668788909912}
39
+ {"step": 380, "loss": 3.2714550495147705, "nll": 3.2110435962677, "acc": 0.41442155838012695, "lr": 0.00025, "sec_per_step": 6.164, "gnorm": 6.51394510269165}
40
+ {"step": 390, "loss": 3.1979150772094727, "nll": 3.139848232269287, "acc": 0.43052393198013306, "lr": 0.00025, "sec_per_step": 6.121, "gnorm": 4.636814594268799}
41
+ {"step": 400, "loss": 2.9832358360290527, "nll": 2.9123218059539795, "acc": 0.4618380069732666, "lr": 0.00025, "sec_per_step": 5.989, "gnorm": 4.325001239776611}
42
+ {"step": 410, "loss": 3.38590407371521, "nll": 3.2992231845855713, "acc": 0.4131641685962677, "lr": 0.00025, "sec_per_step": 5.876, "gnorm": 5.986806392669678}
43
+ {"step": 420, "loss": 2.9860775470733643, "nll": 2.897034168243408, "acc": 0.4553093910217285, "lr": 0.00025, "sec_per_step": 7.482, "gnorm": 4.825826644897461}
44
+ {"step": 430, "loss": 2.791691780090332, "nll": 2.719435214996338, "acc": 0.485871285200119, "lr": 0.00025, "sec_per_step": 6.29, "gnorm": 4.470316410064697}
45
+ {"step": 440, "loss": 3.3054935932159424, "nll": 3.228121042251587, "acc": 0.3485757112503052, "lr": 0.00025, "sec_per_step": 6.228, "gnorm": 5.163248062133789}
46
+ {"step": 450, "loss": 2.6202685832977295, "nll": 2.535158395767212, "acc": 0.5271317958831787, "lr": 0.00025, "sec_per_step": 6.05, "gnorm": 4.609095096588135}
47
+ {"step": 460, "loss": 2.8274412155151367, "nll": 2.754194736480713, "acc": 0.5169230699539185, "lr": 0.00025, "sec_per_step": 6.25, "gnorm": 5.817952632904053}
48
+ {"step": 470, "loss": 3.3049192428588867, "nll": 3.2267391681671143, "acc": 0.3978576958179474, "lr": 0.00025, "sec_per_step": 5.822, "gnorm": 5.276814937591553}
49
+ {"step": 480, "loss": 2.734286069869995, "nll": 2.6532654762268066, "acc": 0.48876839876174927, "lr": 0.00025, "sec_per_step": 6.129, "gnorm": 4.163163661956787}
50
+ {"step": 490, "loss": 3.161760091781616, "nll": 3.0504703521728516, "acc": 0.41040000319480896, "lr": 0.00025, "sec_per_step": 5.932, "gnorm": 5.252750873565674}
51
+ {"step": 500, "loss": 2.9351284503936768, "nll": 2.848027467727661, "acc": 0.47575056552886963, "lr": 0.00025, "sec_per_step": 5.878, "gnorm": 6.60810661315918}
52
+ {"step": 500, "ema_val_nll": 4.757295727729797, "ema_val_acc": 0.3992054760456085, "raw_val_nll": 2.869568943977356, "raw_val_acc": 0.4603290930390358}
53
+ {"step": 510, "loss": 3.00405216217041, "nll": 2.918951988220215, "acc": 0.45122918486595154, "lr": 0.00025, "sec_per_step": 6.425, "gnorm": 5.543540000915527}
54
+ {"step": 520, "loss": 3.0181691646575928, "nll": 2.906599760055542, "acc": 0.37817639112472534, "lr": 0.00025, "sec_per_step": 6.259, "gnorm": 5.1127519607543945}
55
+ {"step": 530, "loss": 2.5579750537872314, "nll": 2.4807825088500977, "acc": 0.507876992225647, "lr": 0.00025, "sec_per_step": 6.225, "gnorm": 4.181877136230469}
56
+ {"step": 540, "loss": 2.5971875190734863, "nll": 2.504199743270874, "acc": 0.5093914270401001, "lr": 0.00025, "sec_per_step": 6.481, "gnorm": 4.407857418060303}
57
+ {"step": 550, "loss": 2.922457456588745, "nll": 2.835352897644043, "acc": 0.447408527135849, "lr": 0.00025, "sec_per_step": 6.384, "gnorm": 5.971846580505371}
58
+ {"step": 560, "loss": 3.397606611251831, "nll": 3.27708101272583, "acc": 0.3853580057621002, "lr": 0.00025, "sec_per_step": 6.407, "gnorm": 7.557126045227051}
59
+ {"step": 570, "loss": 2.498903512954712, "nll": 2.415919303894043, "acc": 0.5505279302597046, "lr": 0.00025, "sec_per_step": 6.452, "gnorm": 3.9383254051208496}
60
+ {"step": 580, "loss": 3.0961005687713623, "nll": 3.017442226409912, "acc": 0.4386100471019745, "lr": 0.00025, "sec_per_step": 6.23, "gnorm": 5.793766021728516}
61
+ {"step": 590, "loss": 3.157524347305298, "nll": 3.046320915222168, "acc": 0.40030792355537415, "lr": 0.00025, "sec_per_step": 5.893, "gnorm": 5.688717842102051}
62
+ {"step": 600, "loss": 2.3616044521331787, "nll": 2.268296957015991, "acc": 0.5714285969734192, "lr": 0.00025, "sec_per_step": 6.051, "gnorm": 6.217453479766846}
63
+ {"step": 610, "loss": 2.9710419178009033, "nll": 2.88120174407959, "acc": 0.4688957929611206, "lr": 0.00025, "sec_per_step": 6.14, "gnorm": 6.2860870361328125}
64
+ {"step": 620, "loss": 2.986084461212158, "nll": 2.878966808319092, "acc": 0.3944530189037323, "lr": 0.00025, "sec_per_step": 6.3, "gnorm": 6.300302505493164}
65
+ {"step": 630, "loss": 2.5614638328552246, "nll": 2.4820749759674072, "acc": 0.5108024477958679, "lr": 0.00025, "sec_per_step": 6.298, "gnorm": 5.761435508728027}
66
+ {"step": 640, "loss": 3.084238052368164, "nll": 2.9953041076660156, "acc": 0.45324283838272095, "lr": 0.00025, "sec_per_step": 6.518, "gnorm": 5.899290084838867}
67
+ {"step": 650, "loss": 3.2054970264434814, "nll": 3.1445343494415283, "acc": 0.3931357264518738, "lr": 0.00025, "sec_per_step": 6.168, "gnorm": 6.1849799156188965}
68
+ {"step": 660, "loss": 2.0578274726867676, "nll": 1.9782261848449707, "acc": 0.6058282256126404, "lr": 0.00025, "sec_per_step": 6.434, "gnorm": 4.426883220672607}
69
+ {"step": 670, "loss": 2.946725606918335, "nll": 2.8325183391571045, "acc": 0.4192807972431183, "lr": 0.00025, "sec_per_step": 6.409, "gnorm": 4.88676118850708}
70
+ {"step": 680, "loss": 2.393976926803589, "nll": 2.304959535598755, "acc": 0.5380557775497437, "lr": 0.00025, "sec_per_step": 5.937, "gnorm": 3.898456573486328}
71
+ {"step": 690, "loss": 3.166815757751465, "nll": 3.0621657371520996, "acc": 0.39906832575798035, "lr": 0.00025, "sec_per_step": 6.558, "gnorm": 5.971321105957031}
72
+ {"step": 700, "loss": 2.7465415000915527, "nll": 2.62947940826416, "acc": 0.4311145544052124, "lr": 0.00025, "sec_per_step": 6.339, "gnorm": 5.578491687774658}
73
+ {"step": 710, "loss": 2.4950203895568848, "nll": 2.4207072257995605, "acc": 0.5385220050811768, "lr": 0.00025, "sec_per_step": 6.264, "gnorm": 3.6538729667663574}
74
+ {"step": 720, "loss": 2.5446856021881104, "nll": 2.4633123874664307, "acc": 0.5325264930725098, "lr": 0.00025, "sec_per_step": 6.018, "gnorm": 4.902586936950684}
75
+ {"step": 730, "loss": 2.8988325595855713, "nll": 2.799029588699341, "acc": 0.4440919756889343, "lr": 0.00025, "sec_per_step": 6.047, "gnorm": 5.226912975311279}
76
+ {"step": 740, "loss": 2.6665353775024414, "nll": 2.5732717514038086, "acc": 0.5003952383995056, "lr": 0.00025, "sec_per_step": 6.122, "gnorm": 4.964861869812012}
77
+ {"step": 750, "loss": 3.175828695297241, "nll": 3.05306077003479, "acc": 0.40813136100769043, "lr": 0.00025, "sec_per_step": 6.156, "gnorm": 5.22468900680542}
78
+ {"step": 750, "ema_val_nll": 4.113529145717621, "ema_val_acc": 0.40666238963603973, "raw_val_nll": 2.643985390663147, "raw_val_acc": 0.4819430932402611}
79
+ {"step": 760, "loss": 3.002694845199585, "nll": 2.898425340652466, "acc": 0.4198295772075653, "lr": 0.00025, "sec_per_step": 7.212, "gnorm": 4.94859504699707}
80
+ {"step": 770, "loss": 2.7087178230285645, "nll": 2.623079538345337, "acc": 0.4950343668460846, "lr": 0.00025, "sec_per_step": 6.125, "gnorm": 4.606741905212402}
81
+ {"step": 780, "loss": 2.5839145183563232, "nll": 2.4882335662841797, "acc": 0.518324613571167, "lr": 0.00025, "sec_per_step": 5.895, "gnorm": 3.720069408416748}
82
+ {"step": 790, "loss": 2.465515613555908, "nll": 2.3439218997955322, "acc": 0.5054263472557068, "lr": 0.00025, "sec_per_step": 6.3, "gnorm": 4.490382671356201}
83
+ {"step": 800, "loss": 2.7122833728790283, "nll": 2.60189151763916, "acc": 0.4909803867340088, "lr": 0.00025, "sec_per_step": 6.147, "gnorm": 4.550020694732666}
84
+ {"step": 810, "loss": 2.675985097885132, "nll": 2.5788652896881104, "acc": 0.5086477994918823, "lr": 0.00025, "sec_per_step": 6.241, "gnorm": 4.516785621643066}
85
+ {"step": 820, "loss": 2.798062562942505, "nll": 2.6930816173553467, "acc": 0.48151060938835144, "lr": 0.00025, "sec_per_step": 6.345, "gnorm": 5.3493971824646}
86
+ {"step": 830, "loss": 2.7418172359466553, "nll": 2.628443956375122, "acc": 0.4716981053352356, "lr": 0.00025, "sec_per_step": 6.246, "gnorm": 5.233100891113281}
87
+ {"step": 840, "loss": 2.781986951828003, "nll": 2.6775472164154053, "acc": 0.44110479950904846, "lr": 0.00025, "sec_per_step": 5.871, "gnorm": 5.794379234313965}
88
+ {"step": 850, "loss": 3.0547821521759033, "nll": 2.9467966556549072, "acc": 0.4636785089969635, "lr": 0.00025, "sec_per_step": 6.361, "gnorm": 6.320489883422852}
89
+ {"step": 860, "loss": 2.9305124282836914, "nll": 2.7980825901031494, "acc": 0.4084821343421936, "lr": 0.00025, "sec_per_step": 6.358, "gnorm": 4.6363043785095215}
90
+ {"step": 870, "loss": 2.394909620285034, "nll": 2.2805004119873047, "acc": 0.5445935130119324, "lr": 0.00025, "sec_per_step": 6.242, "gnorm": 4.042086601257324}
91
+ {"step": 880, "loss": 2.4840915203094482, "nll": 2.368605375289917, "acc": 0.4941084086894989, "lr": 0.00025, "sec_per_step": 6.399, "gnorm": 5.068140506744385}
92
+ {"step": 890, "loss": 2.506272792816162, "nll": 2.4257686138153076, "acc": 0.5312265157699585, "lr": 0.00025, "sec_per_step": 6.348, "gnorm": 4.804419040679932}
93
+ {"step": 900, "loss": 2.5496621131896973, "nll": 2.4423022270202637, "acc": 0.5185185074806213, "lr": 0.00025, "sec_per_step": 6.004, "gnorm": 5.5875349044799805}
94
+ {"step": 910, "loss": 2.7930030822753906, "nll": 2.7017321586608887, "acc": 0.4925606846809387, "lr": 0.00025, "sec_per_step": 6.159, "gnorm": 4.833495616912842}
95
+ {"step": 920, "loss": 2.495145797729492, "nll": 2.3928442001342773, "acc": 0.5393258333206177, "lr": 0.00025, "sec_per_step": 5.959, "gnorm": 3.727670192718506}
96
+ {"step": 930, "loss": 3.3178422451019287, "nll": 3.1799910068511963, "acc": 0.4032633900642395, "lr": 0.00025, "sec_per_step": 6.723, "gnorm": 6.969775676727295}
97
+ {"step": 940, "loss": 3.0164005756378174, "nll": 2.8922157287597656, "acc": 0.3935331106185913, "lr": 0.00025, "sec_per_step": 5.885, "gnorm": 5.34336519241333}
98
+ {"step": 950, "loss": 2.8784775733947754, "nll": 2.7882721424102783, "acc": 0.43485212326049805, "lr": 0.00025, "sec_per_step": 6.02, "gnorm": 5.112490653991699}
99
+ {"step": 960, "loss": 2.7010087966918945, "nll": 2.5866334438323975, "acc": 0.4752163589000702, "lr": 0.00025, "sec_per_step": 6.199, "gnorm": 5.5563645362854}
100
+ {"step": 970, "loss": 2.844118595123291, "nll": 2.7358219623565674, "acc": 0.46790409088134766, "lr": 0.00025, "sec_per_step": 5.841, "gnorm": 5.387760162353516}
101
+ {"step": 980, "loss": 2.6960089206695557, "nll": 2.5920751094818115, "acc": 0.48721805214881897, "lr": 0.00025, "sec_per_step": 6.272, "gnorm": 4.827456951141357}
102
+ {"step": 990, "loss": 2.5970065593719482, "nll": 2.484112024307251, "acc": 0.48900681734085083, "lr": 0.00025, "sec_per_step": 6.287, "gnorm": 4.624963760375977}
103
+ {"step": 1000, "loss": 2.3499960899353027, "nll": 2.2429587841033936, "acc": 0.5277127027511597, "lr": 0.00025, "sec_per_step": 6.107, "gnorm": 4.778969764709473}
104
+ {"step": 1000, "ema_val_nll": 3.6114672422409058, "ema_val_acc": 0.4406821057200432, "raw_val_nll": 2.666260600090027, "raw_val_acc": 0.4742664396762848}
105
+ {"step": 1010, "loss": 2.7779881954193115, "nll": 2.6650407314300537, "acc": 0.4598715901374817, "lr": 0.00025, "sec_per_step": 8.415, "gnorm": 5.046490669250488}
106
+ {"step": 1020, "loss": 2.4154486656188965, "nll": 2.325665235519409, "acc": 0.539177656173706, "lr": 0.00025, "sec_per_step": 6.235, "gnorm": 4.5242228507995605}
107
+ {"step": 1030, "loss": 2.5797035694122314, "nll": 2.485356330871582, "acc": 0.5289193391799927, "lr": 0.00025, "sec_per_step": 6.513, "gnorm": 4.961014270782471}
108
+ {"step": 1040, "loss": 2.6177289485931396, "nll": 2.5035901069641113, "acc": 0.5226011276245117, "lr": 0.00025, "sec_per_step": 6.029, "gnorm": 4.298247337341309}
109
+ {"step": 1050, "loss": 2.256521701812744, "nll": 2.1624436378479004, "acc": 0.6048083901405334, "lr": 0.00025, "sec_per_step": 6.097, "gnorm": 3.6198694705963135}
110
+ {"step": 1060, "loss": 2.7656686305999756, "nll": 2.6518473625183105, "acc": 0.4772382378578186, "lr": 0.00025, "sec_per_step": 6.203, "gnorm": 4.267014980316162}
111
+ {"step": 1070, "loss": 2.9220519065856934, "nll": 2.8266348838806152, "acc": 0.45517775416374207, "lr": 0.00025, "sec_per_step": 6.151, "gnorm": 3.8635213375091553}
112
+ {"step": 1080, "loss": 2.689377546310425, "nll": 2.5772922039031982, "acc": 0.4730856716632843, "lr": 0.00025, "sec_per_step": 5.973, "gnorm": 4.459869861602783}
113
+ {"step": 1090, "loss": 2.34483003616333, "nll": 2.2445740699768066, "acc": 0.545592725276947, "lr": 0.00025, "sec_per_step": 6.188, "gnorm": 4.070357322692871}
114
+ {"step": 1100, "loss": 2.853635311126709, "nll": 2.735430955886841, "acc": 0.44164037704467773, "lr": 0.00025, "sec_per_step": 6.181, "gnorm": 4.984803676605225}
115
+ {"step": 1110, "loss": 2.3250839710235596, "nll": 2.2021560668945312, "acc": 0.5528771281242371, "lr": 0.00025, "sec_per_step": 5.84, "gnorm": 4.130050182342529}
116
+ {"step": 1120, "loss": 2.395454168319702, "nll": 2.250638484954834, "acc": 0.5038520693778992, "lr": 0.00025, "sec_per_step": 6.079, "gnorm": 3.9396791458129883}
117
+ {"step": 1130, "loss": 3.1179373264312744, "nll": 3.0062990188598633, "acc": 0.41455939412117004, "lr": 0.00025, "sec_per_step": 6.23, "gnorm": 5.461380481719971}
118
+ {"step": 1140, "loss": 2.6154885292053223, "nll": 2.4895212650299072, "acc": 0.5114678740501404, "lr": 0.00025, "sec_per_step": 6.159, "gnorm": 4.283007621765137}
119
+ {"step": 1150, "loss": 2.4366607666015625, "nll": 2.3064374923706055, "acc": 0.5144193172454834, "lr": 0.00025, "sec_per_step": 6.121, "gnorm": 5.6990227699279785}
120
+ {"step": 1160, "loss": 2.505553722381592, "nll": 2.39853835105896, "acc": 0.5068492889404297, "lr": 0.00025, "sec_per_step": 6.078, "gnorm": 4.561639785766602}
121
+ {"step": 1170, "loss": 2.403393030166626, "nll": 2.287985324859619, "acc": 0.5042080879211426, "lr": 0.00025, "sec_per_step": 6.208, "gnorm": 4.251239776611328}
122
+ {"step": 1180, "loss": 2.3850104808807373, "nll": 2.2874596118927, "acc": 0.5426356792449951, "lr": 0.00025, "sec_per_step": 6.152, "gnorm": 5.599863052368164}
123
+ {"step": 1190, "loss": 2.782942295074463, "nll": 2.6919305324554443, "acc": 0.4758364260196686, "lr": 0.00025, "sec_per_step": 6.28, "gnorm": 5.388584136962891}
124
+ {"step": 1200, "loss": 2.4973490238189697, "nll": 2.3614211082458496, "acc": 0.5364543199539185, "lr": 0.00025, "sec_per_step": 6.34, "gnorm": 4.672701835632324}
125
+ {"step": 1210, "loss": 2.744319200515747, "nll": 2.60308575630188, "acc": 0.4330826997756958, "lr": 0.00025, "sec_per_step": 6.317, "gnorm": 5.39306116104126}
126
+ {"step": 1220, "loss": 2.6432716846466064, "nll": 2.517979621887207, "acc": 0.49802058935165405, "lr": 0.00025, "sec_per_step": 6.218, "gnorm": 4.594691753387451}
127
+ {"step": 1230, "loss": 2.746180534362793, "nll": 2.6465775966644287, "acc": 0.4908953011035919, "lr": 0.00025, "sec_per_step": 6.299, "gnorm": 5.096075534820557}
128
+ {"step": 1240, "loss": 2.739842653274536, "nll": 2.6066157817840576, "acc": 0.46555638313293457, "lr": 0.00025, "sec_per_step": 5.874, "gnorm": 5.038521766662598}
129
+ {"step": 1250, "loss": 2.1190388202667236, "nll": 2.018712043762207, "acc": 0.5824345350265503, "lr": 0.00025, "sec_per_step": 6.281, "gnorm": 4.792925834655762}
130
+ {"step": 1250, "ema_val_nll": 3.211992383003235, "ema_val_acc": 0.4749576523900032, "raw_val_nll": 2.4866853952407837, "raw_val_acc": 0.49806810915470123}
131
+ {"step": 1260, "loss": 3.0065832138061523, "nll": 2.8991711139678955, "acc": 0.4409448802471161, "lr": 0.00025, "sec_per_step": 7.317, "gnorm": 4.6214680671691895}
132
+ {"step": 1270, "loss": 2.6068108081817627, "nll": 2.4872145652770996, "acc": 0.46268656849861145, "lr": 0.00025, "sec_per_step": 6.016, "gnorm": 5.460693359375}
133
+ {"step": 1280, "loss": 2.9487662315368652, "nll": 2.828066825866699, "acc": 0.43990573287010193, "lr": 0.00025, "sec_per_step": 6.076, "gnorm": 4.329925060272217}
134
+ {"step": 1290, "loss": 2.550868272781372, "nll": 2.4090263843536377, "acc": 0.4996282458305359, "lr": 0.00025, "sec_per_step": 6.159, "gnorm": 4.255090236663818}
135
+ {"step": 1300, "loss": 2.4514379501342773, "nll": 2.347454071044922, "acc": 0.5015174746513367, "lr": 0.00025, "sec_per_step": 6.068, "gnorm": 5.65173864364624}
136
+ {"step": 1310, "loss": 2.480661153793335, "nll": 2.361290216445923, "acc": 0.4915003776550293, "lr": 0.00025, "sec_per_step": 6.154, "gnorm": 4.723772048950195}
137
+ {"step": 1320, "loss": 2.580076217651367, "nll": 2.4763495922088623, "acc": 0.5211701393127441, "lr": 0.00025, "sec_per_step": 6.095, "gnorm": 4.933023452758789}
138
+ {"step": 1330, "loss": 2.862834930419922, "nll": 2.7394986152648926, "acc": 0.4516380727291107, "lr": 0.00025, "sec_per_step": 6.372, "gnorm": 4.52992582321167}
139
+ {"step": 1340, "loss": 2.0927963256835938, "nll": 2.003164291381836, "acc": 0.5794117450714111, "lr": 0.00025, "sec_per_step": 6.307, "gnorm": 3.553281784057617}
140
+ {"step": 1350, "loss": 2.3785457611083984, "nll": 2.26143479347229, "acc": 0.5496987700462341, "lr": 0.00025, "sec_per_step": 6.146, "gnorm": 4.909267902374268}
141
+ {"step": 1360, "loss": 2.4946482181549072, "nll": 2.3889689445495605, "acc": 0.5175096988677979, "lr": 0.00025, "sec_per_step": 6.075, "gnorm": 4.5072340965271}
142
+ {"step": 1370, "loss": 2.6881964206695557, "nll": 2.546342134475708, "acc": 0.4414556920528412, "lr": 0.00025, "sec_per_step": 6.424, "gnorm": 6.146117687225342}
143
+ {"step": 1380, "loss": 2.326253652572632, "nll": 2.199773073196411, "acc": 0.5414527058601379, "lr": 0.00025, "sec_per_step": 6.49, "gnorm": 4.503512382507324}
144
+ {"step": 1390, "loss": 2.906200647354126, "nll": 2.7529072761535645, "acc": 0.46252837777137756, "lr": 0.00025, "sec_per_step": 6.226, "gnorm": 4.7287211418151855}
145
+ {"step": 1400, "loss": 2.4027457237243652, "nll": 2.2472972869873047, "acc": 0.5239937901496887, "lr": 0.00025, "sec_per_step": 6.423, "gnorm": 5.4805474281311035}
146
+ {"step": 1410, "loss": 2.4080331325531006, "nll": 2.2458603382110596, "acc": 0.46352583169937134, "lr": 0.00025, "sec_per_step": 6.33, "gnorm": 5.683468818664551}
147
+ {"step": 1420, "loss": 2.3063762187957764, "nll": 2.1755499839782715, "acc": 0.5429864525794983, "lr": 0.00025, "sec_per_step": 6.386, "gnorm": 5.139619827270508}
148
+ {"step": 1430, "loss": 2.491290330886841, "nll": 2.3421034812927246, "acc": 0.4966088831424713, "lr": 0.00025, "sec_per_step": 6.315, "gnorm": 4.2525129318237305}
149
+ {"step": 1440, "loss": 2.152965784072876, "nll": 2.0535247325897217, "acc": 0.5964912176132202, "lr": 0.00025, "sec_per_step": 6.17, "gnorm": 3.590088129043579}
150
+ {"step": 1450, "loss": 2.95025897026062, "nll": 2.827056646347046, "acc": 0.414219468832016, "lr": 0.00025, "sec_per_step": 6.41, "gnorm": 5.303553581237793}
151
+ {"step": 1460, "loss": 2.8419055938720703, "nll": 2.7361035346984863, "acc": 0.4551831781864166, "lr": 0.00025, "sec_per_step": 6.336, "gnorm": 5.055415153503418}
152
+ {"step": 1470, "loss": 2.92997407913208, "nll": 2.8064730167388916, "acc": 0.4229337275028229, "lr": 0.00025, "sec_per_step": 6.254, "gnorm": 5.053462982177734}
153
+ {"step": 1480, "loss": 2.549100399017334, "nll": 2.419140338897705, "acc": 0.511904776096344, "lr": 0.00025, "sec_per_step": 6.215, "gnorm": 4.3006205558776855}
154
+ {"step": 1490, "loss": 2.893587112426758, "nll": 2.7813587188720703, "acc": 0.4875583350658417, "lr": 0.00025, "sec_per_step": 6.132, "gnorm": 5.892554759979248}
155
+ {"step": 1500, "loss": 2.6392624378204346, "nll": 2.5167980194091797, "acc": 0.48127925395965576, "lr": 0.00025, "sec_per_step": 6.339, "gnorm": 5.114858150482178}
156
+ {"step": 1500, "ema_val_nll": 2.950586438179016, "ema_val_acc": 0.4897710531949997, "raw_val_nll": 2.484873652458191, "raw_val_acc": 0.5019320175051689}
157
+ {"step": 1510, "loss": 2.5321125984191895, "nll": 2.433446168899536, "acc": 0.5450918078422546, "lr": 0.00025, "sec_per_step": 8.25, "gnorm": 3.91320538520813}
158
+ {"step": 1520, "loss": 2.616060972213745, "nll": 2.5097177028656006, "acc": 0.5171624422073364, "lr": 0.00025, "sec_per_step": 6.419, "gnorm": 4.143905162811279}
159
+ {"step": 1530, "loss": 3.1057634353637695, "nll": 2.9970128536224365, "acc": 0.4374045729637146, "lr": 0.00025, "sec_per_step": 6.011, "gnorm": 6.740103721618652}
160
+ {"step": 1540, "loss": 2.6444315910339355, "nll": 2.5181338787078857, "acc": 0.5045385956764221, "lr": 0.00025, "sec_per_step": 6.482, "gnorm": 4.590264320373535}
161
+ {"step": 1550, "loss": 2.076510190963745, "nll": 1.9581280946731567, "acc": 0.5774980187416077, "lr": 0.00025, "sec_per_step": 6.308, "gnorm": 4.635385990142822}
162
+ {"step": 1560, "loss": 2.5528955459594727, "nll": 2.4293298721313477, "acc": 0.5011746287345886, "lr": 0.00025, "sec_per_step": 6.263, "gnorm": 4.992447853088379}
163
+ {"step": 1570, "loss": 2.8153975009918213, "nll": 2.6946966648101807, "acc": 0.44435974955558777, "lr": 0.00025, "sec_per_step": 6.228, "gnorm": 6.261963844299316}
164
+ {"step": 1580, "loss": 2.534857749938965, "nll": 2.419583797454834, "acc": 0.5106545090675354, "lr": 0.00025, "sec_per_step": 6.673, "gnorm": 5.6587347984313965}
165
+ {"step": 1590, "loss": 2.804851531982422, "nll": 2.683345079421997, "acc": 0.48448145389556885, "lr": 0.00025, "sec_per_step": 6.528, "gnorm": 4.932685375213623}
166
+ {"step": 1600, "loss": 2.805572032928467, "nll": 2.6489508152008057, "acc": 0.45264846086502075, "lr": 0.00025, "sec_per_step": 6.35, "gnorm": 6.264685153961182}
167
+ {"step": 1610, "loss": 2.7004661560058594, "nll": 2.5743415355682373, "acc": 0.4812312424182892, "lr": 0.00025, "sec_per_step": 5.881, "gnorm": 6.80185604095459}
168
+ {"step": 1620, "loss": 2.419762372970581, "nll": 2.2805428504943848, "acc": 0.5216741561889648, "lr": 0.00025, "sec_per_step": 6.344, "gnorm": 4.487758636474609}
169
+ {"step": 1630, "loss": 2.575009346008301, "nll": 2.4346442222595215, "acc": 0.486821711063385, "lr": 0.00025, "sec_per_step": 6.45, "gnorm": 5.625582695007324}
170
+ {"step": 1640, "loss": 2.6458613872528076, "nll": 2.5290637016296387, "acc": 0.5316363573074341, "lr": 0.00025, "sec_per_step": 6.287, "gnorm": 3.472202777862549}
171
+ {"step": 1650, "loss": 2.4841203689575195, "nll": 2.3537588119506836, "acc": 0.5102505683898926, "lr": 0.00025, "sec_per_step": 6.008, "gnorm": 4.373164653778076}
172
+ {"step": 1660, "loss": 2.651214599609375, "nll": 2.5295002460479736, "acc": 0.4582701027393341, "lr": 0.00025, "sec_per_step": 5.934, "gnorm": 5.1948137283325195}
173
+ {"step": 1670, "loss": 2.6843926906585693, "nll": 2.5529730319976807, "acc": 0.46102941036224365, "lr": 0.00025, "sec_per_step": 6.049, "gnorm": 5.177816867828369}
174
+ {"step": 1680, "loss": 2.5292937755584717, "nll": 2.3909285068511963, "acc": 0.4948939383029938, "lr": 0.00025, "sec_per_step": 6.28, "gnorm": 4.215677738189697}
175
+ {"step": 1690, "loss": 2.4495229721069336, "nll": 2.3191332817077637, "acc": 0.530809223651886, "lr": 0.00025, "sec_per_step": 5.761, "gnorm": 3.945343255996704}
176
+ {"step": 1700, "loss": 2.617029905319214, "nll": 2.466738224029541, "acc": 0.47542187571525574, "lr": 0.00025, "sec_per_step": 6.277, "gnorm": 4.773387432098389}
177
+ {"step": 1710, "loss": 2.5217444896698, "nll": 2.3812272548675537, "acc": 0.49837398529052734, "lr": 0.00025, "sec_per_step": 6.32, "gnorm": 4.3939595222473145}
178
+ {"step": 1720, "loss": 2.846107006072998, "nll": 2.687957286834717, "acc": 0.4266567528247833, "lr": 0.00025, "sec_per_step": 6.239, "gnorm": 6.500346660614014}
179
+ {"step": 1730, "loss": 2.7188565731048584, "nll": 2.5787158012390137, "acc": 0.49103662371635437, "lr": 0.00025, "sec_per_step": 6.068, "gnorm": 4.834630012512207}
180
+ {"step": 1740, "loss": 2.459904193878174, "nll": 2.310497283935547, "acc": 0.509476900100708, "lr": 0.00025, "sec_per_step": 6.306, "gnorm": 4.794856071472168}
181
+ {"step": 1750, "loss": 2.819455623626709, "nll": 2.679036855697632, "acc": 0.4685039222240448, "lr": 0.00025, "sec_per_step": 6.202, "gnorm": 4.270397663116455}
182
+ {"step": 1750, "ema_val_nll": 2.7671797275543213, "ema_val_acc": 0.5013483017683029, "raw_val_nll": 2.4472358226776123, "raw_val_acc": 0.5091685876250267}
183
+ {"step": 1760, "loss": 2.1412112712860107, "nll": 2.036386489868164, "acc": 0.5695817470550537, "lr": 0.00025, "sec_per_step": 7.043, "gnorm": 5.377252578735352}
184
+ {"step": 1770, "loss": 2.5172595977783203, "nll": 2.411311149597168, "acc": 0.527930736541748, "lr": 0.00025, "sec_per_step": 6.318, "gnorm": 4.626220703125}
185
+ {"step": 1780, "loss": 2.6562397480010986, "nll": 2.4928221702575684, "acc": 0.501886785030365, "lr": 0.00025, "sec_per_step": 6.121, "gnorm": 4.809992790222168}
186
+ {"step": 1790, "loss": 2.3886077404022217, "nll": 2.2686352729797363, "acc": 0.5409711599349976, "lr": 0.00025, "sec_per_step": 6.397, "gnorm": 3.8020665645599365}
187
+ {"step": 1800, "loss": 2.6781694889068604, "nll": 2.530588388442993, "acc": 0.48704269528388977, "lr": 0.00025, "sec_per_step": 6.233, "gnorm": 4.668972492218018}
188
+ {"step": 1810, "loss": 2.15527606010437, "nll": 2.005056142807007, "acc": 0.5967492461204529, "lr": 0.00025, "sec_per_step": 5.59, "gnorm": 5.025992393493652}
189
+ {"step": 1820, "loss": 2.5151400566101074, "nll": 2.4142086505889893, "acc": 0.5522388219833374, "lr": 0.00025, "sec_per_step": 6.031, "gnorm": 4.32218074798584}
190
+ {"step": 1830, "loss": 2.6874396800994873, "nll": 2.5723373889923096, "acc": 0.4754098355770111, "lr": 0.00025, "sec_per_step": 5.866, "gnorm": 5.34877347946167}
191
+ {"step": 1840, "loss": 2.763144016265869, "nll": 2.6047627925872803, "acc": 0.4909909963607788, "lr": 0.00025, "sec_per_step": 5.903, "gnorm": 4.591948509216309}
192
+ {"step": 1850, "loss": 2.4946372509002686, "nll": 2.345001220703125, "acc": 0.5270379185676575, "lr": 0.00025, "sec_per_step": 6.223, "gnorm": 5.389904022216797}
193
+ {"step": 1860, "loss": 2.732940673828125, "nll": 2.615614414215088, "acc": 0.48630136251449585, "lr": 0.00025, "sec_per_step": 6.253, "gnorm": 4.7220025062561035}
194
+ {"step": 1870, "loss": 2.449154853820801, "nll": 2.285970687866211, "acc": 0.5331734418869019, "lr": 0.00025, "sec_per_step": 5.931, "gnorm": 6.002699375152588}
195
+ {"step": 1880, "loss": 2.2676377296447754, "nll": 2.1399571895599365, "acc": 0.5431165099143982, "lr": 0.00025, "sec_per_step": 6.419, "gnorm": 5.448880672454834}
196
+ {"step": 1890, "loss": 2.97053861618042, "nll": 2.838926076889038, "acc": 0.4377880096435547, "lr": 0.00025, "sec_per_step": 6.276, "gnorm": 5.652556419372559}
197
+ {"step": 1900, "loss": 2.379473924636841, "nll": 2.2505381107330322, "acc": 0.549730122089386, "lr": 0.00025, "sec_per_step": 5.922, "gnorm": 3.900559425354004}
198
+ {"step": 1910, "loss": 2.437931776046753, "nll": 2.3125457763671875, "acc": 0.5287008881568909, "lr": 0.00025, "sec_per_step": 6.051, "gnorm": 3.7915947437286377}
199
+ {"step": 1920, "loss": 2.351442813873291, "nll": 2.212263822555542, "acc": 0.5736263990402222, "lr": 0.00025, "sec_per_step": 6.284, "gnorm": 3.1446502208709717}
200
+ {"step": 1930, "loss": 2.6715996265411377, "nll": 2.5253102779388428, "acc": 0.48139533400535583, "lr": 0.00024479166666666665, "sec_per_step": 6.33, "gnorm": 6.262954235076904}
201
+ {"step": 1940, "loss": 2.6839425563812256, "nll": 2.54409122467041, "acc": 0.5152904987335205, "lr": 0.00023958333333333335, "sec_per_step": 6.287, "gnorm": 4.908402442932129}
202
+ {"step": 1950, "loss": 2.4449639320373535, "nll": 2.3473141193389893, "acc": 0.5391941666603088, "lr": 0.000234375, "sec_per_step": 6.298, "gnorm": 3.5283448696136475}
203
+ {"step": 1960, "loss": 2.43481707572937, "nll": 2.3232977390289307, "acc": 0.5366614460945129, "lr": 0.00022916666666666666, "sec_per_step": 6.065, "gnorm": 4.786221981048584}
204
+ {"step": 1970, "loss": 2.6724231243133545, "nll": 2.545682668685913, "acc": 0.5007727742195129, "lr": 0.00022395833333333336, "sec_per_step": 6.055, "gnorm": 4.052018165588379}
205
+ {"step": 1980, "loss": 2.4458563327789307, "nll": 2.3175594806671143, "acc": 0.5416666865348816, "lr": 0.00021875, "sec_per_step": 6.476, "gnorm": 5.176480293273926}
206
+ {"step": 1990, "loss": 2.3839120864868164, "nll": 2.2538390159606934, "acc": 0.5558121800422668, "lr": 0.00021354166666666665, "sec_per_step": 6.128, "gnorm": 4.680918216705322}
207
+ {"step": 2000, "loss": 2.9278793334960938, "nll": 2.78448486328125, "acc": 0.4787735939025879, "lr": 0.00020833333333333335, "sec_per_step": 6.083, "gnorm": 6.968480587005615}
208
+ {"step": 2000, "ema_val_nll": 2.6115826964378357, "ema_val_acc": 0.5086340084671974, "raw_val_nll": 2.296378254890442, "raw_val_acc": 0.5319766253232956}
209
+ {"step": 2010, "loss": 2.7372467517852783, "nll": 2.5916860103607178, "acc": 0.48620128631591797, "lr": 0.00020312500000000002, "sec_per_step": 8.423, "gnorm": 4.617999076843262}
210
+ {"step": 2020, "loss": 2.4059858322143555, "nll": 2.270644187927246, "acc": 0.5312976837158203, "lr": 0.00019791666666666666, "sec_per_step": 6.384, "gnorm": 6.127459526062012}
211
+ {"step": 2030, "loss": 2.1576266288757324, "nll": 1.98272705078125, "acc": 0.55513596534729, "lr": 0.00019270833333333336, "sec_per_step": 6.406, "gnorm": 4.933815956115723}
212
+ {"step": 2040, "loss": 2.548945426940918, "nll": 2.3902101516723633, "acc": 0.5061255693435669, "lr": 0.0001875, "sec_per_step": 5.928, "gnorm": 5.75822639465332}
213
+ {"step": 2050, "loss": 2.2869479656219482, "nll": 2.1253886222839355, "acc": 0.5607333779335022, "lr": 0.0001822916666666667, "sec_per_step": 6.069, "gnorm": 3.870657444000244}
214
+ {"step": 2060, "loss": 2.933115243911743, "nll": 2.8382081985473633, "acc": 0.4946889281272888, "lr": 0.00017708333333333332, "sec_per_step": 5.885, "gnorm": 3.6446266174316406}
215
+ {"step": 2070, "loss": 2.4248807430267334, "nll": 2.273270845413208, "acc": 0.5530303120613098, "lr": 0.000171875, "sec_per_step": 6.1, "gnorm": 3.4939122200012207}
216
+ {"step": 2080, "loss": 1.9751367568969727, "nll": 1.8419090509414673, "acc": 0.57421875, "lr": 0.0001666666666666667, "sec_per_step": 5.892, "gnorm": 5.098540306091309}
217
+ {"step": 2090, "loss": 2.5112316608428955, "nll": 2.3503072261810303, "acc": 0.4948778450489044, "lr": 0.0001614583333333333, "sec_per_step": 6.317, "gnorm": 4.346006393432617}
218
+ {"step": 2100, "loss": 2.2872812747955322, "nll": 2.1516096591949463, "acc": 0.5481203198432922, "lr": 0.00015625, "sec_per_step": 6.508, "gnorm": 4.3317108154296875}
219
+ {"step": 2110, "loss": 2.7115135192871094, "nll": 2.5750491619110107, "acc": 0.4988797605037689, "lr": 0.0001510416666666667, "sec_per_step": 5.76, "gnorm": 3.901121139526367}
220
+ {"step": 2120, "loss": 2.5074963569641113, "nll": 2.393440008163452, "acc": 0.540123462677002, "lr": 0.00014583333333333332, "sec_per_step": 5.955, "gnorm": 5.327136039733887}
221
+ {"step": 2130, "loss": 2.687457323074341, "nll": 2.557091474533081, "acc": 0.5162287354469299, "lr": 0.00014062500000000002, "sec_per_step": 6.048, "gnorm": 4.342340469360352}
222
+ {"step": 2140, "loss": 1.7849406003952026, "nll": 1.64743971824646, "acc": 0.6392067074775696, "lr": 0.0001354166666666667, "sec_per_step": 6.081, "gnorm": 4.134641647338867}
223
+ {"step": 2150, "loss": 2.2291746139526367, "nll": 2.0813710689544678, "acc": 0.5711026787757874, "lr": 0.0001302083333333333, "sec_per_step": 6.057, "gnorm": 5.770716190338135}
224
+ {"step": 2160, "loss": 2.1470110416412354, "nll": 2.018789768218994, "acc": 0.5789865851402283, "lr": 0.000125, "sec_per_step": 5.865, "gnorm": 3.632281541824341}
225
+ {"step": 2170, "loss": 2.294814109802246, "nll": 2.1122381687164307, "acc": 0.5601532459259033, "lr": 0.00011979166666666666, "sec_per_step": 6.031, "gnorm": 4.823059558868408}
226
+ {"step": 2180, "loss": 2.255075454711914, "nll": 2.1100776195526123, "acc": 0.5873749256134033, "lr": 0.00011458333333333334, "sec_per_step": 6.292, "gnorm": 5.419332027435303}
227
+ {"step": 2190, "loss": 2.466179132461548, "nll": 2.3219850063323975, "acc": 0.5212598443031311, "lr": 0.000109375, "sec_per_step": 6.017, "gnorm": 5.503530025482178}
228
+ {"step": 2200, "loss": 2.6158857345581055, "nll": 2.506026029586792, "acc": 0.5175037980079651, "lr": 0.00010416666666666666, "sec_per_step": 5.97, "gnorm": 3.886592388153076}
229
+ {"step": 2210, "loss": 2.5306787490844727, "nll": 2.370680332183838, "acc": 0.4883720874786377, "lr": 9.895833333333334e-05, "sec_per_step": 6.301, "gnorm": 4.414287090301514}
230
+ {"step": 2220, "loss": 2.51403546333313, "nll": 2.3920834064483643, "acc": 0.5277989506721497, "lr": 9.375e-05, "sec_per_step": 6.322, "gnorm": 4.095523357391357}
231
+ {"step": 2230, "loss": 2.5394654273986816, "nll": 2.3612449169158936, "acc": 0.5, "lr": 8.854166666666666e-05, "sec_per_step": 5.919, "gnorm": 5.0627570152282715}
232
+ {"step": 2240, "loss": 2.494802951812744, "nll": 2.3610856533050537, "acc": 0.508009135723114, "lr": 8.333333333333334e-05, "sec_per_step": 5.903, "gnorm": 4.964487552642822}
233
+ {"step": 2250, "loss": 1.8854117393493652, "nll": 1.7896050214767456, "acc": 0.6266159415245056, "lr": 7.8125e-05, "sec_per_step": 6.097, "gnorm": 4.4996466636657715}
234
+ {"step": 2250, "ema_val_nll": 2.4679396748542786, "ema_val_acc": 0.5299465879797935, "raw_val_nll": 2.1637205481529236, "raw_val_acc": 0.5578823387622833}
235
+ {"step": 2260, "loss": 2.230721950531006, "nll": 2.091271162033081, "acc": 0.5658198595046997, "lr": 7.291666666666666e-05, "sec_per_step": 7.034, "gnorm": 4.909244537353516}
236
+ {"step": 2270, "loss": 2.035552740097046, "nll": 1.8922446966171265, "acc": 0.6076335906982422, "lr": 6.770833333333334e-05, "sec_per_step": 6.018, "gnorm": 3.9589521884918213}
237
+ {"step": 2280, "loss": 2.838111400604248, "nll": 2.701313018798828, "acc": 0.4763810932636261, "lr": 6.25e-05, "sec_per_step": 5.892, "gnorm": 5.069949150085449}
238
+ {"step": 2290, "loss": 2.502668619155884, "nll": 2.3781075477600098, "acc": 0.5141545534133911, "lr": 5.729166666666666e-05, "sec_per_step": 6.362, "gnorm": 4.266213893890381}
239
+ {"step": 2300, "loss": 2.645368814468384, "nll": 2.5022597312927246, "acc": 0.49233129620552063, "lr": 5.2083333333333343e-05, "sec_per_step": 6.165, "gnorm": 4.9644622802734375}
240
+ {"step": 2310, "loss": 2.61704421043396, "nll": 2.5010693073272705, "acc": 0.5034013390541077, "lr": 4.6875e-05, "sec_per_step": 6.549, "gnorm": 3.7004849910736084}
241
+ {"step": 2320, "loss": 2.2191402912139893, "nll": 2.054413318634033, "acc": 0.5559936761856079, "lr": 4.166666666666666e-05, "sec_per_step": 5.904, "gnorm": 5.543429374694824}
242
+ {"step": 2330, "loss": 2.2716104984283447, "nll": 2.1119349002838135, "acc": 0.5815176367759705, "lr": 3.645833333333334e-05, "sec_per_step": 6.088, "gnorm": 4.0455451011657715}
243
+ {"step": 2340, "loss": 2.360276460647583, "nll": 2.187093496322632, "acc": 0.5592452883720398, "lr": 3.125e-05, "sec_per_step": 6.027, "gnorm": 4.779704570770264}
244
+ {"step": 2350, "loss": 1.95784330368042, "nll": 1.7860734462738037, "acc": 0.6135488748550415, "lr": 2.6041666666666658e-05, "sec_per_step": 6.292, "gnorm": 3.6473851203918457}
245
+ {"step": 2360, "loss": 2.1754679679870605, "nll": 1.9920741319656372, "acc": 0.5615686178207397, "lr": 2.0833333333333343e-05, "sec_per_step": 6.08, "gnorm": 4.795201301574707}
246
+ {"step": 2370, "loss": 2.4577841758728027, "nll": 2.3091211318969727, "acc": 0.5495638251304626, "lr": 1.5625e-05, "sec_per_step": 5.973, "gnorm": 4.615086078643799}
247
+ {"step": 2380, "loss": 1.8841142654418945, "nll": 1.7523480653762817, "acc": 0.6373873949050903, "lr": 1.0416666666666658e-05, "sec_per_step": 5.942, "gnorm": 4.416830062866211}
248
+ {"step": 2390, "loss": 2.4632983207702637, "nll": 2.296647548675537, "acc": 0.4739263951778412, "lr": 5.2083333333333425e-06, "sec_per_step": 6.363, "gnorm": 4.186145305633545}
249
+ {"step": 2400, "loss": 2.47607159614563, "nll": 2.3211183547973633, "acc": 0.5543219447135925, "lr": 0.0, "sec_per_step": 6.358, "gnorm": 4.649677753448486}
250
+ {"step": 2400, "ema_val_nll": 2.445981442928314, "ema_val_acc": 0.5343381315469742, "raw_val_nll": 2.176288068294525, "raw_val_acc": 0.545825183391571}
model_config.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "vocab_size": 512,
3
+ "grid_size": 16,
4
+ "dim": 512,
5
+ "n_heads": 8,
6
+ "depth_in": 2,
7
+ "depth_core": 8,
8
+ "depth_out": 2,
9
+ "patch": 2,
10
+ "mlp_ratio": 3.0,
11
+ "n_registers": 4,
12
+ "text_dim": 512,
13
+ "text_len": 24,
14
+ "rope_theta": 100.0,
15
+ "conv_ffn": true,
16
+ "color_freqs": 6,
17
+ "self_cond": true
18
+ }
showcase.png ADDED
train_config.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "batch_size": 8,
3
+ "grad_accum": 1,
4
+ "total_steps": 2400,
5
+ "lr": 0.00025,
6
+ "min_lr_ratio": 0.0,
7
+ "warmup_steps": 100,
8
+ "decay_frac": 0.2,
9
+ "weight_decay": 0.05,
10
+ "beta1": 0.9,
11
+ "beta2": 0.95,
12
+ "grad_clip": 1.0,
13
+ "optimizer": "muon",
14
+ "muon_momentum": 0.95,
15
+ "ema_decay": 0.999,
16
+ "mask_schedule": "cosine",
17
+ "loss_weighting": "uniform",
18
+ "stratified_t": true,
19
+ "smooth_mask_prob": 0.25,
20
+ "label_smoothing": 0.1,
21
+ "color_tau": 0.05,
22
+ "self_cond_prob": 0.5,
23
+ "self_cond_lookback": 0.3,
24
+ "cond_drop_prob": 0.1,
25
+ "subject_prob": 0.05,
26
+ "hflip_prob": 0.5,
27
+ "seed": 1234,
28
+ "log_every": 10,
29
+ "eval_every": 250,
30
+ "eval_batches": 4,
31
+ "eval_batch": 16,
32
+ "sample_every": 500,
33
+ "ckpt_every_steps": 400,
34
+ "ckpt_every_minutes": 4.0,
35
+ "sample_steps": 16,
36
+ "sample_cfg": 3.0,
37
+ "sample_prompts": [
38
+ "red health potion bottle",
39
+ "golden crown with blue gems",
40
+ "green slime monster",
41
+ "iron sword with a leather hilt",
42
+ "wooden treasure chest",
43
+ "pink cherry blossom tree",
44
+ "blue magic spellbook",
45
+ "orange fox head",
46
+ "glowing purple crystal",
47
+ "cheeseburger",
48
+ "skull with a candle",
49
+ "rocket ship",
50
+ "brown leather boots",
51
+ "yellow rubber duck",
52
+ "ice cream cone with strawberry scoop",
53
+ "black cat"
54
+ ]
55
+ }
trout-fp16.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9d2ffe91522aede1965db4441780c7bc7d67dc0d966f0e289c508f02793f665
3
+ size 95251680
trout.py ADDED
@@ -0,0 +1,1017 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ trout.py -- a from-scratch diffusion language model for 16x16 pixel art.
3
+
4
+ Trout treats a 16x16 sprite as a 256-token "sentence" over a learned 512-color
5
+ palette (token 0 = transparent). A bidirectional transformer denoiser starts
6
+ from a fully masked canvas and unmasks all pixels in parallel over ~16 steps,
7
+ conditioned on a frozen CLIP text tower. No VAE, no latent space, no
8
+ autoregression -- absorbing-state masked (discrete) diffusion.
9
+
10
+ Architecture (hourglass dual-resolution denoiser, 47.6M params for the small
11
+ preset), reimplemented from scratch following the design published with
12
+ Tessera-small by Jagrat Patel (TheJagStudio), whose model card and open demo
13
+ space document every choice below:
14
+
15
+ * joint [registers | text tokens | pixel tokens] bidirectional attention
16
+ * hourglass: depth_in pixel-level blocks -> 2x2 patch-merged core blocks
17
+ -> depth_out pixel-level blocks, with a residual skip around the core
18
+ * QK-RMSNorm, 2D axial RoPE (theta=100) with sub-pixel-aligned patch centers
19
+ * Conv-SwiGLU: depthwise 3x3 conv on the gate branch over image tokens
20
+ * adaLN-single conditioning: shared MLP on (pooled text + mask ratio
21
+ Fourier features) with per-block learned modulation offsets and
22
+ zero-initialized gates (blocks start as identity)
23
+ * color-grounded token embeddings: learned vector + MLP(Fourier(OKLab))
24
+ * 4 register tokens in the prefix
25
+ * self-conditioning: the previous step's predicted color distribution is
26
+ projected and added at masked positions
27
+
28
+ Theory sources (the "knowledge" this model is built on):
29
+ * Absorbing-state / masked discrete diffusion:
30
+ Sahoo et al., "Simple and Effective Masked Diffusion Language Models",
31
+ NeurIPS 2024 (arXiv:2406.07524) -- MDLM
32
+ Shi et al., "Simplified and Generalized Masked Diffusion for Discrete
33
+ Data", NeurIPS 2024 (arXiv:2406.04329)
34
+ Labadie & Ye, "Absorbing State Discrete Diffusion" survey, 2025.
35
+ * Parallel confidence-ordered decoding: Chang et al., "MaskGIT: Masked
36
+ Generative Image Transformer", CVPR 2022.
37
+ * Hourglass transformer: Blecher et al., "Nougat" (arXiv:2308.13418).
38
+ * adaLN-single: Peebles & Xie, "DiT" (arXiv:2212.09748); PixArt-alpha.
39
+ * Registers: Darcet et al., "Vision Transformers Need Registers" (ICLR 2024).
40
+ * QK-norm: Gemma/Henry et al. "Training Transformers with ... QK-Norm".
41
+ * Self-conditioning: Chen, "Analog Bits" (arXiv:2208.04202).
42
+ * Muon optimizer: Jordan et al. (arXiv:2502.16982) and Moonshot
43
+ "Muon is Scalable" (arXiv:2508.02535) for the 0.2*sqrt(max(m,n)) scale.
44
+ * OKLab: Bjorn Ottosson, "A perceptual color space for image processing"
45
+ (2020) -- public-domain reference formulas.
46
+
47
+ Written by Z.ai for the user's "Trout" project; trained code lives in
48
+ train_trout.py. Apache-2.0.
49
+ """
50
+
51
+ from __future__ import annotations
52
+
53
+ import json
54
+ import math
55
+ import os
56
+ from dataclasses import asdict, dataclass, field
57
+ from typing import Any, Dict, Iterator, List, Optional, Sequence, Tuple
58
+
59
+ import numpy as np
60
+ import torch
61
+ import torch.nn as nn
62
+ import torch.nn.functional as F
63
+
64
+ DEFAULT_TEXT_ENCODER = "openai/clip-vit-base-patch32"
65
+
66
+
67
+ # ---------------------------------------------------------------------------
68
+ # configuration
69
+ # ---------------------------------------------------------------------------
70
+
71
+ @dataclass
72
+ class TroutConfig:
73
+ """Architecture hyper-parameters for the Trout denoiser."""
74
+ vocab_size: int = 512 # palette entries incl. transparent (index 0); MASK id == vocab_size
75
+ grid_size: int = 16 # image is grid_size x grid_size tokens
76
+ dim: int = 512 # residual width
77
+ n_heads: int = 8
78
+ depth_in: int = 2 # pixel-level blocks before the 2x2 merge
79
+ depth_core: int = 8 # patch-level blocks on merged (grid/patch)^2 tokens
80
+ depth_out: int = 2 # pixel-level blocks after the unmerge
81
+ patch: int = 2 # hourglass merge factor (1 = flat model)
82
+ mlp_ratio: float = 3.0 # SwiGLU hidden = mlp_ratio * dim
83
+ n_registers: int = 4 # learnable register tokens in the prefix
84
+ text_dim: int = 512 # hidden size of the frozen CLIP text tower
85
+ text_len: int = 24 # text tokens fed to the denoiser
86
+ rope_theta: float = 100.0 # base of the 2D axial RoPE frequencies
87
+ conv_ffn: bool = True # depthwise 3x3 conv on the gate branch
88
+ color_freqs: int = 6 # Fourier frequencies for OKLab color embeddings
89
+ self_cond: bool = True # feed previous-step predictions back as input
90
+
91
+ @property
92
+ def mask_id(self) -> int:
93
+ return self.vocab_size
94
+
95
+ @property
96
+ def num_tokens(self) -> int:
97
+ return self.grid_size * self.grid_size
98
+
99
+ @property
100
+ def hourglass(self) -> bool:
101
+ return self.patch > 1 and self.depth_core > 0
102
+
103
+ @property
104
+ def head_dim(self) -> int:
105
+ return self.dim // self.n_heads
106
+
107
+ @property
108
+ def mlp_hidden(self) -> int:
109
+ return int(self.dim * self.mlp_ratio)
110
+
111
+ def validate(self) -> None:
112
+ assert self.dim % self.n_heads == 0
113
+ assert self.head_dim % 4 == 0, "head_dim must be divisible by 4 (2D axial RoPE)"
114
+ if self.hourglass:
115
+ assert self.grid_size % self.patch == 0
116
+
117
+ def to_dict(self) -> Dict[str, Any]:
118
+ return asdict(self)
119
+
120
+ @classmethod
121
+ def from_dict(cls, d: Dict[str, Any]) -> "TroutConfig":
122
+ import dataclasses as _dc
123
+ names = {f.name for f in _dc.fields(cls)}
124
+ keep = {k: v for k, v in d.items() if k in names}
125
+ cfg = cls(**keep)
126
+ cfg.validate()
127
+ return cfg
128
+
129
+
130
+ SMALL_PRESET: Dict[str, Any] = dict(
131
+ dim=512, n_heads=8, depth_in=2, depth_core=8, depth_out=2,
132
+ ) # matches Tessera-small; ~47.6M parameters
133
+
134
+
135
+ # ---------------------------------------------------------------------------
136
+ # OKLab color math (Bjorn Ottosson's public-domain formulas)
137
+ # ---------------------------------------------------------------------------
138
+
139
+ def _srgb_to_linear_np(c):
140
+ return np.where(c <= 0.04045, c / 12.92, ((c + 0.055) / 1.055) ** 2.4)
141
+
142
+
143
+ def _linear_to_srgb_np(c):
144
+ c = np.clip(c, 0.0, 1.0)
145
+ return np.where(c <= 0.0031308, 12.92 * c, 1.055 * np.maximum(c, 1e-12) ** (1 / 2.4) - 0.055)
146
+
147
+
148
+ def srgb_to_oklab(rgb: np.ndarray) -> np.ndarray:
149
+ """rgb: (..., 3) in [0,255]. Returns OKLab (..., 3) float32."""
150
+ x = np.asarray(rgb, dtype=np.float64) / 255.0
151
+ lin = _srgb_to_linear_np(x)
152
+ r, g, b = lin[..., 0], lin[..., 1], lin[..., 2]
153
+ l = 0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b
154
+ m = 0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b
155
+ s = 0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b
156
+ l_, m_, s_ = np.cbrt(np.maximum(l, 0)), np.cbrt(np.maximum(m, 0)), np.cbrt(np.maximum(s, 0))
157
+ L = 0.2104542553 * l_ + 0.7936177850 * m_ - 0.0040720468 * s_
158
+ A = 1.9779984951 * l_ - 2.4285922050 * m_ + 0.4505937099 * s_
159
+ B = 0.0259040371 * l_ + 0.7827717662 * m_ - 0.8086757660 * s_
160
+ return np.stack([L, A, B], -1).astype(np.float32)
161
+
162
+
163
+ def _torch_srgb_to_linear(c):
164
+ return torch.where(c <= 0.04045, c / 12.92, ((c + 0.055) / 1.055) ** 2.4)
165
+
166
+
167
+ def srgb_to_oklab_t(rgb: torch.Tensor) -> torch.Tensor:
168
+ x = rgb.float() / 255.0
169
+ lin = _torch_srgb_to_linear(x)
170
+ r, g, b = lin[..., 0], lin[..., 1], lin[..., 2]
171
+ l = 0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b
172
+ m = 0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b
173
+ s = 0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b
174
+ l_ = torch.pow(l.clamp(min=0), 1 / 3)
175
+ m_ = torch.pow(m.clamp(min=0), 1 / 3)
176
+ s_ = torch.pow(s.clamp(min=0), 1 / 3)
177
+ L = 0.2104542553 * l_ + 0.7936177850 * m_ - 0.0040720468 * s_
178
+ A = 1.9779984951 * l_ - 2.4285922050 * m_ + 0.4505937099 * s_
179
+ B = 0.0259040371 * l_ + 0.7827717662 * m_ - 0.8086757660 * s_
180
+ return torch.stack([L, A, B], -1).float()
181
+
182
+
183
+ def rgb_to_key(rgb: np.ndarray) -> np.ndarray:
184
+ rgb = rgb.astype(np.int64)
185
+ return (rgb[..., 0] << 16) | (rgb[..., 1] << 8) | rgb[..., 2]
186
+
187
+
188
+ def key_to_rgb(key: np.ndarray) -> np.ndarray:
189
+ key = np.asarray(key, dtype=np.int64)
190
+ return np.stack([(key >> 16) & 255, (key >> 8) & 255, key & 255], -1).astype(np.uint8)
191
+
192
+
193
+ # ---------------------------------------------------------------------------
194
+ # palette: 512 colors, frequency-weighted k-means in OKLab, snapped to data
195
+ # ---------------------------------------------------------------------------
196
+
197
+ def _weighted_kmeans(x: torch.Tensor, w: torch.Tensor, k: int, iters: int = 60,
198
+ seed: int = 0) -> torch.Tensor:
199
+ """Weighted k-means with k-means++ init. x: (U,3), w: (U,). Returns (k,3)."""
200
+ g = torch.Generator().manual_seed(seed)
201
+ U = x.shape[0]
202
+ w = w.float()
203
+ first = torch.multinomial(w, 1, generator=g).item()
204
+ centers = [x[first]]
205
+ d2 = ((x - x[first]) ** 2).sum(1)
206
+ for _ in range(1, k):
207
+ p = (w * d2)
208
+ if p.sum() <= 0:
209
+ j = int(torch.randint(U, (1,), generator=g))
210
+ else:
211
+ j = torch.multinomial(p.clamp(min=0), 1, generator=g).item()
212
+ centers.append(x[j])
213
+ d2 = torch.minimum(d2, ((x - x[j]) ** 2).sum(1))
214
+ C = torch.stack(centers).clone()
215
+ for _ in range(iters):
216
+ # nearest assignment, chunked to bound memory
217
+ idx = torch.empty(U, dtype=torch.long)
218
+ mind = torch.empty(U)
219
+ c2 = (C * C).sum(1)
220
+ for s in range(0, U, 1 << 18):
221
+ p = x[s:s + (1 << 18)]
222
+ d = (p * p).sum(1, keepdim=True) - 2 * p @ C.T + c2[None]
223
+ dv, di = d.min(1)
224
+ idx[s:s + p.shape[0]] = di
225
+ mind[s:s + p.shape[0]] = dv.clamp(min=0)
226
+ sums = torch.zeros_like(C).index_add_(0, idx, x * w[:, None])
227
+ ws = torch.zeros(k).index_add_(0, idx, w)
228
+ newC = torch.where((ws <= 0)[:, None], C, sums / ws.clamp(min=1e-12)[:, None])
229
+ nempty = int((ws <= 0).sum())
230
+ if nempty > 0:
231
+ worst = torch.topk(w * mind, nempty).indices
232
+ newC[ws <= 0] = x[worst]
233
+ shift = (newC - C).norm(dim=1).max().item()
234
+ C = newC
235
+ if shift < 1e-7:
236
+ break
237
+ return C
238
+
239
+
240
+ class Palette:
241
+ """A fixed list of colors. Index 0 = transparent; 1..K-1 opaque sRGB."""
242
+
243
+ def __init__(self, colors: np.ndarray, name: str = "custom"):
244
+ colors = np.asarray(colors, dtype=np.uint8).reshape(-1, 3).copy()
245
+ colors[0] = 0
246
+ self.colors = colors
247
+ self.name = name
248
+ lab = srgb_to_oklab(colors)
249
+ lab[0] = 0.0
250
+ self.lab = lab
251
+
252
+ @property
253
+ def size(self) -> int:
254
+ return int(self.colors.shape[0])
255
+
256
+ def __len__(self) -> int:
257
+ return self.size
258
+
259
+ def rgba_table(self) -> np.ndarray:
260
+ t = np.concatenate([self.colors, np.full((self.size, 1), 255, np.uint8)], 1)
261
+ t[0] = 0
262
+ return t
263
+
264
+ def tokens_to_rgba(self, tokens) -> np.ndarray:
265
+ if isinstance(tokens, torch.Tensor):
266
+ tokens = tokens.detach().cpu().numpy()
267
+ tokens = np.asarray(tokens).astype(np.int64)
268
+ table = np.concatenate([self.rgba_table(), np.array([[255, 0, 255, 255]], np.uint8)], 0)
269
+ return table[np.clip(tokens, 0, self.size)]
270
+
271
+ def nearest_index(self, rgb: np.ndarray) -> np.ndarray:
272
+ """Nearest opaque entry (1..K-1) in OKLab for (M,3) uint8."""
273
+ if rgb.shape[0] == 0:
274
+ return np.zeros(0, np.int64)
275
+ pts = torch.from_numpy(srgb_to_oklab(rgb))
276
+ cen = torch.from_numpy(self.lab[1:])
277
+ idx, _ = self._nearest(pts, cen)
278
+ return idx.numpy() + 1
279
+
280
+ @staticmethod
281
+ def _nearest(points: torch.Tensor, centers: torch.Tensor, chunk: int = 1 << 18):
282
+ idx = torch.empty(points.shape[0], dtype=torch.long)
283
+ dist = torch.empty(points.shape[0])
284
+ c2 = (centers * centers).sum(1)
285
+ for s in range(0, points.shape[0], chunk):
286
+ p = points[s:s + chunk]
287
+ d = (p * p).sum(1, keepdim=True) - 2 * p @ centers.T + c2[None]
288
+ dv, di = d.min(1)
289
+ idx[s:s + p.shape[0]] = di
290
+ dist[s:s + p.shape[0]] = dv.clamp(min=0)
291
+ return idx, dist
292
+
293
+ def build_lut(self, keys: np.ndarray) -> Dict[str, np.ndarray]:
294
+ keys = np.unique(np.asarray(keys, dtype=np.int64))
295
+ idx = self.nearest_index(key_to_rgb(keys))
296
+ return {"keys": keys, "idx": idx.astype(np.uint16)}
297
+
298
+ def quantize_rgba(self, rgba: np.ndarray, alpha_threshold: int = 128,
299
+ lut: Optional[dict] = None) -> np.ndarray:
300
+ """uint8 RGBA (...,4) -> uint16 tokens (...); alpha<thresh -> transparent."""
301
+ rgba = np.asarray(rgba)
302
+ shape = rgba.shape[:-1]
303
+ flat = rgba.reshape(-1, 4)
304
+ opaque = flat[:, 3] >= alpha_threshold
305
+ out = np.zeros(flat.shape[0], np.uint16)
306
+ if opaque.any():
307
+ keys = rgb_to_key(flat[opaque, :3])
308
+ if lut is None:
309
+ lut = self.build_lut(keys)
310
+ pos = np.searchsorted(lut["keys"], keys)
311
+ pos = np.clip(pos, 0, len(lut["keys"]) - 1)
312
+ hit = lut["keys"][pos] == keys
313
+ vals = lut["idx"][pos]
314
+ if not hit.all():
315
+ miss = ~hit
316
+ vals = vals.copy()
317
+ vals[miss] = self.nearest_index(key_to_rgb(keys[miss]))
318
+ out[opaque] = vals
319
+ return out.reshape(shape)
320
+
321
+ def soft_targets(self, tau: float = 0.05) -> torch.Tensor:
322
+ """(K,K) row-stochastic matrix for perceptual label smoothing:
323
+ row c = softmax over opaque colors of -dE(c,.)^2 / tau^2."""
324
+ lab = torch.from_numpy(self.lab).double()
325
+ d2 = torch.cdist(lab, lab) ** 2
326
+ logits = -d2 / (tau * tau)
327
+ logits[:, 0] = -math.inf
328
+ S = torch.softmax(logits, dim=1)
329
+ S[0] = 0.0
330
+ S[0, 0] = 1.0
331
+ return S.float()
332
+
333
+ def to_dict(self) -> Dict[str, Any]:
334
+ return {"name": self.name, "colors": self.colors.tolist()}
335
+
336
+ @classmethod
337
+ def from_dict(cls, d: Dict[str, Any]) -> "Palette":
338
+ return cls(np.array(d["colors"], dtype=np.uint8), name=d.get("name", "custom"))
339
+
340
+ def save(self, path: str) -> None:
341
+ with open(path, "w") as f:
342
+ json.dump(self.to_dict(), f)
343
+
344
+ @classmethod
345
+ def load(cls, path: str) -> "Palette":
346
+ with open(path) as f:
347
+ return cls.from_dict(json.load(f))
348
+
349
+ def preview(self, cell: int = 12, cols: int = 32) -> "Image":
350
+ from PIL import Image
351
+ rows = math.ceil(self.size / cols)
352
+ img = np.full((rows * cell, cols * cell, 3), 255, np.uint8)
353
+ for i in range(self.size):
354
+ r, c = divmod(i, cols)
355
+ if i == 0:
356
+ yy, xx = np.mgrid[0:cell, 0:cell]
357
+ img[r * cell:(r + 1) * cell, c * cell:(c + 1) * cell] = np.where(
358
+ (((yy // 3) + (xx // 3)) % 2 == 0)[..., None], 200, 140)
359
+ else:
360
+ img[r * cell:(r + 1) * cell, c * cell:(c + 1) * cell] = self.colors[i]
361
+ return Image.fromarray(img)
362
+
363
+ @classmethod
364
+ def build_kmeans(cls, keys: np.ndarray, counts: np.ndarray, size: int = 512,
365
+ weight_power: float = 0.5, iters: int = 60, seed: int = 0) -> "Palette":
366
+ """Data-driven palette: 1 transparent + size-1 opaque colors chosen by
367
+ frequency-weighted (tempered by weight_power) k-means in OKLab, with
368
+ centroids snapped to real dataset colors so exact hues survive."""
369
+ keys = np.asarray(keys, dtype=np.int64)
370
+ counts = np.asarray(counts, dtype=np.float64)
371
+ k = size - 1
372
+ rgb = key_to_rgb(keys)
373
+ if keys.shape[0] <= k:
374
+ colors = np.concatenate([np.zeros((1, 3), np.uint8), rgb], 0)
375
+ return cls._sorted(colors, name=f"exact{len(colors)}")
376
+ x = torch.from_numpy(srgb_to_oklab(rgb))
377
+ w = torch.from_numpy(counts ** weight_power)
378
+ C = _weighted_kmeans(x, w, k, iters=iters, seed=seed)
379
+ snap_idx, _ = cls._nearest(C, x)
380
+ snap_idx = snap_idx.numpy()
381
+ chosen = list(dict.fromkeys(snap_idx.tolist()))
382
+ if len(chosen) < k: # fill duplicate slots with worst-represented colors
383
+ cen = x[torch.as_tensor(chosen)]
384
+ _, dist = cls._nearest(x, cen)
385
+ order = torch.argsort(w * dist, descending=True).numpy()
386
+ for j in order:
387
+ if len(chosen) >= k:
388
+ break
389
+ if int(j) not in chosen:
390
+ chosen.append(int(j))
391
+ colors = np.concatenate([np.zeros((1, 3), np.uint8), rgb[np.array(chosen[:k])]], 0)
392
+ return cls._sorted(colors, name=f"kmeans{size}")
393
+
394
+ @classmethod
395
+ def _sorted(cls, colors: np.ndarray, name: str) -> "Palette":
396
+ """Order opaque colors by hue bucket then lightness (readable palette)."""
397
+ opaque = colors[1:]
398
+ lab = srgb_to_oklab(opaque)
399
+ chroma = np.sqrt(lab[:, 1] ** 2 + lab[:, 2] ** 2)
400
+ hue = np.arctan2(lab[:, 2], lab[:, 1])
401
+ bucket = np.where(chroma < 0.03, -1, np.floor((hue + np.pi) / (2 * np.pi) * 12)).astype(np.int64)
402
+ order = np.lexsort((lab[:, 0], bucket))
403
+ return cls(np.concatenate([colors[:1], opaque[order]], 0), name=name)
404
+
405
+
406
+ def quantization_report(palette: Palette, rgba: np.ndarray, tokens: np.ndarray,
407
+ max_pixels: int = 2_000_000) -> Dict[str, float]:
408
+ """Mean/p95 OKLab error introduced by quantization on opaque pixels."""
409
+ flat = rgba.reshape(-1, 4)
410
+ tok = tokens.reshape(-1)
411
+ if flat.shape[0] > max_pixels:
412
+ sel = np.random.default_rng(0).integers(0, flat.shape[0], max_pixels)
413
+ flat, tok = flat[sel], tok[sel]
414
+ op = flat[:, 3] >= 128
415
+ if not op.any():
416
+ return {"mean_dE": 0.0, "p95_dE": 0.0, "exact_frac": 1.0}
417
+ src = srgb_to_oklab(flat[op, :3])
418
+ dst = palette.lab[tok[op].astype(np.int64)]
419
+ d = np.linalg.norm(src - dst, axis=1)
420
+ exact = (palette.colors[tok[op].astype(np.int64)] == flat[op, :3]).all(1)
421
+ return {"mean_dE": float(d.mean()), "p95_dE": float(np.percentile(d, 95)),
422
+ "max_dE": float(d.max()), "exact_frac": float(exact.mean())}
423
+
424
+
425
+ # ---------------------------------------------------------------------------
426
+ # model building blocks
427
+ # ---------------------------------------------------------------------------
428
+
429
+ class RMSNorm(nn.Module):
430
+ def __init__(self, dim: int, eps: float = 1e-6, affine: bool = True):
431
+ super().__init__()
432
+ self.eps = eps
433
+ self.weight = nn.Parameter(torch.ones(dim)) if affine else None
434
+
435
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
436
+ dtype = x.dtype
437
+ x32 = x.float()
438
+ y = x32 * torch.rsqrt(x32.pow(2).mean(-1, keepdim=True) + self.eps)
439
+ if self.weight is not None:
440
+ y = y * self.weight.float()
441
+ return y.to(dtype)
442
+
443
+
444
+ def _rope_freqs(head_dim: int, theta: float) -> torch.Tensor:
445
+ n = head_dim // 4 # rotation pairs per axis
446
+ return 1.0 / (theta ** (torch.arange(n, dtype=torch.float64) / n))
447
+
448
+
449
+ def build_rope(n_prefix: int, grid: int, head_dim: int, theta: float,
450
+ stride: int = 1) -> Tuple[torch.Tensor, torch.Tensor]:
451
+ """cos/sin tables for 2D axial RoPE. Image token (i,j) at a level with
452
+ `stride` pixels per token sits at pixel-space centre
453
+ (i*stride + (stride-1)/2, j*stride + (stride-1)/2); prefix angle 0."""
454
+ f = _rope_freqs(head_dim, theta)
455
+ ii, jj = torch.meshgrid(torch.arange(grid, dtype=torch.float64),
456
+ torch.arange(grid, dtype=torch.float64), indexing="ij")
457
+ off = (stride - 1) / 2.0
458
+ rows = (ii.reshape(-1) * stride + off)[:, None] * f[None]
459
+ cols = (jj.reshape(-1) * stride + off)[:, None] * f[None]
460
+ ang = torch.cat([rows, cols], 1) # (grid^2, head_dim/2)
461
+ ang = torch.cat([torch.zeros(n_prefix, ang.shape[1], dtype=ang.dtype), ang], 0)
462
+ return ang.cos().float(), ang.sin().float()
463
+
464
+
465
+ def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
466
+ # x: (B,H,N,hd); cos/sin: (N, hd/2)
467
+ x1, x2 = x.float().chunk(2, dim=-1)
468
+ out = torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1)
469
+ return out.to(x.dtype)
470
+
471
+
472
+ def fourier_features(x: torch.Tensor, n_freqs: int) -> torch.Tensor:
473
+ """x: (...,C) roughly in [-1,1] -> (..., C*2*n_freqs) sin/cos features."""
474
+ freqs = (2.0 ** torch.arange(n_freqs, dtype=torch.float32)) * math.pi
475
+ ang = x.float().unsqueeze(-1) * freqs
476
+ return torch.cat([ang.sin(), ang.cos()], dim=-1).flatten(-2)
477
+
478
+
479
+ class Attention(nn.Module):
480
+ def __init__(self, dim: int, n_heads: int):
481
+ super().__init__()
482
+ self.n_heads = n_heads
483
+ self.head_dim = dim // n_heads
484
+ self.qkv = nn.Linear(dim, 3 * dim, bias=False)
485
+ self.proj = nn.Linear(dim, dim, bias=False)
486
+ self.q_norm = RMSNorm(self.head_dim)
487
+ self.k_norm = RMSNorm(self.head_dim)
488
+
489
+ def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
490
+ B, N, D = x.shape
491
+ qkv = self.qkv(x).view(B, N, 3, self.n_heads, self.head_dim).permute(2, 0, 3, 1, 4)
492
+ q, k, v = qkv.unbind(0)
493
+ q = apply_rope(self.q_norm(q), cos, sin)
494
+ k = apply_rope(self.k_norm(k), cos, sin)
495
+ o = F.scaled_dot_product_attention(q, k, v)
496
+ return self.proj(o.transpose(1, 2).reshape(B, N, D))
497
+
498
+
499
+ class ConvSwiGLU(nn.Module):
500
+ """SwiGLU whose gate branch gets a residual depthwise 3x3 conv over the
501
+ image grid (local pixel structure)."""
502
+
503
+ def __init__(self, dim: int, hidden: int, conv: bool = True):
504
+ super().__init__()
505
+ self.w12 = nn.Linear(dim, 2 * hidden, bias=False)
506
+ self.w3 = nn.Linear(hidden, dim, bias=False)
507
+ self.dw = nn.Conv2d(hidden, hidden, 3, padding=1, groups=hidden, bias=True) if conv else None
508
+
509
+ def forward(self, x: torch.Tensor, n_prefix: int, hw: Tuple[int, int]) -> torch.Tensor:
510
+ a, g = self.w12(x).chunk(2, dim=-1)
511
+ if self.dw is not None:
512
+ B, _, C = g.shape
513
+ H, W = hw
514
+ g_pre, g_img = g[:, :n_prefix], g[:, n_prefix:]
515
+ conv = self.dw(g_img.transpose(1, 2).reshape(B, C, H, W)).flatten(2).transpose(1, 2)
516
+ g = torch.cat([g_pre, g_img + conv.to(g_img.dtype)], dim=1)
517
+ return self.w3(a * F.silu(g))
518
+
519
+
520
+ class Block(nn.Module):
521
+ def __init__(self, cfg: TroutConfig):
522
+ super().__init__()
523
+ self.norm1 = RMSNorm(cfg.dim, affine=False)
524
+ self.attn = Attention(cfg.dim, cfg.n_heads)
525
+ self.norm2 = RMSNorm(cfg.dim, affine=False)
526
+ self.mlp = ConvSwiGLU(cfg.dim, cfg.mlp_hidden, conv=cfg.conv_ffn)
527
+ # per-block offsets added to the shared adaLN-single modulation
528
+ # (shift/scale/gate) x (attn/mlp)
529
+ self.mod_bias = nn.Parameter(torch.zeros(6, cfg.dim))
530
+
531
+ def forward(self, x, mod, cos, sin, n_prefix: int, hw: Tuple[int, int]):
532
+ m = (mod + self.mod_bias).unsqueeze(2) # (B,6,1,D)
533
+ sh1, sc1, g1, sh2, sc2, g2 = m.unbind(1)
534
+ h = self.norm1(x) * (1 + sc1) + sh1
535
+ x = x + g1 * self.attn(h, cos, sin)
536
+ h = self.norm2(x) * (1 + sc2) + sh2
537
+ x = x + g2 * self.mlp(h, n_prefix, hw)
538
+ return x
539
+
540
+
541
+ # ---------------------------------------------------------------------------
542
+ # the model
543
+ # ---------------------------------------------------------------------------
544
+
545
+ class TroutModel(nn.Module):
546
+ """Dual-resolution (hourglass) masked-diffusion transformer denoiser."""
547
+
548
+ def __init__(self, cfg: TroutConfig, palette_lab: Optional[torch.Tensor] = None,
549
+ palette_rgb: Optional[torch.Tensor] = None):
550
+ super().__init__()
551
+ cfg.validate()
552
+ self.cfg = cfg
553
+ K, D, G = cfg.vocab_size, cfg.dim, cfg.grid_size
554
+ self.mask_id = cfg.mask_id
555
+ if palette_lab is None:
556
+ palette_lab = torch.zeros(K, 3)
557
+ if palette_rgb is None:
558
+ palette_rgb = torch.zeros(K, 3, dtype=torch.uint8)
559
+ assert palette_lab.shape == (K, 3), f"palette has {palette_lab.shape[0]} colors, config says {K}"
560
+ # the palette travels with the checkpoint
561
+ self.register_buffer("palette_lab", palette_lab.float().clone())
562
+ self.register_buffer("palette_rgb", palette_rgb.to(torch.uint8).clone())
563
+
564
+ # ---- token embeddings (color-grounded) ----
565
+ self.tok_emb = nn.Embedding(K + 1, D) # +1 for MASK
566
+ n_feat = 3 + 3 * 2 * cfg.color_freqs + 1
567
+ self.color_mlp = nn.Sequential(nn.Linear(n_feat, D), nn.SiLU(), nn.Linear(D, D))
568
+ self.pos_emb = nn.Parameter(torch.zeros(1, G * G, D))
569
+ self.sc_proj = nn.Linear(K, D, bias=False) if cfg.self_cond else None
570
+ # ---- text conditioning ----
571
+ self.text_norm = RMSNorm(cfg.text_dim)
572
+ self.text_proj = nn.Linear(cfg.text_dim, D)
573
+ self.null_text = nn.Parameter(torch.zeros(1, cfg.text_len, D))
574
+ self.pool_mlp = nn.Sequential(RMSNorm(cfg.text_dim), nn.Linear(cfg.text_dim, D), nn.SiLU(), nn.Linear(D, D))
575
+ self.null_cond = nn.Parameter(torch.zeros(1, D))
576
+ self.ratio_mlp = nn.Sequential(nn.Linear(2 * 8, D), nn.SiLU(), nn.Linear(D, D))
577
+ self.registers = nn.Parameter(torch.zeros(1, cfg.n_registers, D))
578
+ self.t_block = nn.Sequential(nn.SiLU(), nn.Linear(D, 6 * D))
579
+ # ---- hourglass backbone ----
580
+ p = cfg.patch
581
+ self.blocks_in = nn.ModuleList([Block(cfg) for _ in range(cfg.depth_in)])
582
+ self.down = nn.Sequential(RMSNorm(p * p * D), nn.Linear(p * p * D, D))
583
+ self.blocks_core = nn.ModuleList([Block(cfg) for _ in range(cfg.depth_core)])
584
+ self.up = nn.Linear(D, p * p * D)
585
+ self.blocks_out = nn.ModuleList([Block(cfg) for _ in range(cfg.depth_out)])
586
+ self.norm_out = RMSNorm(D, affine=False)
587
+ self.final_mod = nn.Sequential(nn.SiLU(), nn.Linear(D, 2 * D))
588
+ self.head = nn.Linear(D, K)
589
+ # ---- fixed tables ----
590
+ n_pre = cfg.n_registers + cfg.text_len
591
+ self.n_prefix = n_pre
592
+ cf, sf = build_rope(n_pre, G, cfg.head_dim, cfg.rope_theta, stride=1)
593
+ self.register_buffer("rope_cos", cf, persistent=False)
594
+ self.register_buffer("rope_sin", sf, persistent=False)
595
+ cc, sc = build_rope(n_pre, G // cfg.patch, cfg.head_dim, cfg.rope_theta, stride=cfg.patch)
596
+ self.register_buffer("rope_cos_core", cc, persistent=False)
597
+ self.register_buffer("rope_sin_core", sc, persistent=False)
598
+ self.register_buffer("color_feats", self._color_features(self.palette_lab), persistent=False)
599
+ self.grad_checkpointing = False
600
+ self._init_weights()
601
+
602
+ # ------------------------------------------------------------------
603
+ def _color_features(self, lab: torch.Tensor) -> torch.Tensor:
604
+ x = torch.stack([lab[:, 0] * 2 - 1, lab[:, 1] / 0.4, lab[:, 2] / 0.4], -1)
605
+ alpha = torch.ones(lab.shape[0], 1)
606
+ alpha[0] = 0.0 # token 0 = transparent
607
+ x = x * alpha
608
+ return torch.cat([x, fourier_features(x, self.cfg.color_freqs), alpha], -1)
609
+
610
+ def set_palette(self, palette_lab: torch.Tensor, palette_rgb: torch.Tensor) -> None:
611
+ self.palette_lab.copy_(palette_lab.float())
612
+ self.palette_rgb.copy_(palette_rgb.to(torch.uint8))
613
+ self.color_feats = self._color_features(self.palette_lab.detach().cpu()).to(self.palette_lab.device)
614
+
615
+ def _load_from_state_dict(self, state_dict, prefix, *args, **kwargs):
616
+ super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
617
+ self.color_feats = self._color_features(self.palette_lab.detach().cpu()).to(self.palette_lab.device)
618
+
619
+ def _init_weights(self) -> None:
620
+ def init(m):
621
+ if isinstance(m, nn.Linear):
622
+ nn.init.trunc_normal_(m.weight, std=0.02)
623
+ if m.bias is not None:
624
+ nn.init.zeros_(m.bias)
625
+ elif isinstance(m, nn.Embedding):
626
+ nn.init.normal_(m.weight, std=0.02)
627
+ self.apply(init)
628
+ nn.init.trunc_normal_(self.pos_emb, std=0.02)
629
+ nn.init.normal_(self.null_text, std=0.02)
630
+ nn.init.normal_(self.null_cond, std=0.02)
631
+ nn.init.normal_(self.registers, std=0.02)
632
+ # zero-init: conditioning projections, gates, head, self-cond, up path
633
+ for m in [self.t_block[-1], self.final_mod[-1], self.head]:
634
+ nn.init.zeros_(m.weight)
635
+ nn.init.zeros_(m.bias)
636
+ if self.sc_proj is not None:
637
+ nn.init.zeros_(self.sc_proj.weight)
638
+ nn.init.zeros_(self.up.weight)
639
+ nn.init.zeros_(self.up.bias)
640
+ for blk in list(self.blocks_in) + list(self.blocks_core) + list(self.blocks_out):
641
+ if blk.mlp.dw is not None:
642
+ nn.init.zeros_(blk.mlp.dw.weight)
643
+ nn.init.zeros_(blk.mlp.dw.bias)
644
+
645
+ # ------------------------------------------------------------------
646
+ def _merge(self, h: torch.Tensor) -> torch.Tensor:
647
+ B, N, D = h.shape
648
+ G, p = self.cfg.grid_size, self.cfg.patch
649
+ g = G // p
650
+ h = h.view(B, g, p, g, p, D).permute(0, 1, 3, 2, 4, 5).reshape(B, g * g, p * p * D)
651
+ return self.down(h)
652
+
653
+ def _split(self, h: torch.Tensor) -> torch.Tensor:
654
+ B = h.shape[0]
655
+ G, p, D = self.cfg.grid_size, self.cfg.patch, self.cfg.dim
656
+ g = G // p
657
+ h = self.up(h).view(B, g, g, p, p, D).permute(0, 1, 3, 2, 4, 5).reshape(B, G * G, D)
658
+ return h
659
+
660
+ def _run(self, blk: Block, xs, mod, cos, sin, n_pre: int, hw):
661
+ if self.grad_checkpointing and self.training and torch.is_grad_enabled():
662
+ return torch.utils.checkpoint.checkpoint(blk, xs, mod, cos, sin, n_pre, hw,
663
+ use_reentrant=False)
664
+ return blk(xs, mod, cos, sin, n_pre, hw)
665
+
666
+ def embed_table(self) -> torch.Tensor:
667
+ K = self.cfg.vocab_size
668
+ w = self.tok_emb.weight
669
+ colors = w[:K] + self.color_mlp(self.color_feats).to(w.dtype)
670
+ return torch.cat([colors, w[K:]], 0)
671
+
672
+ def forward(self, x: torch.Tensor, text_states: torch.Tensor, text_pooled: torch.Tensor,
673
+ mask_ratio: Optional[torch.Tensor] = None,
674
+ sc_probs: Optional[torch.Tensor] = None,
675
+ drop_text: Optional[torch.Tensor] = None) -> torch.Tensor:
676
+ """
677
+ x: (B,N) long palette ids or mask_id
678
+ text_states: (B,L,text_dim) frozen text-encoder token states
679
+ text_pooled: (B,text_dim) pooled text embedding
680
+ mask_ratio: (B,) fraction masked (computed from x if None)
681
+ sc_probs: (B,N,K) previous-step predicted distribution or None
682
+ drop_text: (B,) bool -> use learned null condition (CFG)
683
+ returns logits (B,N,K) float32
684
+ """
685
+ cfg = self.cfg
686
+ B, N = x.shape
687
+ D, G = cfg.dim, cfg.grid_size
688
+ is_mask = x == self.mask_id
689
+ if mask_ratio is None:
690
+ mask_ratio = is_mask.float().mean(1)
691
+ # ---- image tokens ----
692
+ h = F.embedding(x, self.embed_table()).float() + self.pos_emb
693
+ if self.sc_proj is not None and sc_probs is not None:
694
+ h = h + self.sc_proj(sc_probs.to(self.sc_proj.weight.dtype)).float() * is_mask.unsqueeze(-1)
695
+ # ---- text tokens + global conditioning ----
696
+ t = self.text_proj(self.text_norm(text_states.float())).float()
697
+ c = self.pool_mlp(text_pooled.float()).float()
698
+ if drop_text is not None:
699
+ t = torch.where(drop_text.view(B, 1, 1), self.null_text, t)
700
+ c = torch.where(drop_text.view(B, 1), self.null_cond, c)
701
+ r = mask_ratio.float().view(B, 1) * 2 - 1
702
+ g = c + self.ratio_mlp(fourier_features(r, 8).view(B, -1)).float()
703
+ mod = self.t_block(g).float().view(B, 6, D)
704
+ prefix = torch.cat([self.registers.expand(B, -1, -1), t], 1)
705
+ n_pre = prefix.shape[1]
706
+ xs = torch.cat([prefix, h], 1)
707
+ # ---- pixel level (in) ----
708
+ for blk in self.blocks_in:
709
+ xs = self._run(blk, xs, mod, self.rope_cos, self.rope_sin, n_pre, (G, G))
710
+ # ---- patch level (core) ----
711
+ pre, img = xs[:, :n_pre], xs[:, n_pre:]
712
+ skip = img
713
+ xs = torch.cat([pre, self._merge(img).float()], 1)
714
+ gc = G // cfg.patch
715
+ for blk in self.blocks_core:
716
+ xs = self._run(blk, xs, mod, self.rope_cos_core, self.rope_sin_core, n_pre, (gc, gc))
717
+ pre, core = xs[:, :n_pre], xs[:, n_pre:]
718
+ xs = torch.cat([pre, skip + self._split(core).float()], 1)
719
+ # ---- pixel level (out) ----
720
+ for blk in self.blocks_out:
721
+ xs = self._run(blk, xs, mod, self.rope_cos, self.rope_sin, n_pre, (G, G))
722
+ img = xs[:, n_pre:]
723
+ shift, scale = self.final_mod(g).float().unsqueeze(1).chunk(2, dim=-1)
724
+ img = self.norm_out(img) * (1 + scale) + shift
725
+ return self.head(img).float()
726
+
727
+ def num_params(self, trainable_only: bool = True) -> int:
728
+ return sum(p.numel() for p in self.parameters() if p.requires_grad or not trainable_only)
729
+
730
+
731
+ # ---------------------------------------------------------------------------
732
+ # sampler: parallel confidence-ordered unmasking (MaskGIT-style) with CFG,
733
+ # Gumbel-annealed choice, spatial NMS and self-conditioning feedback
734
+ # ---------------------------------------------------------------------------
735
+
736
+ def filter_logits(logits: torch.Tensor, top_k: int = 0, top_p: float = 1.0,
737
+ min_p: float = 0.0) -> torch.Tensor:
738
+ if top_k and top_k > 0 and top_k < logits.shape[-1]:
739
+ kth = torch.topk(logits, top_k, dim=-1).values[..., -1:]
740
+ logits = logits.masked_fill(logits < kth, -math.inf)
741
+ if min_p and min_p > 0:
742
+ p = logits.softmax(-1)
743
+ logits = logits.masked_fill(p < min_p * p.amax(-1, keepdim=True), -math.inf)
744
+ if top_p is not None and top_p < 1.0:
745
+ sorted_l, idx = torch.sort(logits, dim=-1, descending=True)
746
+ sp = sorted_l.softmax(-1)
747
+ remove = (sp.cumsum(-1) - sp) > top_p
748
+ sorted_l = sorted_l.masked_fill(remove, -math.inf)
749
+ logits = torch.full_like(logits, -math.inf).scatter(-1, idx, sorted_l)
750
+ return logits
751
+
752
+
753
+ def cfg_weight(scale: float, schedule: str, frac: float) -> float:
754
+ if schedule == "constant":
755
+ return scale
756
+ if schedule == "linear":
757
+ return 1.0 + (scale - 1.0) * frac
758
+ if schedule == "cosine":
759
+ return 1.0 + (scale - 1.0) * (1 - math.cos(math.pi * frac)) / 2
760
+ raise ValueError(f"unknown cfg schedule {schedule!r}")
761
+
762
+
763
+ def _gumbel(shape, device, generator):
764
+ u = torch.rand(shape, device=device, generator=generator).clamp_(1e-10, 1 - 1e-10)
765
+ return -torch.log(-torch.log(u))
766
+
767
+
768
+ def _nms_eligible(score: torch.Tensor, masked: torch.Tensor, certain: torch.Tensor,
769
+ grid: int) -> torch.Tensor:
770
+ B, N = score.shape
771
+ cand = masked & ~certain
772
+ s = torch.where(cand, score, torch.full_like(score, -math.inf)).view(B, 1, grid, grid)
773
+ local_max = F.max_pool2d(s, 3, stride=1, padding=1).view(B, N)
774
+ return certain | (cand & (score >= local_max))
775
+
776
+
777
+ @torch.no_grad()
778
+ def sample(model: TroutModel, text_states: torch.Tensor, text_pooled: torch.Tensor,
779
+ steps: int = 16, cfg_scale: float = 3.0, cfg_schedule: str = "constant",
780
+ temperature: float = 1.0, temperature_end: Optional[float] = None,
781
+ top_k: int = 0, top_p: float = 1.0, min_p: float = 0.0,
782
+ choice_temperature: float = 2.0, spatial_nms: bool = True,
783
+ nms_threshold: float = 0.9, polish_steps: int = 0, polish_frac: float = 0.08,
784
+ init_tokens: Optional[torch.Tensor] = None, keep_mask: Optional[torch.Tensor] = None,
785
+ self_cond: bool = True, generator: Optional[torch.Generator] = None,
786
+ return_history: bool = False):
787
+ """Generate palette-token canvases (B, N); inpainting via init_tokens+keep_mask."""
788
+ cfg = model.cfg
789
+ device = text_pooled.device
790
+ B, N, K, M, G = text_pooled.shape[0], cfg.num_tokens, cfg.vocab_size, cfg.mask_id, cfg.grid_size
791
+ if init_tokens is not None and keep_mask is not None:
792
+ init_tokens = init_tokens.to(device).long()
793
+ x = torch.where(keep_mask.to(device).bool(), init_tokens, torch.full_like(init_tokens, M))
794
+ else:
795
+ x = torch.full((B, N), M, dtype=torch.long, device=device)
796
+ masked = x == M
797
+ conf_map = torch.where(masked, torch.zeros(B, N, device=device),
798
+ torch.full((B, N), math.inf, device=device))
799
+ n_start = masked.sum(1)
800
+ use_cfg = cfg_scale is not None and abs(cfg_scale - 1.0) > 1e-6
801
+ if use_cfg:
802
+ ts2 = torch.cat([text_states, text_states])
803
+ tp2 = torch.cat([text_pooled, text_pooled])
804
+ drop2 = torch.cat([torch.zeros(B, dtype=torch.bool, device=device),
805
+ torch.ones(B, dtype=torch.bool, device=device)])
806
+ sc = torch.zeros(B, N, K, device=device) if (cfg.self_cond and self_cond) else None
807
+ history = []
808
+
809
+ def predict(x_in, sc_in, frac):
810
+ ratio = (x_in == M).float().mean(1)
811
+ if use_cfg:
812
+ sc2 = torch.cat([sc_in, sc_in]) if sc_in is not None else None
813
+ out = model(torch.cat([x_in, x_in]), ts2, tp2, torch.cat([ratio, ratio]),
814
+ sc2, drop2).float()
815
+ lc, lu = out.chunk(2)
816
+ w = cfg_weight(cfg_scale, cfg_schedule, frac)
817
+ return lu + w * (lc - lu)
818
+ return model(x_in, text_states, text_pooled, ratio, sc_in, None).float()
819
+
820
+ for k in range(steps):
821
+ if not masked.any():
822
+ break
823
+ frac = (k + 1) / steps
824
+ logits = predict(x, sc, frac)
825
+ probs_full = logits.softmax(-1)
826
+ T = temperature if temperature_end is None else temperature + (temperature_end - temperature) * (k / max(1, steps - 1))
827
+ if T <= 0:
828
+ sampled = logits.argmax(-1)
829
+ p_sel = probs_full.gather(-1, sampled.unsqueeze(-1)).squeeze(-1)
830
+ else:
831
+ probs = filter_logits(logits / T, top_k, top_p, min_p).softmax(-1)
832
+ sampled = torch.multinomial(probs.view(-1, K), 1, generator=generator).view(B, N)
833
+ p_sel = probs.gather(-1, sampled.unsqueeze(-1)).squeeze(-1)
834
+ n_masked = masked.sum(1)
835
+ remaining = torch.floor(n_start.float() * math.cos(math.pi / 2 * frac)).long()
836
+ if k == steps - 1:
837
+ remaining = torch.zeros_like(remaining)
838
+ n_commit = torch.minimum((n_masked - remaining).clamp(min=1), n_masked)
839
+ score = torch.log(p_sel.clamp(min=1e-12))
840
+ if choice_temperature > 0: # Gumbel-annealed choice (MaskGIT)
841
+ score = score + choice_temperature * (1.0 - frac) * _gumbel(score.shape, device, generator)
842
+ priority = score
843
+ if spatial_nms: # don't commit two uncertain neighbours in one step
844
+ elig = _nms_eligible(score, masked, p_sel >= nms_threshold, G)
845
+ priority = score + elig.float() * 1e4
846
+ priority = torch.where(masked, priority, torch.full_like(priority, -math.inf))
847
+ ranks = priority.argsort(1, descending=True).argsort(1)
848
+ commit = (ranks < n_commit.unsqueeze(1)) & masked
849
+ x = torch.where(commit, sampled, x)
850
+ conf_map = torch.where(commit, p_sel, conf_map)
851
+ masked = masked & ~commit
852
+ if sc is not None:
853
+ sc = probs_full
854
+ if return_history:
855
+ history.append(x.clone())
856
+
857
+ # ---- optional polish: re-mask the least confident committed pixels ----
858
+ n_re = max(1, int(round(polish_frac * N)))
859
+ for _ in range(polish_steps):
860
+ low = torch.topk(-conf_map, n_re, dim=1).indices
861
+ remask = torch.zeros(B, N, dtype=torch.bool, device=device).scatter_(1, low, True)
862
+ remask &= torch.isfinite(conf_map)
863
+ x_in = torch.where(remask, torch.full_like(x, M), x)
864
+ logits = predict(x_in, sc, 1.0)
865
+ probs_full = logits.softmax(-1)
866
+ new = logits.argmax(-1)
867
+ x = torch.where(remask, new, x)
868
+ conf_map = torch.where(remask, probs_full.gather(-1, new.unsqueeze(-1)).squeeze(-1), conf_map)
869
+ if sc is not None:
870
+ sc = probs_full
871
+ if return_history:
872
+ history.append(x.clone())
873
+
874
+ if return_history:
875
+ return x, history
876
+ return x
877
+
878
+
879
+ # ---------------------------------------------------------------------------
880
+ # checkpoint helpers (same schema semantics as the Tessera distribution:
881
+ # ema weights + model_config + palette + text-encoder id + step)
882
+ # ---------------------------------------------------------------------------
883
+
884
+ def save_checkpoint(path: str, model: TroutModel, palette: Palette, step: int,
885
+ ema_state: Optional[Dict[str, torch.Tensor]] = None,
886
+ raw_state: Optional[Dict[str, torch.Tensor]] = None,
887
+ text_encoder: str = DEFAULT_TEXT_ENCODER, half: bool = False) -> None:
888
+ def prep(sd):
889
+ out = {}
890
+ for k, v in sd.items():
891
+ out[k] = (v.half() if half and v.is_floating_point() else v).cpu()
892
+ return out
893
+ ckpt = {"model_config": model.cfg.to_dict(), "palette": palette.to_dict(),
894
+ "text_encoder": text_encoder, "step": step}
895
+ if ema_state is not None:
896
+ ckpt["ema"] = prep(ema_state)
897
+ if raw_state is not None:
898
+ ckpt["model"] = prep(raw_state)
899
+ torch.save(ckpt, path)
900
+
901
+
902
+ def load_model_from_checkpoint(path: str, device="cpu", use_ema: bool = True):
903
+ ckpt = torch.load(path, map_location="cpu", weights_only=False)
904
+ mcfg = TroutConfig.from_dict(ckpt["model_config"])
905
+ palette = Palette.from_dict(ckpt["palette"])
906
+ model = TroutModel(mcfg, torch.from_numpy(palette.lab), torch.from_numpy(palette.colors))
907
+ key = "ema" if (use_ema and "ema" in ckpt) else "model"
908
+ sd = {k: v.float() if v.is_floating_point() else v for k, v in ckpt[key].items()}
909
+ model.load_state_dict(sd)
910
+ model.to(device).eval().requires_grad_(False)
911
+ return model, palette, ckpt
912
+
913
+
914
+ # ---------------------------------------------------------------------------
915
+ # visualization helpers
916
+ # ---------------------------------------------------------------------------
917
+
918
+ def tokens_to_images(tokens, palette: Palette, grid: int = 16) -> List:
919
+ from PIL import Image
920
+ if isinstance(tokens, torch.Tensor):
921
+ tokens = tokens.detach().cpu().numpy()
922
+ toks = np.asarray(tokens).reshape(-1, grid * grid)
923
+ imgs = []
924
+ for row in toks:
925
+ rgba = palette.tokens_to_rgba(row.reshape(grid, grid))
926
+ imgs.append(Image.fromarray(rgba, mode="RGBA"))
927
+ return imgs
928
+
929
+
930
+ def upscale(image, scale: int = 16, checker: bool = True):
931
+ """Nearest-neighbour upscale with an optional checkerboard background."""
932
+ from PIL import Image
933
+ w, h = image.size
934
+ big = image.resize((w * scale, h * scale), Image.NEAREST)
935
+ if not checker:
936
+ return big
937
+ bg = Image.new("RGBA", big.size, (255, 255, 255, 255))
938
+ s = 4 * 2 # checker cell in big pixels
939
+ yy, xx = np.mgrid[0:big.size[1], 0:big.size[0]]
940
+ mask = (((yy // s) + (xx // s)) % 2 == 0)
941
+ light = Image.fromarray(np.where(mask[..., None], np.array([238, 238, 238, 255], np.uint8),
942
+ np.array([204, 204, 204, 255], np.uint8)), "RGBA")
943
+ light.alpha_composite(big)
944
+ return light
945
+
946
+
947
+ def make_grid(images, ncols: int = 8, scale: int = 8, checker: bool = True):
948
+ from PIL import Image
949
+ cells = [upscale(im, scale, checker) for im in images]
950
+ cw, ch = cells[0].size
951
+ rows = math.ceil(len(cells) / ncols)
952
+ out = Image.new("RGBA", (ncols * cw, rows * ch), (255, 255, 255, 255))
953
+ for i, c in enumerate(cells):
954
+ out.alpha_composite(c, ((i % ncols) * cw, (i // ncols) * ch))
955
+ return out
956
+
957
+
958
+ # ---------------------------------------------------------------------------
959
+ # frozen CLIP text tower wrapper
960
+ # ---------------------------------------------------------------------------
961
+
962
+ def load_tokenizer(name: str):
963
+ from transformers import AutoTokenizer
964
+ return AutoTokenizer.from_pretrained(name)
965
+
966
+
967
+ def tokenize_texts(tokenizer, texts: Sequence[str], text_len: int) -> Tuple[np.ndarray, np.ndarray]:
968
+ """Fixed-length tokenization; pad with pad id; over-long prompts truncated
969
+ while keeping the final EOS token."""
970
+ eos = tokenizer.eos_token_id
971
+ pad = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else eos
972
+ n = len(texts)
973
+ out = np.full((n, text_len), pad, dtype=np.int32)
974
+ lengths = np.zeros(n, dtype=np.int32)
975
+ enc = tokenizer(list(texts), add_special_tokens=True, truncation=False)["input_ids"]
976
+ for i, ids in enumerate(enc):
977
+ lengths[i] = len(ids)
978
+ if len(ids) > text_len:
979
+ ids = list(ids[:text_len - 1]) + [eos]
980
+ out[i, :len(ids)] = ids
981
+ return out, lengths
982
+
983
+
984
+ class FrozenTextEncoder:
985
+ """Wraps a Hugging Face CLIP text tower. Not an nn.Module on purpose so its
986
+ weights never end up in the trainable model, optimizer or checkpoints."""
987
+
988
+ def __init__(self, name: str = DEFAULT_TEXT_ENCODER, text_len: int = 24,
989
+ device="cpu", dtype: torch.dtype = torch.float32):
990
+ from transformers import CLIPTextModel
991
+ from transformers.utils import logging as hf_logging
992
+ self.name = name
993
+ self.text_len = text_len
994
+ self.device = torch.device(device)
995
+ self.dtype = dtype
996
+ self.tokenizer = load_tokenizer(name)
997
+ prev = hf_logging.get_verbosity()
998
+ hf_logging.set_verbosity_error()
999
+ try:
1000
+ model = CLIPTextModel.from_pretrained(name)
1001
+ finally:
1002
+ hf_logging.set_verbosity(prev)
1003
+ self.model = model.to(device=self.device, dtype=dtype).eval().requires_grad_(False)
1004
+ self.dim = int(model.config.hidden_size)
1005
+
1006
+ def tokenize(self, texts: Sequence[str]) -> torch.Tensor:
1007
+ ids, _ = tokenize_texts(self.tokenizer, list(texts), self.text_len)
1008
+ return torch.from_numpy(ids).long().to(self.device)
1009
+
1010
+ @torch.no_grad()
1011
+ def __call__(self, ids: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
1012
+ out = self.model(input_ids=ids.long().to(self.device))
1013
+ return out.last_hidden_state, out.pooler_output
1014
+
1015
+ @torch.no_grad()
1016
+ def encode(self, texts: Sequence[str]) -> Tuple[torch.Tensor, torch.Tensor]:
1017
+ return self(self.tokenize(texts))