Trout: from-scratch 47.6M masked-diffusion pixel-art model
Browse files- LICENSE +22 -0
- README.md +109 -0
- clip_score.json +6 -0
- metrics.jsonl +250 -0
- model_config.json +18 -0
- showcase.png +0 -0
- train_config.json +55 -0
- trout-fp16.pt +3 -0
- trout.py +1017 -0
LICENSE
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Apache License
|
| 2 |
+
Version 2.0, January 2004
|
| 3 |
+
http://www.apache.org/licenses/
|
| 4 |
+
|
| 5 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 6 |
+
you may not use this file except in compliance with the License.
|
| 7 |
+
You may obtain a copy of the License at
|
| 8 |
+
|
| 9 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
| 10 |
+
|
| 11 |
+
Unless required by applicable law or agreed to in writing, software
|
| 12 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 13 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 14 |
+
See the License for the specific language governing permissions and
|
| 15 |
+
limitations under the License.
|
| 16 |
+
|
| 17 |
+
----
|
| 18 |
+
|
| 19 |
+
Trout was trained from scratch on the Pixel Art Atlas 1.4M dataset
|
| 20 |
+
(Angais, CC0 1.0). The CLIP text encoder used for conditioning
|
| 21 |
+
(openai/clip-vit-base-patch32) keeps its own MIT license and is not
|
| 22 |
+
redistributed with these weights.
|
README.md
ADDED
|
@@ -0,0 +1,109 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
pipeline_tag: text-to-image
|
| 5 |
+
datasets:
|
| 6 |
+
- Angais/pixel-art-atlas-1.4m
|
| 7 |
+
tags:
|
| 8 |
+
- pixel-art
|
| 9 |
+
- text-to-image
|
| 10 |
+
- masked-diffusion
|
| 11 |
+
- discrete-diffusion
|
| 12 |
+
- diffusion-lm
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# 🐟 Trout: a diffusion language model for 16×16 pixel art
|
| 16 |
+
|
| 17 |
+
**Trout** is a 47.6M-parameter absorbing-state **masked diffusion** model that treats each
|
| 18 |
+
16×16 sprite as a 256-token "sentence" over a learned **512-color palette**
|
| 19 |
+
(token 0 = transparent). A bidirectional hourglass transformer starts from a
|
| 20 |
+
fully masked canvas and unmasks all 256 pixels **in parallel in about 16 steps**,
|
| 21 |
+
conditioned on a frozen CLIP text encoder. No VAE, no latent space, no autoregression.
|
| 22 |
+
|
| 23 |
+
Trout was **built and trained from scratch** (architecture after
|
| 24 |
+
[Tessera-small](https://huggingface.co/thejagstudio/tessera-small) by Jagrat Patel
|
| 25 |
+
(TheJagStudio), whose open model card and demo space document the design) on a
|
| 26 |
+
2-vCPU CPU-only sandbox. All numbers below are real and reproducible from
|
| 27 |
+
`metrics.jsonl`.
|
| 28 |
+
|
| 29 |
+
## Model
|
| 30 |
+
|
| 31 |
+
| | |
|
| 32 |
+
|---|---|
|
| 33 |
+
| Parameters | 47,597,568 (Tessera-small: 47.6M) |
|
| 34 |
+
| Architecture | joint text+pixel bidirectional transformer, hourglass 2 + 8 + 2 blocks (2×2 patch-merged core), width 512, 8 heads, QK-RMSNorm, 2D axial RoPE (θ=100), Conv-SwiGLU, adaLN-single, 4 registers, self-conditioning, learned pos-emb |
|
| 35 |
+
| Text encoder | frozen `openai/clip-vit-base-patch32` text tower, 24 tokens (not included, MIT) |
|
| 36 |
+
| Vocabulary | 512 palette tokens (frequency-weighted OKLab k-means, mean ΔE 0.0099, snapped to dataset colors) + `<MASK>` |
|
| 37 |
+
| Objective | absorbing-state masked diffusion, cosine mask schedule, stratified t, blob masks (25%), perceptual OKLab label smoothing (ε=0.1, τ=0.05), CFG dropout (10%), hflip aug |
|
| 38 |
+
| Sampler | parallel confidence-ordered unmasking with CFG, Gumbel-annealed choice (T=2), spatial NMS (3×3, 0.9), self-conditioning feedback; inpainting supported |
|
| 39 |
+
| Optimizer | Muon (batched Newton-Schulz, RMS-matched 0.2·√max(m,n)) on hidden matrices + AdamW elsewhere, Warmup-Stable-Decay, EMA |
|
| 40 |
+
|
| 41 |
+
## Training — and the honest compute gap
|
| 42 |
+
|
| 43 |
+
Tessera-small was trained for **82,342 steps × 512 sprites** on an RTX PRO 6000
|
| 44 |
+
(≈5.5 h). Trout was trained **from scratch on 2 vCPUs / 4 GB RAM** for
|
| 45 |
+
**2,400 steps × 8 sprites** (~19k sprite-views of a 60k-sprite
|
| 46 |
+
subset of the same dataset) in CPU fp32. The recipe (schedule, smoothing,
|
| 47 |
+
self-conditioning, CFG dropout, Muon+AdamW, WSD, EMA) is followed faithfully;
|
| 48 |
+
only the compute budget is smaller by ~4 orders of magnitude, so Trout's
|
| 49 |
+
metrics are correspondingly weaker than Tessera's — this is a genuine
|
| 50 |
+
budget-limited from-scratch run, not a distilled or fine-tuned copy.
|
| 51 |
+
|
| 52 |
+
## Metrics (exported weights)
|
| 53 |
+
|
| 54 |
+
| Metric | Trout (this run) | Tessera-small (reference) |
|
| 55 |
+
|---|---|---|
|
| 56 |
+
| Validation NLL | 2.176 nats/pixel (perplexity 8.81 of 512) | 0.859 nats/pixel (2.36) |
|
| 57 |
+
| Validation accuracy | 54.6% of masked pixels | 78.8% |
|
| 58 |
+
| CLIP score (ViT-B/32) | 0.23 generated vs 0.26 real sprites |
|
| 59 |
+
|
| 60 |
+
The full per-step log is in `metrics.jsonl`.
|
| 61 |
+
|
| 62 |
+
## Usage
|
| 63 |
+
|
| 64 |
+
```python
|
| 65 |
+
from huggingface_hub import hf_hub_download
|
| 66 |
+
from trout import load_model_from_checkpoint, FrozenTextEncoder, sample, tokens_to_images, make_grid
|
| 67 |
+
|
| 68 |
+
ckpt = hf_hub_download("FrameXlabs/Trout", "trout-fp16.pt")
|
| 69 |
+
model, palette, meta = load_model_from_checkpoint(ckpt, device="cpu")
|
| 70 |
+
enc = FrozenTextEncoder(meta["text_encoder"], model.cfg.text_len, "cpu")
|
| 71 |
+
|
| 72 |
+
states, pooled = enc.encode(["golden crown with blue gems"] * 4)
|
| 73 |
+
toks = sample(model, states, pooled, steps=16, cfg_scale=3.0,
|
| 74 |
+
generator=torch.Generator().manual_seed(0))
|
| 75 |
+
images = tokens_to_images(toks, palette)
|
| 76 |
+
make_grid(images, ncols=4).save("crowns.png") # or try the live demo Space
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
## Files
|
| 80 |
+
|
| 81 |
+
| File | Contents |
|
| 82 |
+
|---|---|
|
| 83 |
+
| `trout-fp16.pt` | exported weights (fp16; raw/EMA chosen by validation NLL), model config, palette, text-encoder id, step |
|
| 84 |
+
| `trout.py` | single-file from-scratch implementation (model, palette, sampler, pipeline helpers) |
|
| 85 |
+
| `model_config.json`, `train_config.json` | architecture + training hyperparameters |
|
| 86 |
+
| `metrics.jsonl` | full training and validation log |
|
| 87 |
+
| `showcase.png` | sample grid (16 prompts × 4 samples, steps=16, cfg=3.0) |
|
| 88 |
+
|
| 89 |
+
## Sources & credits (the knowledge this model is built on)
|
| 90 |
+
|
| 91 |
+
- **Tessera-small** by Jagrat Patel ([TheJagStudio](https://huggingface.co/thejagstudio)) —
|
| 92 |
+
the architecture this from-scratch reimplementation follows; its open model card and demo
|
| 93 |
+
space were the design reference.
|
| 94 |
+
- **Pixel Art Atlas 1.4M** by [Angais](https://huggingface.co/Angais) (CC0) — training data.
|
| 95 |
+
- Sahoo et al., *Simple and Effective Masked Diffusion Language Models*, NeurIPS 2024 (arXiv:2406.07524).
|
| 96 |
+
- Shi et al., *Simplified and Generalized Masked Diffusion for Discrete Data*, NeurIPS 2024 (arXiv:2406.04329).
|
| 97 |
+
- Chang et al., *MaskGIT: Masked Generative Image Transformer*, CVPR 2022 (arXiv:2202.04200).
|
| 98 |
+
- Blecher et al., *Nougat* hourglass transformer (arXiv:2308.13418).
|
| 99 |
+
- Peebles & Xie, *DiT* adaLN (arXiv:2212.09748); PixArt-α adaLN-single.
|
| 100 |
+
- Darcet et al., *Vision Transformers Need Registers*, ICLR 2024.
|
| 101 |
+
- Chen, *Analog Bits* self-conditioning (arXiv:2208.04202).
|
| 102 |
+
- Jordan et al., *Muon* (arXiv:2502.16982); Moonshot, *Muon is Scalable* (arXiv:2508.02535).
|
| 103 |
+
- Ottosson, *OKLab* perceptual color space (2020).
|
| 104 |
+
|
| 105 |
+
## License
|
| 106 |
+
|
| 107 |
+
Apache-2.0 (Trout weights and code; trained on CC0 data; CLIP text tower keeps
|
| 108 |
+
its own MIT license and is not redistributed here). Architecture credit to
|
| 109 |
+
Tessera-small as noted above.
|
clip_score.json
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"clip_score_generated": 0.2284971207845956,
|
| 3 |
+
"clip_score_real": 0.25555067602545023,
|
| 4 |
+
"n": 64,
|
| 5 |
+
"note": "CLIP ViT-B/32 cos-sim, 16-step cfg=3.0 samples vs real val sprites"
|
| 6 |
+
}
|
metrics.jsonl
ADDED
|
@@ -0,0 +1,250 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"step": 10, "loss": 6.200986862182617, "nll": 6.200520038604736, "acc": 0.3830431401729584, "lr": 2.7499999999999998e-05, "sec_per_step": 6.398, "gnorm": 6.468178749084473}
|
| 2 |
+
{"step": 20, "loss": 6.085262298583984, "nll": 6.084070682525635, "acc": 0.34061792492866516, "lr": 5.25e-05, "sec_per_step": 6.11, "gnorm": 6.487669944763184}
|
| 3 |
+
{"step": 30, "loss": 5.7887115478515625, "nll": 5.784521579742432, "acc": 0.34792807698249817, "lr": 7.75e-05, "sec_per_step": 6.32, "gnorm": 8.065339088439941}
|
| 4 |
+
{"step": 40, "loss": 5.292694091796875, "nll": 5.2869391441345215, "acc": 0.3832145631313324, "lr": 0.0001025, "sec_per_step": 6.314, "gnorm": 10.66641902923584}
|
| 5 |
+
{"step": 50, "loss": 4.211287021636963, "nll": 4.189800262451172, "acc": 0.41723355650901794, "lr": 0.0001275, "sec_per_step": 6.067, "gnorm": 13.290396690368652}
|
| 6 |
+
{"step": 60, "loss": 4.039217472076416, "nll": 4.010549545288086, "acc": 0.3768569231033325, "lr": 0.0001525, "sec_per_step": 5.947, "gnorm": 4.629364013671875}
|
| 7 |
+
{"step": 70, "loss": 4.080512046813965, "nll": 4.0558061599731445, "acc": 0.3692913353443146, "lr": 0.00017750000000000003, "sec_per_step": 5.796, "gnorm": 4.4744062423706055}
|
| 8 |
+
{"step": 80, "loss": 3.756932497024536, "nll": 3.736483097076416, "acc": 0.422374427318573, "lr": 0.00020250000000000002, "sec_per_step": 5.894, "gnorm": 3.615117311477661}
|
| 9 |
+
{"step": 90, "loss": 4.03838586807251, "nll": 4.009660243988037, "acc": 0.35103243589401245, "lr": 0.0002275, "sec_per_step": 5.949, "gnorm": 9.545635223388672}
|
| 10 |
+
{"step": 100, "loss": 3.926302671432495, "nll": 3.896104335784912, "acc": 0.35153844952583313, "lr": 0.00025, "sec_per_step": 5.966, "gnorm": 5.883094787597656}
|
| 11 |
+
{"step": 110, "loss": 3.689901828765869, "nll": 3.681730031967163, "acc": 0.44033873081207275, "lr": 0.00025, "sec_per_step": 6.052, "gnorm": 5.032026767730713}
|
| 12 |
+
{"step": 120, "loss": 2.9494550228118896, "nll": 2.9360105991363525, "acc": 0.5653495192527771, "lr": 0.00025, "sec_per_step": 6.461, "gnorm": 3.8694350719451904}
|
| 13 |
+
{"step": 130, "loss": 3.870502233505249, "nll": 3.849687337875366, "acc": 0.37245696783065796, "lr": 0.00025, "sec_per_step": 6.388, "gnorm": 3.324218273162842}
|
| 14 |
+
{"step": 140, "loss": 3.858419418334961, "nll": 3.8451130390167236, "acc": 0.3631415367126465, "lr": 0.00025, "sec_per_step": 5.897, "gnorm": 2.8710551261901855}
|
| 15 |
+
{"step": 150, "loss": 3.7835071086883545, "nll": 3.757303237915039, "acc": 0.3560490012168884, "lr": 0.00025, "sec_per_step": 6.176, "gnorm": 3.1728737354278564}
|
| 16 |
+
{"step": 160, "loss": 3.879119873046875, "nll": 3.8416779041290283, "acc": 0.3188854455947876, "lr": 0.00025, "sec_per_step": 6.097, "gnorm": 6.751031398773193}
|
| 17 |
+
{"step": 170, "loss": 3.833686351776123, "nll": 3.8111066818237305, "acc": 0.3591117858886719, "lr": 0.00025, "sec_per_step": 6.261, "gnorm": 3.194800615310669}
|
| 18 |
+
{"step": 180, "loss": 3.9325289726257324, "nll": 3.9077517986297607, "acc": 0.32421875, "lr": 0.00025, "sec_per_step": 6.429, "gnorm": 5.764603614807129}
|
| 19 |
+
{"step": 190, "loss": 3.7423737049102783, "nll": 3.7159183025360107, "acc": 0.3640483319759369, "lr": 0.00025, "sec_per_step": 5.951, "gnorm": 3.4916608333587646}
|
| 20 |
+
{"step": 200, "loss": 3.4003562927246094, "nll": 3.387075662612915, "acc": 0.44687265157699585, "lr": 0.00025, "sec_per_step": 6.264, "gnorm": 3.4254374504089355}
|
| 21 |
+
{"step": 210, "loss": 3.4553167819976807, "nll": 3.4268157482147217, "acc": 0.3961038887500763, "lr": 0.00025, "sec_per_step": 6.379, "gnorm": 3.3052334785461426}
|
| 22 |
+
{"step": 220, "loss": 3.253338098526001, "nll": 3.226210594177246, "acc": 0.4332015812397003, "lr": 0.00025, "sec_per_step": 5.931, "gnorm": 4.595219135284424}
|
| 23 |
+
{"step": 230, "loss": 3.795321226119995, "nll": 3.771483898162842, "acc": 0.3651289939880371, "lr": 0.00025, "sec_per_step": 6.087, "gnorm": 5.327313423156738}
|
| 24 |
+
{"step": 240, "loss": 3.2495949268341064, "nll": 3.2198522090911865, "acc": 0.4190329909324646, "lr": 0.00025, "sec_per_step": 6.114, "gnorm": 5.86907958984375}
|
| 25 |
+
{"step": 250, "loss": 3.48093581199646, "nll": 3.4433395862579346, "acc": 0.38876405358314514, "lr": 0.00025, "sec_per_step": 6.339, "gnorm": 3.7719953060150146}
|
| 26 |
+
{"step": 250, "ema_val_nll": 5.711481094360352, "ema_val_acc": 0.399124912917614, "raw_val_nll": 3.4989233016967773, "raw_val_acc": 0.396411269903183}
|
| 27 |
+
{"step": 260, "loss": 3.0223655700683594, "nll": 2.9900503158569336, "acc": 0.4696296155452728, "lr": 0.00025, "sec_per_step": 6.07, "gnorm": 3.129960298538208}
|
| 28 |
+
{"step": 270, "loss": 3.310396432876587, "nll": 3.2859034538269043, "acc": 0.4375951290130615, "lr": 0.00025, "sec_per_step": 5.982, "gnorm": 6.224829196929932}
|
| 29 |
+
{"step": 280, "loss": 3.3965539932250977, "nll": 3.3703815937042236, "acc": 0.444947212934494, "lr": 0.00025, "sec_per_step": 6.051, "gnorm": 5.335007667541504}
|
| 30 |
+
{"step": 290, "loss": 3.239400863647461, "nll": 3.199361801147461, "acc": 0.4470762312412262, "lr": 0.00025, "sec_per_step": 5.781, "gnorm": 5.158711910247803}
|
| 31 |
+
{"step": 300, "loss": 2.912628173828125, "nll": 2.868650197982788, "acc": 0.489393949508667, "lr": 0.00025, "sec_per_step": 6.085, "gnorm": 3.7020480632781982}
|
| 32 |
+
{"step": 310, "loss": 3.418224573135376, "nll": 3.376781463623047, "acc": 0.38988327980041504, "lr": 0.00025, "sec_per_step": 5.879, "gnorm": 4.393625736236572}
|
| 33 |
+
{"step": 320, "loss": 3.3609254360198975, "nll": 3.3164608478546143, "acc": 0.38576778769493103, "lr": 0.00025, "sec_per_step": 5.719, "gnorm": 4.6255059242248535}
|
| 34 |
+
{"step": 330, "loss": 2.9795143604278564, "nll": 2.921900749206543, "acc": 0.47132351994514465, "lr": 0.00025, "sec_per_step": 6.111, "gnorm": 3.5180745124816895}
|
| 35 |
+
{"step": 340, "loss": 3.5651557445526123, "nll": 3.5129222869873047, "acc": 0.3803863227367401, "lr": 0.00025, "sec_per_step": 6.429, "gnorm": 6.238819122314453}
|
| 36 |
+
{"step": 350, "loss": 3.0168468952178955, "nll": 2.974374771118164, "acc": 0.46493300795555115, "lr": 0.00025, "sec_per_step": 6.299, "gnorm": 5.946975231170654}
|
| 37 |
+
{"step": 360, "loss": 2.665656566619873, "nll": 2.6218910217285156, "acc": 0.5297216176986694, "lr": 0.00025, "sec_per_step": 6.026, "gnorm": 4.707192420959473}
|
| 38 |
+
{"step": 370, "loss": 3.0414562225341797, "nll": 2.9841203689575195, "acc": 0.43898695707321167, "lr": 0.00025, "sec_per_step": 6.234, "gnorm": 5.299668788909912}
|
| 39 |
+
{"step": 380, "loss": 3.2714550495147705, "nll": 3.2110435962677, "acc": 0.41442155838012695, "lr": 0.00025, "sec_per_step": 6.164, "gnorm": 6.51394510269165}
|
| 40 |
+
{"step": 390, "loss": 3.1979150772094727, "nll": 3.139848232269287, "acc": 0.43052393198013306, "lr": 0.00025, "sec_per_step": 6.121, "gnorm": 4.636814594268799}
|
| 41 |
+
{"step": 400, "loss": 2.9832358360290527, "nll": 2.9123218059539795, "acc": 0.4618380069732666, "lr": 0.00025, "sec_per_step": 5.989, "gnorm": 4.325001239776611}
|
| 42 |
+
{"step": 410, "loss": 3.38590407371521, "nll": 3.2992231845855713, "acc": 0.4131641685962677, "lr": 0.00025, "sec_per_step": 5.876, "gnorm": 5.986806392669678}
|
| 43 |
+
{"step": 420, "loss": 2.9860775470733643, "nll": 2.897034168243408, "acc": 0.4553093910217285, "lr": 0.00025, "sec_per_step": 7.482, "gnorm": 4.825826644897461}
|
| 44 |
+
{"step": 430, "loss": 2.791691780090332, "nll": 2.719435214996338, "acc": 0.485871285200119, "lr": 0.00025, "sec_per_step": 6.29, "gnorm": 4.470316410064697}
|
| 45 |
+
{"step": 440, "loss": 3.3054935932159424, "nll": 3.228121042251587, "acc": 0.3485757112503052, "lr": 0.00025, "sec_per_step": 6.228, "gnorm": 5.163248062133789}
|
| 46 |
+
{"step": 450, "loss": 2.6202685832977295, "nll": 2.535158395767212, "acc": 0.5271317958831787, "lr": 0.00025, "sec_per_step": 6.05, "gnorm": 4.609095096588135}
|
| 47 |
+
{"step": 460, "loss": 2.8274412155151367, "nll": 2.754194736480713, "acc": 0.5169230699539185, "lr": 0.00025, "sec_per_step": 6.25, "gnorm": 5.817952632904053}
|
| 48 |
+
{"step": 470, "loss": 3.3049192428588867, "nll": 3.2267391681671143, "acc": 0.3978576958179474, "lr": 0.00025, "sec_per_step": 5.822, "gnorm": 5.276814937591553}
|
| 49 |
+
{"step": 480, "loss": 2.734286069869995, "nll": 2.6532654762268066, "acc": 0.48876839876174927, "lr": 0.00025, "sec_per_step": 6.129, "gnorm": 4.163163661956787}
|
| 50 |
+
{"step": 490, "loss": 3.161760091781616, "nll": 3.0504703521728516, "acc": 0.41040000319480896, "lr": 0.00025, "sec_per_step": 5.932, "gnorm": 5.252750873565674}
|
| 51 |
+
{"step": 500, "loss": 2.9351284503936768, "nll": 2.848027467727661, "acc": 0.47575056552886963, "lr": 0.00025, "sec_per_step": 5.878, "gnorm": 6.60810661315918}
|
| 52 |
+
{"step": 500, "ema_val_nll": 4.757295727729797, "ema_val_acc": 0.3992054760456085, "raw_val_nll": 2.869568943977356, "raw_val_acc": 0.4603290930390358}
|
| 53 |
+
{"step": 510, "loss": 3.00405216217041, "nll": 2.918951988220215, "acc": 0.45122918486595154, "lr": 0.00025, "sec_per_step": 6.425, "gnorm": 5.543540000915527}
|
| 54 |
+
{"step": 520, "loss": 3.0181691646575928, "nll": 2.906599760055542, "acc": 0.37817639112472534, "lr": 0.00025, "sec_per_step": 6.259, "gnorm": 5.1127519607543945}
|
| 55 |
+
{"step": 530, "loss": 2.5579750537872314, "nll": 2.4807825088500977, "acc": 0.507876992225647, "lr": 0.00025, "sec_per_step": 6.225, "gnorm": 4.181877136230469}
|
| 56 |
+
{"step": 540, "loss": 2.5971875190734863, "nll": 2.504199743270874, "acc": 0.5093914270401001, "lr": 0.00025, "sec_per_step": 6.481, "gnorm": 4.407857418060303}
|
| 57 |
+
{"step": 550, "loss": 2.922457456588745, "nll": 2.835352897644043, "acc": 0.447408527135849, "lr": 0.00025, "sec_per_step": 6.384, "gnorm": 5.971846580505371}
|
| 58 |
+
{"step": 560, "loss": 3.397606611251831, "nll": 3.27708101272583, "acc": 0.3853580057621002, "lr": 0.00025, "sec_per_step": 6.407, "gnorm": 7.557126045227051}
|
| 59 |
+
{"step": 570, "loss": 2.498903512954712, "nll": 2.415919303894043, "acc": 0.5505279302597046, "lr": 0.00025, "sec_per_step": 6.452, "gnorm": 3.9383254051208496}
|
| 60 |
+
{"step": 580, "loss": 3.0961005687713623, "nll": 3.017442226409912, "acc": 0.4386100471019745, "lr": 0.00025, "sec_per_step": 6.23, "gnorm": 5.793766021728516}
|
| 61 |
+
{"step": 590, "loss": 3.157524347305298, "nll": 3.046320915222168, "acc": 0.40030792355537415, "lr": 0.00025, "sec_per_step": 5.893, "gnorm": 5.688717842102051}
|
| 62 |
+
{"step": 600, "loss": 2.3616044521331787, "nll": 2.268296957015991, "acc": 0.5714285969734192, "lr": 0.00025, "sec_per_step": 6.051, "gnorm": 6.217453479766846}
|
| 63 |
+
{"step": 610, "loss": 2.9710419178009033, "nll": 2.88120174407959, "acc": 0.4688957929611206, "lr": 0.00025, "sec_per_step": 6.14, "gnorm": 6.2860870361328125}
|
| 64 |
+
{"step": 620, "loss": 2.986084461212158, "nll": 2.878966808319092, "acc": 0.3944530189037323, "lr": 0.00025, "sec_per_step": 6.3, "gnorm": 6.300302505493164}
|
| 65 |
+
{"step": 630, "loss": 2.5614638328552246, "nll": 2.4820749759674072, "acc": 0.5108024477958679, "lr": 0.00025, "sec_per_step": 6.298, "gnorm": 5.761435508728027}
|
| 66 |
+
{"step": 640, "loss": 3.084238052368164, "nll": 2.9953041076660156, "acc": 0.45324283838272095, "lr": 0.00025, "sec_per_step": 6.518, "gnorm": 5.899290084838867}
|
| 67 |
+
{"step": 650, "loss": 3.2054970264434814, "nll": 3.1445343494415283, "acc": 0.3931357264518738, "lr": 0.00025, "sec_per_step": 6.168, "gnorm": 6.1849799156188965}
|
| 68 |
+
{"step": 660, "loss": 2.0578274726867676, "nll": 1.9782261848449707, "acc": 0.6058282256126404, "lr": 0.00025, "sec_per_step": 6.434, "gnorm": 4.426883220672607}
|
| 69 |
+
{"step": 670, "loss": 2.946725606918335, "nll": 2.8325183391571045, "acc": 0.4192807972431183, "lr": 0.00025, "sec_per_step": 6.409, "gnorm": 4.88676118850708}
|
| 70 |
+
{"step": 680, "loss": 2.393976926803589, "nll": 2.304959535598755, "acc": 0.5380557775497437, "lr": 0.00025, "sec_per_step": 5.937, "gnorm": 3.898456573486328}
|
| 71 |
+
{"step": 690, "loss": 3.166815757751465, "nll": 3.0621657371520996, "acc": 0.39906832575798035, "lr": 0.00025, "sec_per_step": 6.558, "gnorm": 5.971321105957031}
|
| 72 |
+
{"step": 700, "loss": 2.7465415000915527, "nll": 2.62947940826416, "acc": 0.4311145544052124, "lr": 0.00025, "sec_per_step": 6.339, "gnorm": 5.578491687774658}
|
| 73 |
+
{"step": 710, "loss": 2.4950203895568848, "nll": 2.4207072257995605, "acc": 0.5385220050811768, "lr": 0.00025, "sec_per_step": 6.264, "gnorm": 3.6538729667663574}
|
| 74 |
+
{"step": 720, "loss": 2.5446856021881104, "nll": 2.4633123874664307, "acc": 0.5325264930725098, "lr": 0.00025, "sec_per_step": 6.018, "gnorm": 4.902586936950684}
|
| 75 |
+
{"step": 730, "loss": 2.8988325595855713, "nll": 2.799029588699341, "acc": 0.4440919756889343, "lr": 0.00025, "sec_per_step": 6.047, "gnorm": 5.226912975311279}
|
| 76 |
+
{"step": 740, "loss": 2.6665353775024414, "nll": 2.5732717514038086, "acc": 0.5003952383995056, "lr": 0.00025, "sec_per_step": 6.122, "gnorm": 4.964861869812012}
|
| 77 |
+
{"step": 750, "loss": 3.175828695297241, "nll": 3.05306077003479, "acc": 0.40813136100769043, "lr": 0.00025, "sec_per_step": 6.156, "gnorm": 5.22468900680542}
|
| 78 |
+
{"step": 750, "ema_val_nll": 4.113529145717621, "ema_val_acc": 0.40666238963603973, "raw_val_nll": 2.643985390663147, "raw_val_acc": 0.4819430932402611}
|
| 79 |
+
{"step": 760, "loss": 3.002694845199585, "nll": 2.898425340652466, "acc": 0.4198295772075653, "lr": 0.00025, "sec_per_step": 7.212, "gnorm": 4.94859504699707}
|
| 80 |
+
{"step": 770, "loss": 2.7087178230285645, "nll": 2.623079538345337, "acc": 0.4950343668460846, "lr": 0.00025, "sec_per_step": 6.125, "gnorm": 4.606741905212402}
|
| 81 |
+
{"step": 780, "loss": 2.5839145183563232, "nll": 2.4882335662841797, "acc": 0.518324613571167, "lr": 0.00025, "sec_per_step": 5.895, "gnorm": 3.720069408416748}
|
| 82 |
+
{"step": 790, "loss": 2.465515613555908, "nll": 2.3439218997955322, "acc": 0.5054263472557068, "lr": 0.00025, "sec_per_step": 6.3, "gnorm": 4.490382671356201}
|
| 83 |
+
{"step": 800, "loss": 2.7122833728790283, "nll": 2.60189151763916, "acc": 0.4909803867340088, "lr": 0.00025, "sec_per_step": 6.147, "gnorm": 4.550020694732666}
|
| 84 |
+
{"step": 810, "loss": 2.675985097885132, "nll": 2.5788652896881104, "acc": 0.5086477994918823, "lr": 0.00025, "sec_per_step": 6.241, "gnorm": 4.516785621643066}
|
| 85 |
+
{"step": 820, "loss": 2.798062562942505, "nll": 2.6930816173553467, "acc": 0.48151060938835144, "lr": 0.00025, "sec_per_step": 6.345, "gnorm": 5.3493971824646}
|
| 86 |
+
{"step": 830, "loss": 2.7418172359466553, "nll": 2.628443956375122, "acc": 0.4716981053352356, "lr": 0.00025, "sec_per_step": 6.246, "gnorm": 5.233100891113281}
|
| 87 |
+
{"step": 840, "loss": 2.781986951828003, "nll": 2.6775472164154053, "acc": 0.44110479950904846, "lr": 0.00025, "sec_per_step": 5.871, "gnorm": 5.794379234313965}
|
| 88 |
+
{"step": 850, "loss": 3.0547821521759033, "nll": 2.9467966556549072, "acc": 0.4636785089969635, "lr": 0.00025, "sec_per_step": 6.361, "gnorm": 6.320489883422852}
|
| 89 |
+
{"step": 860, "loss": 2.9305124282836914, "nll": 2.7980825901031494, "acc": 0.4084821343421936, "lr": 0.00025, "sec_per_step": 6.358, "gnorm": 4.6363043785095215}
|
| 90 |
+
{"step": 870, "loss": 2.394909620285034, "nll": 2.2805004119873047, "acc": 0.5445935130119324, "lr": 0.00025, "sec_per_step": 6.242, "gnorm": 4.042086601257324}
|
| 91 |
+
{"step": 880, "loss": 2.4840915203094482, "nll": 2.368605375289917, "acc": 0.4941084086894989, "lr": 0.00025, "sec_per_step": 6.399, "gnorm": 5.068140506744385}
|
| 92 |
+
{"step": 890, "loss": 2.506272792816162, "nll": 2.4257686138153076, "acc": 0.5312265157699585, "lr": 0.00025, "sec_per_step": 6.348, "gnorm": 4.804419040679932}
|
| 93 |
+
{"step": 900, "loss": 2.5496621131896973, "nll": 2.4423022270202637, "acc": 0.5185185074806213, "lr": 0.00025, "sec_per_step": 6.004, "gnorm": 5.5875349044799805}
|
| 94 |
+
{"step": 910, "loss": 2.7930030822753906, "nll": 2.7017321586608887, "acc": 0.4925606846809387, "lr": 0.00025, "sec_per_step": 6.159, "gnorm": 4.833495616912842}
|
| 95 |
+
{"step": 920, "loss": 2.495145797729492, "nll": 2.3928442001342773, "acc": 0.5393258333206177, "lr": 0.00025, "sec_per_step": 5.959, "gnorm": 3.727670192718506}
|
| 96 |
+
{"step": 930, "loss": 3.3178422451019287, "nll": 3.1799910068511963, "acc": 0.4032633900642395, "lr": 0.00025, "sec_per_step": 6.723, "gnorm": 6.969775676727295}
|
| 97 |
+
{"step": 940, "loss": 3.0164005756378174, "nll": 2.8922157287597656, "acc": 0.3935331106185913, "lr": 0.00025, "sec_per_step": 5.885, "gnorm": 5.34336519241333}
|
| 98 |
+
{"step": 950, "loss": 2.8784775733947754, "nll": 2.7882721424102783, "acc": 0.43485212326049805, "lr": 0.00025, "sec_per_step": 6.02, "gnorm": 5.112490653991699}
|
| 99 |
+
{"step": 960, "loss": 2.7010087966918945, "nll": 2.5866334438323975, "acc": 0.4752163589000702, "lr": 0.00025, "sec_per_step": 6.199, "gnorm": 5.5563645362854}
|
| 100 |
+
{"step": 970, "loss": 2.844118595123291, "nll": 2.7358219623565674, "acc": 0.46790409088134766, "lr": 0.00025, "sec_per_step": 5.841, "gnorm": 5.387760162353516}
|
| 101 |
+
{"step": 980, "loss": 2.6960089206695557, "nll": 2.5920751094818115, "acc": 0.48721805214881897, "lr": 0.00025, "sec_per_step": 6.272, "gnorm": 4.827456951141357}
|
| 102 |
+
{"step": 990, "loss": 2.5970065593719482, "nll": 2.484112024307251, "acc": 0.48900681734085083, "lr": 0.00025, "sec_per_step": 6.287, "gnorm": 4.624963760375977}
|
| 103 |
+
{"step": 1000, "loss": 2.3499960899353027, "nll": 2.2429587841033936, "acc": 0.5277127027511597, "lr": 0.00025, "sec_per_step": 6.107, "gnorm": 4.778969764709473}
|
| 104 |
+
{"step": 1000, "ema_val_nll": 3.6114672422409058, "ema_val_acc": 0.4406821057200432, "raw_val_nll": 2.666260600090027, "raw_val_acc": 0.4742664396762848}
|
| 105 |
+
{"step": 1010, "loss": 2.7779881954193115, "nll": 2.6650407314300537, "acc": 0.4598715901374817, "lr": 0.00025, "sec_per_step": 8.415, "gnorm": 5.046490669250488}
|
| 106 |
+
{"step": 1020, "loss": 2.4154486656188965, "nll": 2.325665235519409, "acc": 0.539177656173706, "lr": 0.00025, "sec_per_step": 6.235, "gnorm": 4.5242228507995605}
|
| 107 |
+
{"step": 1030, "loss": 2.5797035694122314, "nll": 2.485356330871582, "acc": 0.5289193391799927, "lr": 0.00025, "sec_per_step": 6.513, "gnorm": 4.961014270782471}
|
| 108 |
+
{"step": 1040, "loss": 2.6177289485931396, "nll": 2.5035901069641113, "acc": 0.5226011276245117, "lr": 0.00025, "sec_per_step": 6.029, "gnorm": 4.298247337341309}
|
| 109 |
+
{"step": 1050, "loss": 2.256521701812744, "nll": 2.1624436378479004, "acc": 0.6048083901405334, "lr": 0.00025, "sec_per_step": 6.097, "gnorm": 3.6198694705963135}
|
| 110 |
+
{"step": 1060, "loss": 2.7656686305999756, "nll": 2.6518473625183105, "acc": 0.4772382378578186, "lr": 0.00025, "sec_per_step": 6.203, "gnorm": 4.267014980316162}
|
| 111 |
+
{"step": 1070, "loss": 2.9220519065856934, "nll": 2.8266348838806152, "acc": 0.45517775416374207, "lr": 0.00025, "sec_per_step": 6.151, "gnorm": 3.8635213375091553}
|
| 112 |
+
{"step": 1080, "loss": 2.689377546310425, "nll": 2.5772922039031982, "acc": 0.4730856716632843, "lr": 0.00025, "sec_per_step": 5.973, "gnorm": 4.459869861602783}
|
| 113 |
+
{"step": 1090, "loss": 2.34483003616333, "nll": 2.2445740699768066, "acc": 0.545592725276947, "lr": 0.00025, "sec_per_step": 6.188, "gnorm": 4.070357322692871}
|
| 114 |
+
{"step": 1100, "loss": 2.853635311126709, "nll": 2.735430955886841, "acc": 0.44164037704467773, "lr": 0.00025, "sec_per_step": 6.181, "gnorm": 4.984803676605225}
|
| 115 |
+
{"step": 1110, "loss": 2.3250839710235596, "nll": 2.2021560668945312, "acc": 0.5528771281242371, "lr": 0.00025, "sec_per_step": 5.84, "gnorm": 4.130050182342529}
|
| 116 |
+
{"step": 1120, "loss": 2.395454168319702, "nll": 2.250638484954834, "acc": 0.5038520693778992, "lr": 0.00025, "sec_per_step": 6.079, "gnorm": 3.9396791458129883}
|
| 117 |
+
{"step": 1130, "loss": 3.1179373264312744, "nll": 3.0062990188598633, "acc": 0.41455939412117004, "lr": 0.00025, "sec_per_step": 6.23, "gnorm": 5.461380481719971}
|
| 118 |
+
{"step": 1140, "loss": 2.6154885292053223, "nll": 2.4895212650299072, "acc": 0.5114678740501404, "lr": 0.00025, "sec_per_step": 6.159, "gnorm": 4.283007621765137}
|
| 119 |
+
{"step": 1150, "loss": 2.4366607666015625, "nll": 2.3064374923706055, "acc": 0.5144193172454834, "lr": 0.00025, "sec_per_step": 6.121, "gnorm": 5.6990227699279785}
|
| 120 |
+
{"step": 1160, "loss": 2.505553722381592, "nll": 2.39853835105896, "acc": 0.5068492889404297, "lr": 0.00025, "sec_per_step": 6.078, "gnorm": 4.561639785766602}
|
| 121 |
+
{"step": 1170, "loss": 2.403393030166626, "nll": 2.287985324859619, "acc": 0.5042080879211426, "lr": 0.00025, "sec_per_step": 6.208, "gnorm": 4.251239776611328}
|
| 122 |
+
{"step": 1180, "loss": 2.3850104808807373, "nll": 2.2874596118927, "acc": 0.5426356792449951, "lr": 0.00025, "sec_per_step": 6.152, "gnorm": 5.599863052368164}
|
| 123 |
+
{"step": 1190, "loss": 2.782942295074463, "nll": 2.6919305324554443, "acc": 0.4758364260196686, "lr": 0.00025, "sec_per_step": 6.28, "gnorm": 5.388584136962891}
|
| 124 |
+
{"step": 1200, "loss": 2.4973490238189697, "nll": 2.3614211082458496, "acc": 0.5364543199539185, "lr": 0.00025, "sec_per_step": 6.34, "gnorm": 4.672701835632324}
|
| 125 |
+
{"step": 1210, "loss": 2.744319200515747, "nll": 2.60308575630188, "acc": 0.4330826997756958, "lr": 0.00025, "sec_per_step": 6.317, "gnorm": 5.39306116104126}
|
| 126 |
+
{"step": 1220, "loss": 2.6432716846466064, "nll": 2.517979621887207, "acc": 0.49802058935165405, "lr": 0.00025, "sec_per_step": 6.218, "gnorm": 4.594691753387451}
|
| 127 |
+
{"step": 1230, "loss": 2.746180534362793, "nll": 2.6465775966644287, "acc": 0.4908953011035919, "lr": 0.00025, "sec_per_step": 6.299, "gnorm": 5.096075534820557}
|
| 128 |
+
{"step": 1240, "loss": 2.739842653274536, "nll": 2.6066157817840576, "acc": 0.46555638313293457, "lr": 0.00025, "sec_per_step": 5.874, "gnorm": 5.038521766662598}
|
| 129 |
+
{"step": 1250, "loss": 2.1190388202667236, "nll": 2.018712043762207, "acc": 0.5824345350265503, "lr": 0.00025, "sec_per_step": 6.281, "gnorm": 4.792925834655762}
|
| 130 |
+
{"step": 1250, "ema_val_nll": 3.211992383003235, "ema_val_acc": 0.4749576523900032, "raw_val_nll": 2.4866853952407837, "raw_val_acc": 0.49806810915470123}
|
| 131 |
+
{"step": 1260, "loss": 3.0065832138061523, "nll": 2.8991711139678955, "acc": 0.4409448802471161, "lr": 0.00025, "sec_per_step": 7.317, "gnorm": 4.6214680671691895}
|
| 132 |
+
{"step": 1270, "loss": 2.6068108081817627, "nll": 2.4872145652770996, "acc": 0.46268656849861145, "lr": 0.00025, "sec_per_step": 6.016, "gnorm": 5.460693359375}
|
| 133 |
+
{"step": 1280, "loss": 2.9487662315368652, "nll": 2.828066825866699, "acc": 0.43990573287010193, "lr": 0.00025, "sec_per_step": 6.076, "gnorm": 4.329925060272217}
|
| 134 |
+
{"step": 1290, "loss": 2.550868272781372, "nll": 2.4090263843536377, "acc": 0.4996282458305359, "lr": 0.00025, "sec_per_step": 6.159, "gnorm": 4.255090236663818}
|
| 135 |
+
{"step": 1300, "loss": 2.4514379501342773, "nll": 2.347454071044922, "acc": 0.5015174746513367, "lr": 0.00025, "sec_per_step": 6.068, "gnorm": 5.65173864364624}
|
| 136 |
+
{"step": 1310, "loss": 2.480661153793335, "nll": 2.361290216445923, "acc": 0.4915003776550293, "lr": 0.00025, "sec_per_step": 6.154, "gnorm": 4.723772048950195}
|
| 137 |
+
{"step": 1320, "loss": 2.580076217651367, "nll": 2.4763495922088623, "acc": 0.5211701393127441, "lr": 0.00025, "sec_per_step": 6.095, "gnorm": 4.933023452758789}
|
| 138 |
+
{"step": 1330, "loss": 2.862834930419922, "nll": 2.7394986152648926, "acc": 0.4516380727291107, "lr": 0.00025, "sec_per_step": 6.372, "gnorm": 4.52992582321167}
|
| 139 |
+
{"step": 1340, "loss": 2.0927963256835938, "nll": 2.003164291381836, "acc": 0.5794117450714111, "lr": 0.00025, "sec_per_step": 6.307, "gnorm": 3.553281784057617}
|
| 140 |
+
{"step": 1350, "loss": 2.3785457611083984, "nll": 2.26143479347229, "acc": 0.5496987700462341, "lr": 0.00025, "sec_per_step": 6.146, "gnorm": 4.909267902374268}
|
| 141 |
+
{"step": 1360, "loss": 2.4946482181549072, "nll": 2.3889689445495605, "acc": 0.5175096988677979, "lr": 0.00025, "sec_per_step": 6.075, "gnorm": 4.5072340965271}
|
| 142 |
+
{"step": 1370, "loss": 2.6881964206695557, "nll": 2.546342134475708, "acc": 0.4414556920528412, "lr": 0.00025, "sec_per_step": 6.424, "gnorm": 6.146117687225342}
|
| 143 |
+
{"step": 1380, "loss": 2.326253652572632, "nll": 2.199773073196411, "acc": 0.5414527058601379, "lr": 0.00025, "sec_per_step": 6.49, "gnorm": 4.503512382507324}
|
| 144 |
+
{"step": 1390, "loss": 2.906200647354126, "nll": 2.7529072761535645, "acc": 0.46252837777137756, "lr": 0.00025, "sec_per_step": 6.226, "gnorm": 4.7287211418151855}
|
| 145 |
+
{"step": 1400, "loss": 2.4027457237243652, "nll": 2.2472972869873047, "acc": 0.5239937901496887, "lr": 0.00025, "sec_per_step": 6.423, "gnorm": 5.4805474281311035}
|
| 146 |
+
{"step": 1410, "loss": 2.4080331325531006, "nll": 2.2458603382110596, "acc": 0.46352583169937134, "lr": 0.00025, "sec_per_step": 6.33, "gnorm": 5.683468818664551}
|
| 147 |
+
{"step": 1420, "loss": 2.3063762187957764, "nll": 2.1755499839782715, "acc": 0.5429864525794983, "lr": 0.00025, "sec_per_step": 6.386, "gnorm": 5.139619827270508}
|
| 148 |
+
{"step": 1430, "loss": 2.491290330886841, "nll": 2.3421034812927246, "acc": 0.4966088831424713, "lr": 0.00025, "sec_per_step": 6.315, "gnorm": 4.2525129318237305}
|
| 149 |
+
{"step": 1440, "loss": 2.152965784072876, "nll": 2.0535247325897217, "acc": 0.5964912176132202, "lr": 0.00025, "sec_per_step": 6.17, "gnorm": 3.590088129043579}
|
| 150 |
+
{"step": 1450, "loss": 2.95025897026062, "nll": 2.827056646347046, "acc": 0.414219468832016, "lr": 0.00025, "sec_per_step": 6.41, "gnorm": 5.303553581237793}
|
| 151 |
+
{"step": 1460, "loss": 2.8419055938720703, "nll": 2.7361035346984863, "acc": 0.4551831781864166, "lr": 0.00025, "sec_per_step": 6.336, "gnorm": 5.055415153503418}
|
| 152 |
+
{"step": 1470, "loss": 2.92997407913208, "nll": 2.8064730167388916, "acc": 0.4229337275028229, "lr": 0.00025, "sec_per_step": 6.254, "gnorm": 5.053462982177734}
|
| 153 |
+
{"step": 1480, "loss": 2.549100399017334, "nll": 2.419140338897705, "acc": 0.511904776096344, "lr": 0.00025, "sec_per_step": 6.215, "gnorm": 4.3006205558776855}
|
| 154 |
+
{"step": 1490, "loss": 2.893587112426758, "nll": 2.7813587188720703, "acc": 0.4875583350658417, "lr": 0.00025, "sec_per_step": 6.132, "gnorm": 5.892554759979248}
|
| 155 |
+
{"step": 1500, "loss": 2.6392624378204346, "nll": 2.5167980194091797, "acc": 0.48127925395965576, "lr": 0.00025, "sec_per_step": 6.339, "gnorm": 5.114858150482178}
|
| 156 |
+
{"step": 1500, "ema_val_nll": 2.950586438179016, "ema_val_acc": 0.4897710531949997, "raw_val_nll": 2.484873652458191, "raw_val_acc": 0.5019320175051689}
|
| 157 |
+
{"step": 1510, "loss": 2.5321125984191895, "nll": 2.433446168899536, "acc": 0.5450918078422546, "lr": 0.00025, "sec_per_step": 8.25, "gnorm": 3.91320538520813}
|
| 158 |
+
{"step": 1520, "loss": 2.616060972213745, "nll": 2.5097177028656006, "acc": 0.5171624422073364, "lr": 0.00025, "sec_per_step": 6.419, "gnorm": 4.143905162811279}
|
| 159 |
+
{"step": 1530, "loss": 3.1057634353637695, "nll": 2.9970128536224365, "acc": 0.4374045729637146, "lr": 0.00025, "sec_per_step": 6.011, "gnorm": 6.740103721618652}
|
| 160 |
+
{"step": 1540, "loss": 2.6444315910339355, "nll": 2.5181338787078857, "acc": 0.5045385956764221, "lr": 0.00025, "sec_per_step": 6.482, "gnorm": 4.590264320373535}
|
| 161 |
+
{"step": 1550, "loss": 2.076510190963745, "nll": 1.9581280946731567, "acc": 0.5774980187416077, "lr": 0.00025, "sec_per_step": 6.308, "gnorm": 4.635385990142822}
|
| 162 |
+
{"step": 1560, "loss": 2.5528955459594727, "nll": 2.4293298721313477, "acc": 0.5011746287345886, "lr": 0.00025, "sec_per_step": 6.263, "gnorm": 4.992447853088379}
|
| 163 |
+
{"step": 1570, "loss": 2.8153975009918213, "nll": 2.6946966648101807, "acc": 0.44435974955558777, "lr": 0.00025, "sec_per_step": 6.228, "gnorm": 6.261963844299316}
|
| 164 |
+
{"step": 1580, "loss": 2.534857749938965, "nll": 2.419583797454834, "acc": 0.5106545090675354, "lr": 0.00025, "sec_per_step": 6.673, "gnorm": 5.6587347984313965}
|
| 165 |
+
{"step": 1590, "loss": 2.804851531982422, "nll": 2.683345079421997, "acc": 0.48448145389556885, "lr": 0.00025, "sec_per_step": 6.528, "gnorm": 4.932685375213623}
|
| 166 |
+
{"step": 1600, "loss": 2.805572032928467, "nll": 2.6489508152008057, "acc": 0.45264846086502075, "lr": 0.00025, "sec_per_step": 6.35, "gnorm": 6.264685153961182}
|
| 167 |
+
{"step": 1610, "loss": 2.7004661560058594, "nll": 2.5743415355682373, "acc": 0.4812312424182892, "lr": 0.00025, "sec_per_step": 5.881, "gnorm": 6.80185604095459}
|
| 168 |
+
{"step": 1620, "loss": 2.419762372970581, "nll": 2.2805428504943848, "acc": 0.5216741561889648, "lr": 0.00025, "sec_per_step": 6.344, "gnorm": 4.487758636474609}
|
| 169 |
+
{"step": 1630, "loss": 2.575009346008301, "nll": 2.4346442222595215, "acc": 0.486821711063385, "lr": 0.00025, "sec_per_step": 6.45, "gnorm": 5.625582695007324}
|
| 170 |
+
{"step": 1640, "loss": 2.6458613872528076, "nll": 2.5290637016296387, "acc": 0.5316363573074341, "lr": 0.00025, "sec_per_step": 6.287, "gnorm": 3.472202777862549}
|
| 171 |
+
{"step": 1650, "loss": 2.4841203689575195, "nll": 2.3537588119506836, "acc": 0.5102505683898926, "lr": 0.00025, "sec_per_step": 6.008, "gnorm": 4.373164653778076}
|
| 172 |
+
{"step": 1660, "loss": 2.651214599609375, "nll": 2.5295002460479736, "acc": 0.4582701027393341, "lr": 0.00025, "sec_per_step": 5.934, "gnorm": 5.1948137283325195}
|
| 173 |
+
{"step": 1670, "loss": 2.6843926906585693, "nll": 2.5529730319976807, "acc": 0.46102941036224365, "lr": 0.00025, "sec_per_step": 6.049, "gnorm": 5.177816867828369}
|
| 174 |
+
{"step": 1680, "loss": 2.5292937755584717, "nll": 2.3909285068511963, "acc": 0.4948939383029938, "lr": 0.00025, "sec_per_step": 6.28, "gnorm": 4.215677738189697}
|
| 175 |
+
{"step": 1690, "loss": 2.4495229721069336, "nll": 2.3191332817077637, "acc": 0.530809223651886, "lr": 0.00025, "sec_per_step": 5.761, "gnorm": 3.945343255996704}
|
| 176 |
+
{"step": 1700, "loss": 2.617029905319214, "nll": 2.466738224029541, "acc": 0.47542187571525574, "lr": 0.00025, "sec_per_step": 6.277, "gnorm": 4.773387432098389}
|
| 177 |
+
{"step": 1710, "loss": 2.5217444896698, "nll": 2.3812272548675537, "acc": 0.49837398529052734, "lr": 0.00025, "sec_per_step": 6.32, "gnorm": 4.3939595222473145}
|
| 178 |
+
{"step": 1720, "loss": 2.846107006072998, "nll": 2.687957286834717, "acc": 0.4266567528247833, "lr": 0.00025, "sec_per_step": 6.239, "gnorm": 6.500346660614014}
|
| 179 |
+
{"step": 1730, "loss": 2.7188565731048584, "nll": 2.5787158012390137, "acc": 0.49103662371635437, "lr": 0.00025, "sec_per_step": 6.068, "gnorm": 4.834630012512207}
|
| 180 |
+
{"step": 1740, "loss": 2.459904193878174, "nll": 2.310497283935547, "acc": 0.509476900100708, "lr": 0.00025, "sec_per_step": 6.306, "gnorm": 4.794856071472168}
|
| 181 |
+
{"step": 1750, "loss": 2.819455623626709, "nll": 2.679036855697632, "acc": 0.4685039222240448, "lr": 0.00025, "sec_per_step": 6.202, "gnorm": 4.270397663116455}
|
| 182 |
+
{"step": 1750, "ema_val_nll": 2.7671797275543213, "ema_val_acc": 0.5013483017683029, "raw_val_nll": 2.4472358226776123, "raw_val_acc": 0.5091685876250267}
|
| 183 |
+
{"step": 1760, "loss": 2.1412112712860107, "nll": 2.036386489868164, "acc": 0.5695817470550537, "lr": 0.00025, "sec_per_step": 7.043, "gnorm": 5.377252578735352}
|
| 184 |
+
{"step": 1770, "loss": 2.5172595977783203, "nll": 2.411311149597168, "acc": 0.527930736541748, "lr": 0.00025, "sec_per_step": 6.318, "gnorm": 4.626220703125}
|
| 185 |
+
{"step": 1780, "loss": 2.6562397480010986, "nll": 2.4928221702575684, "acc": 0.501886785030365, "lr": 0.00025, "sec_per_step": 6.121, "gnorm": 4.809992790222168}
|
| 186 |
+
{"step": 1790, "loss": 2.3886077404022217, "nll": 2.2686352729797363, "acc": 0.5409711599349976, "lr": 0.00025, "sec_per_step": 6.397, "gnorm": 3.8020665645599365}
|
| 187 |
+
{"step": 1800, "loss": 2.6781694889068604, "nll": 2.530588388442993, "acc": 0.48704269528388977, "lr": 0.00025, "sec_per_step": 6.233, "gnorm": 4.668972492218018}
|
| 188 |
+
{"step": 1810, "loss": 2.15527606010437, "nll": 2.005056142807007, "acc": 0.5967492461204529, "lr": 0.00025, "sec_per_step": 5.59, "gnorm": 5.025992393493652}
|
| 189 |
+
{"step": 1820, "loss": 2.5151400566101074, "nll": 2.4142086505889893, "acc": 0.5522388219833374, "lr": 0.00025, "sec_per_step": 6.031, "gnorm": 4.32218074798584}
|
| 190 |
+
{"step": 1830, "loss": 2.6874396800994873, "nll": 2.5723373889923096, "acc": 0.4754098355770111, "lr": 0.00025, "sec_per_step": 5.866, "gnorm": 5.34877347946167}
|
| 191 |
+
{"step": 1840, "loss": 2.763144016265869, "nll": 2.6047627925872803, "acc": 0.4909909963607788, "lr": 0.00025, "sec_per_step": 5.903, "gnorm": 4.591948509216309}
|
| 192 |
+
{"step": 1850, "loss": 2.4946372509002686, "nll": 2.345001220703125, "acc": 0.5270379185676575, "lr": 0.00025, "sec_per_step": 6.223, "gnorm": 5.389904022216797}
|
| 193 |
+
{"step": 1860, "loss": 2.732940673828125, "nll": 2.615614414215088, "acc": 0.48630136251449585, "lr": 0.00025, "sec_per_step": 6.253, "gnorm": 4.7220025062561035}
|
| 194 |
+
{"step": 1870, "loss": 2.449154853820801, "nll": 2.285970687866211, "acc": 0.5331734418869019, "lr": 0.00025, "sec_per_step": 5.931, "gnorm": 6.002699375152588}
|
| 195 |
+
{"step": 1880, "loss": 2.2676377296447754, "nll": 2.1399571895599365, "acc": 0.5431165099143982, "lr": 0.00025, "sec_per_step": 6.419, "gnorm": 5.448880672454834}
|
| 196 |
+
{"step": 1890, "loss": 2.97053861618042, "nll": 2.838926076889038, "acc": 0.4377880096435547, "lr": 0.00025, "sec_per_step": 6.276, "gnorm": 5.652556419372559}
|
| 197 |
+
{"step": 1900, "loss": 2.379473924636841, "nll": 2.2505381107330322, "acc": 0.549730122089386, "lr": 0.00025, "sec_per_step": 5.922, "gnorm": 3.900559425354004}
|
| 198 |
+
{"step": 1910, "loss": 2.437931776046753, "nll": 2.3125457763671875, "acc": 0.5287008881568909, "lr": 0.00025, "sec_per_step": 6.051, "gnorm": 3.7915947437286377}
|
| 199 |
+
{"step": 1920, "loss": 2.351442813873291, "nll": 2.212263822555542, "acc": 0.5736263990402222, "lr": 0.00025, "sec_per_step": 6.284, "gnorm": 3.1446502208709717}
|
| 200 |
+
{"step": 1930, "loss": 2.6715996265411377, "nll": 2.5253102779388428, "acc": 0.48139533400535583, "lr": 0.00024479166666666665, "sec_per_step": 6.33, "gnorm": 6.262954235076904}
|
| 201 |
+
{"step": 1940, "loss": 2.6839425563812256, "nll": 2.54409122467041, "acc": 0.5152904987335205, "lr": 0.00023958333333333335, "sec_per_step": 6.287, "gnorm": 4.908402442932129}
|
| 202 |
+
{"step": 1950, "loss": 2.4449639320373535, "nll": 2.3473141193389893, "acc": 0.5391941666603088, "lr": 0.000234375, "sec_per_step": 6.298, "gnorm": 3.5283448696136475}
|
| 203 |
+
{"step": 1960, "loss": 2.43481707572937, "nll": 2.3232977390289307, "acc": 0.5366614460945129, "lr": 0.00022916666666666666, "sec_per_step": 6.065, "gnorm": 4.786221981048584}
|
| 204 |
+
{"step": 1970, "loss": 2.6724231243133545, "nll": 2.545682668685913, "acc": 0.5007727742195129, "lr": 0.00022395833333333336, "sec_per_step": 6.055, "gnorm": 4.052018165588379}
|
| 205 |
+
{"step": 1980, "loss": 2.4458563327789307, "nll": 2.3175594806671143, "acc": 0.5416666865348816, "lr": 0.00021875, "sec_per_step": 6.476, "gnorm": 5.176480293273926}
|
| 206 |
+
{"step": 1990, "loss": 2.3839120864868164, "nll": 2.2538390159606934, "acc": 0.5558121800422668, "lr": 0.00021354166666666665, "sec_per_step": 6.128, "gnorm": 4.680918216705322}
|
| 207 |
+
{"step": 2000, "loss": 2.9278793334960938, "nll": 2.78448486328125, "acc": 0.4787735939025879, "lr": 0.00020833333333333335, "sec_per_step": 6.083, "gnorm": 6.968480587005615}
|
| 208 |
+
{"step": 2000, "ema_val_nll": 2.6115826964378357, "ema_val_acc": 0.5086340084671974, "raw_val_nll": 2.296378254890442, "raw_val_acc": 0.5319766253232956}
|
| 209 |
+
{"step": 2010, "loss": 2.7372467517852783, "nll": 2.5916860103607178, "acc": 0.48620128631591797, "lr": 0.00020312500000000002, "sec_per_step": 8.423, "gnorm": 4.617999076843262}
|
| 210 |
+
{"step": 2020, "loss": 2.4059858322143555, "nll": 2.270644187927246, "acc": 0.5312976837158203, "lr": 0.00019791666666666666, "sec_per_step": 6.384, "gnorm": 6.127459526062012}
|
| 211 |
+
{"step": 2030, "loss": 2.1576266288757324, "nll": 1.98272705078125, "acc": 0.55513596534729, "lr": 0.00019270833333333336, "sec_per_step": 6.406, "gnorm": 4.933815956115723}
|
| 212 |
+
{"step": 2040, "loss": 2.548945426940918, "nll": 2.3902101516723633, "acc": 0.5061255693435669, "lr": 0.0001875, "sec_per_step": 5.928, "gnorm": 5.75822639465332}
|
| 213 |
+
{"step": 2050, "loss": 2.2869479656219482, "nll": 2.1253886222839355, "acc": 0.5607333779335022, "lr": 0.0001822916666666667, "sec_per_step": 6.069, "gnorm": 3.870657444000244}
|
| 214 |
+
{"step": 2060, "loss": 2.933115243911743, "nll": 2.8382081985473633, "acc": 0.4946889281272888, "lr": 0.00017708333333333332, "sec_per_step": 5.885, "gnorm": 3.6446266174316406}
|
| 215 |
+
{"step": 2070, "loss": 2.4248807430267334, "nll": 2.273270845413208, "acc": 0.5530303120613098, "lr": 0.000171875, "sec_per_step": 6.1, "gnorm": 3.4939122200012207}
|
| 216 |
+
{"step": 2080, "loss": 1.9751367568969727, "nll": 1.8419090509414673, "acc": 0.57421875, "lr": 0.0001666666666666667, "sec_per_step": 5.892, "gnorm": 5.098540306091309}
|
| 217 |
+
{"step": 2090, "loss": 2.5112316608428955, "nll": 2.3503072261810303, "acc": 0.4948778450489044, "lr": 0.0001614583333333333, "sec_per_step": 6.317, "gnorm": 4.346006393432617}
|
| 218 |
+
{"step": 2100, "loss": 2.2872812747955322, "nll": 2.1516096591949463, "acc": 0.5481203198432922, "lr": 0.00015625, "sec_per_step": 6.508, "gnorm": 4.3317108154296875}
|
| 219 |
+
{"step": 2110, "loss": 2.7115135192871094, "nll": 2.5750491619110107, "acc": 0.4988797605037689, "lr": 0.0001510416666666667, "sec_per_step": 5.76, "gnorm": 3.901121139526367}
|
| 220 |
+
{"step": 2120, "loss": 2.5074963569641113, "nll": 2.393440008163452, "acc": 0.540123462677002, "lr": 0.00014583333333333332, "sec_per_step": 5.955, "gnorm": 5.327136039733887}
|
| 221 |
+
{"step": 2130, "loss": 2.687457323074341, "nll": 2.557091474533081, "acc": 0.5162287354469299, "lr": 0.00014062500000000002, "sec_per_step": 6.048, "gnorm": 4.342340469360352}
|
| 222 |
+
{"step": 2140, "loss": 1.7849406003952026, "nll": 1.64743971824646, "acc": 0.6392067074775696, "lr": 0.0001354166666666667, "sec_per_step": 6.081, "gnorm": 4.134641647338867}
|
| 223 |
+
{"step": 2150, "loss": 2.2291746139526367, "nll": 2.0813710689544678, "acc": 0.5711026787757874, "lr": 0.0001302083333333333, "sec_per_step": 6.057, "gnorm": 5.770716190338135}
|
| 224 |
+
{"step": 2160, "loss": 2.1470110416412354, "nll": 2.018789768218994, "acc": 0.5789865851402283, "lr": 0.000125, "sec_per_step": 5.865, "gnorm": 3.632281541824341}
|
| 225 |
+
{"step": 2170, "loss": 2.294814109802246, "nll": 2.1122381687164307, "acc": 0.5601532459259033, "lr": 0.00011979166666666666, "sec_per_step": 6.031, "gnorm": 4.823059558868408}
|
| 226 |
+
{"step": 2180, "loss": 2.255075454711914, "nll": 2.1100776195526123, "acc": 0.5873749256134033, "lr": 0.00011458333333333334, "sec_per_step": 6.292, "gnorm": 5.419332027435303}
|
| 227 |
+
{"step": 2190, "loss": 2.466179132461548, "nll": 2.3219850063323975, "acc": 0.5212598443031311, "lr": 0.000109375, "sec_per_step": 6.017, "gnorm": 5.503530025482178}
|
| 228 |
+
{"step": 2200, "loss": 2.6158857345581055, "nll": 2.506026029586792, "acc": 0.5175037980079651, "lr": 0.00010416666666666666, "sec_per_step": 5.97, "gnorm": 3.886592388153076}
|
| 229 |
+
{"step": 2210, "loss": 2.5306787490844727, "nll": 2.370680332183838, "acc": 0.4883720874786377, "lr": 9.895833333333334e-05, "sec_per_step": 6.301, "gnorm": 4.414287090301514}
|
| 230 |
+
{"step": 2220, "loss": 2.51403546333313, "nll": 2.3920834064483643, "acc": 0.5277989506721497, "lr": 9.375e-05, "sec_per_step": 6.322, "gnorm": 4.095523357391357}
|
| 231 |
+
{"step": 2230, "loss": 2.5394654273986816, "nll": 2.3612449169158936, "acc": 0.5, "lr": 8.854166666666666e-05, "sec_per_step": 5.919, "gnorm": 5.0627570152282715}
|
| 232 |
+
{"step": 2240, "loss": 2.494802951812744, "nll": 2.3610856533050537, "acc": 0.508009135723114, "lr": 8.333333333333334e-05, "sec_per_step": 5.903, "gnorm": 4.964487552642822}
|
| 233 |
+
{"step": 2250, "loss": 1.8854117393493652, "nll": 1.7896050214767456, "acc": 0.6266159415245056, "lr": 7.8125e-05, "sec_per_step": 6.097, "gnorm": 4.4996466636657715}
|
| 234 |
+
{"step": 2250, "ema_val_nll": 2.4679396748542786, "ema_val_acc": 0.5299465879797935, "raw_val_nll": 2.1637205481529236, "raw_val_acc": 0.5578823387622833}
|
| 235 |
+
{"step": 2260, "loss": 2.230721950531006, "nll": 2.091271162033081, "acc": 0.5658198595046997, "lr": 7.291666666666666e-05, "sec_per_step": 7.034, "gnorm": 4.909244537353516}
|
| 236 |
+
{"step": 2270, "loss": 2.035552740097046, "nll": 1.8922446966171265, "acc": 0.6076335906982422, "lr": 6.770833333333334e-05, "sec_per_step": 6.018, "gnorm": 3.9589521884918213}
|
| 237 |
+
{"step": 2280, "loss": 2.838111400604248, "nll": 2.701313018798828, "acc": 0.4763810932636261, "lr": 6.25e-05, "sec_per_step": 5.892, "gnorm": 5.069949150085449}
|
| 238 |
+
{"step": 2290, "loss": 2.502668619155884, "nll": 2.3781075477600098, "acc": 0.5141545534133911, "lr": 5.729166666666666e-05, "sec_per_step": 6.362, "gnorm": 4.266213893890381}
|
| 239 |
+
{"step": 2300, "loss": 2.645368814468384, "nll": 2.5022597312927246, "acc": 0.49233129620552063, "lr": 5.2083333333333343e-05, "sec_per_step": 6.165, "gnorm": 4.9644622802734375}
|
| 240 |
+
{"step": 2310, "loss": 2.61704421043396, "nll": 2.5010693073272705, "acc": 0.5034013390541077, "lr": 4.6875e-05, "sec_per_step": 6.549, "gnorm": 3.7004849910736084}
|
| 241 |
+
{"step": 2320, "loss": 2.2191402912139893, "nll": 2.054413318634033, "acc": 0.5559936761856079, "lr": 4.166666666666666e-05, "sec_per_step": 5.904, "gnorm": 5.543429374694824}
|
| 242 |
+
{"step": 2330, "loss": 2.2716104984283447, "nll": 2.1119349002838135, "acc": 0.5815176367759705, "lr": 3.645833333333334e-05, "sec_per_step": 6.088, "gnorm": 4.0455451011657715}
|
| 243 |
+
{"step": 2340, "loss": 2.360276460647583, "nll": 2.187093496322632, "acc": 0.5592452883720398, "lr": 3.125e-05, "sec_per_step": 6.027, "gnorm": 4.779704570770264}
|
| 244 |
+
{"step": 2350, "loss": 1.95784330368042, "nll": 1.7860734462738037, "acc": 0.6135488748550415, "lr": 2.6041666666666658e-05, "sec_per_step": 6.292, "gnorm": 3.6473851203918457}
|
| 245 |
+
{"step": 2360, "loss": 2.1754679679870605, "nll": 1.9920741319656372, "acc": 0.5615686178207397, "lr": 2.0833333333333343e-05, "sec_per_step": 6.08, "gnorm": 4.795201301574707}
|
| 246 |
+
{"step": 2370, "loss": 2.4577841758728027, "nll": 2.3091211318969727, "acc": 0.5495638251304626, "lr": 1.5625e-05, "sec_per_step": 5.973, "gnorm": 4.615086078643799}
|
| 247 |
+
{"step": 2380, "loss": 1.8841142654418945, "nll": 1.7523480653762817, "acc": 0.6373873949050903, "lr": 1.0416666666666658e-05, "sec_per_step": 5.942, "gnorm": 4.416830062866211}
|
| 248 |
+
{"step": 2390, "loss": 2.4632983207702637, "nll": 2.296647548675537, "acc": 0.4739263951778412, "lr": 5.2083333333333425e-06, "sec_per_step": 6.363, "gnorm": 4.186145305633545}
|
| 249 |
+
{"step": 2400, "loss": 2.47607159614563, "nll": 2.3211183547973633, "acc": 0.5543219447135925, "lr": 0.0, "sec_per_step": 6.358, "gnorm": 4.649677753448486}
|
| 250 |
+
{"step": 2400, "ema_val_nll": 2.445981442928314, "ema_val_acc": 0.5343381315469742, "raw_val_nll": 2.176288068294525, "raw_val_acc": 0.545825183391571}
|
model_config.json
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"vocab_size": 512,
|
| 3 |
+
"grid_size": 16,
|
| 4 |
+
"dim": 512,
|
| 5 |
+
"n_heads": 8,
|
| 6 |
+
"depth_in": 2,
|
| 7 |
+
"depth_core": 8,
|
| 8 |
+
"depth_out": 2,
|
| 9 |
+
"patch": 2,
|
| 10 |
+
"mlp_ratio": 3.0,
|
| 11 |
+
"n_registers": 4,
|
| 12 |
+
"text_dim": 512,
|
| 13 |
+
"text_len": 24,
|
| 14 |
+
"rope_theta": 100.0,
|
| 15 |
+
"conv_ffn": true,
|
| 16 |
+
"color_freqs": 6,
|
| 17 |
+
"self_cond": true
|
| 18 |
+
}
|
showcase.png
ADDED
|
train_config.json
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"batch_size": 8,
|
| 3 |
+
"grad_accum": 1,
|
| 4 |
+
"total_steps": 2400,
|
| 5 |
+
"lr": 0.00025,
|
| 6 |
+
"min_lr_ratio": 0.0,
|
| 7 |
+
"warmup_steps": 100,
|
| 8 |
+
"decay_frac": 0.2,
|
| 9 |
+
"weight_decay": 0.05,
|
| 10 |
+
"beta1": 0.9,
|
| 11 |
+
"beta2": 0.95,
|
| 12 |
+
"grad_clip": 1.0,
|
| 13 |
+
"optimizer": "muon",
|
| 14 |
+
"muon_momentum": 0.95,
|
| 15 |
+
"ema_decay": 0.999,
|
| 16 |
+
"mask_schedule": "cosine",
|
| 17 |
+
"loss_weighting": "uniform",
|
| 18 |
+
"stratified_t": true,
|
| 19 |
+
"smooth_mask_prob": 0.25,
|
| 20 |
+
"label_smoothing": 0.1,
|
| 21 |
+
"color_tau": 0.05,
|
| 22 |
+
"self_cond_prob": 0.5,
|
| 23 |
+
"self_cond_lookback": 0.3,
|
| 24 |
+
"cond_drop_prob": 0.1,
|
| 25 |
+
"subject_prob": 0.05,
|
| 26 |
+
"hflip_prob": 0.5,
|
| 27 |
+
"seed": 1234,
|
| 28 |
+
"log_every": 10,
|
| 29 |
+
"eval_every": 250,
|
| 30 |
+
"eval_batches": 4,
|
| 31 |
+
"eval_batch": 16,
|
| 32 |
+
"sample_every": 500,
|
| 33 |
+
"ckpt_every_steps": 400,
|
| 34 |
+
"ckpt_every_minutes": 4.0,
|
| 35 |
+
"sample_steps": 16,
|
| 36 |
+
"sample_cfg": 3.0,
|
| 37 |
+
"sample_prompts": [
|
| 38 |
+
"red health potion bottle",
|
| 39 |
+
"golden crown with blue gems",
|
| 40 |
+
"green slime monster",
|
| 41 |
+
"iron sword with a leather hilt",
|
| 42 |
+
"wooden treasure chest",
|
| 43 |
+
"pink cherry blossom tree",
|
| 44 |
+
"blue magic spellbook",
|
| 45 |
+
"orange fox head",
|
| 46 |
+
"glowing purple crystal",
|
| 47 |
+
"cheeseburger",
|
| 48 |
+
"skull with a candle",
|
| 49 |
+
"rocket ship",
|
| 50 |
+
"brown leather boots",
|
| 51 |
+
"yellow rubber duck",
|
| 52 |
+
"ice cream cone with strawberry scoop",
|
| 53 |
+
"black cat"
|
| 54 |
+
]
|
| 55 |
+
}
|
trout-fp16.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d9d2ffe91522aede1965db4441780c7bc7d67dc0d966f0e289c508f02793f665
|
| 3 |
+
size 95251680
|
trout.py
ADDED
|
@@ -0,0 +1,1017 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
trout.py -- a from-scratch diffusion language model for 16x16 pixel art.
|
| 3 |
+
|
| 4 |
+
Trout treats a 16x16 sprite as a 256-token "sentence" over a learned 512-color
|
| 5 |
+
palette (token 0 = transparent). A bidirectional transformer denoiser starts
|
| 6 |
+
from a fully masked canvas and unmasks all pixels in parallel over ~16 steps,
|
| 7 |
+
conditioned on a frozen CLIP text tower. No VAE, no latent space, no
|
| 8 |
+
autoregression -- absorbing-state masked (discrete) diffusion.
|
| 9 |
+
|
| 10 |
+
Architecture (hourglass dual-resolution denoiser, 47.6M params for the small
|
| 11 |
+
preset), reimplemented from scratch following the design published with
|
| 12 |
+
Tessera-small by Jagrat Patel (TheJagStudio), whose model card and open demo
|
| 13 |
+
space document every choice below:
|
| 14 |
+
|
| 15 |
+
* joint [registers | text tokens | pixel tokens] bidirectional attention
|
| 16 |
+
* hourglass: depth_in pixel-level blocks -> 2x2 patch-merged core blocks
|
| 17 |
+
-> depth_out pixel-level blocks, with a residual skip around the core
|
| 18 |
+
* QK-RMSNorm, 2D axial RoPE (theta=100) with sub-pixel-aligned patch centers
|
| 19 |
+
* Conv-SwiGLU: depthwise 3x3 conv on the gate branch over image tokens
|
| 20 |
+
* adaLN-single conditioning: shared MLP on (pooled text + mask ratio
|
| 21 |
+
Fourier features) with per-block learned modulation offsets and
|
| 22 |
+
zero-initialized gates (blocks start as identity)
|
| 23 |
+
* color-grounded token embeddings: learned vector + MLP(Fourier(OKLab))
|
| 24 |
+
* 4 register tokens in the prefix
|
| 25 |
+
* self-conditioning: the previous step's predicted color distribution is
|
| 26 |
+
projected and added at masked positions
|
| 27 |
+
|
| 28 |
+
Theory sources (the "knowledge" this model is built on):
|
| 29 |
+
* Absorbing-state / masked discrete diffusion:
|
| 30 |
+
Sahoo et al., "Simple and Effective Masked Diffusion Language Models",
|
| 31 |
+
NeurIPS 2024 (arXiv:2406.07524) -- MDLM
|
| 32 |
+
Shi et al., "Simplified and Generalized Masked Diffusion for Discrete
|
| 33 |
+
Data", NeurIPS 2024 (arXiv:2406.04329)
|
| 34 |
+
Labadie & Ye, "Absorbing State Discrete Diffusion" survey, 2025.
|
| 35 |
+
* Parallel confidence-ordered decoding: Chang et al., "MaskGIT: Masked
|
| 36 |
+
Generative Image Transformer", CVPR 2022.
|
| 37 |
+
* Hourglass transformer: Blecher et al., "Nougat" (arXiv:2308.13418).
|
| 38 |
+
* adaLN-single: Peebles & Xie, "DiT" (arXiv:2212.09748); PixArt-alpha.
|
| 39 |
+
* Registers: Darcet et al., "Vision Transformers Need Registers" (ICLR 2024).
|
| 40 |
+
* QK-norm: Gemma/Henry et al. "Training Transformers with ... QK-Norm".
|
| 41 |
+
* Self-conditioning: Chen, "Analog Bits" (arXiv:2208.04202).
|
| 42 |
+
* Muon optimizer: Jordan et al. (arXiv:2502.16982) and Moonshot
|
| 43 |
+
"Muon is Scalable" (arXiv:2508.02535) for the 0.2*sqrt(max(m,n)) scale.
|
| 44 |
+
* OKLab: Bjorn Ottosson, "A perceptual color space for image processing"
|
| 45 |
+
(2020) -- public-domain reference formulas.
|
| 46 |
+
|
| 47 |
+
Written by Z.ai for the user's "Trout" project; trained code lives in
|
| 48 |
+
train_trout.py. Apache-2.0.
|
| 49 |
+
"""
|
| 50 |
+
|
| 51 |
+
from __future__ import annotations
|
| 52 |
+
|
| 53 |
+
import json
|
| 54 |
+
import math
|
| 55 |
+
import os
|
| 56 |
+
from dataclasses import asdict, dataclass, field
|
| 57 |
+
from typing import Any, Dict, Iterator, List, Optional, Sequence, Tuple
|
| 58 |
+
|
| 59 |
+
import numpy as np
|
| 60 |
+
import torch
|
| 61 |
+
import torch.nn as nn
|
| 62 |
+
import torch.nn.functional as F
|
| 63 |
+
|
| 64 |
+
DEFAULT_TEXT_ENCODER = "openai/clip-vit-base-patch32"
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
# ---------------------------------------------------------------------------
|
| 68 |
+
# configuration
|
| 69 |
+
# ---------------------------------------------------------------------------
|
| 70 |
+
|
| 71 |
+
@dataclass
|
| 72 |
+
class TroutConfig:
|
| 73 |
+
"""Architecture hyper-parameters for the Trout denoiser."""
|
| 74 |
+
vocab_size: int = 512 # palette entries incl. transparent (index 0); MASK id == vocab_size
|
| 75 |
+
grid_size: int = 16 # image is grid_size x grid_size tokens
|
| 76 |
+
dim: int = 512 # residual width
|
| 77 |
+
n_heads: int = 8
|
| 78 |
+
depth_in: int = 2 # pixel-level blocks before the 2x2 merge
|
| 79 |
+
depth_core: int = 8 # patch-level blocks on merged (grid/patch)^2 tokens
|
| 80 |
+
depth_out: int = 2 # pixel-level blocks after the unmerge
|
| 81 |
+
patch: int = 2 # hourglass merge factor (1 = flat model)
|
| 82 |
+
mlp_ratio: float = 3.0 # SwiGLU hidden = mlp_ratio * dim
|
| 83 |
+
n_registers: int = 4 # learnable register tokens in the prefix
|
| 84 |
+
text_dim: int = 512 # hidden size of the frozen CLIP text tower
|
| 85 |
+
text_len: int = 24 # text tokens fed to the denoiser
|
| 86 |
+
rope_theta: float = 100.0 # base of the 2D axial RoPE frequencies
|
| 87 |
+
conv_ffn: bool = True # depthwise 3x3 conv on the gate branch
|
| 88 |
+
color_freqs: int = 6 # Fourier frequencies for OKLab color embeddings
|
| 89 |
+
self_cond: bool = True # feed previous-step predictions back as input
|
| 90 |
+
|
| 91 |
+
@property
|
| 92 |
+
def mask_id(self) -> int:
|
| 93 |
+
return self.vocab_size
|
| 94 |
+
|
| 95 |
+
@property
|
| 96 |
+
def num_tokens(self) -> int:
|
| 97 |
+
return self.grid_size * self.grid_size
|
| 98 |
+
|
| 99 |
+
@property
|
| 100 |
+
def hourglass(self) -> bool:
|
| 101 |
+
return self.patch > 1 and self.depth_core > 0
|
| 102 |
+
|
| 103 |
+
@property
|
| 104 |
+
def head_dim(self) -> int:
|
| 105 |
+
return self.dim // self.n_heads
|
| 106 |
+
|
| 107 |
+
@property
|
| 108 |
+
def mlp_hidden(self) -> int:
|
| 109 |
+
return int(self.dim * self.mlp_ratio)
|
| 110 |
+
|
| 111 |
+
def validate(self) -> None:
|
| 112 |
+
assert self.dim % self.n_heads == 0
|
| 113 |
+
assert self.head_dim % 4 == 0, "head_dim must be divisible by 4 (2D axial RoPE)"
|
| 114 |
+
if self.hourglass:
|
| 115 |
+
assert self.grid_size % self.patch == 0
|
| 116 |
+
|
| 117 |
+
def to_dict(self) -> Dict[str, Any]:
|
| 118 |
+
return asdict(self)
|
| 119 |
+
|
| 120 |
+
@classmethod
|
| 121 |
+
def from_dict(cls, d: Dict[str, Any]) -> "TroutConfig":
|
| 122 |
+
import dataclasses as _dc
|
| 123 |
+
names = {f.name for f in _dc.fields(cls)}
|
| 124 |
+
keep = {k: v for k, v in d.items() if k in names}
|
| 125 |
+
cfg = cls(**keep)
|
| 126 |
+
cfg.validate()
|
| 127 |
+
return cfg
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
SMALL_PRESET: Dict[str, Any] = dict(
|
| 131 |
+
dim=512, n_heads=8, depth_in=2, depth_core=8, depth_out=2,
|
| 132 |
+
) # matches Tessera-small; ~47.6M parameters
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
# ---------------------------------------------------------------------------
|
| 136 |
+
# OKLab color math (Bjorn Ottosson's public-domain formulas)
|
| 137 |
+
# ---------------------------------------------------------------------------
|
| 138 |
+
|
| 139 |
+
def _srgb_to_linear_np(c):
|
| 140 |
+
return np.where(c <= 0.04045, c / 12.92, ((c + 0.055) / 1.055) ** 2.4)
|
| 141 |
+
|
| 142 |
+
|
| 143 |
+
def _linear_to_srgb_np(c):
|
| 144 |
+
c = np.clip(c, 0.0, 1.0)
|
| 145 |
+
return np.where(c <= 0.0031308, 12.92 * c, 1.055 * np.maximum(c, 1e-12) ** (1 / 2.4) - 0.055)
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
def srgb_to_oklab(rgb: np.ndarray) -> np.ndarray:
|
| 149 |
+
"""rgb: (..., 3) in [0,255]. Returns OKLab (..., 3) float32."""
|
| 150 |
+
x = np.asarray(rgb, dtype=np.float64) / 255.0
|
| 151 |
+
lin = _srgb_to_linear_np(x)
|
| 152 |
+
r, g, b = lin[..., 0], lin[..., 1], lin[..., 2]
|
| 153 |
+
l = 0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b
|
| 154 |
+
m = 0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b
|
| 155 |
+
s = 0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b
|
| 156 |
+
l_, m_, s_ = np.cbrt(np.maximum(l, 0)), np.cbrt(np.maximum(m, 0)), np.cbrt(np.maximum(s, 0))
|
| 157 |
+
L = 0.2104542553 * l_ + 0.7936177850 * m_ - 0.0040720468 * s_
|
| 158 |
+
A = 1.9779984951 * l_ - 2.4285922050 * m_ + 0.4505937099 * s_
|
| 159 |
+
B = 0.0259040371 * l_ + 0.7827717662 * m_ - 0.8086757660 * s_
|
| 160 |
+
return np.stack([L, A, B], -1).astype(np.float32)
|
| 161 |
+
|
| 162 |
+
|
| 163 |
+
def _torch_srgb_to_linear(c):
|
| 164 |
+
return torch.where(c <= 0.04045, c / 12.92, ((c + 0.055) / 1.055) ** 2.4)
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
def srgb_to_oklab_t(rgb: torch.Tensor) -> torch.Tensor:
|
| 168 |
+
x = rgb.float() / 255.0
|
| 169 |
+
lin = _torch_srgb_to_linear(x)
|
| 170 |
+
r, g, b = lin[..., 0], lin[..., 1], lin[..., 2]
|
| 171 |
+
l = 0.4122214708 * r + 0.5363325363 * g + 0.0514459929 * b
|
| 172 |
+
m = 0.2119034982 * r + 0.6806995451 * g + 0.1073969566 * b
|
| 173 |
+
s = 0.0883024619 * r + 0.2817188376 * g + 0.6299787005 * b
|
| 174 |
+
l_ = torch.pow(l.clamp(min=0), 1 / 3)
|
| 175 |
+
m_ = torch.pow(m.clamp(min=0), 1 / 3)
|
| 176 |
+
s_ = torch.pow(s.clamp(min=0), 1 / 3)
|
| 177 |
+
L = 0.2104542553 * l_ + 0.7936177850 * m_ - 0.0040720468 * s_
|
| 178 |
+
A = 1.9779984951 * l_ - 2.4285922050 * m_ + 0.4505937099 * s_
|
| 179 |
+
B = 0.0259040371 * l_ + 0.7827717662 * m_ - 0.8086757660 * s_
|
| 180 |
+
return torch.stack([L, A, B], -1).float()
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
def rgb_to_key(rgb: np.ndarray) -> np.ndarray:
|
| 184 |
+
rgb = rgb.astype(np.int64)
|
| 185 |
+
return (rgb[..., 0] << 16) | (rgb[..., 1] << 8) | rgb[..., 2]
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def key_to_rgb(key: np.ndarray) -> np.ndarray:
|
| 189 |
+
key = np.asarray(key, dtype=np.int64)
|
| 190 |
+
return np.stack([(key >> 16) & 255, (key >> 8) & 255, key & 255], -1).astype(np.uint8)
|
| 191 |
+
|
| 192 |
+
|
| 193 |
+
# ---------------------------------------------------------------------------
|
| 194 |
+
# palette: 512 colors, frequency-weighted k-means in OKLab, snapped to data
|
| 195 |
+
# ---------------------------------------------------------------------------
|
| 196 |
+
|
| 197 |
+
def _weighted_kmeans(x: torch.Tensor, w: torch.Tensor, k: int, iters: int = 60,
|
| 198 |
+
seed: int = 0) -> torch.Tensor:
|
| 199 |
+
"""Weighted k-means with k-means++ init. x: (U,3), w: (U,). Returns (k,3)."""
|
| 200 |
+
g = torch.Generator().manual_seed(seed)
|
| 201 |
+
U = x.shape[0]
|
| 202 |
+
w = w.float()
|
| 203 |
+
first = torch.multinomial(w, 1, generator=g).item()
|
| 204 |
+
centers = [x[first]]
|
| 205 |
+
d2 = ((x - x[first]) ** 2).sum(1)
|
| 206 |
+
for _ in range(1, k):
|
| 207 |
+
p = (w * d2)
|
| 208 |
+
if p.sum() <= 0:
|
| 209 |
+
j = int(torch.randint(U, (1,), generator=g))
|
| 210 |
+
else:
|
| 211 |
+
j = torch.multinomial(p.clamp(min=0), 1, generator=g).item()
|
| 212 |
+
centers.append(x[j])
|
| 213 |
+
d2 = torch.minimum(d2, ((x - x[j]) ** 2).sum(1))
|
| 214 |
+
C = torch.stack(centers).clone()
|
| 215 |
+
for _ in range(iters):
|
| 216 |
+
# nearest assignment, chunked to bound memory
|
| 217 |
+
idx = torch.empty(U, dtype=torch.long)
|
| 218 |
+
mind = torch.empty(U)
|
| 219 |
+
c2 = (C * C).sum(1)
|
| 220 |
+
for s in range(0, U, 1 << 18):
|
| 221 |
+
p = x[s:s + (1 << 18)]
|
| 222 |
+
d = (p * p).sum(1, keepdim=True) - 2 * p @ C.T + c2[None]
|
| 223 |
+
dv, di = d.min(1)
|
| 224 |
+
idx[s:s + p.shape[0]] = di
|
| 225 |
+
mind[s:s + p.shape[0]] = dv.clamp(min=0)
|
| 226 |
+
sums = torch.zeros_like(C).index_add_(0, idx, x * w[:, None])
|
| 227 |
+
ws = torch.zeros(k).index_add_(0, idx, w)
|
| 228 |
+
newC = torch.where((ws <= 0)[:, None], C, sums / ws.clamp(min=1e-12)[:, None])
|
| 229 |
+
nempty = int((ws <= 0).sum())
|
| 230 |
+
if nempty > 0:
|
| 231 |
+
worst = torch.topk(w * mind, nempty).indices
|
| 232 |
+
newC[ws <= 0] = x[worst]
|
| 233 |
+
shift = (newC - C).norm(dim=1).max().item()
|
| 234 |
+
C = newC
|
| 235 |
+
if shift < 1e-7:
|
| 236 |
+
break
|
| 237 |
+
return C
|
| 238 |
+
|
| 239 |
+
|
| 240 |
+
class Palette:
|
| 241 |
+
"""A fixed list of colors. Index 0 = transparent; 1..K-1 opaque sRGB."""
|
| 242 |
+
|
| 243 |
+
def __init__(self, colors: np.ndarray, name: str = "custom"):
|
| 244 |
+
colors = np.asarray(colors, dtype=np.uint8).reshape(-1, 3).copy()
|
| 245 |
+
colors[0] = 0
|
| 246 |
+
self.colors = colors
|
| 247 |
+
self.name = name
|
| 248 |
+
lab = srgb_to_oklab(colors)
|
| 249 |
+
lab[0] = 0.0
|
| 250 |
+
self.lab = lab
|
| 251 |
+
|
| 252 |
+
@property
|
| 253 |
+
def size(self) -> int:
|
| 254 |
+
return int(self.colors.shape[0])
|
| 255 |
+
|
| 256 |
+
def __len__(self) -> int:
|
| 257 |
+
return self.size
|
| 258 |
+
|
| 259 |
+
def rgba_table(self) -> np.ndarray:
|
| 260 |
+
t = np.concatenate([self.colors, np.full((self.size, 1), 255, np.uint8)], 1)
|
| 261 |
+
t[0] = 0
|
| 262 |
+
return t
|
| 263 |
+
|
| 264 |
+
def tokens_to_rgba(self, tokens) -> np.ndarray:
|
| 265 |
+
if isinstance(tokens, torch.Tensor):
|
| 266 |
+
tokens = tokens.detach().cpu().numpy()
|
| 267 |
+
tokens = np.asarray(tokens).astype(np.int64)
|
| 268 |
+
table = np.concatenate([self.rgba_table(), np.array([[255, 0, 255, 255]], np.uint8)], 0)
|
| 269 |
+
return table[np.clip(tokens, 0, self.size)]
|
| 270 |
+
|
| 271 |
+
def nearest_index(self, rgb: np.ndarray) -> np.ndarray:
|
| 272 |
+
"""Nearest opaque entry (1..K-1) in OKLab for (M,3) uint8."""
|
| 273 |
+
if rgb.shape[0] == 0:
|
| 274 |
+
return np.zeros(0, np.int64)
|
| 275 |
+
pts = torch.from_numpy(srgb_to_oklab(rgb))
|
| 276 |
+
cen = torch.from_numpy(self.lab[1:])
|
| 277 |
+
idx, _ = self._nearest(pts, cen)
|
| 278 |
+
return idx.numpy() + 1
|
| 279 |
+
|
| 280 |
+
@staticmethod
|
| 281 |
+
def _nearest(points: torch.Tensor, centers: torch.Tensor, chunk: int = 1 << 18):
|
| 282 |
+
idx = torch.empty(points.shape[0], dtype=torch.long)
|
| 283 |
+
dist = torch.empty(points.shape[0])
|
| 284 |
+
c2 = (centers * centers).sum(1)
|
| 285 |
+
for s in range(0, points.shape[0], chunk):
|
| 286 |
+
p = points[s:s + chunk]
|
| 287 |
+
d = (p * p).sum(1, keepdim=True) - 2 * p @ centers.T + c2[None]
|
| 288 |
+
dv, di = d.min(1)
|
| 289 |
+
idx[s:s + p.shape[0]] = di
|
| 290 |
+
dist[s:s + p.shape[0]] = dv.clamp(min=0)
|
| 291 |
+
return idx, dist
|
| 292 |
+
|
| 293 |
+
def build_lut(self, keys: np.ndarray) -> Dict[str, np.ndarray]:
|
| 294 |
+
keys = np.unique(np.asarray(keys, dtype=np.int64))
|
| 295 |
+
idx = self.nearest_index(key_to_rgb(keys))
|
| 296 |
+
return {"keys": keys, "idx": idx.astype(np.uint16)}
|
| 297 |
+
|
| 298 |
+
def quantize_rgba(self, rgba: np.ndarray, alpha_threshold: int = 128,
|
| 299 |
+
lut: Optional[dict] = None) -> np.ndarray:
|
| 300 |
+
"""uint8 RGBA (...,4) -> uint16 tokens (...); alpha<thresh -> transparent."""
|
| 301 |
+
rgba = np.asarray(rgba)
|
| 302 |
+
shape = rgba.shape[:-1]
|
| 303 |
+
flat = rgba.reshape(-1, 4)
|
| 304 |
+
opaque = flat[:, 3] >= alpha_threshold
|
| 305 |
+
out = np.zeros(flat.shape[0], np.uint16)
|
| 306 |
+
if opaque.any():
|
| 307 |
+
keys = rgb_to_key(flat[opaque, :3])
|
| 308 |
+
if lut is None:
|
| 309 |
+
lut = self.build_lut(keys)
|
| 310 |
+
pos = np.searchsorted(lut["keys"], keys)
|
| 311 |
+
pos = np.clip(pos, 0, len(lut["keys"]) - 1)
|
| 312 |
+
hit = lut["keys"][pos] == keys
|
| 313 |
+
vals = lut["idx"][pos]
|
| 314 |
+
if not hit.all():
|
| 315 |
+
miss = ~hit
|
| 316 |
+
vals = vals.copy()
|
| 317 |
+
vals[miss] = self.nearest_index(key_to_rgb(keys[miss]))
|
| 318 |
+
out[opaque] = vals
|
| 319 |
+
return out.reshape(shape)
|
| 320 |
+
|
| 321 |
+
def soft_targets(self, tau: float = 0.05) -> torch.Tensor:
|
| 322 |
+
"""(K,K) row-stochastic matrix for perceptual label smoothing:
|
| 323 |
+
row c = softmax over opaque colors of -dE(c,.)^2 / tau^2."""
|
| 324 |
+
lab = torch.from_numpy(self.lab).double()
|
| 325 |
+
d2 = torch.cdist(lab, lab) ** 2
|
| 326 |
+
logits = -d2 / (tau * tau)
|
| 327 |
+
logits[:, 0] = -math.inf
|
| 328 |
+
S = torch.softmax(logits, dim=1)
|
| 329 |
+
S[0] = 0.0
|
| 330 |
+
S[0, 0] = 1.0
|
| 331 |
+
return S.float()
|
| 332 |
+
|
| 333 |
+
def to_dict(self) -> Dict[str, Any]:
|
| 334 |
+
return {"name": self.name, "colors": self.colors.tolist()}
|
| 335 |
+
|
| 336 |
+
@classmethod
|
| 337 |
+
def from_dict(cls, d: Dict[str, Any]) -> "Palette":
|
| 338 |
+
return cls(np.array(d["colors"], dtype=np.uint8), name=d.get("name", "custom"))
|
| 339 |
+
|
| 340 |
+
def save(self, path: str) -> None:
|
| 341 |
+
with open(path, "w") as f:
|
| 342 |
+
json.dump(self.to_dict(), f)
|
| 343 |
+
|
| 344 |
+
@classmethod
|
| 345 |
+
def load(cls, path: str) -> "Palette":
|
| 346 |
+
with open(path) as f:
|
| 347 |
+
return cls.from_dict(json.load(f))
|
| 348 |
+
|
| 349 |
+
def preview(self, cell: int = 12, cols: int = 32) -> "Image":
|
| 350 |
+
from PIL import Image
|
| 351 |
+
rows = math.ceil(self.size / cols)
|
| 352 |
+
img = np.full((rows * cell, cols * cell, 3), 255, np.uint8)
|
| 353 |
+
for i in range(self.size):
|
| 354 |
+
r, c = divmod(i, cols)
|
| 355 |
+
if i == 0:
|
| 356 |
+
yy, xx = np.mgrid[0:cell, 0:cell]
|
| 357 |
+
img[r * cell:(r + 1) * cell, c * cell:(c + 1) * cell] = np.where(
|
| 358 |
+
(((yy // 3) + (xx // 3)) % 2 == 0)[..., None], 200, 140)
|
| 359 |
+
else:
|
| 360 |
+
img[r * cell:(r + 1) * cell, c * cell:(c + 1) * cell] = self.colors[i]
|
| 361 |
+
return Image.fromarray(img)
|
| 362 |
+
|
| 363 |
+
@classmethod
|
| 364 |
+
def build_kmeans(cls, keys: np.ndarray, counts: np.ndarray, size: int = 512,
|
| 365 |
+
weight_power: float = 0.5, iters: int = 60, seed: int = 0) -> "Palette":
|
| 366 |
+
"""Data-driven palette: 1 transparent + size-1 opaque colors chosen by
|
| 367 |
+
frequency-weighted (tempered by weight_power) k-means in OKLab, with
|
| 368 |
+
centroids snapped to real dataset colors so exact hues survive."""
|
| 369 |
+
keys = np.asarray(keys, dtype=np.int64)
|
| 370 |
+
counts = np.asarray(counts, dtype=np.float64)
|
| 371 |
+
k = size - 1
|
| 372 |
+
rgb = key_to_rgb(keys)
|
| 373 |
+
if keys.shape[0] <= k:
|
| 374 |
+
colors = np.concatenate([np.zeros((1, 3), np.uint8), rgb], 0)
|
| 375 |
+
return cls._sorted(colors, name=f"exact{len(colors)}")
|
| 376 |
+
x = torch.from_numpy(srgb_to_oklab(rgb))
|
| 377 |
+
w = torch.from_numpy(counts ** weight_power)
|
| 378 |
+
C = _weighted_kmeans(x, w, k, iters=iters, seed=seed)
|
| 379 |
+
snap_idx, _ = cls._nearest(C, x)
|
| 380 |
+
snap_idx = snap_idx.numpy()
|
| 381 |
+
chosen = list(dict.fromkeys(snap_idx.tolist()))
|
| 382 |
+
if len(chosen) < k: # fill duplicate slots with worst-represented colors
|
| 383 |
+
cen = x[torch.as_tensor(chosen)]
|
| 384 |
+
_, dist = cls._nearest(x, cen)
|
| 385 |
+
order = torch.argsort(w * dist, descending=True).numpy()
|
| 386 |
+
for j in order:
|
| 387 |
+
if len(chosen) >= k:
|
| 388 |
+
break
|
| 389 |
+
if int(j) not in chosen:
|
| 390 |
+
chosen.append(int(j))
|
| 391 |
+
colors = np.concatenate([np.zeros((1, 3), np.uint8), rgb[np.array(chosen[:k])]], 0)
|
| 392 |
+
return cls._sorted(colors, name=f"kmeans{size}")
|
| 393 |
+
|
| 394 |
+
@classmethod
|
| 395 |
+
def _sorted(cls, colors: np.ndarray, name: str) -> "Palette":
|
| 396 |
+
"""Order opaque colors by hue bucket then lightness (readable palette)."""
|
| 397 |
+
opaque = colors[1:]
|
| 398 |
+
lab = srgb_to_oklab(opaque)
|
| 399 |
+
chroma = np.sqrt(lab[:, 1] ** 2 + lab[:, 2] ** 2)
|
| 400 |
+
hue = np.arctan2(lab[:, 2], lab[:, 1])
|
| 401 |
+
bucket = np.where(chroma < 0.03, -1, np.floor((hue + np.pi) / (2 * np.pi) * 12)).astype(np.int64)
|
| 402 |
+
order = np.lexsort((lab[:, 0], bucket))
|
| 403 |
+
return cls(np.concatenate([colors[:1], opaque[order]], 0), name=name)
|
| 404 |
+
|
| 405 |
+
|
| 406 |
+
def quantization_report(palette: Palette, rgba: np.ndarray, tokens: np.ndarray,
|
| 407 |
+
max_pixels: int = 2_000_000) -> Dict[str, float]:
|
| 408 |
+
"""Mean/p95 OKLab error introduced by quantization on opaque pixels."""
|
| 409 |
+
flat = rgba.reshape(-1, 4)
|
| 410 |
+
tok = tokens.reshape(-1)
|
| 411 |
+
if flat.shape[0] > max_pixels:
|
| 412 |
+
sel = np.random.default_rng(0).integers(0, flat.shape[0], max_pixels)
|
| 413 |
+
flat, tok = flat[sel], tok[sel]
|
| 414 |
+
op = flat[:, 3] >= 128
|
| 415 |
+
if not op.any():
|
| 416 |
+
return {"mean_dE": 0.0, "p95_dE": 0.0, "exact_frac": 1.0}
|
| 417 |
+
src = srgb_to_oklab(flat[op, :3])
|
| 418 |
+
dst = palette.lab[tok[op].astype(np.int64)]
|
| 419 |
+
d = np.linalg.norm(src - dst, axis=1)
|
| 420 |
+
exact = (palette.colors[tok[op].astype(np.int64)] == flat[op, :3]).all(1)
|
| 421 |
+
return {"mean_dE": float(d.mean()), "p95_dE": float(np.percentile(d, 95)),
|
| 422 |
+
"max_dE": float(d.max()), "exact_frac": float(exact.mean())}
|
| 423 |
+
|
| 424 |
+
|
| 425 |
+
# ---------------------------------------------------------------------------
|
| 426 |
+
# model building blocks
|
| 427 |
+
# ---------------------------------------------------------------------------
|
| 428 |
+
|
| 429 |
+
class RMSNorm(nn.Module):
|
| 430 |
+
def __init__(self, dim: int, eps: float = 1e-6, affine: bool = True):
|
| 431 |
+
super().__init__()
|
| 432 |
+
self.eps = eps
|
| 433 |
+
self.weight = nn.Parameter(torch.ones(dim)) if affine else None
|
| 434 |
+
|
| 435 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 436 |
+
dtype = x.dtype
|
| 437 |
+
x32 = x.float()
|
| 438 |
+
y = x32 * torch.rsqrt(x32.pow(2).mean(-1, keepdim=True) + self.eps)
|
| 439 |
+
if self.weight is not None:
|
| 440 |
+
y = y * self.weight.float()
|
| 441 |
+
return y.to(dtype)
|
| 442 |
+
|
| 443 |
+
|
| 444 |
+
def _rope_freqs(head_dim: int, theta: float) -> torch.Tensor:
|
| 445 |
+
n = head_dim // 4 # rotation pairs per axis
|
| 446 |
+
return 1.0 / (theta ** (torch.arange(n, dtype=torch.float64) / n))
|
| 447 |
+
|
| 448 |
+
|
| 449 |
+
def build_rope(n_prefix: int, grid: int, head_dim: int, theta: float,
|
| 450 |
+
stride: int = 1) -> Tuple[torch.Tensor, torch.Tensor]:
|
| 451 |
+
"""cos/sin tables for 2D axial RoPE. Image token (i,j) at a level with
|
| 452 |
+
`stride` pixels per token sits at pixel-space centre
|
| 453 |
+
(i*stride + (stride-1)/2, j*stride + (stride-1)/2); prefix angle 0."""
|
| 454 |
+
f = _rope_freqs(head_dim, theta)
|
| 455 |
+
ii, jj = torch.meshgrid(torch.arange(grid, dtype=torch.float64),
|
| 456 |
+
torch.arange(grid, dtype=torch.float64), indexing="ij")
|
| 457 |
+
off = (stride - 1) / 2.0
|
| 458 |
+
rows = (ii.reshape(-1) * stride + off)[:, None] * f[None]
|
| 459 |
+
cols = (jj.reshape(-1) * stride + off)[:, None] * f[None]
|
| 460 |
+
ang = torch.cat([rows, cols], 1) # (grid^2, head_dim/2)
|
| 461 |
+
ang = torch.cat([torch.zeros(n_prefix, ang.shape[1], dtype=ang.dtype), ang], 0)
|
| 462 |
+
return ang.cos().float(), ang.sin().float()
|
| 463 |
+
|
| 464 |
+
|
| 465 |
+
def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
|
| 466 |
+
# x: (B,H,N,hd); cos/sin: (N, hd/2)
|
| 467 |
+
x1, x2 = x.float().chunk(2, dim=-1)
|
| 468 |
+
out = torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1)
|
| 469 |
+
return out.to(x.dtype)
|
| 470 |
+
|
| 471 |
+
|
| 472 |
+
def fourier_features(x: torch.Tensor, n_freqs: int) -> torch.Tensor:
|
| 473 |
+
"""x: (...,C) roughly in [-1,1] -> (..., C*2*n_freqs) sin/cos features."""
|
| 474 |
+
freqs = (2.0 ** torch.arange(n_freqs, dtype=torch.float32)) * math.pi
|
| 475 |
+
ang = x.float().unsqueeze(-1) * freqs
|
| 476 |
+
return torch.cat([ang.sin(), ang.cos()], dim=-1).flatten(-2)
|
| 477 |
+
|
| 478 |
+
|
| 479 |
+
class Attention(nn.Module):
|
| 480 |
+
def __init__(self, dim: int, n_heads: int):
|
| 481 |
+
super().__init__()
|
| 482 |
+
self.n_heads = n_heads
|
| 483 |
+
self.head_dim = dim // n_heads
|
| 484 |
+
self.qkv = nn.Linear(dim, 3 * dim, bias=False)
|
| 485 |
+
self.proj = nn.Linear(dim, dim, bias=False)
|
| 486 |
+
self.q_norm = RMSNorm(self.head_dim)
|
| 487 |
+
self.k_norm = RMSNorm(self.head_dim)
|
| 488 |
+
|
| 489 |
+
def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
|
| 490 |
+
B, N, D = x.shape
|
| 491 |
+
qkv = self.qkv(x).view(B, N, 3, self.n_heads, self.head_dim).permute(2, 0, 3, 1, 4)
|
| 492 |
+
q, k, v = qkv.unbind(0)
|
| 493 |
+
q = apply_rope(self.q_norm(q), cos, sin)
|
| 494 |
+
k = apply_rope(self.k_norm(k), cos, sin)
|
| 495 |
+
o = F.scaled_dot_product_attention(q, k, v)
|
| 496 |
+
return self.proj(o.transpose(1, 2).reshape(B, N, D))
|
| 497 |
+
|
| 498 |
+
|
| 499 |
+
class ConvSwiGLU(nn.Module):
|
| 500 |
+
"""SwiGLU whose gate branch gets a residual depthwise 3x3 conv over the
|
| 501 |
+
image grid (local pixel structure)."""
|
| 502 |
+
|
| 503 |
+
def __init__(self, dim: int, hidden: int, conv: bool = True):
|
| 504 |
+
super().__init__()
|
| 505 |
+
self.w12 = nn.Linear(dim, 2 * hidden, bias=False)
|
| 506 |
+
self.w3 = nn.Linear(hidden, dim, bias=False)
|
| 507 |
+
self.dw = nn.Conv2d(hidden, hidden, 3, padding=1, groups=hidden, bias=True) if conv else None
|
| 508 |
+
|
| 509 |
+
def forward(self, x: torch.Tensor, n_prefix: int, hw: Tuple[int, int]) -> torch.Tensor:
|
| 510 |
+
a, g = self.w12(x).chunk(2, dim=-1)
|
| 511 |
+
if self.dw is not None:
|
| 512 |
+
B, _, C = g.shape
|
| 513 |
+
H, W = hw
|
| 514 |
+
g_pre, g_img = g[:, :n_prefix], g[:, n_prefix:]
|
| 515 |
+
conv = self.dw(g_img.transpose(1, 2).reshape(B, C, H, W)).flatten(2).transpose(1, 2)
|
| 516 |
+
g = torch.cat([g_pre, g_img + conv.to(g_img.dtype)], dim=1)
|
| 517 |
+
return self.w3(a * F.silu(g))
|
| 518 |
+
|
| 519 |
+
|
| 520 |
+
class Block(nn.Module):
|
| 521 |
+
def __init__(self, cfg: TroutConfig):
|
| 522 |
+
super().__init__()
|
| 523 |
+
self.norm1 = RMSNorm(cfg.dim, affine=False)
|
| 524 |
+
self.attn = Attention(cfg.dim, cfg.n_heads)
|
| 525 |
+
self.norm2 = RMSNorm(cfg.dim, affine=False)
|
| 526 |
+
self.mlp = ConvSwiGLU(cfg.dim, cfg.mlp_hidden, conv=cfg.conv_ffn)
|
| 527 |
+
# per-block offsets added to the shared adaLN-single modulation
|
| 528 |
+
# (shift/scale/gate) x (attn/mlp)
|
| 529 |
+
self.mod_bias = nn.Parameter(torch.zeros(6, cfg.dim))
|
| 530 |
+
|
| 531 |
+
def forward(self, x, mod, cos, sin, n_prefix: int, hw: Tuple[int, int]):
|
| 532 |
+
m = (mod + self.mod_bias).unsqueeze(2) # (B,6,1,D)
|
| 533 |
+
sh1, sc1, g1, sh2, sc2, g2 = m.unbind(1)
|
| 534 |
+
h = self.norm1(x) * (1 + sc1) + sh1
|
| 535 |
+
x = x + g1 * self.attn(h, cos, sin)
|
| 536 |
+
h = self.norm2(x) * (1 + sc2) + sh2
|
| 537 |
+
x = x + g2 * self.mlp(h, n_prefix, hw)
|
| 538 |
+
return x
|
| 539 |
+
|
| 540 |
+
|
| 541 |
+
# ---------------------------------------------------------------------------
|
| 542 |
+
# the model
|
| 543 |
+
# ---------------------------------------------------------------------------
|
| 544 |
+
|
| 545 |
+
class TroutModel(nn.Module):
|
| 546 |
+
"""Dual-resolution (hourglass) masked-diffusion transformer denoiser."""
|
| 547 |
+
|
| 548 |
+
def __init__(self, cfg: TroutConfig, palette_lab: Optional[torch.Tensor] = None,
|
| 549 |
+
palette_rgb: Optional[torch.Tensor] = None):
|
| 550 |
+
super().__init__()
|
| 551 |
+
cfg.validate()
|
| 552 |
+
self.cfg = cfg
|
| 553 |
+
K, D, G = cfg.vocab_size, cfg.dim, cfg.grid_size
|
| 554 |
+
self.mask_id = cfg.mask_id
|
| 555 |
+
if palette_lab is None:
|
| 556 |
+
palette_lab = torch.zeros(K, 3)
|
| 557 |
+
if palette_rgb is None:
|
| 558 |
+
palette_rgb = torch.zeros(K, 3, dtype=torch.uint8)
|
| 559 |
+
assert palette_lab.shape == (K, 3), f"palette has {palette_lab.shape[0]} colors, config says {K}"
|
| 560 |
+
# the palette travels with the checkpoint
|
| 561 |
+
self.register_buffer("palette_lab", palette_lab.float().clone())
|
| 562 |
+
self.register_buffer("palette_rgb", palette_rgb.to(torch.uint8).clone())
|
| 563 |
+
|
| 564 |
+
# ---- token embeddings (color-grounded) ----
|
| 565 |
+
self.tok_emb = nn.Embedding(K + 1, D) # +1 for MASK
|
| 566 |
+
n_feat = 3 + 3 * 2 * cfg.color_freqs + 1
|
| 567 |
+
self.color_mlp = nn.Sequential(nn.Linear(n_feat, D), nn.SiLU(), nn.Linear(D, D))
|
| 568 |
+
self.pos_emb = nn.Parameter(torch.zeros(1, G * G, D))
|
| 569 |
+
self.sc_proj = nn.Linear(K, D, bias=False) if cfg.self_cond else None
|
| 570 |
+
# ---- text conditioning ----
|
| 571 |
+
self.text_norm = RMSNorm(cfg.text_dim)
|
| 572 |
+
self.text_proj = nn.Linear(cfg.text_dim, D)
|
| 573 |
+
self.null_text = nn.Parameter(torch.zeros(1, cfg.text_len, D))
|
| 574 |
+
self.pool_mlp = nn.Sequential(RMSNorm(cfg.text_dim), nn.Linear(cfg.text_dim, D), nn.SiLU(), nn.Linear(D, D))
|
| 575 |
+
self.null_cond = nn.Parameter(torch.zeros(1, D))
|
| 576 |
+
self.ratio_mlp = nn.Sequential(nn.Linear(2 * 8, D), nn.SiLU(), nn.Linear(D, D))
|
| 577 |
+
self.registers = nn.Parameter(torch.zeros(1, cfg.n_registers, D))
|
| 578 |
+
self.t_block = nn.Sequential(nn.SiLU(), nn.Linear(D, 6 * D))
|
| 579 |
+
# ---- hourglass backbone ----
|
| 580 |
+
p = cfg.patch
|
| 581 |
+
self.blocks_in = nn.ModuleList([Block(cfg) for _ in range(cfg.depth_in)])
|
| 582 |
+
self.down = nn.Sequential(RMSNorm(p * p * D), nn.Linear(p * p * D, D))
|
| 583 |
+
self.blocks_core = nn.ModuleList([Block(cfg) for _ in range(cfg.depth_core)])
|
| 584 |
+
self.up = nn.Linear(D, p * p * D)
|
| 585 |
+
self.blocks_out = nn.ModuleList([Block(cfg) for _ in range(cfg.depth_out)])
|
| 586 |
+
self.norm_out = RMSNorm(D, affine=False)
|
| 587 |
+
self.final_mod = nn.Sequential(nn.SiLU(), nn.Linear(D, 2 * D))
|
| 588 |
+
self.head = nn.Linear(D, K)
|
| 589 |
+
# ---- fixed tables ----
|
| 590 |
+
n_pre = cfg.n_registers + cfg.text_len
|
| 591 |
+
self.n_prefix = n_pre
|
| 592 |
+
cf, sf = build_rope(n_pre, G, cfg.head_dim, cfg.rope_theta, stride=1)
|
| 593 |
+
self.register_buffer("rope_cos", cf, persistent=False)
|
| 594 |
+
self.register_buffer("rope_sin", sf, persistent=False)
|
| 595 |
+
cc, sc = build_rope(n_pre, G // cfg.patch, cfg.head_dim, cfg.rope_theta, stride=cfg.patch)
|
| 596 |
+
self.register_buffer("rope_cos_core", cc, persistent=False)
|
| 597 |
+
self.register_buffer("rope_sin_core", sc, persistent=False)
|
| 598 |
+
self.register_buffer("color_feats", self._color_features(self.palette_lab), persistent=False)
|
| 599 |
+
self.grad_checkpointing = False
|
| 600 |
+
self._init_weights()
|
| 601 |
+
|
| 602 |
+
# ------------------------------------------------------------------
|
| 603 |
+
def _color_features(self, lab: torch.Tensor) -> torch.Tensor:
|
| 604 |
+
x = torch.stack([lab[:, 0] * 2 - 1, lab[:, 1] / 0.4, lab[:, 2] / 0.4], -1)
|
| 605 |
+
alpha = torch.ones(lab.shape[0], 1)
|
| 606 |
+
alpha[0] = 0.0 # token 0 = transparent
|
| 607 |
+
x = x * alpha
|
| 608 |
+
return torch.cat([x, fourier_features(x, self.cfg.color_freqs), alpha], -1)
|
| 609 |
+
|
| 610 |
+
def set_palette(self, palette_lab: torch.Tensor, palette_rgb: torch.Tensor) -> None:
|
| 611 |
+
self.palette_lab.copy_(palette_lab.float())
|
| 612 |
+
self.palette_rgb.copy_(palette_rgb.to(torch.uint8))
|
| 613 |
+
self.color_feats = self._color_features(self.palette_lab.detach().cpu()).to(self.palette_lab.device)
|
| 614 |
+
|
| 615 |
+
def _load_from_state_dict(self, state_dict, prefix, *args, **kwargs):
|
| 616 |
+
super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
|
| 617 |
+
self.color_feats = self._color_features(self.palette_lab.detach().cpu()).to(self.palette_lab.device)
|
| 618 |
+
|
| 619 |
+
def _init_weights(self) -> None:
|
| 620 |
+
def init(m):
|
| 621 |
+
if isinstance(m, nn.Linear):
|
| 622 |
+
nn.init.trunc_normal_(m.weight, std=0.02)
|
| 623 |
+
if m.bias is not None:
|
| 624 |
+
nn.init.zeros_(m.bias)
|
| 625 |
+
elif isinstance(m, nn.Embedding):
|
| 626 |
+
nn.init.normal_(m.weight, std=0.02)
|
| 627 |
+
self.apply(init)
|
| 628 |
+
nn.init.trunc_normal_(self.pos_emb, std=0.02)
|
| 629 |
+
nn.init.normal_(self.null_text, std=0.02)
|
| 630 |
+
nn.init.normal_(self.null_cond, std=0.02)
|
| 631 |
+
nn.init.normal_(self.registers, std=0.02)
|
| 632 |
+
# zero-init: conditioning projections, gates, head, self-cond, up path
|
| 633 |
+
for m in [self.t_block[-1], self.final_mod[-1], self.head]:
|
| 634 |
+
nn.init.zeros_(m.weight)
|
| 635 |
+
nn.init.zeros_(m.bias)
|
| 636 |
+
if self.sc_proj is not None:
|
| 637 |
+
nn.init.zeros_(self.sc_proj.weight)
|
| 638 |
+
nn.init.zeros_(self.up.weight)
|
| 639 |
+
nn.init.zeros_(self.up.bias)
|
| 640 |
+
for blk in list(self.blocks_in) + list(self.blocks_core) + list(self.blocks_out):
|
| 641 |
+
if blk.mlp.dw is not None:
|
| 642 |
+
nn.init.zeros_(blk.mlp.dw.weight)
|
| 643 |
+
nn.init.zeros_(blk.mlp.dw.bias)
|
| 644 |
+
|
| 645 |
+
# ------------------------------------------------------------------
|
| 646 |
+
def _merge(self, h: torch.Tensor) -> torch.Tensor:
|
| 647 |
+
B, N, D = h.shape
|
| 648 |
+
G, p = self.cfg.grid_size, self.cfg.patch
|
| 649 |
+
g = G // p
|
| 650 |
+
h = h.view(B, g, p, g, p, D).permute(0, 1, 3, 2, 4, 5).reshape(B, g * g, p * p * D)
|
| 651 |
+
return self.down(h)
|
| 652 |
+
|
| 653 |
+
def _split(self, h: torch.Tensor) -> torch.Tensor:
|
| 654 |
+
B = h.shape[0]
|
| 655 |
+
G, p, D = self.cfg.grid_size, self.cfg.patch, self.cfg.dim
|
| 656 |
+
g = G // p
|
| 657 |
+
h = self.up(h).view(B, g, g, p, p, D).permute(0, 1, 3, 2, 4, 5).reshape(B, G * G, D)
|
| 658 |
+
return h
|
| 659 |
+
|
| 660 |
+
def _run(self, blk: Block, xs, mod, cos, sin, n_pre: int, hw):
|
| 661 |
+
if self.grad_checkpointing and self.training and torch.is_grad_enabled():
|
| 662 |
+
return torch.utils.checkpoint.checkpoint(blk, xs, mod, cos, sin, n_pre, hw,
|
| 663 |
+
use_reentrant=False)
|
| 664 |
+
return blk(xs, mod, cos, sin, n_pre, hw)
|
| 665 |
+
|
| 666 |
+
def embed_table(self) -> torch.Tensor:
|
| 667 |
+
K = self.cfg.vocab_size
|
| 668 |
+
w = self.tok_emb.weight
|
| 669 |
+
colors = w[:K] + self.color_mlp(self.color_feats).to(w.dtype)
|
| 670 |
+
return torch.cat([colors, w[K:]], 0)
|
| 671 |
+
|
| 672 |
+
def forward(self, x: torch.Tensor, text_states: torch.Tensor, text_pooled: torch.Tensor,
|
| 673 |
+
mask_ratio: Optional[torch.Tensor] = None,
|
| 674 |
+
sc_probs: Optional[torch.Tensor] = None,
|
| 675 |
+
drop_text: Optional[torch.Tensor] = None) -> torch.Tensor:
|
| 676 |
+
"""
|
| 677 |
+
x: (B,N) long palette ids or mask_id
|
| 678 |
+
text_states: (B,L,text_dim) frozen text-encoder token states
|
| 679 |
+
text_pooled: (B,text_dim) pooled text embedding
|
| 680 |
+
mask_ratio: (B,) fraction masked (computed from x if None)
|
| 681 |
+
sc_probs: (B,N,K) previous-step predicted distribution or None
|
| 682 |
+
drop_text: (B,) bool -> use learned null condition (CFG)
|
| 683 |
+
returns logits (B,N,K) float32
|
| 684 |
+
"""
|
| 685 |
+
cfg = self.cfg
|
| 686 |
+
B, N = x.shape
|
| 687 |
+
D, G = cfg.dim, cfg.grid_size
|
| 688 |
+
is_mask = x == self.mask_id
|
| 689 |
+
if mask_ratio is None:
|
| 690 |
+
mask_ratio = is_mask.float().mean(1)
|
| 691 |
+
# ---- image tokens ----
|
| 692 |
+
h = F.embedding(x, self.embed_table()).float() + self.pos_emb
|
| 693 |
+
if self.sc_proj is not None and sc_probs is not None:
|
| 694 |
+
h = h + self.sc_proj(sc_probs.to(self.sc_proj.weight.dtype)).float() * is_mask.unsqueeze(-1)
|
| 695 |
+
# ---- text tokens + global conditioning ----
|
| 696 |
+
t = self.text_proj(self.text_norm(text_states.float())).float()
|
| 697 |
+
c = self.pool_mlp(text_pooled.float()).float()
|
| 698 |
+
if drop_text is not None:
|
| 699 |
+
t = torch.where(drop_text.view(B, 1, 1), self.null_text, t)
|
| 700 |
+
c = torch.where(drop_text.view(B, 1), self.null_cond, c)
|
| 701 |
+
r = mask_ratio.float().view(B, 1) * 2 - 1
|
| 702 |
+
g = c + self.ratio_mlp(fourier_features(r, 8).view(B, -1)).float()
|
| 703 |
+
mod = self.t_block(g).float().view(B, 6, D)
|
| 704 |
+
prefix = torch.cat([self.registers.expand(B, -1, -1), t], 1)
|
| 705 |
+
n_pre = prefix.shape[1]
|
| 706 |
+
xs = torch.cat([prefix, h], 1)
|
| 707 |
+
# ---- pixel level (in) ----
|
| 708 |
+
for blk in self.blocks_in:
|
| 709 |
+
xs = self._run(blk, xs, mod, self.rope_cos, self.rope_sin, n_pre, (G, G))
|
| 710 |
+
# ---- patch level (core) ----
|
| 711 |
+
pre, img = xs[:, :n_pre], xs[:, n_pre:]
|
| 712 |
+
skip = img
|
| 713 |
+
xs = torch.cat([pre, self._merge(img).float()], 1)
|
| 714 |
+
gc = G // cfg.patch
|
| 715 |
+
for blk in self.blocks_core:
|
| 716 |
+
xs = self._run(blk, xs, mod, self.rope_cos_core, self.rope_sin_core, n_pre, (gc, gc))
|
| 717 |
+
pre, core = xs[:, :n_pre], xs[:, n_pre:]
|
| 718 |
+
xs = torch.cat([pre, skip + self._split(core).float()], 1)
|
| 719 |
+
# ---- pixel level (out) ----
|
| 720 |
+
for blk in self.blocks_out:
|
| 721 |
+
xs = self._run(blk, xs, mod, self.rope_cos, self.rope_sin, n_pre, (G, G))
|
| 722 |
+
img = xs[:, n_pre:]
|
| 723 |
+
shift, scale = self.final_mod(g).float().unsqueeze(1).chunk(2, dim=-1)
|
| 724 |
+
img = self.norm_out(img) * (1 + scale) + shift
|
| 725 |
+
return self.head(img).float()
|
| 726 |
+
|
| 727 |
+
def num_params(self, trainable_only: bool = True) -> int:
|
| 728 |
+
return sum(p.numel() for p in self.parameters() if p.requires_grad or not trainable_only)
|
| 729 |
+
|
| 730 |
+
|
| 731 |
+
# ---------------------------------------------------------------------------
|
| 732 |
+
# sampler: parallel confidence-ordered unmasking (MaskGIT-style) with CFG,
|
| 733 |
+
# Gumbel-annealed choice, spatial NMS and self-conditioning feedback
|
| 734 |
+
# ---------------------------------------------------------------------------
|
| 735 |
+
|
| 736 |
+
def filter_logits(logits: torch.Tensor, top_k: int = 0, top_p: float = 1.0,
|
| 737 |
+
min_p: float = 0.0) -> torch.Tensor:
|
| 738 |
+
if top_k and top_k > 0 and top_k < logits.shape[-1]:
|
| 739 |
+
kth = torch.topk(logits, top_k, dim=-1).values[..., -1:]
|
| 740 |
+
logits = logits.masked_fill(logits < kth, -math.inf)
|
| 741 |
+
if min_p and min_p > 0:
|
| 742 |
+
p = logits.softmax(-1)
|
| 743 |
+
logits = logits.masked_fill(p < min_p * p.amax(-1, keepdim=True), -math.inf)
|
| 744 |
+
if top_p is not None and top_p < 1.0:
|
| 745 |
+
sorted_l, idx = torch.sort(logits, dim=-1, descending=True)
|
| 746 |
+
sp = sorted_l.softmax(-1)
|
| 747 |
+
remove = (sp.cumsum(-1) - sp) > top_p
|
| 748 |
+
sorted_l = sorted_l.masked_fill(remove, -math.inf)
|
| 749 |
+
logits = torch.full_like(logits, -math.inf).scatter(-1, idx, sorted_l)
|
| 750 |
+
return logits
|
| 751 |
+
|
| 752 |
+
|
| 753 |
+
def cfg_weight(scale: float, schedule: str, frac: float) -> float:
|
| 754 |
+
if schedule == "constant":
|
| 755 |
+
return scale
|
| 756 |
+
if schedule == "linear":
|
| 757 |
+
return 1.0 + (scale - 1.0) * frac
|
| 758 |
+
if schedule == "cosine":
|
| 759 |
+
return 1.0 + (scale - 1.0) * (1 - math.cos(math.pi * frac)) / 2
|
| 760 |
+
raise ValueError(f"unknown cfg schedule {schedule!r}")
|
| 761 |
+
|
| 762 |
+
|
| 763 |
+
def _gumbel(shape, device, generator):
|
| 764 |
+
u = torch.rand(shape, device=device, generator=generator).clamp_(1e-10, 1 - 1e-10)
|
| 765 |
+
return -torch.log(-torch.log(u))
|
| 766 |
+
|
| 767 |
+
|
| 768 |
+
def _nms_eligible(score: torch.Tensor, masked: torch.Tensor, certain: torch.Tensor,
|
| 769 |
+
grid: int) -> torch.Tensor:
|
| 770 |
+
B, N = score.shape
|
| 771 |
+
cand = masked & ~certain
|
| 772 |
+
s = torch.where(cand, score, torch.full_like(score, -math.inf)).view(B, 1, grid, grid)
|
| 773 |
+
local_max = F.max_pool2d(s, 3, stride=1, padding=1).view(B, N)
|
| 774 |
+
return certain | (cand & (score >= local_max))
|
| 775 |
+
|
| 776 |
+
|
| 777 |
+
@torch.no_grad()
|
| 778 |
+
def sample(model: TroutModel, text_states: torch.Tensor, text_pooled: torch.Tensor,
|
| 779 |
+
steps: int = 16, cfg_scale: float = 3.0, cfg_schedule: str = "constant",
|
| 780 |
+
temperature: float = 1.0, temperature_end: Optional[float] = None,
|
| 781 |
+
top_k: int = 0, top_p: float = 1.0, min_p: float = 0.0,
|
| 782 |
+
choice_temperature: float = 2.0, spatial_nms: bool = True,
|
| 783 |
+
nms_threshold: float = 0.9, polish_steps: int = 0, polish_frac: float = 0.08,
|
| 784 |
+
init_tokens: Optional[torch.Tensor] = None, keep_mask: Optional[torch.Tensor] = None,
|
| 785 |
+
self_cond: bool = True, generator: Optional[torch.Generator] = None,
|
| 786 |
+
return_history: bool = False):
|
| 787 |
+
"""Generate palette-token canvases (B, N); inpainting via init_tokens+keep_mask."""
|
| 788 |
+
cfg = model.cfg
|
| 789 |
+
device = text_pooled.device
|
| 790 |
+
B, N, K, M, G = text_pooled.shape[0], cfg.num_tokens, cfg.vocab_size, cfg.mask_id, cfg.grid_size
|
| 791 |
+
if init_tokens is not None and keep_mask is not None:
|
| 792 |
+
init_tokens = init_tokens.to(device).long()
|
| 793 |
+
x = torch.where(keep_mask.to(device).bool(), init_tokens, torch.full_like(init_tokens, M))
|
| 794 |
+
else:
|
| 795 |
+
x = torch.full((B, N), M, dtype=torch.long, device=device)
|
| 796 |
+
masked = x == M
|
| 797 |
+
conf_map = torch.where(masked, torch.zeros(B, N, device=device),
|
| 798 |
+
torch.full((B, N), math.inf, device=device))
|
| 799 |
+
n_start = masked.sum(1)
|
| 800 |
+
use_cfg = cfg_scale is not None and abs(cfg_scale - 1.0) > 1e-6
|
| 801 |
+
if use_cfg:
|
| 802 |
+
ts2 = torch.cat([text_states, text_states])
|
| 803 |
+
tp2 = torch.cat([text_pooled, text_pooled])
|
| 804 |
+
drop2 = torch.cat([torch.zeros(B, dtype=torch.bool, device=device),
|
| 805 |
+
torch.ones(B, dtype=torch.bool, device=device)])
|
| 806 |
+
sc = torch.zeros(B, N, K, device=device) if (cfg.self_cond and self_cond) else None
|
| 807 |
+
history = []
|
| 808 |
+
|
| 809 |
+
def predict(x_in, sc_in, frac):
|
| 810 |
+
ratio = (x_in == M).float().mean(1)
|
| 811 |
+
if use_cfg:
|
| 812 |
+
sc2 = torch.cat([sc_in, sc_in]) if sc_in is not None else None
|
| 813 |
+
out = model(torch.cat([x_in, x_in]), ts2, tp2, torch.cat([ratio, ratio]),
|
| 814 |
+
sc2, drop2).float()
|
| 815 |
+
lc, lu = out.chunk(2)
|
| 816 |
+
w = cfg_weight(cfg_scale, cfg_schedule, frac)
|
| 817 |
+
return lu + w * (lc - lu)
|
| 818 |
+
return model(x_in, text_states, text_pooled, ratio, sc_in, None).float()
|
| 819 |
+
|
| 820 |
+
for k in range(steps):
|
| 821 |
+
if not masked.any():
|
| 822 |
+
break
|
| 823 |
+
frac = (k + 1) / steps
|
| 824 |
+
logits = predict(x, sc, frac)
|
| 825 |
+
probs_full = logits.softmax(-1)
|
| 826 |
+
T = temperature if temperature_end is None else temperature + (temperature_end - temperature) * (k / max(1, steps - 1))
|
| 827 |
+
if T <= 0:
|
| 828 |
+
sampled = logits.argmax(-1)
|
| 829 |
+
p_sel = probs_full.gather(-1, sampled.unsqueeze(-1)).squeeze(-1)
|
| 830 |
+
else:
|
| 831 |
+
probs = filter_logits(logits / T, top_k, top_p, min_p).softmax(-1)
|
| 832 |
+
sampled = torch.multinomial(probs.view(-1, K), 1, generator=generator).view(B, N)
|
| 833 |
+
p_sel = probs.gather(-1, sampled.unsqueeze(-1)).squeeze(-1)
|
| 834 |
+
n_masked = masked.sum(1)
|
| 835 |
+
remaining = torch.floor(n_start.float() * math.cos(math.pi / 2 * frac)).long()
|
| 836 |
+
if k == steps - 1:
|
| 837 |
+
remaining = torch.zeros_like(remaining)
|
| 838 |
+
n_commit = torch.minimum((n_masked - remaining).clamp(min=1), n_masked)
|
| 839 |
+
score = torch.log(p_sel.clamp(min=1e-12))
|
| 840 |
+
if choice_temperature > 0: # Gumbel-annealed choice (MaskGIT)
|
| 841 |
+
score = score + choice_temperature * (1.0 - frac) * _gumbel(score.shape, device, generator)
|
| 842 |
+
priority = score
|
| 843 |
+
if spatial_nms: # don't commit two uncertain neighbours in one step
|
| 844 |
+
elig = _nms_eligible(score, masked, p_sel >= nms_threshold, G)
|
| 845 |
+
priority = score + elig.float() * 1e4
|
| 846 |
+
priority = torch.where(masked, priority, torch.full_like(priority, -math.inf))
|
| 847 |
+
ranks = priority.argsort(1, descending=True).argsort(1)
|
| 848 |
+
commit = (ranks < n_commit.unsqueeze(1)) & masked
|
| 849 |
+
x = torch.where(commit, sampled, x)
|
| 850 |
+
conf_map = torch.where(commit, p_sel, conf_map)
|
| 851 |
+
masked = masked & ~commit
|
| 852 |
+
if sc is not None:
|
| 853 |
+
sc = probs_full
|
| 854 |
+
if return_history:
|
| 855 |
+
history.append(x.clone())
|
| 856 |
+
|
| 857 |
+
# ---- optional polish: re-mask the least confident committed pixels ----
|
| 858 |
+
n_re = max(1, int(round(polish_frac * N)))
|
| 859 |
+
for _ in range(polish_steps):
|
| 860 |
+
low = torch.topk(-conf_map, n_re, dim=1).indices
|
| 861 |
+
remask = torch.zeros(B, N, dtype=torch.bool, device=device).scatter_(1, low, True)
|
| 862 |
+
remask &= torch.isfinite(conf_map)
|
| 863 |
+
x_in = torch.where(remask, torch.full_like(x, M), x)
|
| 864 |
+
logits = predict(x_in, sc, 1.0)
|
| 865 |
+
probs_full = logits.softmax(-1)
|
| 866 |
+
new = logits.argmax(-1)
|
| 867 |
+
x = torch.where(remask, new, x)
|
| 868 |
+
conf_map = torch.where(remask, probs_full.gather(-1, new.unsqueeze(-1)).squeeze(-1), conf_map)
|
| 869 |
+
if sc is not None:
|
| 870 |
+
sc = probs_full
|
| 871 |
+
if return_history:
|
| 872 |
+
history.append(x.clone())
|
| 873 |
+
|
| 874 |
+
if return_history:
|
| 875 |
+
return x, history
|
| 876 |
+
return x
|
| 877 |
+
|
| 878 |
+
|
| 879 |
+
# ---------------------------------------------------------------------------
|
| 880 |
+
# checkpoint helpers (same schema semantics as the Tessera distribution:
|
| 881 |
+
# ema weights + model_config + palette + text-encoder id + step)
|
| 882 |
+
# ---------------------------------------------------------------------------
|
| 883 |
+
|
| 884 |
+
def save_checkpoint(path: str, model: TroutModel, palette: Palette, step: int,
|
| 885 |
+
ema_state: Optional[Dict[str, torch.Tensor]] = None,
|
| 886 |
+
raw_state: Optional[Dict[str, torch.Tensor]] = None,
|
| 887 |
+
text_encoder: str = DEFAULT_TEXT_ENCODER, half: bool = False) -> None:
|
| 888 |
+
def prep(sd):
|
| 889 |
+
out = {}
|
| 890 |
+
for k, v in sd.items():
|
| 891 |
+
out[k] = (v.half() if half and v.is_floating_point() else v).cpu()
|
| 892 |
+
return out
|
| 893 |
+
ckpt = {"model_config": model.cfg.to_dict(), "palette": palette.to_dict(),
|
| 894 |
+
"text_encoder": text_encoder, "step": step}
|
| 895 |
+
if ema_state is not None:
|
| 896 |
+
ckpt["ema"] = prep(ema_state)
|
| 897 |
+
if raw_state is not None:
|
| 898 |
+
ckpt["model"] = prep(raw_state)
|
| 899 |
+
torch.save(ckpt, path)
|
| 900 |
+
|
| 901 |
+
|
| 902 |
+
def load_model_from_checkpoint(path: str, device="cpu", use_ema: bool = True):
|
| 903 |
+
ckpt = torch.load(path, map_location="cpu", weights_only=False)
|
| 904 |
+
mcfg = TroutConfig.from_dict(ckpt["model_config"])
|
| 905 |
+
palette = Palette.from_dict(ckpt["palette"])
|
| 906 |
+
model = TroutModel(mcfg, torch.from_numpy(palette.lab), torch.from_numpy(palette.colors))
|
| 907 |
+
key = "ema" if (use_ema and "ema" in ckpt) else "model"
|
| 908 |
+
sd = {k: v.float() if v.is_floating_point() else v for k, v in ckpt[key].items()}
|
| 909 |
+
model.load_state_dict(sd)
|
| 910 |
+
model.to(device).eval().requires_grad_(False)
|
| 911 |
+
return model, palette, ckpt
|
| 912 |
+
|
| 913 |
+
|
| 914 |
+
# ---------------------------------------------------------------------------
|
| 915 |
+
# visualization helpers
|
| 916 |
+
# ---------------------------------------------------------------------------
|
| 917 |
+
|
| 918 |
+
def tokens_to_images(tokens, palette: Palette, grid: int = 16) -> List:
|
| 919 |
+
from PIL import Image
|
| 920 |
+
if isinstance(tokens, torch.Tensor):
|
| 921 |
+
tokens = tokens.detach().cpu().numpy()
|
| 922 |
+
toks = np.asarray(tokens).reshape(-1, grid * grid)
|
| 923 |
+
imgs = []
|
| 924 |
+
for row in toks:
|
| 925 |
+
rgba = palette.tokens_to_rgba(row.reshape(grid, grid))
|
| 926 |
+
imgs.append(Image.fromarray(rgba, mode="RGBA"))
|
| 927 |
+
return imgs
|
| 928 |
+
|
| 929 |
+
|
| 930 |
+
def upscale(image, scale: int = 16, checker: bool = True):
|
| 931 |
+
"""Nearest-neighbour upscale with an optional checkerboard background."""
|
| 932 |
+
from PIL import Image
|
| 933 |
+
w, h = image.size
|
| 934 |
+
big = image.resize((w * scale, h * scale), Image.NEAREST)
|
| 935 |
+
if not checker:
|
| 936 |
+
return big
|
| 937 |
+
bg = Image.new("RGBA", big.size, (255, 255, 255, 255))
|
| 938 |
+
s = 4 * 2 # checker cell in big pixels
|
| 939 |
+
yy, xx = np.mgrid[0:big.size[1], 0:big.size[0]]
|
| 940 |
+
mask = (((yy // s) + (xx // s)) % 2 == 0)
|
| 941 |
+
light = Image.fromarray(np.where(mask[..., None], np.array([238, 238, 238, 255], np.uint8),
|
| 942 |
+
np.array([204, 204, 204, 255], np.uint8)), "RGBA")
|
| 943 |
+
light.alpha_composite(big)
|
| 944 |
+
return light
|
| 945 |
+
|
| 946 |
+
|
| 947 |
+
def make_grid(images, ncols: int = 8, scale: int = 8, checker: bool = True):
|
| 948 |
+
from PIL import Image
|
| 949 |
+
cells = [upscale(im, scale, checker) for im in images]
|
| 950 |
+
cw, ch = cells[0].size
|
| 951 |
+
rows = math.ceil(len(cells) / ncols)
|
| 952 |
+
out = Image.new("RGBA", (ncols * cw, rows * ch), (255, 255, 255, 255))
|
| 953 |
+
for i, c in enumerate(cells):
|
| 954 |
+
out.alpha_composite(c, ((i % ncols) * cw, (i // ncols) * ch))
|
| 955 |
+
return out
|
| 956 |
+
|
| 957 |
+
|
| 958 |
+
# ---------------------------------------------------------------------------
|
| 959 |
+
# frozen CLIP text tower wrapper
|
| 960 |
+
# ---------------------------------------------------------------------------
|
| 961 |
+
|
| 962 |
+
def load_tokenizer(name: str):
|
| 963 |
+
from transformers import AutoTokenizer
|
| 964 |
+
return AutoTokenizer.from_pretrained(name)
|
| 965 |
+
|
| 966 |
+
|
| 967 |
+
def tokenize_texts(tokenizer, texts: Sequence[str], text_len: int) -> Tuple[np.ndarray, np.ndarray]:
|
| 968 |
+
"""Fixed-length tokenization; pad with pad id; over-long prompts truncated
|
| 969 |
+
while keeping the final EOS token."""
|
| 970 |
+
eos = tokenizer.eos_token_id
|
| 971 |
+
pad = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else eos
|
| 972 |
+
n = len(texts)
|
| 973 |
+
out = np.full((n, text_len), pad, dtype=np.int32)
|
| 974 |
+
lengths = np.zeros(n, dtype=np.int32)
|
| 975 |
+
enc = tokenizer(list(texts), add_special_tokens=True, truncation=False)["input_ids"]
|
| 976 |
+
for i, ids in enumerate(enc):
|
| 977 |
+
lengths[i] = len(ids)
|
| 978 |
+
if len(ids) > text_len:
|
| 979 |
+
ids = list(ids[:text_len - 1]) + [eos]
|
| 980 |
+
out[i, :len(ids)] = ids
|
| 981 |
+
return out, lengths
|
| 982 |
+
|
| 983 |
+
|
| 984 |
+
class FrozenTextEncoder:
|
| 985 |
+
"""Wraps a Hugging Face CLIP text tower. Not an nn.Module on purpose so its
|
| 986 |
+
weights never end up in the trainable model, optimizer or checkpoints."""
|
| 987 |
+
|
| 988 |
+
def __init__(self, name: str = DEFAULT_TEXT_ENCODER, text_len: int = 24,
|
| 989 |
+
device="cpu", dtype: torch.dtype = torch.float32):
|
| 990 |
+
from transformers import CLIPTextModel
|
| 991 |
+
from transformers.utils import logging as hf_logging
|
| 992 |
+
self.name = name
|
| 993 |
+
self.text_len = text_len
|
| 994 |
+
self.device = torch.device(device)
|
| 995 |
+
self.dtype = dtype
|
| 996 |
+
self.tokenizer = load_tokenizer(name)
|
| 997 |
+
prev = hf_logging.get_verbosity()
|
| 998 |
+
hf_logging.set_verbosity_error()
|
| 999 |
+
try:
|
| 1000 |
+
model = CLIPTextModel.from_pretrained(name)
|
| 1001 |
+
finally:
|
| 1002 |
+
hf_logging.set_verbosity(prev)
|
| 1003 |
+
self.model = model.to(device=self.device, dtype=dtype).eval().requires_grad_(False)
|
| 1004 |
+
self.dim = int(model.config.hidden_size)
|
| 1005 |
+
|
| 1006 |
+
def tokenize(self, texts: Sequence[str]) -> torch.Tensor:
|
| 1007 |
+
ids, _ = tokenize_texts(self.tokenizer, list(texts), self.text_len)
|
| 1008 |
+
return torch.from_numpy(ids).long().to(self.device)
|
| 1009 |
+
|
| 1010 |
+
@torch.no_grad()
|
| 1011 |
+
def __call__(self, ids: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
|
| 1012 |
+
out = self.model(input_ids=ids.long().to(self.device))
|
| 1013 |
+
return out.last_hidden_state, out.pooler_output
|
| 1014 |
+
|
| 1015 |
+
@torch.no_grad()
|
| 1016 |
+
def encode(self, texts: Sequence[str]) -> Tuple[torch.Tensor, torch.Tensor]:
|
| 1017 |
+
return self(self.tokenize(texts))
|