model-collapse commited on
Commit
84fc8a9
·
verified ·
1 Parent(s): 587d3c6

pixmoji v2.2-fast + v2.1: weights, palette, inference code, model card

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ sample_sheet.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,122 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ library_name: pytorch
4
+ pipeline_tag: text-to-image
5
+ tags:
6
+ - pixel-art
7
+ - emoji
8
+ - 16x16
9
+ - maskgit
10
+ - text-to-image
11
+ datasets:
12
+ - Angais/pixel-art-atlas-1.4m
13
+ ---
14
+
15
+ # pixmoji — type a phrase, get 16×16 pixel-art emojis
16
+
17
+ pixmoji generates small **16×16 pixel-art sprites** from short text ("lovely sheep", "cat wearing a tiny crown",
18
+ "three blue apples") fast enough to run as you type. Every image is **generated** by the model; nothing is retrieved.
19
+
20
+ ![sample sheet: 30 prompts × 10 generated emojis](sample_sheet.png)
21
+
22
+ | file | what it is |
23
+ |---|---|
24
+ | `pixmoji-v2.2-fast.safetensors` | the fast model (43.6M params): single pass per step, used for all prompts without a count |
25
+ | `pixmoji-v2.1.safetensors` | the base model (43.6M), used for prompts with a count ≥ 2 ("three blue apples") |
26
+ | `palette.npy` | the 4096-colour palette (token → RGB; token 0 = transparent) |
27
+ | `pixmoji/` | self-contained inference code (model, sampler, reranker, selection) |
28
+
29
+ ## Quick start
30
+
31
+ ```bash
32
+ pip install torch transformers safetensors huggingface_hub pillow numpy
33
+ git clone https://huggingface.co/model-collapse/pixmoji && cd pixmoji
34
+ ```
35
+
36
+ ```python
37
+ import torch, pixmoji
38
+ from PIL import Image
39
+
40
+ gen = pixmoji.load(".") # or pixmoji.load() to download from the Hub; GPU if available
41
+ # on CPUs with AVX-512 BF16 (e.g. AMD Zen 4, Intel Sapphire Rapids), ~2x faster:
42
+ # gen = pixmoji.load(".", device="cpu", dtype=torch.bfloat16)
43
+ emojis = gen.generate_emojis("cat wearing a tiny crown", k=10, seed=0)
44
+ for i, e in enumerate(emojis): # e.rgba: (16, 16, 4) uint8, e.score: TinyCLIP text-image similarity
45
+ Image.fromarray(e.rgba, "RGBA").resize((128, 128), Image.NEAREST).save(f"emoji_{i}.png")
46
+ ```
47
+
48
+ `generate_emojis` returns up to `k` distinct emojis (near-duplicates are dropped, so a simple prompt can return fewer).
49
+
50
+ ## How it works
51
+
52
+ - **Generator:** a 12-layer, 512-wide bidirectional transformer over the 256 pixels of a 16×16 sprite, each pixel a token in a
53
+ 4096-colour palette (+ transparent). It fills the sprite in over 8–12 parallel decoding steps (MaskGIT-style), starting fully
54
+ masked and fixing the most confident pixels each step.
55
+ - **Conditioning:** the frozen text tower of [TinyCLIP-39M-PixelArt](https://huggingface.co/tomaarsen/TinyCLIP-39M-PixelArt)
56
+ (pooled embedding + per-word features) and a count token parsed from the prompt (one…four).
57
+ - **Selection:** 24 candidates are generated and reranked by TinyCLIP image–text similarity; up to 10 diverse ones are returned.
58
+ - **Two models, routed by prompt:**
59
+ - `pixmoji-v2.1` uses classifier-free guidance (two passes per step; guidance 5 for counts). Prompts with a count ≥ 2 use it.
60
+ - `pixmoji-v2.2-fast` is v2.1 **guidance-distilled** so one pass per step reproduces v2.1's guided output, trained on
61
+ states the sampler actually visits (192k recorded v2.1 sampling runs). With 8 steps it needs **one third of v2.1's
62
+ compute** at the same measured quality.
63
+
64
+ ## Training data
65
+
66
+ - [Angais/pixel-art-atlas-1.4m](https://huggingface.co/datasets/Angais/pixel-art-atlas-1.4m) (CC0): 1.4M 16×16 sprites with
67
+ captions, itself generated with Krea 2 and a pixel-art LoRA. Captions were corrected for object counts with Anthropic Claude
68
+ Haiku 5.5.
69
+ - ~38k additional sprites for counts and combinations, generated with Krea 2 Turbo + the
70
+ [Angais/Krea-2-PixelArt-16x16](https://huggingface.co/Angais/Krea-2-PixelArt-16x16) LoRA and filtered by Claude Haiku 5.5
71
+ (count check, "shows both parts" check) and TinyCLIP.
72
+ - Composites of 2–4 sprites for count training. Evaluation concepts were blocked from the generated training data.
73
+
74
+ pixmoji is a separately trained model; it does not contain Krea 2 weights.
75
+
76
+ ## Evaluation
77
+
78
+ Judges: **Anthropic Claude Sonnet 5.5** and **OpenAI GPT-6.1 Terra**, both via Amazon Bedrock, each image pair shown in both
79
+ orders. "Re-seed" = v2.1 run again with other seeds (identical quality by construction), the reference for noise.
80
+
81
+ | | v2.1 (base) |
82
+ |---|---|
83
+ | Novel combinations depicted (30 prompts × 10) | 0.56 |
84
+ | Count accuracy, 1–4 objects (Terra / Sonnet) | 0.76 / 0.67 |
85
+ | Count accuracy, 3–4 objects | 0.49–0.65 |
86
+ | Judged better than the real dataset sprite (parity = 0.50; Sonnet / Terra) | 0.48 / 0.25 |
87
+
88
+ pixmoji-v2.2-fast vs v2.1 (300 fresh single-object captions, all 10 shown emojis judged; positive = v2.1 preferred):
89
+
90
+ | | re-seed | **v2.2-fast, 8 steps** |
91
+ |---|---|---|
92
+ | Δ share of emojis depicting the caption (Sonnet / Terra) | +0.9 / −1.3 pts | +1.0 / −1.5 pts |
93
+ | set head-to-head margin (Sonnet / Terra) | +4.7 / +9.0 | +4.0 / +12.7 |
94
+ | novel combinations depicted (Sonnet / Terra; v2.1: 0.53 / 0.62) | 0.56 / 0.56 | 0.53 / 0.61 |
95
+
96
+ Also within noise in two independent 720-caption head-to-head rounds. Fewer steps (6), fewer candidates, smaller or pruned
97
+ students, int4 weights and low-rank layers were all measured to lose quality and are not used.
98
+
99
+ ## Speed (10 emojis per request)
100
+
101
+ | hardware | prompt without a count (v2.2-fast) | count prompt (v2.1) |
102
+ |---|---|---|
103
+ | NVIDIA A100 (shared with another job), fp16 | ~0.13 s | ~0.25–0.30 s |
104
+ | AMD EPYC 9R14, 64 cores, bf16, threads bound to one socket | 0.93 s | 3.3 s (with `cpu_fast="counts"`) |
105
+ | same CPU, 8 cores | 4.6 s | 12.4 s |
106
+
107
+ CPU tips: `OMP_PROC_BIND=close OMP_PLACES=cores`, keep threads within one socket, `dtype=torch.bfloat16` on CPUs with
108
+ AVX-512 BF16 (the fast model keeps its final colour layer in fp32, which the evaluation showed is needed).
109
+
110
+ ## Limitations
111
+
112
+ - 16×16 only; small details (an egg next to a hen, a helmet on a penguin) are often missing.
113
+ - Counts of 3–4 objects are right only about half to two thirds of the time.
114
+ - Spelling matters: misspelled words ("helicorpter") can produce unrelated objects.
115
+ - Simple prompts sometimes yield fewer than 10 distinct emojis.
116
+ - Images are AI-generated; the model can reflect biases of its training data and of the teacher models.
117
+
118
+ ## Intended use and license
119
+
120
+ Weights and code are released under **CC BY-NC 4.0** (non-commercial use, attribution required). Intended for demos, research
121
+ and personal projects such as chat emojis. Not intended for commercial use, impersonation, or generating hateful or sexual
122
+ content. Credit: Tom Aarsen (TinyCLIP-PixelArt, Pixel Atlas), Angais (Pixel Atlas dataset and Krea-2-PixelArt-16x16 LoRA).
palette.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bad6e5a7e34f663c0233119807a9840dd966b6c4f1b847f6fc166bf476fdf6dc
3
+ size 12419
pixmoji-v2.1.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "pixmoji-v2.1",
3
+ "config": {
4
+ "vocab": 4097,
5
+ "d_model": 512,
6
+ "n_layers": 12,
7
+ "n_heads": 8,
8
+ "mlp_ratio": 4,
9
+ "cond_dim": 512,
10
+ "n_cond_tokens": 4,
11
+ "n_counts": 5,
12
+ "dropout": 0.0,
13
+ "text_tokens": 24,
14
+ "text_dim": 512
15
+ },
16
+ "threshold": -1.0
17
+ }
pixmoji-v2.1.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f7f637d5d86304966d004b5d2d907bb27a5e2841ce363ec225e78b52535adced
3
+ size 174571316
pixmoji-v2.2-fast.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "pixmoji-v2.2-fast",
3
+ "config": {
4
+ "vocab": 4097,
5
+ "d_model": 512,
6
+ "n_layers": 12,
7
+ "n_heads": 8,
8
+ "mlp_ratio": 4,
9
+ "cond_dim": 512,
10
+ "n_cond_tokens": 4,
11
+ "n_counts": 5,
12
+ "dropout": 0.0,
13
+ "text_tokens": 24,
14
+ "text_dim": 512
15
+ },
16
+ "threshold": -1.0
17
+ }
pixmoji-v2.2-fast.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:017353adbd9f39cdffa2f07f445e5b4584ed8b18de7a0086f6dfe2e6acb974b3
3
+ size 174571316
pixmoji/__init__.py ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ """pixmoji: text -> 16x16 pixel-art emojis (masked-token transformer, TinyCLIP-conditioned).
2
+
3
+ import pixmoji
4
+ gen = pixmoji.load() # downloads model-collapse/pixmoji (CPU by default)
5
+ for e in gen.generate_emojis("cat wearing a tiny crown", k=10, seed=0):
6
+ e.rgba # (16, 16, 4) uint8
7
+ """
8
+ from pixmoji.pipeline import Emoji, EmojiGenerator, load
9
+
10
+ __all__ = ["Emoji", "EmojiGenerator", "load"]
pixmoji/counts.py ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Count parsing (0 = unspecified, 1-4) and simple English pluralization."""
2
+ from __future__ import annotations
3
+
4
+ import re
5
+
6
+ NUM_WORDS = {1: "one", 2: "two", 3: "three", 4: "four"}
7
+ _WORD_VALUES = {"one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8,
8
+ "nine": 9, "ten": 10, "eleven": 11, "twelve": 12, "dozen": 12}
9
+ _PAIR = re.compile(r"(?<![\w-])(?:a\s+)?(?:pair|couple)\s+of(?![\w-])", re.I)
10
+ _NUM = re.compile(r"(?<![\w-])(\d+|" + "|".join(_WORD_VALUES) + r")(?![\w-])", re.I)
11
+
12
+
13
+ _HEAD_END = re.compile(r"\s+(?:with|in|on|inside|under|holding|wearing|and|for|from)\s+", re.I)
14
+
15
+
16
+ def head_phrase(text: str) -> str:
17
+ """The main noun phrase: text before the first 'with/in/on/...' clause ('crown key with three gems' -> 'crown key')."""
18
+ return _HEAD_END.split(text, maxsplit=1)[0]
19
+
20
+
21
+ def strip_count(text: str) -> str:
22
+ """Remove a leading count expression from the head phrase ('three red apples' -> 'red apples')."""
23
+ head = head_phrase(text); rest = text[len(head):]
24
+ head = _PAIR.sub("", head, count=1)
25
+ head = _NUM.sub("", head, count=1)
26
+ return (re.sub(r"\s+", " ", head).strip() + rest).strip()
27
+
28
+
29
+ def parse_count(text: str) -> int:
30
+ """Count of the main objects (0 = unspecified), read from the head phrase only."""
31
+ text = head_phrase(text)
32
+ if _PAIR.search(text):
33
+ return 2
34
+ m = _NUM.search(text)
35
+ if not m:
36
+ return 0
37
+ tok = m.group(1).lower()
38
+ n = int(tok) if tok.isdigit() else _WORD_VALUES[tok]
39
+ return n if 1 <= n <= 4 else 0
40
+
41
+
42
+ _IRREGULAR = {"man": "men", "woman": "women", "child": "children", "mouse": "mice", "foot": "feet",
43
+ "tooth": "teeth", "knife": "knives", "leaf": "leaves", "wolf": "wolves",
44
+ "sheep": "sheep", "fish": "fish", "deer": "deer"}
45
+
46
+
47
+ _NOT_MAN = {"human", "shaman", "talisman", "caiman", "ottoman", "german", "walkman"}
48
+
49
+
50
+ def _plural_word(w: str) -> str:
51
+ lw = w.lower()
52
+ if lw not in _NOT_MAN:
53
+ for sing, plur in _IRREGULAR.items(): # suffix match covers compounds: snowman, werewolf, goldfish
54
+ if lw.endswith(sing):
55
+ return w[: len(w) - len(sing)] + plur
56
+ if re.search(r"(s|x|z|ch|sh)$", lw):
57
+ return w + "es"
58
+ if re.search(r"[^aeiou]y$", lw):
59
+ return w[:-1] + "ies"
60
+ return w + "s"
61
+
62
+
63
+ _HEAD_SPLIT = re.compile(r"(\s+(?:of|with|in|on|for)\s+|-(?:at|of|in)-)")
64
+
65
+
66
+ def _already_plural(w: str) -> bool:
67
+ lw = w.lower()
68
+ return lw.endswith("s") and not re.search(r"(ss|us|is|os|as)$", lw)
69
+
70
+
71
+ def pluralize(noun_phrase: str) -> str:
72
+ """Pluralize the head noun: the last word before 'of/with/in/on/for' or a hyphenated '-at-' phrase."""
73
+ m = _HEAD_SPLIT.search(noun_phrase)
74
+ head, tail = (noun_phrase[: m.start()], noun_phrase[m.start():]) if m else (noun_phrase, "")
75
+ parts = head.rsplit(" ", 1)
76
+ last = parts[-1]
77
+ if not _already_plural(last):
78
+ last = _plural_word(last)
79
+ return (parts[0] + " " if len(parts) == 2 else "") + last + tail
pixmoji/model.py ADDED
@@ -0,0 +1,184 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """MaskGIT-style bidirectional transformer over 256 palette-index pixels, conditioned on text embedding + count."""
2
+ from __future__ import annotations
3
+
4
+ import math
5
+ from dataclasses import asdict, dataclass
6
+
7
+ import torch
8
+ import torch.nn as nn
9
+ import torch.nn.functional as F
10
+
11
+ MASK_ID = 4097
12
+
13
+
14
+ @dataclass
15
+ class ModelConfig:
16
+ vocab: int = 4097
17
+ d_model: int = 384
18
+ n_layers: int = 8
19
+ n_heads: int = 6
20
+ mlp_ratio: int = 4
21
+ cond_dim: int = 512
22
+ n_cond_tokens: int = 4
23
+ n_counts: int = 5
24
+ dropout: float = 0.0
25
+ text_tokens: int = 0 # >0: word-level conditioning on this many TinyCLIP token features (v2)
26
+ text_dim: int = 512
27
+
28
+ def to_dict(self) -> dict:
29
+ return asdict(self)
30
+
31
+
32
+ TINY = ModelConfig(d_model=64, n_layers=2, n_heads=2)
33
+ SMALL = ModelConfig(d_model=320, n_layers=6, n_heads=5)
34
+ V2 = ModelConfig(d_model=512, n_layers=12, n_heads=8, text_tokens=24, text_dim=512)
35
+ V2S = ModelConfig(d_model=512, n_layers=8, n_heads=8, text_tokens=24, text_dim=512) # lighter fallback if V2 misses the latency gate
36
+ V2S6 = ModelConfig(d_model=512, n_layers=6, n_heads=8, text_tokens=24, text_dim=512) # CPU student (layer-drop init)
37
+ V2XS = ModelConfig(d_model=384, n_layers=8, n_heads=6, text_tokens=24, text_dim=512) # CPU student (fresh init), ~2.7x fewer FLOPs
38
+ V3 = ModelConfig(d_model=768, n_layers=12, n_heads=12, text_tokens=24, text_dim=512) # capacity probe (~92M), g7e-only serving
39
+
40
+
41
+ class Block(nn.Module):
42
+ def __init__(self, c: ModelConfig):
43
+ super().__init__()
44
+ self.n_heads = c.n_heads
45
+ self.ln1, self.ln2 = nn.LayerNorm(c.d_model), nn.LayerNorm(c.d_model)
46
+ self.qkv = nn.Linear(c.d_model, 3 * c.d_model)
47
+ self.proj = nn.Linear(c.d_model, c.d_model)
48
+ self.mlp = nn.Sequential(nn.Linear(c.d_model, c.mlp_ratio * c.d_model), nn.GELU(), nn.Linear(c.mlp_ratio * c.d_model, c.d_model))
49
+ self.drop = nn.Dropout(c.dropout)
50
+
51
+ def forward(self, x: torch.Tensor, attn_mask: torch.Tensor | None = None) -> torch.Tensor:
52
+ b, t, d = x.shape
53
+ q, k, v = self.qkv(self.ln1(x)).view(b, t, 3, self.n_heads, d // self.n_heads).permute(2, 0, 3, 1, 4)
54
+ a = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask).transpose(1, 2).reshape(b, t, d)
55
+ x = x + self.drop(self.proj(a))
56
+ return x + self.drop(self.mlp(self.ln2(x)))
57
+
58
+
59
+ class PixmojiTransformer(nn.Module):
60
+ def __init__(self, c: ModelConfig = ModelConfig()):
61
+ super().__init__()
62
+ self.cfg = c
63
+ d = c.d_model
64
+ self.cond = nn.Sequential(nn.Linear(c.cond_dim, d), nn.GELU(), nn.Linear(d, c.n_cond_tokens * d))
65
+ self.count_emb = nn.Embedding(c.n_counts, d)
66
+ self.tok_emb = nn.Embedding(c.vocab + 1, d) # + MASK
67
+ self.row_emb, self.col_emb = nn.Parameter(torch.zeros(16, 1, d)), nn.Parameter(torch.zeros(1, 16, d))
68
+ self.n_prefix = c.n_cond_tokens + c.text_tokens + 1
69
+ self.prefix_pos = nn.Parameter(torch.zeros(self.n_prefix, d))
70
+ if c.text_tokens:
71
+ self.word_proj = nn.Linear(c.text_dim, d)
72
+ self.blocks = nn.ModuleList(Block(c) for _ in range(c.n_layers))
73
+ self.ln_f = nn.LayerNorm(d)
74
+ self.head = nn.Linear(d, c.vocab)
75
+ self.apply(self._init)
76
+ for p in (self.row_emb, self.col_emb, self.prefix_pos):
77
+ nn.init.normal_(p, std=0.02)
78
+
79
+ @staticmethod
80
+ def _init(m):
81
+ if isinstance(m, (nn.Linear, nn.Embedding)):
82
+ nn.init.normal_(m.weight, std=0.02)
83
+ if isinstance(m, nn.Linear) and m.bias is not None:
84
+ nn.init.zeros_(m.bias)
85
+
86
+ def forward(self, pixel_tokens: torch.Tensor, text_emb: torch.Tensor, count_ids: torch.Tensor,
87
+ text_feats: torch.Tensor | None = None, text_mask: torch.Tensor | None = None,
88
+ out_pos: torch.Tensor | None = None) -> torch.Tensor:
89
+ """out_pos (b, m): return logits only for these pixel positions (the sampler only needs still-masked ones)."""
90
+ b = pixel_tokens.shape[0]; c = self.cfg; dt = self.tok_emb.weight.dtype # (Embedding: stays float under int8 Linear quantization)
91
+ cond = self.cond(text_emb.to(dt)).view(b, c.n_cond_tokens, c.d_model)
92
+ parts, attn, pos = [cond], None, self.prefix_pos
93
+ if c.text_tokens:
94
+ L = text_feats.shape[1] # may be < text_tokens when trailing padded slots were trimmed (exact: they are masked anyway)
95
+ parts.append(self.word_proj(text_feats.to(dt)))
96
+ nc = c.n_cond_tokens
97
+ pos = torch.cat([self.prefix_pos[:nc + L], self.prefix_pos[-1:]])
98
+ keep = torch.ones(b, nc + L + 1 + 256, dtype=torch.bool, device=pixel_tokens.device)
99
+ keep[:, nc:nc + L] = text_mask
100
+ attn = keep[:, None, None, :] # key padding mask for SDPA
101
+ parts.append(self.count_emb(count_ids)[:, None])
102
+ prefix = torch.cat(parts, 1) + pos
103
+ pix = self.tok_emb(pixel_tokens) + (self.row_emb + self.col_emb).reshape(1, 256, c.d_model)
104
+ x = torch.cat([prefix, pix], 1)
105
+ for blk in self.blocks:
106
+ x = blk(x, attn)
107
+ h = x[:, prefix.shape[1]:]
108
+ if out_pos is not None:
109
+ h = torch.gather(h, 1, out_pos[..., None].expand(-1, -1, h.shape[-1]))
110
+ return self.head(self.ln_f(h.to(self.ln_f.weight.dtype))) # output layer may run in higher precision (mixed bf16)
111
+
112
+
113
+ def null_words(text_feats: torch.Tensor, text_mask: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
114
+ """CFG / cond-drop null condition for word features: all zeros, only the first token visible."""
115
+ m = torch.zeros_like(text_mask); m[:, 0] = True
116
+ return torch.zeros_like(text_feats), m
117
+
118
+
119
+ def masking_loss(model: PixmojiTransformer, tokens: torch.Tensor, text_emb: torch.Tensor, count_ids: torch.Tensor,
120
+ cond_drop: float = 0.1, generator: torch.Generator | None = None, mask: torch.Tensor | None = None,
121
+ text_feats: torch.Tensor | None = None, text_mask: torch.Tensor | None = None) -> torch.Tensor:
122
+ b, n = tokens.shape; dev = tokens.device
123
+ if mask is None:
124
+ u = torch.rand(b, 1, generator=generator).to(dev)
125
+ n_mask = (n * torch.cos(math.pi / 2 * u)).ceil().clamp(1, n) # >= 1 masked position
126
+ rank = torch.rand(b, n, generator=generator).to(dev).argsort(1).argsort(1)
127
+ mask = rank < n_mask
128
+ if cond_drop > 0:
129
+ drop = (torch.rand(b, generator=generator) < cond_drop).to(dev)
130
+ text_emb = text_emb.masked_fill(drop[:, None], 0.0)
131
+ count_ids = count_ids.masked_fill(drop, 0)
132
+ if text_feats is not None:
133
+ nf, nm = null_words(text_feats, text_mask)
134
+ text_feats = torch.where(drop[:, None, None], nf, text_feats); text_mask = torch.where(drop[:, None], nm, text_mask)
135
+ logits = model(tokens.masked_fill(mask, MASK_ID), text_emb, count_ids, text_feats=text_feats, text_mask=text_mask)
136
+ return F.cross_entropy(logits[mask].float(), tokens[mask])
137
+
138
+
139
+ @torch.no_grad()
140
+ def mixed_bf16(model: "PixmojiTransformer") -> "PixmojiTransformer":
141
+ """bf16 everywhere except the final norm + colour head (fp32): fast on AVX-512 BF16 CPUs, keeps logit resolution."""
142
+ model = model.to(torch.bfloat16); model.ln_f.float(); model.head.float(); return model
143
+
144
+
145
+ def guided_logits(teacher: PixmojiTransformer, tokens_in: torch.Tensor, text_emb: torch.Tensor, count_ids: torch.Tensor,
146
+ cfg: float, cfg_count: float, text_feats=None, text_mask=None) -> torch.Tensor:
147
+ """Classifier-free-guided teacher logits (the serving recipe): l_u + w·(l_c − l_u), w = cfg_count for count prompts (2–4)."""
148
+ b = tokens_in.shape[0]
149
+ words = {}
150
+ if text_feats is not None:
151
+ nf, nm = null_words(text_feats, text_mask)
152
+ words = {"text_feats": torch.cat([text_feats, nf]), "text_mask": torch.cat([text_mask, nm])}
153
+ lg = teacher(torch.cat([tokens_in, tokens_in]), torch.cat([text_emb, torch.zeros_like(text_emb)]),
154
+ torch.cat([count_ids, torch.zeros_like(count_ids)]), **words).float()
155
+ l_c, l_u = lg[:b], lg[b:]
156
+ w = torch.where(count_ids >= 2, torch.full_like(count_ids, 0, dtype=torch.float) + cfg_count, torch.full_like(count_ids, 0, dtype=torch.float) + cfg)
157
+ return l_u + w[:, None, None] * (l_c - l_u)
158
+
159
+
160
+ def distill_loss(student: PixmojiTransformer, teacher: PixmojiTransformer, tokens: torch.Tensor, text_emb: torch.Tensor,
161
+ count_ids: torch.Tensor, cfg: float = 2.5, cfg_count: float = 5.0, generator: torch.Generator | None = None,
162
+ text_feats=None, text_mask=None, commit: torch.Tensor | None = None, steps: int = 12,
163
+ final_weight: float = 0.0) -> torch.Tensor:
164
+ """Guidance distillation: the student's single conditional pass learns the teacher's guided distribution
165
+ (KL on masked positions). Off-policy (commit=None): random masks of real sprites, training schedule.
166
+ On-policy (commit = step each position was fixed in a recorded serving-sampler run of the teacher): the state at a
167
+ random step k is the recorded sample with positions fixed at >= k masked, i.e. exactly what the sampler sees."""
168
+ b, n = tokens.shape; dev = tokens.device
169
+ if commit is not None:
170
+ k = torch.randint(0, steps, (b, 1), generator=generator).to(dev)
171
+ mask = commit.to(dev) >= k
172
+ else:
173
+ u = torch.rand(b, 1, generator=generator).to(dev)
174
+ n_mask = (n * torch.cos(math.pi / 2 * u)).ceil().clamp(1, n)
175
+ mask = torch.rand(b, n, generator=generator).to(dev).argsort(1).argsort(1) < n_mask
176
+ inp = tokens.masked_fill(mask, MASK_ID)
177
+ target = torch.log_softmax(guided_logits(teacher, inp, text_emb, count_ids, cfg, cfg_count, text_feats, text_mask), -1)
178
+ pred = torch.log_softmax(student(inp, text_emb, count_ids, text_feats=text_feats, text_mask=text_mask).float(), -1)
179
+ loss = F.kl_div(pred[mask], target[mask], log_target=True, reduction="batchmean")
180
+ if final_weight and commit is not None:
181
+ # step distillation: also predict the tokens the teacher's sampler actually ended with (from the same on-policy state),
182
+ # so the student can commit more pixels per step coherently
183
+ loss = loss + final_weight * F.nll_loss(pred[mask], tokens.to(dev)[mask])
184
+ return loss
pixmoji/pipeline.py ADDED
@@ -0,0 +1,155 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """End-to-end: text -> N sampled sprites -> TinyCLIP scores -> up to k selected Emoji (everything model-generated)."""
2
+ from __future__ import annotations
3
+
4
+ import json
5
+ from dataclasses import dataclass
6
+ from pathlib import Path
7
+
8
+ import numpy as np
9
+ import torch
10
+
11
+ from pixmoji.counts import parse_count
12
+ from pixmoji.model import ModelConfig, PixmojiTransformer
13
+ from pixmoji.sampling import sample
14
+ from pixmoji.select import select
15
+
16
+
17
+ @dataclass
18
+ class Emoji:
19
+ rgba: np.ndarray
20
+ score: float
21
+
22
+
23
+ def tokens_to_rgba_t(tokens: torch.Tensor, palette_t: torch.Tensor) -> torch.Tensor:
24
+ t = tokens.long()
25
+ rgb = palette_t.to(t.device)[t]
26
+ alpha = torch.where(t > 0, 255, 0).to(torch.uint8)[..., None]
27
+ out = torch.cat([rgb.to(torch.uint8), alpha], -1)
28
+ out = out * (t > 0)[..., None].to(torch.uint8)
29
+ return out.reshape(*t.shape[:-1], 16, 16, 4)
30
+
31
+
32
+ def save_checkpoint(model: PixmojiTransformer, path, threshold: float, meta: dict | None = None) -> None:
33
+ torch.save({"config": model.cfg.to_dict(), "state_dict": model.state_dict(), "threshold": float(threshold), "meta": meta or {}}, path)
34
+
35
+
36
+ def load_model(path, device: str = "cpu", dtype: torch.dtype = torch.float32) -> tuple[PixmojiTransformer, dict]:
37
+ path = Path(path)
38
+ if path.suffix == ".safetensors": # release format: weights + sibling <name>.json (config, threshold)
39
+ from safetensors.torch import load_file
40
+ meta = json.load(open(path.with_suffix(".json")))
41
+ ck = {"config": meta["config"], "state_dict": load_file(str(path)), "threshold": meta["threshold"]}
42
+ else:
43
+ ck = torch.load(path, map_location="cpu", weights_only=False)
44
+ m = PixmojiTransformer(ModelConfig(**ck["config"])); m.load_state_dict(ck["state_dict"])
45
+ return m.to(device=device, dtype=dtype).eval(), ck
46
+
47
+
48
+ class EmojiGenerator:
49
+ # Serving defaults chosen by the 2026-11-14 sampler sweep (blind test vs real, g5 latency): 12 steps, T 0.6,
50
+ # guidance 2.5 (cleanest single objects), guidance 5 when the prompt has a count (low guidance loses counts).
51
+ def __init__(self, model_path, palette_path, device: str = "cuda", n: int = 24, steps: int = 12, cfg: float = 2.5,
52
+ threshold: float | None = None, clip=None, compile: bool = False, temperature: float = 0.6, cfg_count: float = 5.0,
53
+ cfg_steps: int | None = None, fast_model_path=None, fast_n: int = 24, fast_temperature: float = 0.5, fast_steps: int | None = 8,
54
+ fast_counts: bool = False, count_n: int | None = None, count_steps: int | None = None, dtype: torch.dtype | None = None,
55
+ cpu_fast: bool | str = False):
56
+ """fast_model_path: the on-policy guidance-distilled model (pixmoji-v2.2-fast; single pass per step) used for prompts
57
+ without a count >= 2, with N 24, 8 steps, T 0.5 (1/3 of the base compute; compression study 2026-11-17, rounds 9-16).
58
+ Count prompts keep the guided base model unchanged (distillation and fewer candidates both lost count accuracy)."""
59
+ # dtype: default fp16 on GPU, fp32 on CPU; torch.bfloat16 on CPUs with AVX-512 BF16 is ~2.2x faster (compression study)
60
+ self.device = device; dtype = dtype or (torch.float16 if device.startswith("cuda") else torch.float32)
61
+ self.model, ck = load_model(model_path, device, dtype)
62
+ if compile:
63
+ self.model = torch.compile(self.model)
64
+ self.fast_model, self.fast_n, self.fast_temperature, self.fast_steps = None, fast_n, fast_temperature, fast_steps
65
+ self.fast_counts, self.count_n, self.count_steps = fast_counts, count_n, count_steps # count-route options (study)
66
+ if fast_model_path is not None:
67
+ self.fast_model, _ = load_model(fast_model_path, device, dtype)
68
+ if compile:
69
+ self.fast_model = torch.compile(self.fast_model)
70
+ if dtype == torch.bfloat16 and self.fast_model is not None:
71
+ # measured (rounds 19-21): the fast route needs its final norm + colour head in fp32 (full bf16 cost Terra -4.8
72
+ # depicts points); the guided count route is within noise in full bf16 over 600 count prompts, so it stays full bf16.
73
+ from pixmoji.model import mixed_bf16
74
+ self.fast_model = mixed_bf16(self.fast_model)
75
+ # cpu_fast: torch.compile with Inductor freezing (oneDNN weight prepack, fused bias/GELU/norms), word slots padded to
76
+ # 3 bucket sizes (exact; bounds recompiles) and logits only for still-masked positions (exact in distribution).
77
+ # Measured (rounds 26-27, CPU, judges): within noise on the count route, borderline on the plain route, so
78
+ # cpu_fast="counts" (default when True is not requested) applies it to the count route only; "all" applies it to both.
79
+ # Pair with OMP_PROC_BIND=close OMP_PLACES=cores and threads within one socket.
80
+ self.sample_opts, self.sample_opts_count = {}, {}
81
+ if cpu_fast:
82
+ import torch._inductor.config as ic
83
+ ic.freezing = True
84
+ opts = {"word_buckets": (8, 16, 24), "masked_only": True}
85
+ self.model = torch.compile(self.model); self.sample_opts_count = opts
86
+ if cpu_fast == "all" and self.fast_model is not None:
87
+ self.fast_model = torch.compile(self.fast_model); self.sample_opts = opts
88
+ self.threshold = ck["threshold"] if threshold is None else threshold
89
+ self.n, self.steps, self.cfg, self.temperature, self.cfg_count, self.cfg_steps = n, steps, cfg, temperature, cfg_count, cfg_steps
90
+ self.palette = torch.from_numpy(np.load(palette_path, allow_pickle=False)).to(device)
91
+ if clip is None:
92
+ from pixmoji.tinyclip import TinyClip
93
+ clip = TinyClip(device=device, dtype=dtype)
94
+ self.text_fn, self.image_fn = clip.torch_modules()
95
+ self.clip = clip; self.text_tokens = ck["config"].get("text_tokens", 0) # >0: v2 word-level model
96
+
97
+ def words_for(self, text: str) -> dict:
98
+ """Word-level features for v2+ models ({} for v1 checkpoints)."""
99
+ if not self.text_tokens:
100
+ return {}
101
+ from pixmoji.text import tokenize_vocab
102
+ ids, lens = tokenize_vocab([text])
103
+ f, m = self.clip.token_features(torch.from_numpy(ids), torch.from_numpy(lens))
104
+ return {"text_feats": f.float(), "text_mask": m}
105
+
106
+ def _encode(self, text: str):
107
+ """(pooled text embedding (1, 512), word features). One text-tower pass when the prompt fits the word slots."""
108
+ if self.text_tokens and hasattr(self.clip, "text_and_tokens"):
109
+ from pixmoji.text import TEXT_TOKENS, tokenize_vocab
110
+ ids, lens = tokenize_vocab([text])
111
+ if lens[0] < TEXT_TOKENS: # not truncated: EOS state == CLIP pooled output
112
+ te, f, m = self.clip.text_and_tokens(torch.from_numpy(ids), torch.from_numpy(lens))
113
+ return te.float(), {"text_feats": f.float(), "text_mask": m}
114
+ return self.text_fn([text]).float(), self.words_for(text)
115
+
116
+ @torch.inference_mode()
117
+ def generate_emojis(self, text: str, k: int = 10, seed: int | None = None) -> list[Emoji]:
118
+ if not text or not text.strip():
119
+ return []
120
+ gen = torch.Generator(device=self.device).manual_seed(seed) if seed is not None else None
121
+ count = parse_count(text); te, words = self._encode(text)
122
+ if self.fast_model is not None and (count < 2 or self.fast_counts):
123
+ toks = sample(self.fast_model, te.to(self.device), count, self.fast_n, steps=self.fast_steps or self.steps, cfg=self.cfg,
124
+ temperature=self.fast_temperature, generator=gen, cfg_steps=0, **words, **self.sample_opts)
125
+ else:
126
+ cnt_route = count >= 2
127
+ toks = sample(self.model, te.to(self.device), count, (self.count_n or self.n) if cnt_route else self.n,
128
+ steps=(self.count_steps or self.steps) if cnt_route else self.steps, cfg=self.cfg_count if count >= 2 else self.cfg,
129
+ temperature=self.temperature, generator=gen, cfg_steps=self.cfg_steps, **words,
130
+ **(self.sample_opts_count if cnt_route else self.sample_opts))
131
+ rgba = tokens_to_rgba_t(toks, self.palette)
132
+ ie = self.image_fn(rgba).float()
133
+ scores = (ie @ te[0].to(ie.device)).cpu().numpy()
134
+ opaque = (toks > 0).float().mean(1).cpu().numpy()
135
+ idx = select(scores, ie.cpu().numpy(), opaque, k=k, threshold=self.threshold)
136
+ rgba_np = rgba.cpu().numpy()
137
+ return [Emoji(rgba_np[i], float(scores[i])) for i in idx]
138
+
139
+
140
+ def _resolve(repo_or_dir, filename):
141
+ p = Path(str(repo_or_dir)) / filename
142
+ if p.exists():
143
+ return p
144
+ from huggingface_hub import hf_hub_download
145
+ return Path(hf_hub_download(str(repo_or_dir), filename))
146
+
147
+
148
+ def load(repo_or_dir="model-collapse/pixmoji", device: str | None = None, dtype: torch.dtype | None = None, **kw) -> "EmojiGenerator":
149
+ """Serving setup from the model card: prompts with a count >= 2 use pixmoji-v2.1 (guided); all others pixmoji-v2.2-fast
150
+ (single pass, 8 steps). On CPUs with AVX-512 BF16 pass dtype=torch.bfloat16 for ~2x speed."""
151
+ device = device or ("cuda" if torch.cuda.is_available() else "cpu")
152
+ for f in ("pixmoji-v2.1.json", "pixmoji-v2.2-fast.safetensors", "pixmoji-v2.2-fast.json"):
153
+ _resolve(repo_or_dir, f)
154
+ return EmojiGenerator(_resolve(repo_or_dir, "pixmoji-v2.1.safetensors"), _resolve(repo_or_dir, "palette.npy"), device=device,
155
+ fast_model_path=_resolve(repo_or_dir, "pixmoji-v2.2-fast.safetensors"), dtype=dtype, **kw)
pixmoji/sampling.py ADDED
@@ -0,0 +1,90 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """MaskGIT parallel decoding with classifier-free guidance.
2
+
3
+ Sampling is memory-bound (n x 256 x 4097 logits per step), so by default tokens are drawn among the top_k colours
4
+ per pixel with the Gumbel-max trick in the logits' dtype; top_k=None reproduces full-vocabulary sampling.
5
+ """
6
+ from __future__ import annotations
7
+
8
+ import math
9
+
10
+ import torch
11
+
12
+ from pixmoji.model import MASK_ID, PixmojiTransformer, null_words
13
+
14
+
15
+ def remaining_masked(k: int, steps: int, n: int = 256) -> int:
16
+ return int(math.floor(n * math.cos(math.pi / 2 * (k + 1) / steps)))
17
+
18
+
19
+ def _gumbel(shape, generator, device) -> torch.Tensor:
20
+ gen_dev = generator.device if generator is not None else device
21
+ u = torch.rand(shape, generator=generator, device=gen_dev).clamp_(1e-10, 1 - 1e-10)
22
+ return (-torch.log(-torch.log(u))).to(device)
23
+
24
+
25
+ def _draw(logits: torch.Tensor, temperature: float, top_k: int | None, generator) -> tuple[torch.Tensor, torch.Tensor]:
26
+ """Sample one token per position; return (tokens, log-prob of the drawn token under the (top-k) distribution)."""
27
+ t = max(temperature, 1e-6)
28
+ if top_k is not None and top_k < logits.shape[-1]:
29
+ vals, idx = logits.topk(top_k, dim=-1)
30
+ vals = vals.float() / t
31
+ else:
32
+ vals, idx = logits.float() / t, None
33
+ logp = torch.log_softmax(vals, -1)
34
+ choice = (logp + _gumbel(logp.shape, generator, logp.device)).argmax(-1, keepdim=True)
35
+ conf = logp.gather(-1, choice)[..., 0]
36
+ drawn = choice[..., 0] if idx is None else idx.gather(-1, choice)[..., 0]
37
+ return drawn, conf
38
+
39
+
40
+ @torch.no_grad()
41
+ def sample(model: PixmojiTransformer, text_emb: torch.Tensor, count_id: int, n: int, steps: int = 6, cfg: float = 3.0,
42
+ temperature: float = 1.0, generator: torch.Generator | None = None, top_k: int | None = 64,
43
+ text_feats: torch.Tensor | None = None, text_mask: torch.Tensor | None = None, cfg_steps: int | None = None, return_commit: bool = False,
44
+ word_buckets: tuple[int, ...] | None = None, masked_only: bool = False):
45
+ """word_buckets: pad the trimmed word slots up to the next bucket size (exact: padded slots are masked) so compiled
46
+ models see only a few shapes. masked_only: compute logits and draws only for still-masked positions (exact in
47
+ distribution; ~30% less head/top-k work over a run, different random stream).
48
+ return_commit: also return (n, 256) the step at which each position was fixed (for on-policy distillation).
49
+ cfg_steps: apply classifier-free guidance only on the first cfg_steps steps (None = all); later steps run the
50
+ conditional pass alone (half the compute for those steps)."""
51
+ dev = next(model.parameters()).device
52
+ e = text_emb.reshape(1, -1).to(dev).expand(n, -1)
53
+ emb = torch.cat([e, torch.zeros_like(e)])
54
+ cnt = torch.cat([torch.full((n,), count_id, device=dev), torch.zeros(n, dtype=torch.long, device=dev)])
55
+ words, words_c = {}, {}
56
+ if text_feats is not None: # word-level conditioning (v2): conditional half gets the words, unconditional half the null words
57
+ mk = text_mask.reshape(1, -1).to(dev)
58
+ L = max(1, int(mk.any(0).nonzero().max()) + 1) if mk.any() else 1 # trim trailing padded slots (exact)
59
+ if word_buckets:
60
+ L = min(next((x for x in word_buckets if x >= L), L), mk.shape[1])
61
+ f = text_feats.reshape(1, *text_feats.shape[-2:])[:, :L].to(dev).expand(n, -1, -1)
62
+ mk = mk[:, :L].expand(n, -1)
63
+ nf, nm = null_words(f, mk)
64
+ words = {"text_feats": torch.cat([f, nf]), "text_mask": torch.cat([mk, nm])}
65
+ words_c = {"text_feats": f, "text_mask": mk}
66
+ tokens = torch.full((n, 256), MASK_ID, dtype=torch.long, device=dev)
67
+ commit = torch.full((n, 256), steps - 1, dtype=torch.long, device=dev)
68
+ for k in range(steps):
69
+ masked = tokens == MASK_ID
70
+ pos = None
71
+ if masked_only: # every row has the same number of masked positions at a given step (deterministic schedule)
72
+ m = int(masked[0].sum()); pos = torch.argsort(masked.to(torch.int8), dim=1, descending=True, stable=True)[:, :m]
73
+ if cfg_steps is None or k < cfg_steps:
74
+ logits = model(torch.cat([tokens, tokens]), emb, cnt, **words, **({"out_pos": torch.cat([pos, pos])} if pos is not None else {}))
75
+ l_c, l_u = logits[:n], logits[n:]
76
+ lg = l_u + cfg * (l_c - l_u) if cfg != 1.0 else l_c
77
+ else:
78
+ lg = model(tokens, emb[:n], cnt[:n], **words_c, **({"out_pos": pos} if pos is not None else {}))
79
+ drawn, conf = _draw(lg, temperature, top_k, generator)
80
+ conf = conf + temperature * (1 - (k + 1) / steps) * _gumbel(conf.shape, generator, dev)
81
+ if pos is not None: # scatter back to all 256 positions; fixed positions keep their token and are never re-masked
82
+ drawn = tokens.scatter(1, pos, drawn); conf = torch.full(tokens.shape, float("inf"), device=dev, dtype=conf.dtype).scatter(1, pos, conf)
83
+ tokens = torch.where(masked, drawn, tokens)
84
+ conf = conf.masked_fill(~masked, float("inf")) # already-fixed positions are never re-masked
85
+ keep_masked = remaining_masked(k, steps)
86
+ if keep_masked > 0:
87
+ tokens.scatter_(1, conf.argsort(1)[:, :keep_masked], MASK_ID)
88
+ if return_commit:
89
+ commit = torch.where(masked & (tokens != MASK_ID), torch.full_like(commit, k), commit)
90
+ return (tokens, commit) if return_commit else tokens
pixmoji/select.py ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Pick up to k candidates: relevance first (threshold + score order), diversity only as a near-tie breaker."""
2
+ from __future__ import annotations
3
+
4
+ import numpy as np
5
+
6
+
7
+ def select(scores: np.ndarray, img_emb: np.ndarray, opaque_frac: np.ndarray, k: int = 10, threshold: float = 0.0,
8
+ div_weight: float = 0.05, dup_cos: float = 0.97, min_opaque: float = 0.05) -> list[int]:
9
+ """Greedy MMR with mmr = score - div_weight * max_cos_to_selected.
10
+
11
+ TinyCLIP scores span ~0.2-0.4 while cosine spans 0-1, so a small div_weight keeps relevance dominant
12
+ (spec: representativeness first, diversity second). Never pads: returns fewer than k if few pass.
13
+ """
14
+ scores = np.asarray(scores, np.float64); emb = np.asarray(img_emb, np.float64)
15
+ cand = [i for i in np.argsort(-scores, kind="stable") if opaque_frac[i] >= min_opaque and scores[i] >= threshold]
16
+ chosen: list[int] = []
17
+ max_cos = np.full(len(scores), -np.inf)
18
+ while cand and len(chosen) < k:
19
+ pen = np.where(np.isfinite(max_cos[cand]), max_cos[cand], 0.0)
20
+ best = cand[int(np.argmax(scores[cand] - div_weight * pen))]
21
+ chosen.append(int(best))
22
+ sims = emb[cand] @ emb[best]
23
+ max_cos[cand] = np.maximum(max_cos[cand], sims)
24
+ cand = [c for c, s in zip(cand, sims) if c != best and s < dup_cos]
25
+ return chosen
pixmoji/text.py ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """CLIP tokenisation for pixmoji's word-level conditioning (same tokenizer as TinyCLIP-PixelArt)."""
2
+ from __future__ import annotations
3
+
4
+ import functools
5
+
6
+ import numpy as np
7
+
8
+ TEXT_TOKENS = 24 # word slots (incl. BOS/EOS)
9
+
10
+
11
+ @functools.lru_cache(maxsize=1)
12
+ def _tokenizer():
13
+ from transformers import CLIPProcessor
14
+ from pixmoji.tinyclip import MODEL_ID, REVISION
15
+ return CLIPProcessor.from_pretrained(MODEL_ID, revision=REVISION).tokenizer
16
+
17
+
18
+ def tokenize_vocab(texts: list[str]) -> tuple[np.ndarray, np.ndarray]:
19
+ tok = _tokenizer(); ids = np.zeros((len(texts), TEXT_TOKENS), np.int32); lens = np.zeros(len(texts), np.int32)
20
+ for i, e in enumerate(tok(list(texts), truncation=True, max_length=TEXT_TOKENS)["input_ids"]):
21
+ ids[i, :len(e)] = e; lens[i] = len(e)
22
+ return ids, lens
pixmoji/tinyclip.py ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """TinyCLIP-PixelArt wrapper: text and 16x16 sprite embeddings (numpy and torch paths).
2
+
3
+ Uses transformers CLIPModel directly, because the checkpoint was saved with sentence-transformers 6.3.0.dev0
4
+ and the installed 5.x crashes in encode_query (see ledger, pre-flight T4).
5
+ """
6
+ from __future__ import annotations
7
+
8
+ import numpy as np
9
+ import torch
10
+ import torch.nn.functional as F
11
+ from PIL import Image
12
+ from transformers import CLIPModel, CLIPProcessor
13
+
14
+ MODEL_ID = "tomaarsen/TinyCLIP-39M-PixelArt"
15
+ REVISION = "d25945a2a2f17be4ffd963adf3666053ff57b281" # pinned
16
+ # Model card: "composite the image onto an RGB (128, 128, 128) background before encoding, matching the training preprocessing".
17
+ IMAGE_BG = (128, 128, 128) # measured on 2,000 val rows: hit@1 0.929 / hit@10 0.999 (white 0.924, black 0.885)
18
+ RESIZE_MODE = "bicubic" # processor_config resample=3; 16 -> 224 shortest edge, center-crop 224 (no-op for squares)
19
+ _MEAN = (0.48145466, 0.4578275, 0.40821073)
20
+ _STD = (0.26862954, 0.26130258, 0.27577711)
21
+
22
+
23
+ def _features(out) -> torch.Tensor:
24
+ """get_*_features returns a tensor in transformers 4.x and a BaseModelOutputWithPooling (.pooler_output) in 5.x."""
25
+ return out if isinstance(out, torch.Tensor) else out.pooler_output
26
+
27
+
28
+ def composite(rgba: np.ndarray) -> np.ndarray:
29
+ a = rgba[..., 3:4].astype(np.float32) / 255.0
30
+ bg = np.array(IMAGE_BG, np.float32)
31
+ return (rgba[..., :3].astype(np.float32) * a + bg * (1 - a)).round().astype(np.uint8)
32
+
33
+
34
+ class TinyClip:
35
+ def __init__(self, device: str = "cpu", dim: int = 512, dtype: torch.dtype = torch.float32):
36
+ self.device, self.dim = device, dim
37
+ self.model = CLIPModel.from_pretrained(MODEL_ID, revision=REVISION, dtype=dtype).to(device).eval()
38
+ self.processor = CLIPProcessor.from_pretrained(MODEL_ID, revision=REVISION)
39
+ self.size = self.processor.image_processor.crop_size["height"]
40
+ self._mean = torch.tensor(_MEAN, device=device, dtype=dtype).view(1, 3, 1, 1)
41
+ self._std = torch.tensor(_STD, device=device, dtype=dtype).view(1, 3, 1, 1)
42
+
43
+ def _norm(self, e: torch.Tensor) -> torch.Tensor:
44
+ return F.normalize(e[:, : self.dim].float(), dim=-1)
45
+
46
+ @torch.no_grad()
47
+ def _text(self, texts: list[str]) -> torch.Tensor:
48
+ tok = self.processor.tokenizer(list(texts), padding=True, truncation=True, max_length=77, return_tensors="pt").to(self.device)
49
+ return self._norm(_features(self.model.get_text_features(**tok)))
50
+
51
+ def encode_text(self, texts: list[str], batch_size: int = 1024) -> np.ndarray:
52
+ out = [self._text(texts[i:i + batch_size]).cpu().numpy() for i in range(0, len(texts), batch_size)]
53
+ return np.concatenate(out).astype(np.float32) if out else np.zeros((0, self.dim), np.float32)
54
+
55
+ @torch.no_grad()
56
+ def encode_images(self, rgba: np.ndarray, batch_size: int = 256) -> np.ndarray:
57
+ out = []
58
+ for i in range(0, len(rgba), batch_size):
59
+ ims = [Image.fromarray(x) for x in composite(rgba[i:i + batch_size])]
60
+ px = self.processor.image_processor(images=ims, return_tensors="pt")["pixel_values"].to(self.device, self.model.dtype)
61
+ out.append(self._norm(_features(self.model.get_image_features(pixel_values=px))).cpu().numpy())
62
+ return np.concatenate(out).astype(np.float32)
63
+
64
+ @torch.no_grad()
65
+ def token_features(self, ids: torch.Tensor, lengths: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
66
+ """Per-token features (B, L, 512) from the text tower's final hidden state (after final layer norm) + key mask (B, L).
67
+
68
+ Padded positions are zeroed. The EOS position equals CLIP's pooled output (before projection).
69
+ """
70
+ ids = ids.to(self.device).long(); L = ids.shape[1]
71
+ mask = torch.arange(L, device=ids.device)[None] < lengths.to(ids.device)[:, None]
72
+ out = self.model.text_model(input_ids=ids, attention_mask=mask.long())
73
+ h = out.last_hidden_state if hasattr(out, "last_hidden_state") else out[0]
74
+ return h.masked_fill(~mask[..., None], 0.0), mask
75
+
76
+ @torch.no_grad()
77
+ def text_and_tokens(self, ids: torch.Tensor, lengths: torch.Tensor):
78
+ """One text-tower pass for both the pooled, normalised text embedding and the per-token features.
79
+ Matches encode_text only when the text was not truncated to TEXT_TOKENS (caller checks lengths < L)."""
80
+ f, mask = self.token_features(ids, lengths)
81
+ eos = f[torch.arange(f.shape[0], device=f.device), lengths.to(f.device).long() - 1]
82
+ return self._norm(self.model.text_projection(eos)), f, mask
83
+
84
+ def torch_modules(self):
85
+ def text_fn(texts: list[str]) -> torch.Tensor:
86
+ return self._text(texts)
87
+
88
+ @torch.no_grad()
89
+ def image_fn(rgba: torch.Tensor) -> torch.Tensor:
90
+ x = rgba.to(self.device).float()
91
+ a = x[..., 3:4] / 255.0
92
+ bg = torch.tensor(IMAGE_BG, device=x.device, dtype=torch.float32)
93
+ rgb = (x[..., :3] * a + bg * (1 - a)).round().permute(0, 3, 1, 2) # n,3,16,16 in 0..255
94
+ rgb = F.interpolate(rgb, size=(self.size, self.size), mode=RESIZE_MODE, align_corners=False).clamp(0, 255)
95
+ px = ((rgb / 255.0).to(self._mean.dtype) - self._mean) / self._std
96
+ return self._norm(_features(self.model.get_image_features(pixel_values=px)))
97
+
98
+ return text_fn, image_fn
sample_sheet.png ADDED

Git LFS Details

  • SHA256: 7e0dd4e220918c303dffca3e7da1d015842d4311372f32ac0ff32788ab2798d8
  • Pointer size: 131 Bytes
  • Size of remote file: 212 kB