File size: 6,582 Bytes
84fc8a9 55c046f 84fc8a9 55c046f 84fc8a9 55c046f 84fc8a9 55c046f 84fc8a9 55c046f 84fc8a9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 | ---
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: text-to-image
tags:
- pixel-art
- emoji
- 16x16
- maskgit
- text-to-image
datasets:
- Angais/pixel-art-atlas-1.4m
---
# pixmoji — type a phrase, get 16×16 pixel-art emojis
pixmoji generates small **16×16 pixel-art sprites** from short text ("lovely sheep", "cat wearing a tiny crown",
"three blue apples") fast enough to run as you type. Every image is **generated** by the model; nothing is retrieved.

| file | what it is |
|---|---|
| `pixmoji-v2.2-fast.safetensors` | the fast model (43.6M params): single pass per step, used for all prompts without a count |
| `pixmoji-v2.1.safetensors` | the base model (43.6M), used for prompts with a count ≥ 2 ("three blue apples") |
| `palette.npy` | the 4096-colour palette (token → RGB; token 0 = transparent) |
| `pixmoji/` | self-contained inference code (model, sampler, reranker, selection) |
## Quick start
```bash
pip install torch transformers safetensors huggingface_hub pillow numpy
```
```python
import sys, torch
from huggingface_hub import snapshot_download
from PIL import Image
repo = snapshot_download("model-collapse/pixmoji") # weights + the small `pixmoji/` inference package
sys.path.insert(0, repo)
import pixmoji
gen = pixmoji.load(repo) # GPU if available
# on CPUs with AVX-512 BF16 (e.g. AMD Zen 4, Intel Sapphire Rapids), ~2x faster:
# gen = pixmoji.load(repo, device="cpu", dtype=torch.bfloat16)
emojis = gen.generate_emojis("cat wearing a tiny crown", k=10, seed=0)
for i, e in enumerate(emojis): # e.rgba: (16, 16, 4) uint8, e.score: TinyCLIP text-image similarity
Image.fromarray(e.rgba, "RGBA").resize((128, 128), Image.NEAREST).save(f"emoji_{i}.png")
```
(To `git clone` the repo instead, install [Git LFS](https://git-lfs.com) first so the weight files are downloaded.)
`generate_emojis` returns up to `k` distinct emojis (near-duplicates are dropped, so a simple prompt can return fewer).
## How it works
- **Generator:** a 12-layer, 512-wide bidirectional transformer over the 256 pixels of a 16×16 sprite, each pixel a token in a
4096-colour palette (+ transparent). It fills the sprite in over 8–12 parallel decoding steps (MaskGIT-style), starting fully
masked and fixing the most confident pixels each step.
- **Conditioning:** the frozen text tower of [TinyCLIP-39M-PixelArt](https://huggingface.co/tomaarsen/TinyCLIP-39M-PixelArt)
(pooled embedding + per-word features) and a count token parsed from the prompt (one…four).
- **Selection:** 24 candidates are generated and reranked by TinyCLIP image–text similarity; up to 10 diverse ones are returned.
- **Two models, routed by prompt:**
- `pixmoji-v2.1` uses classifier-free guidance (two passes per step; guidance 5 for counts). Prompts with a count ≥ 2 use it.
- `pixmoji-v2.2-fast` is v2.1 **guidance-distilled** so one pass per step reproduces v2.1's guided output, trained on
states the sampler actually visits (192k recorded v2.1 sampling runs). With 8 steps it needs **one third of v2.1's
compute** at the same measured quality.
## Training data
- [Angais/pixel-art-atlas-1.4m](https://huggingface.co/datasets/Angais/pixel-art-atlas-1.4m) (CC0): 1.4M 16×16 sprites with
captions, itself generated with Krea 2 and a pixel-art LoRA. Captions were corrected for object counts with Anthropic Claude
Haiku 5.5.
- ~38k additional sprites for counts and combinations, generated with Krea 2 Turbo + the
[Angais/Krea-2-PixelArt-16x16](https://huggingface.co/Angais/Krea-2-PixelArt-16x16) LoRA and filtered by Claude Haiku 5.5
(count check, "shows both parts" check) and TinyCLIP.
- Composites of 2–4 sprites for count training. Evaluation concepts were blocked from the generated training data.
pixmoji is a separately trained model; it does not contain Krea 2 weights.
## Evaluation
Judges: **Anthropic Claude Sonnet 5.5** and **OpenAI GPT-6.1 Terra**, both via Amazon Bedrock, each image pair shown in both
orders. "Re-seed" = v2.1 run again with other seeds (identical quality by construction), the reference for noise.
| | v2.1 (base) |
|---|---|
| Novel combinations depicted (30 prompts × 10) | 0.56 |
| Count accuracy, 1–4 objects (Terra / Sonnet) | 0.76 / 0.67 |
| Count accuracy, 3–4 objects | 0.49–0.65 |
| Judged better than the real dataset sprite (parity = 0.50; Sonnet / Terra) | 0.48 / 0.25 |
pixmoji-v2.2-fast vs v2.1 (300 fresh single-object captions, all 10 shown emojis judged; positive = v2.1 preferred):
| | re-seed | **v2.2-fast, 8 steps** |
|---|---|---|
| Δ share of emojis depicting the caption (Sonnet / Terra) | +0.9 / −1.3 pts | +1.0 / −1.5 pts |
| set head-to-head margin (Sonnet / Terra) | +4.7 / +9.0 | +4.0 / +12.7 |
| novel combinations depicted (Sonnet / Terra; v2.1: 0.53 / 0.62) | 0.56 / 0.56 | 0.53 / 0.61 |
Also within noise in two independent 720-caption head-to-head rounds. Fewer steps (6), fewer candidates, smaller or pruned
students, int4 weights and low-rank layers were all measured to lose quality and are not used.
## Speed (10 emojis per request)
| hardware | prompt without a count (v2.2-fast) | count prompt (v2.1) |
|---|---|---|
| NVIDIA A100 (shared with another job), fp16 | ~0.13 s | ~0.25–0.30 s |
| AMD EPYC 9R14, 64 cores, bf16, threads bound to one socket | 0.93 s | 3.3 s (with `cpu_fast="counts"`) |
| same CPU, 8 cores | 4.6 s | 12.4 s |
CPU tips: `OMP_PROC_BIND=close OMP_PLACES=cores`, keep threads within one socket, `dtype=torch.bfloat16` on CPUs with
AVX-512 BF16 (the fast model keeps its final colour layer in fp32, which the evaluation showed is needed).
## Limitations
- 16×16 only; small details (an egg next to a hen, a helmet on a penguin) are often missing.
- Counts of 3–4 objects are right only about half to two thirds of the time.
- Spelling matters: misspelled words ("helicorpter") can produce unrelated objects.
- Simple prompts sometimes yield fewer than 10 distinct emojis.
- Images are AI-generated; the model can reflect biases of its training data and of the teacher models.
## Intended use and license
Weights and code are released under **CC BY-NC 4.0** (non-commercial use, attribution required). Intended for demos, research
and personal projects such as chat emojis. Not intended for commercial use, impersonation, or generating hateful or sexual
content. Credit: Tom Aarsen (TinyCLIP-PixelArt, Pixel Atlas), Angais (Pixel Atlas dataset and Krea-2-PixelArt-16x16 LoRA).
|