--- license: cc-by-nc-4.0 library_name: pytorch pipeline_tag: text-to-image tags: - pixel-art - emoji - 16x16 - maskgit - text-to-image datasets: - Angais/pixel-art-atlas-1.4m --- # pixmoji — type a phrase, get 16×16 pixel-art emojis pixmoji generates small **16×16 pixel-art sprites** from short text ("lovely sheep", "cat wearing a tiny crown", "three blue apples") fast enough to run as you type. Every image is **generated** by the model; nothing is retrieved. ![sample sheet: 30 prompts × 10 generated emojis](sample_sheet.png) | file | what it is | |---|---| | `pixmoji-v2.2-fast.safetensors` | the fast model (43.6M params): single pass per step, used for all prompts without a count | | `pixmoji-v2.1.safetensors` | the base model (43.6M), used for prompts with a count ≥ 2 ("three blue apples") | | `palette.npy` | the 4096-colour palette (token → RGB; token 0 = transparent) | | `pixmoji/` | self-contained inference code (model, sampler, reranker, selection) | ## Quick start ```bash pip install torch transformers safetensors huggingface_hub pillow numpy ``` ```python import sys, torch from huggingface_hub import snapshot_download from PIL import Image repo = snapshot_download("model-collapse/pixmoji") # weights + the small `pixmoji/` inference package sys.path.insert(0, repo) import pixmoji gen = pixmoji.load(repo) # GPU if available # on CPUs with AVX-512 BF16 (e.g. AMD Zen 4, Intel Sapphire Rapids), ~2x faster: # gen = pixmoji.load(repo, device="cpu", dtype=torch.bfloat16) emojis = gen.generate_emojis("cat wearing a tiny crown", k=10, seed=0) for i, e in enumerate(emojis): # e.rgba: (16, 16, 4) uint8, e.score: TinyCLIP text-image similarity Image.fromarray(e.rgba, "RGBA").resize((128, 128), Image.NEAREST).save(f"emoji_{i}.png") ``` (To `git clone` the repo instead, install [Git LFS](https://git-lfs.com) first so the weight files are downloaded.) `generate_emojis` returns up to `k` distinct emojis (near-duplicates are dropped, so a simple prompt can return fewer). ## How it works - **Generator:** a 12-layer, 512-wide bidirectional transformer over the 256 pixels of a 16×16 sprite, each pixel a token in a 4096-colour palette (+ transparent). It fills the sprite in over 8–12 parallel decoding steps (MaskGIT-style), starting fully masked and fixing the most confident pixels each step. - **Conditioning:** the frozen text tower of [TinyCLIP-39M-PixelArt](https://huggingface.co/tomaarsen/TinyCLIP-39M-PixelArt) (pooled embedding + per-word features) and a count token parsed from the prompt (one…four). - **Selection:** 24 candidates are generated and reranked by TinyCLIP image–text similarity; up to 10 diverse ones are returned. - **Two models, routed by prompt:** - `pixmoji-v2.1` uses classifier-free guidance (two passes per step; guidance 5 for counts). Prompts with a count ≥ 2 use it. - `pixmoji-v2.2-fast` is v2.1 **guidance-distilled** so one pass per step reproduces v2.1's guided output, trained on states the sampler actually visits (192k recorded v2.1 sampling runs). With 8 steps it needs **one third of v2.1's compute** at the same measured quality. ## Training data - [Angais/pixel-art-atlas-1.4m](https://huggingface.co/datasets/Angais/pixel-art-atlas-1.4m) (CC0): 1.4M 16×16 sprites with captions, itself generated with Krea 2 and a pixel-art LoRA. Captions were corrected for object counts with Anthropic Claude Haiku 5.5. - ~38k additional sprites for counts and combinations, generated with Krea 2 Turbo + the [Angais/Krea-2-PixelArt-16x16](https://huggingface.co/Angais/Krea-2-PixelArt-16x16) LoRA and filtered by Claude Haiku 5.5 (count check, "shows both parts" check) and TinyCLIP. - Composites of 2–4 sprites for count training. Evaluation concepts were blocked from the generated training data. pixmoji is a separately trained model; it does not contain Krea 2 weights. ## Evaluation Judges: **Anthropic Claude Sonnet 5.5** and **OpenAI GPT-6.1 Terra**, both via Amazon Bedrock, each image pair shown in both orders. "Re-seed" = v2.1 run again with other seeds (identical quality by construction), the reference for noise. | | v2.1 (base) | |---|---| | Novel combinations depicted (30 prompts × 10) | 0.56 | | Count accuracy, 1–4 objects (Terra / Sonnet) | 0.76 / 0.67 | | Count accuracy, 3–4 objects | 0.49–0.65 | | Judged better than the real dataset sprite (parity = 0.50; Sonnet / Terra) | 0.48 / 0.25 | pixmoji-v2.2-fast vs v2.1 (300 fresh single-object captions, all 10 shown emojis judged; positive = v2.1 preferred): | | re-seed | **v2.2-fast, 8 steps** | |---|---|---| | Δ share of emojis depicting the caption (Sonnet / Terra) | +0.9 / −1.3 pts | +1.0 / −1.5 pts | | set head-to-head margin (Sonnet / Terra) | +4.7 / +9.0 | +4.0 / +12.7 | | novel combinations depicted (Sonnet / Terra; v2.1: 0.53 / 0.62) | 0.56 / 0.56 | 0.53 / 0.61 | Also within noise in two independent 720-caption head-to-head rounds. Fewer steps (6), fewer candidates, smaller or pruned students, int4 weights and low-rank layers were all measured to lose quality and are not used. ## Speed (10 emojis per request) | hardware | prompt without a count (v2.2-fast) | count prompt (v2.1) | |---|---|---| | NVIDIA A100 (shared with another job), fp16 | ~0.13 s | ~0.25–0.30 s | | AMD EPYC 9R14, 64 cores, bf16, threads bound to one socket | 0.93 s | 3.3 s (with `cpu_fast="counts"`) | | same CPU, 8 cores | 4.6 s | 12.4 s | CPU tips: `OMP_PROC_BIND=close OMP_PLACES=cores`, keep threads within one socket, `dtype=torch.bfloat16` on CPUs with AVX-512 BF16 (the fast model keeps its final colour layer in fp32, which the evaluation showed is needed). ## Limitations - 16×16 only; small details (an egg next to a hen, a helmet on a penguin) are often missing. - Counts of 3–4 objects are right only about half to two thirds of the time. - Spelling matters: misspelled words ("helicorpter") can produce unrelated objects. - Simple prompts sometimes yield fewer than 10 distinct emojis. - Images are AI-generated; the model can reflect biases of its training data and of the teacher models. ## Intended use and license Weights and code are released under **CC BY-NC 4.0** (non-commercial use, attribution required). Intended for demos, research and personal projects such as chat emojis. Not intended for commercial use, impersonation, or generating hateful or sexual content. Credit: Tom Aarsen (TinyCLIP-PixelArt, Pixel Atlas), Angais (Pixel Atlas dataset and Krea-2-PixelArt-16x16 LoRA).