|
Download README.md from model-collapse/pixmoji: direct link, hf CLI and curl.
- Browser
- Download file 6.58 kB
-
https://huggingface.co/model-collapse/pixmoji/resolve/main/README.md
- Command line
-
hf download hf://model-collapse/pixmoji/README.md
-
curl -L -o README.md https://huggingface.co/model-collapse/pixmoji/resolve/main/README.md
6.58 kB
| license: cc-by-nc-4.0 | |
| library_name: pytorch | |
| pipeline_tag: text-to-image | |
| tags: | |
| - pixel-art | |
| - emoji | |
| - 16x16 | |
| - maskgit | |
| - text-to-image | |
| datasets: | |
| - Angais/pixel-art-atlas-1.4m | |
| # pixmoji — type a phrase, get 16×16 pixel-art emojis | |
| pixmoji generates small **16×16 pixel-art sprites** from short text ("lovely sheep", "cat wearing a tiny crown", | |
| "three blue apples") fast enough to run as you type. Every image is **generated** by the model; nothing is retrieved. | |
|  | |
| | file | what it is | | |
| |---|---| | |
| | `pixmoji-v2.2-fast.safetensors` | the fast model (43.6M params): single pass per step, used for all prompts without a count | | |
| | `pixmoji-v2.1.safetensors` | the base model (43.6M), used for prompts with a count ≥ 2 ("three blue apples") | | |
| | `palette.npy` | the 4096-colour palette (token → RGB; token 0 = transparent) | | |
| | `pixmoji/` | self-contained inference code (model, sampler, reranker, selection) | | |
| ## Quick start | |
| ```bash | |
| pip install torch transformers safetensors huggingface_hub pillow numpy | |
| ``` | |
| ```python | |
| import sys, torch | |
| from huggingface_hub import snapshot_download | |
| from PIL import Image | |
| repo = snapshot_download("model-collapse/pixmoji") # weights + the small `pixmoji/` inference package | |
| sys.path.insert(0, repo) | |
| import pixmoji | |
| gen = pixmoji.load(repo) # GPU if available | |
| # on CPUs with AVX-512 BF16 (e.g. AMD Zen 4, Intel Sapphire Rapids), ~2x faster: | |
| # gen = pixmoji.load(repo, device="cpu", dtype=torch.bfloat16) | |
| emojis = gen.generate_emojis("cat wearing a tiny crown", k=10, seed=0) | |
| for i, e in enumerate(emojis): # e.rgba: (16, 16, 4) uint8, e.score: TinyCLIP text-image similarity | |
| Image.fromarray(e.rgba, "RGBA").resize((128, 128), Image.NEAREST).save(f"emoji_{i}.png") | |
| ``` | |
| (To `git clone` the repo instead, install [Git LFS](https://git-lfs.com) first so the weight files are downloaded.) | |
| `generate_emojis` returns up to `k` distinct emojis (near-duplicates are dropped, so a simple prompt can return fewer). | |
| ## How it works | |
| - **Generator:** a 12-layer, 512-wide bidirectional transformer over the 256 pixels of a 16×16 sprite, each pixel a token in a | |
| 4096-colour palette (+ transparent). It fills the sprite in over 8–12 parallel decoding steps (MaskGIT-style), starting fully | |
| masked and fixing the most confident pixels each step. | |
| - **Conditioning:** the frozen text tower of [TinyCLIP-39M-PixelArt](https://huggingface.co/tomaarsen/TinyCLIP-39M-PixelArt) | |
| (pooled embedding + per-word features) and a count token parsed from the prompt (one…four). | |
| - **Selection:** 24 candidates are generated and reranked by TinyCLIP image–text similarity; up to 10 diverse ones are returned. | |
| - **Two models, routed by prompt:** | |
| - `pixmoji-v2.1` uses classifier-free guidance (two passes per step; guidance 5 for counts). Prompts with a count ≥ 2 use it. | |
| - `pixmoji-v2.2-fast` is v2.1 **guidance-distilled** so one pass per step reproduces v2.1's guided output, trained on | |
| states the sampler actually visits (192k recorded v2.1 sampling runs). With 8 steps it needs **one third of v2.1's | |
| compute** at the same measured quality. | |
| ## Training data | |
| - [Angais/pixel-art-atlas-1.4m](https://huggingface.co/datasets/Angais/pixel-art-atlas-1.4m) (CC0): 1.4M 16×16 sprites with | |
| captions, itself generated with Krea 2 and a pixel-art LoRA. Captions were corrected for object counts with Anthropic Claude | |
| Haiku 5.5. | |
| - ~38k additional sprites for counts and combinations, generated with Krea 2 Turbo + the | |
| [Angais/Krea-2-PixelArt-16x16](https://huggingface.co/Angais/Krea-2-PixelArt-16x16) LoRA and filtered by Claude Haiku 5.5 | |
| (count check, "shows both parts" check) and TinyCLIP. | |
| - Composites of 2–4 sprites for count training. Evaluation concepts were blocked from the generated training data. | |
| pixmoji is a separately trained model; it does not contain Krea 2 weights. | |
| ## Evaluation | |
| Judges: **Anthropic Claude Sonnet 5.5** and **OpenAI GPT-6.1 Terra**, both via Amazon Bedrock, each image pair shown in both | |
| orders. "Re-seed" = v2.1 run again with other seeds (identical quality by construction), the reference for noise. | |
| | | v2.1 (base) | | |
| |---|---| | |
| | Novel combinations depicted (30 prompts × 10) | 0.56 | | |
| | Count accuracy, 1–4 objects (Terra / Sonnet) | 0.76 / 0.67 | | |
| | Count accuracy, 3–4 objects | 0.49–0.65 | | |
| | Judged better than the real dataset sprite (parity = 0.50; Sonnet / Terra) | 0.48 / 0.25 | | |
| pixmoji-v2.2-fast vs v2.1 (300 fresh single-object captions, all 10 shown emojis judged; positive = v2.1 preferred): | |
| | | re-seed | **v2.2-fast, 8 steps** | | |
| |---|---|---| | |
| | Δ share of emojis depicting the caption (Sonnet / Terra) | +0.9 / −1.3 pts | +1.0 / −1.5 pts | | |
| | set head-to-head margin (Sonnet / Terra) | +4.7 / +9.0 | +4.0 / +12.7 | | |
| | novel combinations depicted (Sonnet / Terra; v2.1: 0.53 / 0.62) | 0.56 / 0.56 | 0.53 / 0.61 | | |
| Also within noise in two independent 720-caption head-to-head rounds. Fewer steps (6), fewer candidates, smaller or pruned | |
| students, int4 weights and low-rank layers were all measured to lose quality and are not used. | |
| ## Speed (10 emojis per request) | |
| | hardware | prompt without a count (v2.2-fast) | count prompt (v2.1) | | |
| |---|---|---| | |
| | NVIDIA A100 (shared with another job), fp16 | ~0.13 s | ~0.25–0.30 s | | |
| | AMD EPYC 9R14, 64 cores, bf16, threads bound to one socket | 0.93 s | 3.3 s (with `cpu_fast="counts"`) | | |
| | same CPU, 8 cores | 4.6 s | 12.4 s | | |
| CPU tips: `OMP_PROC_BIND=close OMP_PLACES=cores`, keep threads within one socket, `dtype=torch.bfloat16` on CPUs with | |
| AVX-512 BF16 (the fast model keeps its final colour layer in fp32, which the evaluation showed is needed). | |
| ## Limitations | |
| - 16×16 only; small details (an egg next to a hen, a helmet on a penguin) are often missing. | |
| - Counts of 3–4 objects are right only about half to two thirds of the time. | |
| - Spelling matters: misspelled words ("helicorpter") can produce unrelated objects. | |
| - Simple prompts sometimes yield fewer than 10 distinct emojis. | |
| - Images are AI-generated; the model can reflect biases of its training data and of the teacher models. | |
| ## Intended use and license | |
| Weights and code are released under **CC BY-NC 4.0** (non-commercial use, attribution required). Intended for demos, research | |
| and personal projects such as chat emojis. Not intended for commercial use, impersonation, or generating hateful or sexual | |
| content. Credit: Tom Aarsen (TinyCLIP-PixelArt, Pixel Atlas), Angais (Pixel Atlas dataset and Krea-2-PixelArt-16x16 LoRA). | |