--- license: mit library_name: pytorch pipeline_tag: text-to-image tags: - pixel-art - weight-space-programming - hand-written-weights - no-training - tiny-model - pico-8 - toy language: - en ---
# πŸ•ΉοΈ PixelGen2 **A text-to-pixel-art model with 6,747 parameters. Zero of them were trained.** Every weight was typed in by hand. It draws 21,866 distinct sprites from 28 templates. 12 bodies Γ— 8 accessories, two colours each, generated by PixelGen2 Every sprite above is one prompt: "<colour> <body> with <colour> <accessory>". No gradients were harmed.
![params](https://img.shields.io/badge/parameters-6%2C747-ff004d?style=flat-square) ![outputs](https://img.shields.io/badge/distinct%20sprites-21%2C866-29adff?style=flat-square) ![training](https://img.shields.io/badge/training%20steps-0-00e436?style=flat-square) ![palette](https://img.shields.io/badge/palette-PICO--8-ffec27?style=flat-square) ![license](https://img.shields.io/badge/license-MIT-83769c?style=flat-square)
--- ## What this is A tiny PyTorch `nn.Module` that turns a prompt like `"red potion"` or `"purple ghost with gold crown"` into a 16Γ—16 pixel-art sprite in the PICO-8 palette, upscaled to 128Γ—128. It's an experiment in **weight-space programming**, inspired by [this post](https://rikiyat.github.io/blog/weight-space-programming/): instead of training a network, you write a program and *compile it into weights*. Linear layers become lookup tables, argmax becomes a decision, a 1Γ—1 conv becomes a palette. The model is a real `nn.Module` β€” you can call `.forward()`, save a `state_dict`, load it back β€” but no optimizer ever touched it. There are two models in the repo: | | `PixelGen` (v1) | `PixelGen2` (v2) | |---|---|---| | prompt | `" "` | `"[small] [with ]"` | | parameters | 4,227 | 6,747 | | hand-drawn templates | 21 bodies | 21 bodies + 7 accessories | | distinct sprites (seed 0) | 189 | **21,866** | | shading | fixed row bands | structural rim light (two 3Γ—3 convs) | v1 is the minimal version, published separately as [DJLougen/pixelgen](https://huggingface.co/DJLougen/pixelgen); v2 is the same thing plus four compositional layers, and is what the CLI uses by default. This repo contains both. Everything below was measured, not estimated β€” scripts in `experiments/`. ## How it works Nine steps for v1. Each is an ordinary tensor op; together they're a program. ``` "red potion" β”‚ β–Ό 1. bag of words prompt β†’ count vector over a 138-word vocab 2. class WTA Linear(138β†’21) β†’ argmax β†’ one-hot body [potion] 3. shape bank one-hot selects a 16Γ—8 half-template; '?' cells flip on when seed-noise β‰₯ 0.5 4. mirror half + horizontal flip β†’ 16Γ—16 body (always symmetric) 5. outline 3Γ—3 dilate(body) βˆ’ body β†’ outline pixels (palette index 1) 6. shading rows 0–6 light, 7–10 mid, 11–15 dark 7. ramp WTA Linear(138β†’9) β†’ argmax β†’ colour ramp [red] 8. one-hot palette every pixel β†’ one-hot over 16 PICO-8 colours 9. palette conv Conv2d(16β†’3, 1Γ—1) β†’ RGB β”‚ β–Ό 16Γ—16 sprite β†’ nearest-neighbour Γ—8 β†’ 128Γ—128 PNG ``` v2 inserts four more fixed-weight layers (`pixelgen/compose.py`): | layer | mechanism | params | |---|---|---| | **accessory** | `Linear(138β†’8)` WTA selects a 16Γ—8 part (crown, halo, horns, helmet, antenna, wings, torch) overlaid on the body with `max`. `none` wins ties. | 1,112 | | **second colour** | the prompt is split at `with`/`wearing`/`holding`/`and`; a second `Linear(138β†’9)` colours the accessory from the words after it. No colour word there β†’ body colour wins via a 0.5 one-hot bonus. | 1,251 | | **structural shading** | two 3Γ—3 convs with a single tap each: a pixel is *light* if its up-left neighbour is empty, *dark* if its down-right neighbour is. Light from the top-left, for every shape, for 18 parameters. | 18 | | **size** | `small`/`tiny`/`mini` β†’ nearest-neighbour 0.75Γ— resample of the body map, bottom-centred. | 139 | The "generative" part is step 3: templates have `?` cells gated by per-seed noise, so the same prompt gives small variations across seeds:
blue invader, seeds 0–7 "blue invader", seeds 0 through 7
## Numbers | | v1 | v2 | |---|---|---| | Registered parameters | 4,227 | **6,747** | | Non-trainable buffers (template / shade / ramp / part banks) | 6,171 | 7,333 | | Total values | 10,398 (β‰ˆ41.6 KB fp32) | 14,080 (β‰ˆ56.3 KB fp32) | | Training steps | **0** | **0** | | Optimizers, backward passes, `requires_grad=True` in the repo | 0 (grep-verified) | 0 | | Bodies / accessories / colour ramps | 21 / – / 9 | 21 / 7 / 9 | | Distinct images at seed 0 | 189 of 189 prompts | 21,866 of 24,192 prompts | | Output | 16Γ—16 β†’ 128Γ—128 RGB, every pixel ∈ PICO-8 | same | | `weights/pixelgen.pt` (v1) | 45.7 KB, SHA-256 `4c5dffd1…540eb`, identical across rebuilds | – |
Parameter breakdown (v2; v1 is the first six rows) | Tensor | Shape | Count | What it is | |---|---|---|---| | `word_to_class.weight` | 21 Γ— 138 | 2,898 | vocab β†’ body scores (synonym table) | | `word_to_class.bias` | 21 | 21 | default body (potion) | | `word_to_ramp.weight` | 9 Γ— 138 | 1,242 | vocab β†’ colour-ramp scores | | `word_to_ramp.bias` | 9 | 9 | default ramp (green) | | `outline.weight` | 1Γ—1Γ—3Γ—3 | 9 | dilation kernel | | `palette_conv.weight` | 3Γ—16Γ—1Γ—1 | 48 | one-hot colour β†’ RGB (exact PICO-8 Γ· 255) | | `word_to_acc.weight` / `.bias` | 8 Γ— 138, 8 | 1,112 | vocab β†’ accessory scores, `none` biased | | `word_to_ramp2.weight` / `.bias` | 9 Γ— 138, 9 | 1,251 | accessory-half vocab β†’ ramp | | `lit.weight`, `shadow.weight` | 1Γ—1Γ—3Γ—3 Γ—2 | 18 | up-left tap, down-right tap | | `word_to_size.weight` / `.bias` | 1 Γ— 138, 1 | 139 | size words β†’ small flag | | **Parameters** | | **6,747** | | | `on_bank` / `free_bank` | 21Γ—16Γ—8 Γ—2 | 5,376 | body templates (`#` solid, `?` noise-controlled) | | `acc_bank` | 8Γ—16Γ—8 | 1,024 | accessory templates | | `shade_bank` | 3Γ—16Γ—16 | 768 | row-band base shading | | `ramp_table` | 9Γ—3 | 27 | palette indices per ramp | | `acc_mask_a` | 138 | 138 | body-class words never mean a hat in the body half | | **Buffers** | | **7,333** | |
## How to use ```bash git clone https://huggingface.co/DJLougen/pixelgen2 && cd pixelgen2 pip install torch numpy pillow # that's the whole dependency list python generate.py "red potion" --out out/red_potion.png python generate.py "green slime with gold crown" --seed 3 --scale 16 python generate.py "small angel skull" --out out/skull.png python generate.py "blue invader" --v1 # original single-part model ``` Or from Python: ```python from pixelgen.compose import PixelGen2 model = PixelGen2() # weights are built in __init__, no file needed img = model.sprite("purple ghost with horns", seed=4, scale=8) # PIL.Image, 128Γ—128 img.save("ghost.png") ``` **Prompt grammar.** `[small] [colour] [with [colour] ]`. Words are matched by a bag-of-words, so order doesn't matter within each half.
Vocabulary **Bodies:** potion (bottle, flask, elixir, vial) Β· sword (blade, dagger) Β· shield (buckler) Β· ship (rocket, spaceship, starship) Β· invader (alien, monster) Β· slime (blob, jelly) Β· skull (bone, skeleton) Β· tree (pine, forest) Β· key Β· heart (love, life) Β· star (sparkle) Β· mushroom (fungus, toadstool, shroom) Β· ghost (spirit, phantom, spooky) Β· bomb (grenade, explosive) Β· coin (token, money, medal) Β· gem (diamond, jewel, crystal) Β· cherry (cherries, berry, fruit) Β· crown Β· bat (vampire) Β· cactus (desert, succulent) Β· robot (droid, android, mech) **Accessories:** crown (king, queen, royal, crowned, prince, princess) Β· halo (angel, angelic, holy, saint) Β· horns (horned, devil, demon, evil) Β· helmet (knight, armor, armour, armored) Β· antenna (antennae, bug, martian) Β· wings (winged, fly, flying, bird, fairy, butterfly) Β· torch (torches, fire, flame, burning) **Colours:** red (crimson, ruby) Β· blue (azure, sapphire) Β· green (emerald, lime) Β· yellow (gold, golden) Β· purple (violet, amethyst) Β· orange (amber) Β· pink (rose) Β· brown (wood, wooden) Β· white (grey, gray, silver, bone) **Size:** small, tiny, mini, little. **Separators:** with, wearing, holding, and, plus. `"crown"` in the body half is the crown object; after a separator it's a hat. `"bone"` is both a skull synonym and a white synonym, so `"bone"` alone is a white skull. `"alien"` is an invader body and `"martian"` is an antenna accessory, so `"martian alien"` is what you'd hope.
## How far can you push it? Four experiments, all in `experiments/` with the scripts that produced them. Full notes in [`experiments/README.md`](experiments/README.md). **1. Adding a class is a pure append.** Freeze the 11-class model, add `mushroom`, diff. +400 values. Diffed by *index*, 84 old values changed; diffed by *meaning* (class name Γ— word), **zero** β€” the raw diff was entirely a column permutation from a `sorted()` vocab. 17/17 old prompts bit-identical over 3 seeds. The same check still passes after +10 classes and the whole accessory vocabulary. **2. Capacity is unbounded; addressing is quadratic.** Synthetic random classes, rebuilt and probed at each size: | classes | params | dense layer share | isolated | |---|---|---|---| | 21 | 2,937 | 22% | βœ“ | | 121 | 51,537 | 57% | βœ“ | | 1,021 | 3.2M | 92% | βœ“ | | 10,021 | 302M | 99.1% | βœ“ | It never breaks β€” the original 21 classes stay bit-identical at 10k. But vocab grows with classes, so `word_to_class` is β‰ˆ3CΒ² and 99.997% zeros. Under 100k params the dense design tops out near **170 classes**. **3. Compressing the class layer costs semantics.** A binary code (`d = ⌈logβ‚‚CβŒ‰`, words emit Β±1 codes, score = `d βˆ’ 2Β·hamming`) shrinks the class layer **528Γ—** at 10k classes with no loss of isolation β€” but 61% of two-body-word prompts (`"sword shield"`) now resolve to a *third* class, vs 0% dense. The additive bag-of-words semantics were the thing that got compressed away. **4. Composition is the lever.** v1 β†’ v2: 1.6Γ— params, **115Γ— distinct outputs**. 781 distinct images per hand-drawn template instead of 9.
row-band shading vs structural rim lighting Top: v1 row bands. Bottom: v2 structural lighting, 18 params.
Quality, honestly: structural lighting helps blobby shapes (potion, heart, mushroom, tree, gem) and **hurts thin ones** (invader, bat, robot) — when every pixel is an edge, rim light becomes speckle. A fixed 3×3 kernel has no way to know a shape is too thin to light. That's the limit of hand-written rules without a learned prior, and exactly the kind of thing a few hundred trained parameters would fix. ## Limitations - **It only knows what was drawn.** 21 bodies, 7 accessories. Prompt for a dragon and you get a potion (the default). Capacity is unbounded; *content* is a human with a text editor. - **Every shape is horizontally symmetric.** The mirror step makes 16×8 templates cheap; it also means `"torch"` is two torches and the key is a top-down key. Lighting (v2) is deliberately *not* symmetric. - **Accessories sit at fixed rows.** A crown floats above short bodies; a helmet merges into tall ones. Fixing that needs per-body anchor points — more drawing, not more mechanism. - **One accessory per sprite.** `"devil knight"` picks one by index order. - **Variation is cosmetic.** Seeds only toggle the `?` cells — a few pixels per sprite. - **Bag of words.** `"sword shield"` is a tie broken by index; `"not red"` is red; within each half, order is ignored. - **Structural shading speckles thin shapes.** See above. - **It's PyTorch for the bit, not for the need.** You could rewrite this in NumPy. Using `nn.Linear` and `nn.Conv2d` is the point of the exercise. - **Art quality is programmer art**, drawn in a text editor by someone who is not a pixel artist. ## Why bother? Because it's a nice demonstration that a lot of what we call "a model" is just *structure*, and structure can be written directly. A linear layer with the right weights *is* a dictionary. A conv with the right kernel *is* an edge detector. Winner-take-all *is* an `if`. The blog post argues this is a useful lens for understanding what trained networks are doing; this repo is the version where you skip the training and just write the weights — and then push on it to see where hand-writing stops being enough. Where it stops: addressing (quadratic), compression (breaks additivity), and quality rules at the edges (shading thin shapes). The natural next step is the hybrid — hand-write the ~95% that's structure, train only the small word→part map. Also it's cool that it worked. ## Repo contents ``` pixelgen/ model.py PixelGen (v1) — all weights set in __init__ compose.py PixelGen2 (v2) — v1 + accessory, second colour, structural shading, size templates.py 21 body half-templates ('#' solid, '?' noise, '.' empty) parts.py 7 accessory half-templates palette.py PICO-8 palette + 9 colour ramps tokenizer.py vocab, synonyms, prompt splitting weights.py save/load helpers build.py writes weights/pixelgen.pt (v1, deterministic) generate.py CLI (v2 default, --v1 for the original) tests/ 16 property tests experiments/ the four experiments above: scripts, frozen baselines, README showcase2.png the grid at the top seeds.png seed variation strip ``` Tests: `python -m pytest tests -q` → 16 passed. ## Citation ```bibtex @misc{pixelgen2_2025, title = {PixelGen2: a compositional hand-weighted pixel-art generator}, author = {Lougen, D. J.}, year = {2025}, url = {https://huggingface.co/DJLougen/pixelgen2}, note = {Weight-space programming experiment; no parameters were trained.} } ``` Inspired by *[Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming/)*.