| # Experiments: how far does hand-written capacity go? |
|
|
| Every number below was produced by a script in this folder. No training anywhere. |
|
|
| | script | question | answer | |
| |---|---|---| |
| | `add_class.py` | what changes when you add one class? | +400 values, 0 semantic edits to old weights, 0 behaviour change on old prompts | |
| | `scale_classes.py` | how many classes before it breaks? | it doesn't; 10,021 classes still bit-isolated — but the word→class matrix is O(C²) | |
| | `coded_classes.py` | can the class layer be compressed? | 528× at 10k classes via binary codes, at the cost of 61% multi-word interference | |
| | `count_outputs.py` | what does composition buy? | 21,866 distinct images from 28 templates; 115× output for 1.6× params | |
|
|
| ## 1. Adding one class is a pure append (in function space) |
|
|
| `before.pt` is the frozen 11-class model. After adding `mushroom`: |
|
|
| ``` |
| tensor shapes on_bank (11,16,8)->(12,16,8) word_to_class (11,59)->(12,63) +400 values |
| raw index diff 84 old values "changed" |
| semantic diff 0 old (class,word) entries changed; shade/ramp/outline/palette identical |
| old prompts 17/17 bit-identical over 3 seeds; only mushroom-word prompts moved (potion -> mushroom) |
| ``` |
|
|
| The 84-value raw diff is entirely a **column permutation**: the vocab is `sorted()`, so inserting `fungus` at column 11 shifts 48 of 59 old words right. Diffed by meaning, nothing moved. Lesson: if you want weight diffs to be legible, make index assignment append-only. |
|
|
| The same check still passes after +10 hand-drawn classes and the accessory vocabulary: v1 renders the original 11 classes bit-identically. |
|
|
| ## 2. Capacity is unbounded; addressing is quadratic |
|
|
| Synthetic random classes (3 nonsense words each), rebuilt and probed at each size: |
|
|
| ``` |
| classes vocab params buffers MB dense% gen ms isolated |
| 21 95 2,937 6,171 0.04 21.9% 0.12 True |
| 121 395 51,537 31,771 0.33 57.4% 0.13 True |
| 1,021 3,095 3,188,937 262,171 13.80 91.6% 0.25 True |
| 3,021 9,095 27,560,937 774,171 113.34 97.0% 1.45 True |
| 10,021 30,095 301,862,937 2,566,171 1217.72 99.1% 11.95 True |
| ``` |
|
|
| `isolated` = the 21 real classes and probe prompts (`red potion`, `dragon`, `""`, `bone`) are bit-identical to the 21-class model, and the newest synthetic class resolves by all its synonyms. It never fails. |
|
|
| But vocab grows with classes, so `word_to_class` is V×C ≈ 3C² — at 10k classes it's 99.1% of the model and 99.997% zeros. Under a 100k-parameter budget the dense design tops out around **170 classes**. |
|
|
| ## 3. Compressing the class layer buys size, costs semantics |
|
|
| Replace `Linear(V,C)` with a binary code: each word emits its class's d=⌈log₂C⌉-bit code as ±1, a `Linear(d,C)` of code rows scores `d − 2·hamming`. |
|
|
| ``` |
| classes d dense params coded params ratio isolated interference |
| 21 5 2,016 601 3.4x True 61% |
| 121 7 47,916 3,733 12.8x True 61% |
| 1,021 10 3,161,016 42,181 74.9x True 61% |
| 3,021 12 27,479,016 148,413 185.2x True 61% |
| 10,021 14 301,592,016 571,645 527.6x True 61% |
| ``` |
|
|
| Isolation survives. But `interference` — the share of `"wordA wordB"` prompts (over the 21 real classes) that resolve to a **third** class — jumps from 0% (dense; ties go to lower index) to 61%, flat across scale. The sum of two codes is nearest to some unrelated code. The bag-of-words layer's additive semantics were the thing being compressed away. Open question: do error-correcting or sparse k-hot codes recover 0% interference, and at what parameter cost? |
|
|
| ## 4. Composition: the lever that actually matters |
|
|
| `PixelGen2` (`pixelgen/compose.py`) adds four fixed-weight layers to the same model: |
|
|
| | layer | mechanism | params | |
| |---|---|---| |
| | accessory | `Linear(V,8)` WTA → 16×8 part overlaid with `max` | 1,112 + 1,162 buffer | |
| | second colour | `Linear(V,9)` over words after `with`/`wearing`/…; body ramp wins ties via a 0.5 one-hot bonus | 1,251 | |
| | structural shading | two 3×3 convs: light where up-left neighbour is empty, dark where down-right is | 18 | |
| | size | `Linear(V,1)` → nearest 0.75× resample, bottom-centred | 139 | |
|
|
| ``` |
| PixelGen : 4,227 params + 6,171 buffers | 189 prompts -> 189 distinct images |
| PixelGen2: 6,747 params + 7,333 buffers | 24,192 prompts -> 21,866 distinct images |
| hand-drawn content: 21 bodies + 7 parts = 28 templates |
| distinct images per template: 9 -> 781 |
| param growth 1.60x output growth 115.7x |
| ``` |
|
|
| (The 2,326 duplicates are prompts whose accessory colour equals the body colour, or whose accessory is hidden behind the body.) |
|
|
| ### Quality, honestly |
|
|
|  |
|
|
| Top: row-band shading (v1). Bottom: structural rim lighting (v2), 18 parameters. It helps blobby shapes — potion, heart, mushroom, tree, gem get a readable highlight and shadow — and **hurts thin ones**: invader, bat, robot turn to speckle because every pixel is an edge. A fixed 3×3 kernel has no notion of "this shape is too thin to light". That's the limit of rule-based shading without a learned prior, and it's exactly the kind of thing a few hundred trained parameters would fix. |
|
|
| Accessories are placed at fixed rows, so a crown floats above short bodies and a helmet merges into tall ones. Fixing that properly needs per-body anchor points — more hand-written content, not more mechanism. |
|
|
| ## What this says about weight-space programming |
|
|
| - **Adding capacity is free and modular.** Hand-written weights are block-diagonal by construction; trained weights are not. Add-a-class is `dict.append`. |
| - **The expensive part is the index, not the content.** Dense lookup is O(C²); the content (templates) is O(C). Compression trades additivity for size. |
| - **Composition beats enumeration by two orders of magnitude**, and it's the only route to "many things" that doesn't scale a human's drawing time linearly. |
| - **Quality rules written by hand are brittle at the edges** — the shading result is the clearest example. The honest endpoint for pure weight-space programming is a very capable procedural generator with legible, auditable failure modes, and zero creativity outside what was drawn. |
| - The natural next step is the hybrid the blog post gestures at: hand-write the structure (parts, compositing, palette, transforms — ~95% of params), and train only the small word→part mapping, so new vocabulary becomes learnable while everything downstream stays readable. |
|
|
