Experiments: how far does hand-written capacity go?
Every number below was produced by a script in this folder. No training anywhere.
| script | question | answer |
|---|---|---|
add_class.py |
what changes when you add one class? | +400 values, 0 semantic edits to old weights, 0 behaviour change on old prompts |
scale_classes.py |
how many classes before it breaks? | it doesn't; 10,021 classes still bit-isolated — but the word→class matrix is O(C²) |
coded_classes.py |
can the class layer be compressed? | 528× at 10k classes via binary codes, at the cost of 61% multi-word interference |
count_outputs.py |
what does composition buy? | 21,866 distinct images from 28 templates; 115× output for 1.6× params |
1. Adding one class is a pure append (in function space)
before.pt is the frozen 11-class model. After adding mushroom:
tensor shapes on_bank (11,16,8)->(12,16,8) word_to_class (11,59)->(12,63) +400 values
raw index diff 84 old values "changed"
semantic diff 0 old (class,word) entries changed; shade/ramp/outline/palette identical
old prompts 17/17 bit-identical over 3 seeds; only mushroom-word prompts moved (potion -> mushroom)
The 84-value raw diff is entirely a column permutation: the vocab is sorted(), so inserting fungus at column 11 shifts 48 of 59 old words right. Diffed by meaning, nothing moved. Lesson: if you want weight diffs to be legible, make index assignment append-only.
The same check still passes after +10 hand-drawn classes and the accessory vocabulary: v1 renders the original 11 classes bit-identically.
2. Capacity is unbounded; addressing is quadratic
Synthetic random classes (3 nonsense words each), rebuilt and probed at each size:
classes vocab params buffers MB dense% gen ms isolated
21 95 2,937 6,171 0.04 21.9% 0.12 True
121 395 51,537 31,771 0.33 57.4% 0.13 True
1,021 3,095 3,188,937 262,171 13.80 91.6% 0.25 True
3,021 9,095 27,560,937 774,171 113.34 97.0% 1.45 True
10,021 30,095 301,862,937 2,566,171 1217.72 99.1% 11.95 True
isolated = the 21 real classes and probe prompts (red potion, dragon, "", bone) are bit-identical to the 21-class model, and the newest synthetic class resolves by all its synonyms. It never fails.
But vocab grows with classes, so word_to_class is V×C ≈ 3C² — at 10k classes it's 99.1% of the model and 99.997% zeros. Under a 100k-parameter budget the dense design tops out around 170 classes.
3. Compressing the class layer buys size, costs semantics
Replace Linear(V,C) with a binary code: each word emits its class's d=⌈log₂C⌉-bit code as ±1, a Linear(d,C) of code rows scores d − 2·hamming.
classes d dense params coded params ratio isolated interference
21 5 2,016 601 3.4x True 61%
121 7 47,916 3,733 12.8x True 61%
1,021 10 3,161,016 42,181 74.9x True 61%
3,021 12 27,479,016 148,413 185.2x True 61%
10,021 14 301,592,016 571,645 527.6x True 61%
Isolation survives. But interference — the share of "wordA wordB" prompts (over the 21 real classes) that resolve to a third class — jumps from 0% (dense; ties go to lower index) to 61%, flat across scale. The sum of two codes is nearest to some unrelated code. The bag-of-words layer's additive semantics were the thing being compressed away. Open question: do error-correcting or sparse k-hot codes recover 0% interference, and at what parameter cost?
4. Composition: the lever that actually matters
PixelGen2 (pixelgen/compose.py) adds four fixed-weight layers to the same model:
| layer | mechanism | params |
|---|---|---|
| accessory | Linear(V,8) WTA → 16×8 part overlaid with max |
1,112 + 1,162 buffer |
| second colour | Linear(V,9) over words after with/wearing/…; body ramp wins ties via a 0.5 one-hot bonus |
1,251 |
| structural shading | two 3×3 convs: light where up-left neighbour is empty, dark where down-right is | 18 |
| size | Linear(V,1) → nearest 0.75× resample, bottom-centred |
139 |
PixelGen : 4,227 params + 6,171 buffers | 189 prompts -> 189 distinct images
PixelGen2: 6,747 params + 7,333 buffers | 24,192 prompts -> 21,866 distinct images
hand-drawn content: 21 bodies + 7 parts = 28 templates
distinct images per template: 9 -> 781
param growth 1.60x output growth 115.7x
(The 2,326 duplicates are prompts whose accessory colour equals the body colour, or whose accessory is hidden behind the body.)
Quality, honestly
Top: row-band shading (v1). Bottom: structural rim lighting (v2), 18 parameters. It helps blobby shapes — potion, heart, mushroom, tree, gem get a readable highlight and shadow — and hurts thin ones: invader, bat, robot turn to speckle because every pixel is an edge. A fixed 3×3 kernel has no notion of "this shape is too thin to light". That's the limit of rule-based shading without a learned prior, and it's exactly the kind of thing a few hundred trained parameters would fix.
Accessories are placed at fixed rows, so a crown floats above short bodies and a helmet merges into tall ones. Fixing that properly needs per-body anchor points — more hand-written content, not more mechanism.
What this says about weight-space programming
- Adding capacity is free and modular. Hand-written weights are block-diagonal by construction; trained weights are not. Add-a-class is
dict.append. - The expensive part is the index, not the content. Dense lookup is O(C²); the content (templates) is O(C). Compression trades additivity for size.
- Composition beats enumeration by two orders of magnitude, and it's the only route to "many things" that doesn't scale a human's drawing time linearly.
- Quality rules written by hand are brittle at the edges — the shading result is the clearest example. The honest endpoint for pure weight-space programming is a very capable procedural generator with legible, auditable failure modes, and zero creativity outside what was drawn.
- The natural next step is the hybrid the blog post gestures at: hand-write the structure (parts, compositing, palette, transforms — ~95% of params), and train only the small word→part mapping, so new vocabulary becomes learnable while everything downstream stays readable.