# Experiments: how far does hand-written capacity go?

Every number below was produced by a script in this folder. No training anywhere.

| script | question | answer |
|---|---|---|
| `add_class.py` | what changes when you add one class? | +400 values, 0 semantic edits to old weights, 0 behaviour change on old prompts |
| `scale_classes.py` | how many classes before it breaks? | it doesn't; 10,021 classes still bit-isolated — but the word→class matrix is O(C²) |
| `coded_classes.py` | can the class layer be compressed? | 528× at 10k classes via binary codes, at the cost of 61% multi-word interference |
| `count_outputs.py` | what does composition buy? | 21,866 distinct images from 28 templates; 115× output for 1.6× params |

## 1. Adding one class is a pure append (in function space)

`before.pt` is the frozen 11-class model. After adding `mushroom`:

```
tensor shapes      on_bank (11,16,8)->(12,16,8)   word_to_class (11,59)->(12,63)   +400 values
raw index diff     84 old values "changed"
semantic diff      0 old (class,word) entries changed; shade/ramp/outline/palette identical
old prompts        17/17 bit-identical over 3 seeds; only mushroom-word prompts moved (potion -> mushroom)
```

The 84-value raw diff is entirely a **column permutation**: the vocab is `sorted()`, so inserting `fungus` at column 11 shifts 48 of 59 old words right. Diffed by meaning, nothing moved. Lesson: if you want weight diffs to be legible, make index assignment append-only.

The same check still passes after +10 hand-drawn classes and the accessory vocabulary: v1 renders the original 11 classes bit-identically.

## 2. Capacity is unbounded; addressing is quadratic

Synthetic random classes (3 nonsense words each), rebuilt and probed at each size:

```
 classes  vocab      params   buffers      MB  dense%  gen ms  isolated
      21     95       2,937     6,171    0.04   21.9%    0.12      True
     121    395      51,537    31,771    0.33   57.4%    0.13      True
   1,021  3,095   3,188,937   262,171   13.80   91.6%    0.25      True
   3,021  9,095  27,560,937   774,171  113.34   97.0%    1.45      True
  10,021 30,095 301,862,937 2,566,171 1217.72   99.1%   11.95      True
```

`isolated` = the 21 real classes and probe prompts (`red potion`, `dragon`, `""`, `bone`) are bit-identical to the 21-class model, and the newest synthetic class resolves by all its synonyms. It never fails.

But vocab grows with classes, so `word_to_class` is V×C ≈ 3C² — at 10k classes it's 99.1% of the model and 99.997% zeros. Under a 100k-parameter budget the dense design tops out around **170 classes**.

## 3. Compressing the class layer buys size, costs semantics

Replace `Linear(V,C)` with a binary code: each word emits its class's d=⌈log₂C⌉-bit code as ±1, a `Linear(d,C)` of code rows scores `d − 2·hamming`.

```
 classes   d   dense params   coded params   ratio  isolated  interference
      21   5          2,016            601    3.4x      True     61%
     121   7         47,916          3,733   12.8x      True     61%
   1,021  10      3,161,016         42,181   74.9x      True     61%
   3,021  12     27,479,016        148,413  185.2x      True     61%
  10,021  14    301,592,016        571,645  527.6x      True     61%
```

Isolation survives. But `interference` — the share of `"wordA wordB"` prompts (over the 21 real classes) that resolve to a **third** class — jumps from 0% (dense; ties go to lower index) to 61%, flat across scale. The sum of two codes is nearest to some unrelated code. The bag-of-words layer's additive semantics were the thing being compressed away. Open question: do error-correcting or sparse k-hot codes recover 0% interference, and at what parameter cost?

## 4. Composition: the lever that actually matters

`PixelGen2` (`pixelgen/compose.py`) adds four fixed-weight layers to the same model:

| layer | mechanism | params |
|---|---|---|
| accessory | `Linear(V,8)` WTA → 16×8 part overlaid with `max` | 1,112 + 1,162 buffer |
| second colour | `Linear(V,9)` over words after `with`/`wearing`/…; body ramp wins ties via a 0.5 one-hot bonus | 1,251 |
| structural shading | two 3×3 convs: light where up-left neighbour is empty, dark where down-right is | 18 |
| size | `Linear(V,1)` → nearest 0.75× resample, bottom-centred | 139 |

```
PixelGen :  4,227 params + 6,171 buffers |    189 prompts ->    189 distinct images
PixelGen2:  6,747 params + 7,333 buffers | 24,192 prompts -> 21,866 distinct images
hand-drawn content: 21 bodies + 7 parts = 28 templates
distinct images per template:  9  ->  781
param growth 1.60x   output growth 115.7x
```

(The 2,326 duplicates are prompts whose accessory colour equals the body colour, or whose accessory is hidden behind the body.)

### Quality, honestly

![shading before/after](shading_before_after.png)

Top: row-band shading (v1). Bottom: structural rim lighting (v2), 18 parameters. It helps blobby shapes — potion, heart, mushroom, tree, gem get a readable highlight and shadow — and **hurts thin ones**: invader, bat, robot turn to speckle because every pixel is an edge. A fixed 3×3 kernel has no notion of "this shape is too thin to light". That's the limit of rule-based shading without a learned prior, and it's exactly the kind of thing a few hundred trained parameters would fix.

Accessories are placed at fixed rows, so a crown floats above short bodies and a helmet merges into tall ones. Fixing that properly needs per-body anchor points — more hand-written content, not more mechanism.

## What this says about weight-space programming

- **Adding capacity is free and modular.** Hand-written weights are block-diagonal by construction; trained weights are not. Add-a-class is `dict.append`.
- **The expensive part is the index, not the content.** Dense lookup is O(C²); the content (templates) is O(C). Compression trades additivity for size.
- **Composition beats enumeration by two orders of magnitude**, and it's the only route to "many things" that doesn't scale a human's drawing time linearly.
- **Quality rules written by hand are brittle at the edges** — the shading result is the clearest example. The honest endpoint for pure weight-space programming is a very capable procedural generator with legible, auditable failure modes, and zero creativity outside what was drawn.
- The natural next step is the hybrid the blog post gestures at: hand-write the structure (parts, compositing, palette, transforms — ~95% of params), and train only the small word→part mapping, so new vocabulary becomes learnable while everything downstream stays readable.