EVAL_REPORT.md

# PXG-Tiny v0.3 — Evaluation Report

**Model:** PXG-Tiny v0.3 · 483,040 params · INT8/fp32 NumPy runtime
**Author:** Chowdhury Tarul Ahsan <tarulahsan@gmail.com>
**Date:** 2026-08-27

## 1. What was measured

* **Teacher-forced parity** — torch reference vs shipped NumPy runtime on
  identical inputs (hard numerical gate before any release).
* **Raw held-out generalization** — 31 English prompts deliberately phrased
  to avoid every training surface form (relative clauses, "made of",
  "of the … kind", "covered in moss", …). Each prompt scored over 3 seeds by
  attribute-grounding checks with **retry disabled** (pure model skill).
* **User-path capability** — same style prompts through the *shipped*
  pipeline (verifier-guided auto-retry, the mode real users get).

## 2. Version journey (same 31-prompt held-out suite)

| Metric | v0.1 | v0.2 | **v0.3** |
|---|---|---|---|
| Val CE / top-1 (teacher-forced) | 0.0533 / 98.14% | 0.0425 / 98.49% | 0.0449 / **98.34%**¹ |
| Corpus size (rows) | 23,451 | 64,088 | 65,022 |
| Raw held-out pass | 48.4% | 66.7% | **77.4%** |
| Strict (3/3 seeds) | 32.3% | 58.1% | **71.0%** |
| Hard-subset held-out | 9.5% | ~57% | **76.2%** |
| User-path (guided retries) | 52.6% | — | **100%** (19/19) |
| INT8 parity gate | FAIL | FAIL | **PASS** |

¹ v0.3 corpus is intentionally harder (non-comma syntax, mid adjectives), so a
similar CE reflects stronger generalization, confirmed by the raw eval.

## 3. Numerical parity (v0.3, shipped bundle)

* fp32 bundle: teacher-forced max |Δlogit| = **0.0000** — bit-exact; argmax
  match 3072/3072 (100%). The NumPy runtime is an exact mirror.
* INT8 bundle (decoder Linears quantized): argmax match > 99%; gate PASS.
* Diagnostic greedy-rollout agreement ≈ 74% on both — autoregressive near-tie
  cascades, expected and immaterial under stochastic sampling (see README
  limitations).

## 4. Final accepted-version gallery (shipped model, guided pipeline)

16 held-out showcase prompts → **16/16 sprites delivered** (13 clean first-or-
early accepts, 3 `accept_degraded` after the full 8-retry budget). Artifacts
in `samples/gallery/` (per-sprite PNG + `gallery.json` + `contact_sheet.png`),
generated in ≈23 s total on 2 CPU cores (~0.5 s per first-attempt sprite).

Highlights that failed in v0.1 and pass cleanly in v0.3:
*“liquid potion of red brew”, “one sword of the crystal kind”, “staff whose
orb is ruby”, “old oak with autumn leaves”, “rocky stone pavement tile”,
“a gold coin seen from the side”.*

## 5. Known weaknesses (honest)

1. **`accept_degraded` tail (~10–20% of hard prompts)** — free-run sampling
   occasionally drifts off-attribute; the self-guided retry usually recovers,
   but the last-resort sprite can miss one attribute (e.g. the gallery's
   "flame orb" rendered purple-ish instead of fiery red).
2. **Resolution is 16×16 only** — larger canvases/spritesheets are refused by
   the ask-first gate by design (roadmap: PXG main program).
3. **23-class vocabulary** — outside objects get a clarifying question, not a
   hallucinated blob.
4. **Exact-alpha only** — sprites are binary-transparent (no soft edges); a
   property of the master-palette representation, not a bug.
5. Self-guided bias nudges palette families; it cannot force *shape* changes
   (orientation) — those rely on the trained encoder alone.

## 6. Reproduction

```bash
python3 train/eval_generalization.py --seeds 3     # raw held-out suite
python3 tests/… ; pytest tests/ -q                 # 22 checks
python3 ../../scripts/make_gallery.py              # rebuild gallery
```

Training: `train/train_model.py` (4,200 steps ≈ 50 min on 2 CPU cores).
Corpus: `train/gen_corpus.py` (65,022 rows, fully procedural, CC0).

---

## 7. v0.4 — 38-class expansion and the full-grid acceptance run

v0.4 expands the corpus to **38 classes / 97,005 rows** (characters: wizard,
knight, archer, skeleton, zombie; animals: cat, dog, bird, fish, deer, mouse,
bat; furniture: chair, table, stool) and re-hardens the whole quality chain.
The acceptance protocol changed from "a held-out prompt list" to a
**complete class × prompt × seed grid**, because a model that passes an
averaged benchmark can still silently fail one class.

### 7.1 Bugs found by the 38-class grid (and fixed)

| Symptom | Root cause | Fix |
|---|---|---|
| "a wizard staff" refused for `no_face` | parser picked the FIRST class noun; head noun is *staff* | head-noun-last resolution with `with`-clause carve-out |
| "skeleton warrior" drew a knight | blanket `warrior→knight` synonym | fighter words are a fallback only, never override a class noun |
| "shining armor" demanded glow pixels | idiom mapped to the glow attribute | glow now requires explicit light words (glowing/shimmering/luminous/radiant) |
| "coins", "grassy ground tile" got clarify-loops | no plurals, missing synonyms | class-noun singularizer + shrub/lawn/stag/dungeon/ocean/brass/bronze… synonyms |
| deer 0/4 with perfect-looking renders | checker bars stricter than GT (legs band, wood fraction excluded deer-tan) | thresholds re-calibrated against GT percentiles (legs band 10-15, deer-tan counts as body) |
| "emerald dagger" 0/3 | recipes never trained emerald daggers — an impossible ask hard-failed | empirical class × material capability map; unsupported combos downgrade to the class default |
| bat→fish, "dining table"→gem renders | weak class anchoring on paraphrases | escalating self-guidance (palette bias → spatial priors → canonical prompt anchor) |
| vials rendered 16 px wide | shape intent not expressible by palette bias | per-position transparency bias on side columns (GT vial width = 6 px) |
| wizards/archers missing faces on rephrasings | same | per-position skin bias in the head box |
| **deer rendered as a legless blob 24/24 seeds** | AR cascade: teacher-forced top-1 = 100%, but the sparse antler rows get skipped once decoding drifts | structural prior: seed the two recipe-constant antler rows (identical across all 1,800 GT deer), then decode freely |

### 7.2 Final acceptance numbers

| Protocol | Result |
|---|---|
| Full grid: 38 classes × 2 prompts × 2 seeds, shipped pipeline | **152/152 = 100%**, all 38 classes |
| Independent fresh-phrasing sweep (39 novel phrasings × 3 seeds) | **116/117 = 99.1%** |
| Showcase gallery through the shipped pipeline | **31/31** |
| Raw held-out suite (46 prompts, verifier disabled) | 59.4% mean pass, 50% strict 3/3 |
| INT8→fp32 parity (teacher-forced) | argmax match 100%, max \|Δlogit\| 2e-05 |
| val top-1 (8,300 steps) | 98.89% (CE 0.0294) |

The raw-vs-guided gap is the point of the verifier: the 483k-param backbone
knows every class (perfect teacher-forcing) but free-run AR sampling still
drifts on hard prompts; the verifier-guided escalation closes that gap for
the shipped user path.

### 7.3 Remaining weaknesses

1. The last fresh-phrasing failure: "steel paladin standing tall" can end on
   a thin knight after 8 attempts (`accept_degraded`) — knights with merged
   legs still win the argmax race occasionally.
2. Structural priors (deer antlers, face box, vial margins) are recipe-derived
   constants; they are honest for a self-contained procedural model but would
   not transfer to a web-crawl-trained model.
3. 16×16, 32-color, 38 classes — resolution and vocabulary growth remain
   roadmap items (see §5).