PXG-Tiny v0.3 — Evaluation Report
Model: PXG-Tiny v0.3 · 483,040 params · INT8/fp32 NumPy runtime Author: Chowdhury Tarul Ahsan tarulahsan@gmail.com Date: 2026-08-27
1. What was measured
- Teacher-forced parity — torch reference vs shipped NumPy runtime on identical inputs (hard numerical gate before any release).
- Raw held-out generalization — 31 English prompts deliberately phrased to avoid every training surface form (relative clauses, "made of", "of the … kind", "covered in moss", …). Each prompt scored over 3 seeds by attribute-grounding checks with retry disabled (pure model skill).
- User-path capability — same style prompts through the shipped pipeline (verifier-guided auto-retry, the mode real users get).
2. Version journey (same 31-prompt held-out suite)
| Metric | v0.1 | v0.2 | v0.3 |
|---|---|---|---|
| Val CE / top-1 (teacher-forced) | 0.0533 / 98.14% | 0.0425 / 98.49% | 0.0449 / **98.34%**¹ |
| Corpus size (rows) | 23,451 | 64,088 | 65,022 |
| Raw held-out pass | 48.4% | 66.7% | 77.4% |
| Strict (3/3 seeds) | 32.3% | 58.1% | 71.0% |
| Hard-subset held-out | 9.5% | ~57% | 76.2% |
| User-path (guided retries) | 52.6% | — | 100% (19/19) |
| INT8 parity gate | FAIL | FAIL | PASS |
¹ v0.3 corpus is intentionally harder (non-comma syntax, mid adjectives), so a similar CE reflects stronger generalization, confirmed by the raw eval.
3. Numerical parity (v0.3, shipped bundle)
- fp32 bundle: teacher-forced max |Δlogit| = 0.0000 — bit-exact; argmax match 3072/3072 (100%). The NumPy runtime is an exact mirror.
- INT8 bundle (decoder Linears quantized): argmax match > 99%; gate PASS.
- Diagnostic greedy-rollout agreement ≈ 74% on both — autoregressive near-tie cascades, expected and immaterial under stochastic sampling (see README limitations).
4. Final accepted-version gallery (shipped model, guided pipeline)
16 held-out showcase prompts → 16/16 sprites delivered (13 clean first-or-
early accepts, 3 accept_degraded after the full 8-retry budget). Artifacts
in samples/gallery/ (per-sprite PNG + gallery.json + contact_sheet.png),
generated in ≈23 s total on 2 CPU cores (~0.5 s per first-attempt sprite).
Highlights that failed in v0.1 and pass cleanly in v0.3: “liquid potion of red brew”, “one sword of the crystal kind”, “staff whose orb is ruby”, “old oak with autumn leaves”, “rocky stone pavement tile”, “a gold coin seen from the side”.
5. Known weaknesses (honest)
accept_degradedtail (~10–20% of hard prompts) — free-run sampling occasionally drifts off-attribute; the self-guided retry usually recovers, but the last-resort sprite can miss one attribute (e.g. the gallery's "flame orb" rendered purple-ish instead of fiery red).- Resolution is 16×16 only — larger canvases/spritesheets are refused by the ask-first gate by design (roadmap: PXG main program).
- 23-class vocabulary — outside objects get a clarifying question, not a hallucinated blob.
- Exact-alpha only — sprites are binary-transparent (no soft edges); a property of the master-palette representation, not a bug.
- Self-guided bias nudges palette families; it cannot force shape changes (orientation) — those rely on the trained encoder alone.
6. Reproduction
python3 train/eval_generalization.py --seeds 3 # raw held-out suite
python3 tests/… ; pytest tests/ -q # 22 checks
python3 ../../scripts/make_gallery.py # rebuild gallery
Training: train/train_model.py (4,200 steps ≈ 50 min on 2 CPU cores).
Corpus: train/gen_corpus.py (65,022 rows, fully procedural, CC0).
7. v0.4 — 38-class expansion and the full-grid acceptance run
v0.4 expands the corpus to 38 classes / 97,005 rows (characters: wizard, knight, archer, skeleton, zombie; animals: cat, dog, bird, fish, deer, mouse, bat; furniture: chair, table, stool) and re-hardens the whole quality chain. The acceptance protocol changed from "a held-out prompt list" to a complete class × prompt × seed grid, because a model that passes an averaged benchmark can still silently fail one class.
7.1 Bugs found by the 38-class grid (and fixed)
| Symptom | Root cause | Fix |
|---|---|---|
"a wizard staff" refused for no_face |
parser picked the FIRST class noun; head noun is staff | head-noun-last resolution with with-clause carve-out |
| "skeleton warrior" drew a knight | blanket warrior→knight synonym |
fighter words are a fallback only, never override a class noun |
| "shining armor" demanded glow pixels | idiom mapped to the glow attribute | glow now requires explicit light words (glowing/shimmering/luminous/radiant) |
| "coins", "grassy ground tile" got clarify-loops | no plurals, missing synonyms | class-noun singularizer + shrub/lawn/stag/dungeon/ocean/brass/bronze… synonyms |
| deer 0/4 with perfect-looking renders | checker bars stricter than GT (legs band, wood fraction excluded deer-tan) | thresholds re-calibrated against GT percentiles (legs band 10-15, deer-tan counts as body) |
| "emerald dagger" 0/3 | recipes never trained emerald daggers — an impossible ask hard-failed | empirical class × material capability map; unsupported combos downgrade to the class default |
| bat→fish, "dining table"→gem renders | weak class anchoring on paraphrases | escalating self-guidance (palette bias → spatial priors → canonical prompt anchor) |
| vials rendered 16 px wide | shape intent not expressible by palette bias | per-position transparency bias on side columns (GT vial width = 6 px) |
| wizards/archers missing faces on rephrasings | same | per-position skin bias in the head box |
| deer rendered as a legless blob 24/24 seeds | AR cascade: teacher-forced top-1 = 100%, but the sparse antler rows get skipped once decoding drifts | structural prior: seed the two recipe-constant antler rows (identical across all 1,800 GT deer), then decode freely |
7.2 Final acceptance numbers
| Protocol | Result |
|---|---|
| Full grid: 38 classes × 2 prompts × 2 seeds, shipped pipeline | 152/152 = 100%, all 38 classes |
| Independent fresh-phrasing sweep (39 novel phrasings × 3 seeds) | 116/117 = 99.1% |
| Showcase gallery through the shipped pipeline | 31/31 |
| Raw held-out suite (46 prompts, verifier disabled) | 59.4% mean pass, 50% strict 3/3 |
| INT8→fp32 parity (teacher-forced) | argmax match 100%, max |Δlogit| 2e-05 |
| val top-1 (8,300 steps) | 98.89% (CE 0.0294) |
The raw-vs-guided gap is the point of the verifier: the 483k-param backbone knows every class (perfect teacher-forcing) but free-run AR sampling still drifts on hard prompts; the verifier-guided escalation closes that gap for the shipped user path.
7.3 Remaining weaknesses
- The last fresh-phrasing failure: "steel paladin standing tall" can end on
a thin knight after 8 attempts (
accept_degraded) — knights with merged legs still win the argmax race occasionally. - Structural priors (deer antlers, face box, vial margins) are recipe-derived constants; they are honest for a self-contained procedural model but would not transfer to a web-crawl-trained model.
- 16×16, 32-color, 38 classes — resolution and vocabulary growth remain roadmap items (see §5).