outputs/eval/baseline/REVIEW.md

The three composites confirm the JSON reviews exactly: teacher (left) is sharp and legible — storefront "THE OPEN PAGE" signage, a mirror-calm snowy mountain lake at dawn, a dew-beaded spider web at sunrise — while the student (right) is in every case a flat mottled paint-like field that preserves only coarse palette and light/dark zoning (warm ochres in 00, top-warm/bottom-cool teal in 20, warm orange-to-cool blotch field in 16) with zero edges, glyphs, geometry, or subject. This is a uniform, catastrophic collapse to low-frequency color statistics.

Here is the consolidated report.

# Warm-Start Compression Degradation Report — 4B → 2.44B Text-to-Image Student

## 1. Overall Verdict

The warm-start-only student is **non-functional as an image generator** across every evaluated capability. The surgery did not produce a degraded-but-recoverable model; it produced a model that has lost the core generative computation entirely. In all 26 reviewed pairs the student outputs collapse to the same failure mode: a low-frequency, paint-like color field that preserves only the teacher's dominant palette and coarse light/dark zoning, with no edges, no objects, no glyphs, no spatial/attribute binding, and frequent chroma-noise/speckle artifacts that the teacher never contains. Severity is a uniform **5/5 across all 7 categories** — there is no "best" capability, only varying flavors of total destruction. This is the expected signature of having severed token-mixing in 12 of 20 single blocks: per-token low-rank+GELU surrogates (warm-start residual rel-err ~0.9) cannot move information between spatial positions, so anything requiring cross-position structure is gone, and only per-token color/luminance statistics survive. Recovery training on this checkpoint is **premature** — the surrogate cannot represent the missing function class, so fine-tuning would be spending compute trying to learn token-mixing through modules architecturally incapable of it.

## 2. Degradation Ranking (worst-first)

All categories are severity 5/5; ranking reflects how completely the *specific* capability is obliterated and how far the student drifts from even the teacher's palette.

| Rank | Capability | Severity | Worst example | Notes |
|------|-----------|----------|---------------|-------|
| 1 | **Text rendering** | 5 | `01_text` (mug text legible in teacher, gone in student) | Glyphs require precise high-frequency token-mixing; zero characters survive. Even the palette echo is faint. Hardest-bound capability, most completely destroyed. |
| 2 | **Texture / fine detail** | 5 | `18_texture` (cable-knit) | Student panels are nearly interchangeable abstract noise; palette even drifts off (neutral grey → orange/pink/teal speckle). High-frequency micro-structure fully erased. |
| 3 | **Spatial / attribute binding** | 5 | `09_spatial` (stacked sphere/cone/cube) | No objects to bind to; additionally shows hue inversion/oversaturation (acid green/magenta) outside teacher palette — worst color drift in the set. |
| 4 | **Style adherence** | 5 | `26_style` (minimalist flat-design sailboat) | "Flat/minimal" target becomes maximal chromatic-fringe noise — maximal contrast between target and output. |
| 5 | **Count / composition** | 5 | `07_count` (chessboard) | High-structure countable content fully smeared; no instances exist to count. |
| 6 | **Face / hands** | 5 | `14_face` (face + raised five-finger hand) | All facial geometry and the hand erased to pastel smear; micro-detail gone. |
| 7 | **Scene / global structure** | 5 | `21_scene` (fireplace living room) | Slightly "best" only because coarse palette + light/dark zoning is most visible (e.g. `20`'s top-warm/bottom-cool gradient), but no geometry, horizon, or object survives. |

**Read-off:** degradation tracks **spatial-frequency content and token-mixing dependence**. Capabilities dominated by high-frequency, position-coupled structure (text, fine texture, attribute binding) are the most destroyed; capabilities where a coarse palette/luminance gradient is a partial proxy for the answer (scenes) retain the most superficial resemblance — but none are usable.

## 3. What the Surgery Preserved vs. Destroyed — and Why

**Preserved (everywhere, and ONLY this):**
- Dominant global color palette / hue distribution.
- Coarse spatial distribution of light vs. dark (low-frequency luminance zoning, e.g. top-warm/bottom-cool in `20`).
- Overall canvas brightness/exposure and image dimensions.

**Destroyed (everywhere):**
- All discrete objects, edges, surfaces, foreground/background separation.
- All text/glyphs and countable instances.
- All spatial relations and attribute binding (which color belongs to which object; on-top-of/beside/left-right).
- All high-frequency detail (skin/hair/fabric/bark micro-texture) and style fidelity.
- Color *purity* in the worst cases: added chroma-noise, rainbow speckle, and oversaturated hue inversion (`09`, `26`, `27`) that the teacher never produced.

**Architectural cause.** The 12 replaced single blocks were attention blocks; their job is **token-mixing** — routing information *between* spatial token positions. The replacement is a **per-token** low-rank+GELU MLP: it maps each token's channel vector independently with no path for one position to read another. This is exactly the function class that cannot approximate attention, which is why warm-start residual rel-err sits at ~0.9 (the surrogate captures almost none of the attention output). The observed behavior is the direct consequence:

- **What survives = what per-token channel maps can carry:** the local color/luminance statistic at each token. That is precisely "palette + coarse light/dark zoning."
- **What dies = everything cross-position:** edges, objects, glyphs, counts, and spatial binding are all *relational* properties constructed by mixing tokens. With 12/20 mixing layers replaced by non-mixing modules, the network's ability to assemble coherent 2D structure collapses, and the 5 double blocks + 8 surviving single blocks cannot reconstruct it downstream.
- **Error compounds, it doesn't average:** a ~0.9 residual at 12 stacked positions means the residual stream is dominated by garbage by mid-network; the speckle/chroma artifacts (`09`, `16`, `26`) are this accumulated, un-mixed noise leaking into the decoder. So this is **not** "60% of capability retained because 8/20 blocks are full" — token-mixing is a serial bottleneck, and severing it at 12 points breaks the whole chain.

## 4. Prioritized Recommendations (do these BEFORE recovery training)

The current checkpoint should **not** go into recovery training as-is: the surrogate is the wrong function class for what was removed. Fix the surgery first.

1. **(Highest priority) Replace per-token surrogates with a cheap *token-mixing* surrogate.** The single most important change: the surrogate must move information between positions. Cheapest options, in order of preference: (a) **linear/low-rank attention** (e.g. softmax-free, kernelized, or Performer/linear-attention) — keeps O(N) cost but restores genuine mixing; (b) **a small grouped/strided local-window attention** (cheap, restores at least neighborhood mixing for edges/texture); (c) failing those, a **token-mixing MLP (MLP-Mixer style)** that mixes along the token axis. Re-measure warm-start residual rel-err per block; target ≤0.3–0.4 before any training. If rel-err stays ~0.9, the surrogate is still wrong — iterate here, not in training.

2. **Keep more single blocks full, and choose *which* to keep by measured importance, not count.** Going from 8→12+ full single blocks is the safest lever. Critically, **select blocks to replace by per-block attention-importance**: ablate/replace each single block one at a time on a calibration set and rank by output rel-err or downstream FID/CLIP drop. Replace only the genuinely low-impact / low-rank-attention blocks; never replace consecutive runs of high-impact mixing blocks (serial token-mixing bottlenecks compound, as seen here). Early and final blocks are usually highest-impact — bias toward keeping those full.

3. **Right-size the rank empirically via SVD of the original attention output, not by a fixed guess.** Per target block, compute the singular-value spectrum of the attention block's output (or its delta) on calibration data and pick the rank that captures ~90–95% energy. The current uniform low rank is almost certainly far below the effective rank of attention's token-mixing operator — hence rel-err 0.9. Allow **non-uniform rank per block** (more rank where importance ranking in #2 says it matters).

4. **Initialize surrogates by distillation/least-squares fit to the teacher block, not naive warm-start copy.** Before global training, solve a cheap per-block regression: fit each surrogate's parameters to reproduce the teacher attention block's outputs on a few thousand calibration token batches (closed-form for the linear parts, short local SGD for the mixing kernel). This converts "warm-start residual 0.9" into a properly fitted init and is far cheaper than full recovery training. Use the resulting per-block rel-err as the **go/no-go gate** for entering recovery training.

5. **Add a residual/skip-preserving bypass so unmodeled signal isn't destroyed.** Structure each surrogate as `out = attn_low_rank_mix(x) + identity_or_teacher_passthrough`, so the portion the surrogate cannot model is at least passed through rather than replaced by noise. This directly attacks the accumulated-speckle artifact (`09/16/26`) and stabilizes the residual stream across the 12 positions.

6. **Set a quantitative acceptance gate before spending recovery compute.** Define pass criteria on the warm-start (pre-training) checkpoint: e.g. mean per-replaced-block output rel-err ≤ 0.3, plus a small held-out eval showing **any** legible edges/objects in scene/texture prompts and **non-zero** OCR-detectable glyphs on text prompts. Only enter recovery training once palette-only collapse is broken; otherwise iterate on #1–#5. This avoids burning training compute to teach token-mixing to modules that structurally cannot perform it.

**Bottom line for the next iteration:** the parameter budget isn't the problem — the *surrogate function class* is. Swap per-token low-rank+GELU for a cheap token-mixing surrogate, choose replaced blocks by measured importance (keeping more single blocks full and avoiding consecutive high-impact mixing layers), size rank from the attention output's SVD spectrum, and gate on per-block residual rel-err before any recovery training.

Grounding composites read: `/root/workspace/outputs/eval/baseline/compare/00_text.png`, `/root/workspace/outputs/eval/baseline/compare/20_scene.png`, `/root/workspace/outputs/eval/baseline/compare/16_texture.png` — all three confirm uniform collapse to palette + coarse light/dark zoning with no edges, glyphs, or geometry.