outputs/eval/baseline/REVIEW.md

The three composites confirm the JSON reviews exactly: teacher (left) is sharp and legible — storefront "THE OPEN PAGE" signage, a mirror-calm snowy mountain lake at dawn, a dew-beaded spider web at sunrise — while the student (right) is in every case a flat mottled paint-like field that preserves only coarse palette and light/dark zoning (warm ochres in 00, top-warm/bottom-cool teal in 20, warm orange-to-cool blotch field in 16) with zero edges, glyphs, geometry, or subject. This is a uniform, catastrophic collapse to low-frequency color statistics.

Here is the consolidated report.

Warm-Start Compression Degradation Report — 4B → 2.44B Text-to-Image Student

1. Overall Verdict

The warm-start-only student is non-functional as an image generator across every evaluated capability. The surgery did not produce a degraded-but-recoverable model; it produced a model that has lost the core generative computation entirely. In all 26 reviewed pairs the student outputs collapse to the same failure mode: a low-frequency, paint-like color field that preserves only the teacher's dominant palette and coarse light/dark zoning, with no edges, no objects, no glyphs, no spatial/attribute binding, and frequent chroma-noise/speckle artifacts that the teacher never contains. Severity is a uniform 5/5 across all 7 categories — there is no "best" capability, only varying flavors of total destruction. This is the expected signature of having severed token-mixing in 12 of 20 single blocks: per-token low-rank+GELU surrogates (warm-start residual rel-err ~0.9) cannot move information between spatial positions, so anything requiring cross-position structure is gone, and only per-token color/luminance statistics survive. Recovery training on this checkpoint is premature — the surrogate cannot represent the missing function class, so fine-tuning would be spending compute trying to learn token-mixing through modules architecturally incapable of it.

2. Degradation Ranking (worst-first)

All categories are severity 5/5; ranking reflects how completely the specific capability is obliterated and how far the student drifts from even the teacher's palette.

Rank	Capability	Severity	Worst example	Notes
1	Text rendering	5	01_text (mug text legible in teacher, gone in student)	Glyphs require precise high-frequency token-mixing; zero characters survive. Even the palette echo is faint. Hardest-bound capability, most completely destroyed.
2	Texture / fine detail	5	18_texture (cable-knit)	Student panels are nearly interchangeable abstract noise; palette even drifts off (neutral grey → orange/pink/teal speckle). High-frequency micro-structure fully erased.
3	Spatial / attribute binding	5	09_spatial (stacked sphere/cone/cube)	No objects to bind to; additionally shows hue inversion/oversaturation (acid green/magenta) outside teacher palette — worst color drift in the set.
4	Style adherence	5	26_style (minimalist flat-design sailboat)	"Flat/minimal" target becomes maximal chromatic-fringe noise — maximal contrast between target and output.
5	Count / composition	5	07_count (chessboard)	High-structure countable content fully smeared; no instances exist to count.
6	Face / hands	5	14_face (face + raised five-finger hand)	All facial geometry and the hand erased to pastel smear; micro-detail gone.
7	Scene / global structure	5	21_scene (fireplace living room)	Slightly "best" only because coarse palette + light/dark zoning is most visible (e.g. 20's top-warm/bottom-cool gradient), but no geometry, horizon, or object survives.

Read-off: degradation tracks spatial-frequency content and token-mixing dependence. Capabilities dominated by high-frequency, position-coupled structure (text, fine texture, attribute binding) are the most destroyed; capabilities where a coarse palette/luminance gradient is a partial proxy for the answer (scenes) retain the most superficial resemblance — but none are usable.

3. What the Surgery Preserved vs. Destroyed — and Why

Preserved (everywhere, and ONLY this):

• Dominant global color palette / hue distribution.
• Coarse spatial distribution of light vs. dark (low-frequency luminance zoning, e.g. top-warm/bottom-cool in 20).
• Overall canvas brightness/exposure and image dimensions.

Destroyed (everywhere):

• All discrete objects, edges, surfaces, foreground/background separation.
• All text/glyphs and countable instances.
• All spatial relations and attribute binding (which color belongs to which object; on-top-of/beside/left-right).
• All high-frequency detail (skin/hair/fabric/bark micro-texture) and style fidelity.
• Color purity in the worst cases: added chroma-noise, rainbow speckle, and oversaturated hue inversion (09, 26, 27) that the teacher never produced.

Architectural cause. The 12 replaced single blocks were attention blocks; their job is token-mixing — routing information between spatial token positions. The replacement is a per-token low-rank+GELU MLP: it maps each token's channel vector independently with no path for one position to read another. This is exactly the function class that cannot approximate attention, which is why warm-start residual rel-err sits at ~0.9 (the surrogate captures almost none of the attention output). The observed behavior is the direct consequence:

• What survives = what per-token channel maps can carry: the local color/luminance statistic at each token. That is precisely "palette + coarse light/dark zoning."
• What dies = everything cross-position: edges, objects, glyphs, counts, and spatial binding are all relational properties constructed by mixing tokens. With 12/20 mixing layers replaced by non-mixing modules, the network's ability to assemble coherent 2D structure collapses, and the 5 double blocks + 8 surviving single blocks cannot reconstruct it downstream.
• Error compounds, it doesn't average: a ~0.9 residual at 12 stacked positions means the residual stream is dominated by garbage by mid-network; the speckle/chroma artifacts (09, 16, 26) are this accumulated, un-mixed noise leaking into the decoder. So this is not "60% of capability retained because 8/20 blocks are full" — token-mixing is a serial bottleneck, and severing it at 12 points breaks the whole chain.

4. Prioritized Recommendations (do these BEFORE recovery training)

The current checkpoint should not go into recovery training as-is: the surrogate is the wrong function class for what was removed. Fix the surgery first.

1. (Highest priority) Replace per-token surrogates with a cheap token-mixing surrogate. The single most important change: the surrogate must move information between positions. Cheapest options, in order of preference: (a) linear/low-rank attention (e.g. softmax-free, kernelized, or Performer/linear-attention) — keeps O(N) cost but restores genuine mixing; (b) a small grouped/strided local-window attention (cheap, restores at least neighborhood mixing for edges/texture); (c) failing those, a token-mixing MLP (MLP-Mixer style) that mixes along the token axis. Re-measure warm-start residual rel-err per block; target ≤0.3–0.4 before any training. If rel-err stays ~0.9, the surrogate is still wrong — iterate here, not in training.

2. Keep more single blocks full, and choose which to keep by measured importance, not count. Going from 8→12+ full single blocks is the safest lever. Critically, select blocks to replace by per-block attention-importance: ablate/replace each single block one at a time on a calibration set and rank by output rel-err or downstream FID/CLIP drop. Replace only the genuinely low-impact / low-rank-attention blocks; never replace consecutive runs of high-impact mixing blocks (serial token-mixing bottlenecks compound, as seen here). Early and final blocks are usually highest-impact — bias toward keeping those full.

3. Right-size the rank empirically via SVD of the original attention output, not by a fixed guess. Per target block, compute the singular-value spectrum of the attention block's output (or its delta) on calibration data and pick the rank that captures ~90–95% energy. The current uniform low rank is almost certainly far below the effective rank of attention's token-mixing operator — hence rel-err 0.9. Allow non-uniform rank per block (more rank where importance ranking in #2 says it matters).

4. Initialize surrogates by distillation/least-squares fit to the teacher block, not naive warm-start copy. Before global training, solve a cheap per-block regression: fit each surrogate's parameters to reproduce the teacher attention block's outputs on a few thousand calibration token batches (closed-form for the linear parts, short local SGD for the mixing kernel). This converts "warm-start residual 0.9" into a properly fitted init and is far cheaper than full recovery training. Use the resulting per-block rel-err as the go/no-go gate for entering recovery training.

5. Add a residual/skip-preserving bypass so unmodeled signal isn't destroyed. Structure each surrogate as out = attn_low_rank_mix(x) + identity_or_teacher_passthrough, so the portion the surrogate cannot model is at least passed through rather than replaced by noise. This directly attacks the accumulated-speckle artifact (09/16/26) and stabilizes the residual stream across the 12 positions.

6. Set a quantitative acceptance gate before spending recovery compute. Define pass criteria on the warm-start (pre-training) checkpoint: e.g. mean per-replaced-block output rel-err ≤ 0.3, plus a small held-out eval showing any legible edges/objects in scene/texture prompts and non-zero OCR-detectable glyphs on text prompts. Only enter recovery training once palette-only collapse is broken; otherwise iterate on #1–#5. This avoids burning training compute to teach token-mixing to modules that structurally cannot perform it.

Bottom line for the next iteration: the parameter budget isn't the problem — the surrogate function class is. Swap per-token low-rank+GELU for a cheap token-mixing surrogate, choose replaced blocks by measured importance (keeping more single blocks full and avoiding consecutive high-impact mixing layers), size rank from the attention output's SVD spectrum, and gate on per-block residual rel-err before any recovery training.

Grounding composites read: /root/workspace/outputs/eval/baseline/compare/00_text.png, /root/workspace/outputs/eval/baseline/compare/20_scene.png, /root/workspace/outputs/eval/baseline/compare/16_texture.png — all three confirm uniform collapse to palette + coarse light/dark zoning with no edges, glyphs, or geometry.