| # A learned layout generator trained on Crello designers (Dev 3, 2026-10-27) |
|
|
| **Question.** The selector is good (judge-distilled critic, tie-breaks: `docs/crello-gains-2026-10-26.md`) but can only choose among |
| what the section 2 generators propose, and on 3,000 designer-made size variants the uniform stretch is closer to the designer |
| than our pick (0.190 vs 0.237 mean distance). Can a small model that sees the source layout and the target size propose better |
| candidates, and do the fitter, hard checks, rule score and critic make use of them? |
|
|
| **Answer in one paragraph.** A 33-input, 2 x 64 tanh MLP (31 KB, numpy) predicts a box per content unit. Alone it beats the stretch |
| clearly on held-out design families (unit distance 0.1886 → 0.1525, element distance 0.1905 → 0.1545; better in 67% of pairs). As a |
| candidate family (`learned`) it is the first of ours that is, on average, closer to the designers than the stretch (0.183 vs 0.195 where |
| it is valid) and the valid candidate closest to the designer in 53% of pairs. Left to the selector alone it ships in 29% of held-out |
| pairs and moves the pick distance 0.2522 → 0.2352 (+0.0169, family-bootstrap interval [+0.0132, +0.0209]) with no hard-check, fallback |
| or floor-count change on our three sets, **but a judge round on its 144 changed picks went only 66 to 44 for it** (section 6b): the family |
| reproduces what designers do, including small text and top-heavy stacks on tall canvases, and the rule score's fidelity term rewards |
| exactly that. **Shipped with an admission guard** (a learned layout replaces the pick made without the family only if its smallest text |
| is within 1 px and its utilization score within 0.05 of that pick; everything else is byte-identical to stage one): the kept picks were |
| judged 54 to 17, 97 picks change on our three sets (all to `learned`; hard checks, fallbacks, floor counts and the mean smallest text |
| unchanged), and on Crello the held-out gain is +0.0093 [+0.0062, +0.0127] with the changed picks' smallest text larger in 392 pairs and |
| smaller in 36. |
|
|
| ## 1. Dataset (`tools/crello/layout_dataset.py`) |
| |
| | | count | |
| |---|---| |
| | raw same-design pairs (Hugging Face) | 37,279 | |
| | both templates in the extracted parquet | 31,153 (6,126 dropped: the pairs file also spans the test split, whose layouts were not extracted) | |
| | same element count | 26,809 (4,344 dropped) | |
| | same type sequence | 22,977 (3,832 dropped) | |
| | same text-length sequence | 18,463 (4,514 dropped) | |
| | at most 30 elements and at least one text | **18,449 usable pairs** (14 dropped), 1,316 families, 5,563 templates | |
| | dropped while converting, analysing or featurising | **0** (no analysis error, no pair without a content unit) | |
| | directed pairs (each pair read both ways) | 36,898 | |
| | **rows** (one per content unit) | **240,227** (6.51 per pair); 120,039 as given, 120,188 swapped | |
| | member elements of those units | 280,676 | |
| | split by family (`pairs.py`, 70 / 30) | train 915 families, 160,997 rows; test 401 families, 79,230 rows (39,620 as given, 6,412 pairs) | |
| | the 3,000-pair sample of `designer_eval.py` | all 3,000 pairs are in the dataset; 1,008 are in test families | |
|
|
| The filters are `pairs.py`'s (`pairs_stats.json`); nothing new was dropped. Every usable pair is used, not just the 3,000 sample. |
| Both directions are included (`rev` column): they read the same designer pair the other way, never straddle the split, and |
| raised held-out quality slightly (exploration, one seed: unit distance 0.1555 as given only, 0.1522 with both). |
|
|
| **Inputs.** `engine.resize._layout_features(analysis, target)`, the function the engine calls at run time, so training and serving |
| cannot differ: the unit's source box and ink box as shares of the source canvas (8), kind flags (text, image, shape, pinned |
| furniture, band, rotated, group), prominence, font size over the source short side, log text length, block index / count / |
| relative position, flow axis, number of content units, source / target / log-ratio aspect, source and target short sides over 1000, |
| rank by area and by font size, left / right alignment, sibling flag: 33 numbers, geometry and text length only (a rename test checks |
| it). **Target.** The union of the designer's drawn bounds of the unit's member elements at the target size, as shares of the target |
| canvas (cx, cy, w, h); the designer layout is `make_tasks.designer_layout` (same elements, the target template's geometry, text |
| heights as our renderer measures the synthetic text). **Baseline.** The uniform stretch, which in these units is the source box |
| itself. The designer's target font size per unit is stored too (`fs_t`) and was used to check text sizing, not as a target. |
|
|
| Not modelled: backgrounds and decorations (the model places content units only; decorations follow the anchor's rules). |
|
|
| ## 2. Models (`tools/crello/train_layout.py`; held-out families only) |
| |
| All models predict the correction to the stretch. Training uses 85% of the train families, early stopping and model selection use the |
| other 15% (a hash of the family id), the 401 test families are only evaluated. Rows are weighted by 1 / sqrt(pairs in the family) (a |
| handful of boilerplate families hold 20% of the pairs; exploration, one seed, unit distance: weight power 0 → 0.1589, 0.5 → 0.1517, |
| 1 → 0.1559). Metrics on the 39,620 test rows (6,412 pairs): MAE of (cx, cy, w, h) as shares of the target canvas; **unit distance** = |
| the engine's `dist` formula on unit boxes; **element distance** = the same on member elements of content units (predicted unit boxes |
| mapped back to elements), pairs pooled and family-balanced. |
| |
| | model | MAE | MAE cx / cy / w / h | unit dist | element dist (pooled) | element dist (family-balanced) | sample test pairs (1,008) | |
| |---|---|---|---|---|---|---| |
| | (a) stretch | 0.0739 | 0.0684 / 0.0657 / 0.1028 / 0.0586 | 0.1886 | 0.1905 | 0.1260 | 0.1842 | |
| | uniform scale min(sx, sy), centres kept | 0.0718 | 0.0684 / 0.0657 / 0.0969 / 0.0561 | 0.1843 | 0.1876 | 0.1250 | 0.1849 | |
| | (b) ridge, squared loss (alpha 10) | 0.0705 | 0.0742 / 0.0685 / 0.0823 / 0.0570 | 0.1828 | 0.1822 | 0.1365 | 0.1774 | |
| | (b') ridge model trained on the distance | 0.0669 | 0.0680 / 0.0653 / 0.0826 / 0.0518 | 0.1742 | 0.1757 | 0.1189 | 0.1692 | |
| | (c) sklearn `MLPRegressor` 2 x 64 tanh, squared loss, 3 seeds | 0.0743 | 0.0914 / 0.0768 / 0.0757 / 0.0533 | 0.1976 (3 seeds 0.1974 ± 0.0003) | 0.1993 | 0.1612 | 0.1955 | |
| | (d) the same + set context (mean of the other units' inputs) | 0.0810 | 0.1018 / 0.0833 / 0.0811 / 0.0579 | 0.2162 | 0.2181 | 0.1754 | 0.2139 | |
| | numpy MLP 2 x 64 tanh, squared loss | 0.0646 | 0.0742 / 0.0672 / 0.0681 / 0.0490 | 0.1707 | 0.1711 | 0.1288 | 0.1667 | |
| | **(c') numpy MLP 2 x 64 tanh on the distance, 5 seeds (shipped)** | **0.0573** | 0.0633 / 0.0611 / 0.0624 / 0.0425 | **0.1525** (5 seeds 0.1524 ± 0.0004) | **0.1545** | **0.1081** | **0.1488** | |
| | (d') the same + set context | 0.0580 | 0.0646 / 0.0612 / 0.0637 / 0.0426 | 0.1541 (5 seeds ± 0.0004) | 0.1562 | 0.1096 | 0.1511 | |
| |
| What the table says: |
| |
| * **The loss is what matters.** Designers keep relative positions, so the stretch's centres are already good (MAE 0.068) and a |
| squared-loss regression, which regresses to the mean, makes them worse (0.074 ridge, 0.091 sklearn MLP) while fixing the sizes. sklearn's |
| `MLPRegressor` cannot train on anything else; a small numpy Adam loop on the metric itself (the engine's distance, smoothed) is what |
| beats the stretch: every output improves except the centres, which it keeps at stretch quality (0.063, 0.061 against 0.068, 0.066). |
| The squared-loss sklearn MLPs lose to the stretch, as does the one with context. |
| * **Capacity and context.** The model is small on purpose: one hidden layer 0.1540, 2 x 32 0.1530, 2 x 64 0.1517, 2 x 128 0.1515, 3 x 64 |
| 0.1505, relu 0.1524 (exploration, validation protocol, one seed: unit distance). A 3-seed ensemble gives 0.1500 for three times the |
| file. Twelve extra features (bounding box of all content, position within it, gaps to the nearest unit in each direction, overlaps, |
| area share) gave 0.1506 against 0.1515, so the 33 features stay. A linear model on the metric is 0.1742: the gain needs the |
| nonlinearity. **Set context does not help** (0.1541 against 0.1525): the simplest model that beats the stretch clearly is the 2 x 64 |
| MLP without it, and that is what ships. |
| * **Where it wins.** Better than the stretch in 67% of held-out pairs: 35% when the aspect changes by under 0.2 (it is not better than |
| the stretch there: 0.0631 against 0.0606; the engine does not generate it below a 3% change anyway), 71% from 0.2 to 0.7, 84% above 0.7. |
| |
| By aspect-change bucket and target class (all 6,412 held-out pairs as given, element distance, pooled / family-balanced): |
| |
| | group | pairs | stretch | uniform | ridge | **mlp** | mlp beats stretch | |
| |---|---|---|---|---|---|---| |
| | all | 6412 | 0.1905 / 0.1260 | 0.1876 / 0.1250 | 0.1822 / 0.1365 | **0.1545 / 0.1081** | 67% | |
| | aspect change 0-0.2 | 1557 | 0.0606 / 0.0633 | 0.0619 / 0.0644 | 0.0892 / 0.0930 | 0.0631 / 0.0656 | 35% | |
| | aspect change 0.2-0.7 | 2707 | 0.1575 / 0.1341 | 0.1581 / 0.1336 | 0.1573 / 0.1383 | **0.1391 / 0.1165** | 71% | |
| | aspect change 0.7+ | 2148 | 0.3263 / 0.3070 | 0.3157 / 0.3010 | 0.2808 / 0.2670 | **0.2402 / 0.2276** | 84% | |
| | target strip | 544 | 0.2939 / 0.2228 | 0.2975 / 0.2234 | 0.2688 / 0.2151 | **0.2271 / 0.1780** | 73% | |
| | target small | 1750 | 0.1686 / 0.1390 | 0.1659 / 0.1370 | 0.1662 / 0.1447 | **0.1432 / 0.1184** | 66% | |
| | target square | 874 | 0.1803 / 0.1518 | 0.1726 / 0.1480 | 0.1732 / 0.1504 | **0.1566 / 0.1302** | 67% | |
| | target vertical | 1693 | 0.1963 / 0.1422 | 0.1971 / 0.1443 | 0.1841 / 0.1466 | **0.1504 / 0.1146** | 62% | |
| | target other | 1551 | 0.1783 / 0.1498 | 0.1714 / 0.1431 | 0.1727 / 0.1543 | **0.1452 / 0.1270** | 70% | |
| |
| Against our current candidates, on the 1,005 sample pairs in test families that have a recorded pick (element distance over the |
| elements of content units, from `candidates.jsonl` of the branch without the model): stretch 0.1845, engine pick 0.2456, engine |
| oracle (the valid candidate closest to the designer) 0.1748, **the model's raw boxes 0.1489**; it beats the stretch in 66% of these |
| pairs, the pick in 86% and the oracle in 76%. By bucket (0-0.2 / 0.2-0.7 / 0.7+): 0.0595 / 0.1362 / 0.2332 against pick 0.0833 / 0.2420 / |
| 0.3784 and oracle 0.0757 / 0.1561 / 0.2730. These are raw predictions (not fitted, checked or clamped), which is why the engine |
| numbers below are smaller. Full per-class table: `output/crello/layout_train.md` (regenerated by the training command). |
|
|
| The shipped file is the best of 5 seeds on validation (seed 0; test unit distance of the 5 seeds 0.1524 ± 0.0004), trained on train |
| families only, 6,596 weights, 31.3 KB. |
|
|
| ## 3. In the engine (`engine/resize.py`: end of section 2, section 5b) |
|
|
| `_learned_candidates(an, target, ctx, existing)` (after `_proportional_candidates` in `generate_candidates`, same ±3% gate): |
| `_layout_model()` (lazy, numpy, no pickle, cached per file state, validates names and shapes; the family is simply not generated if |
| the file is absent, corrupt or inconsistent, so the output is byte-identical to the output without it; `CHAMELEON_LEARNED=0` and |
| `TUNABLES["learned"]["enabled"]` switch it off) → `_layout_predict` (features → boxes in target px) → `_anchor_ladder(ctx, pred=…)`, |
| the anchor's own rung ladder with `_anchor_try(pred=…)`: |
|
|
| * free units are centred on the predicted centre; furniture, bands and satellites are placed as in `prop`; |
| * **text** takes the predicted width (at least its longest word, at most the inner width), the height its wrapped text needs, and a |
| size that starts at what `prop` gives (uniform scale lifted to the floor of the rung) and, at rungs above 0, grows to the largest |
| whole size, up to the anchor's cap, whose wrapped text is at most 1.15 times as tall as the predicted box; |
| * **images, groups and rotated elements** get the largest uniform scale that fits the predicted box and the margins (floor-lifted); |
| a plain shape takes the predicted box; |
| * then the anchor's margin clamp, ink push-apart, strictness per rung, satellites; decorations as in `prop`; |
| * up to 2 layouts (`learned`, `learned:<rung>`, highest rung first), skipping layouts equal to any existing candidate, appended last, |
| so every older candidate is unchanged; never raises (a failure logs `[learned] skipped` and yields none). |
|
|
| **Admission guard (`docs/engine.md` B30, added after judge round 6, section 6b).** `resize()` chooses twice (`_choose`: `_select`, then |
| the critic): over every candidate and over the candidates that are not learned. When the two picks differ, the learned pick ships only if |
| `_learned_admissible` holds: its smallest text is at most `TUNABLES["learned"]["guard_px"]` (1) px under the family-off pick's and its |
| utilization score (S5) at most `guard_util` (0.05) lower; otherwise the family-off pick ships. The log line ends in `learned over <pick>` |
| when a learned layout ships. So the family cannot change the output except through a learned layout that passes the guard: the 8 picks |
| its presence moved through the critic's finalist set (section 5) revert, and without a shipped learned layout the output is byte-identical |
| to stage one's (`tests/test_learned.py`). |
|
|
| Why "box" text: designers' font sizes are the area-preserving size, median 1.0 x sqrt(sx sy) of the source size, against the uniform |
| min(sx, sy) the anchor and `prop` use (designer / uniform: median 1.03 below a 0.2 aspect change, 1.24 from 0.2 to 0.7, 1.76 above 0.7). |
| Taking the size `prop` gives (`text: "prop"`) made `learned` a re-positioned `prop` with small text (median candidate / designer font |
| size 0.80); deriving it from the predicted box gives 0.995 (p10-p90 0.76-1.28, valid candidates above rung 0; `prop` 0.75, anchor 0.78, |
| lanes 1.03 with a wider spread 0.55-1.48; the last-resort rung 0 keeps the `prop` size, 0.77). Measured on the 1,008 test-family pairs, pick distance |
| 0.2522 before: `prop` text 0.2460 and picked in 15%; box text 0.2354 and 29%; box tolerance 1.0 / 1.15 / 1.35: 0.2387 / 0.2354 / |
| 0.2346 (1.15 kept; these three were measured before the rung-0 refinement). Using the prop size at the last-resort rung 0 (crowded canvases, where box-sized text overlapped) raised the |
| validity of generated candidates from 88% to 96% at the same pick distance. These three engine settings were chosen on the 3,000 |
| Crello pairs; the gain is the same on train and test families (below), so little of it can be overfitting, but the 1,008 test pairs |
| were looked at while choosing them. The model itself never saw them. The samples, synthetic and holdout sets were only used to check |
| for regressions. |
|
|
| ## 4. Crello designer pairs: engine without and with the model (`tools/crello/layout_eval.py`) |
| |
| Same 3,000 pairs, same engine (local macOS), `eval_base` = the branch without `layout.npz`, `eval_learned` = with it. |
| `dist` is `designer_eval.py`'s (all non-background elements, decorations included). |
|
|
| **Held-out (test) families, 1,008 pairs:** |
|
|
| | group | pairs | generated | valid | closest valid to the designer | oracle dist before → after | picked | pick dist before → after | stretch dist | picks changed | |
| |---|---|---|---|---|---|---|---|---|---| |
| | all | 1008 | 94% | 96% | 52.8% | 0.1860 → 0.1670 | 28.8% (290) | 0.2522 → **0.2352** | 0.1930 | 319 | |
| | aspect change 0-0.2 | 285 | 78% | 98% | 35.4% | 0.0846 → 0.0820 | 40.4% (115) | 0.0919 → 0.0876 | 0.0587 | 121 | |
| | aspect change 0.2-0.7 | 368 | 100% | 95% | 56.5% | 0.1629 → 0.1477 | 36.1% (133) | 0.2419 → 0.2126 | 0.1541 | 150 | |
| | aspect change 0.7+ | 355 | 100% | 95% | 62.8% | 0.2904 → 0.2548 | 11.8% (42) | 0.3915 → 0.3773 | 0.3410 | 48 | |
| | target strip | 129 | 91% | 93% | 42.6% | 0.2508 → 0.2312 | 24.8% (32) | 0.3265 → 0.3101 | 0.2836 | 35 | |
| | target small | 247 | 95% | 97% | 56.3% | 0.1735 → 0.1560 | 28.7% (71) | 0.2437 → 0.2260 | 0.1678 | 74 | |
| | target square | 120 | 94% | 93% | 45.0% | 0.1880 → 0.1752 | 26.7% (32) | 0.2410 → 0.2283 | 0.1873 | 35 | |
| | target vertical | 273 | 93% | 96% | 55.3% | 0.1743 → 0.1504 | 31.1% (85) | 0.2238 → 0.2054 | 0.1851 | 96 | |
| | target other | 239 | 95% | 96% | 55.6% | 0.1760 → 0.1585 | 29.3% (70) | 0.2589 → 0.2421 | 0.1819 | 79 | |
|
|
| Pick distance gain over the 1,008 test pairs +0.0169, family-bootstrap 95% interval [+0.0132, +0.0209] (232 families); train |
| families (1,992 pairs) +0.0162 [+0.0132, +0.0194]; all 3,000 pairs 0.2489 → 0.2325 (+0.0164 [+0.0140, +0.0190]). Where `learned` is |
| picked, the distance of those pairs goes 0.1909 → 0.1381 (test). Where a valid `learned` candidate exists (904 of 1,008 test pairs), |
| the closest one averages 0.1831 against `prop` 0.2077, anchor 0.2169, lanes 0.334, the stretch 0.195 and the shipped pick 0.241; it is |
| closer than `prop` in 69% of the pairs where both are valid and closer than the stretch in 47%. The selector still ships a |
| layout 0.04 farther from the designers than the stretch (0.2352 against 0.1930), and the oracle (0.1670) shows how much of the gap |
| sits in selection and how much in generation: the rule score prefers lanes layouts (better readability and utilisation, worse |
| fidelity) to the closest `learned` layout in 273 of the 332 test pairs where `learned` is closest but not picked (`prop` 40, anchor 19). |
|
|
| Other effects, all 3,000 pairs: shipped picks valid 2,994 → 2,994, fallbacks 10 → 9; candidates per pair 6.14 → 7.62 (valid 6.09 → |
| 7.50); mean smallest text of the pick 26.40 → 26.82 px; texts below floor per pick 0.661 → 0.613; of the 934 changed picks the smallest text |
| is larger in 432, smaller in 151, equal in 351 (mean 23.56 → 24.93 px). Picked families: lanes 1,699 → 1,411, prop 820 → 373, anchor 469 |
| → 356, learned 0 → 849. |
|
|
| **With the admission guard (shipped; `eval_guard`).** Test families: picked in 23.5% (237 pairs), pick distance 0.2522 → **0.2429** |
| (+0.0093, family bootstrap [+0.0062, +0.0127]); by aspect change 0.0919 → 0.0898 (0 to 0.2, picked 37.5%), 0.2419 → 0.2294 (0.2 to 0.7, |
| 25.5%), 0.3915 → 0.3798 (0.7 and up, 10.1%); train families +0.0082 [+0.0062, +0.0104]; all 3,000 pairs 0.2489 → 0.2403 (+0.0086 |
| [+0.0068, +0.0104]). The guard keeps 699 of the 849 learned picks, gives back about 45% of the unguarded gain, and turns the family into |
| a readability gain as well: of the 699 changed picks the smallest text is larger in 392, smaller in 36, equal in 271 (mean 21.98 → 24.63 |
| px); all pairs: mean smallest text 26.40 → 27.01 px, texts below floor per pick 0.661 → 0.617, fallbacks 10 → 9, picked families lanes |
| 1,699 → 1,549, prop 820 → 396, anchor 469 → 345, learned 699. |
| |
| ## 5. Our three sets (`tools/evaluate.py --no-render --no-invariance`; local macOS, critic on) |
| |
| Baseline = the branch with `CHAMELEON_LEARNED=0`, which reproduces `combined5/` exactly (all 1,162 picks and scores identical). |
| |
| | set | hard checks | fallbacks | all text at floor | texts below floor per pair | smallest text, mean px | mean rule score | picks changed | to `learned` (smaller / larger / equal smallest text) | |
| |---|---|---|---|---|---|---|---|---| |
| | samples (46) | 46/46 → 46/46 | 0 → 0 | 35 → 35 | 0.913 → 0.913 | 22.91 → 22.83 | 0.9215 → 0.9227 | 8 | 5 (2 / 0 / 3) | |
| | synthetic (271) | 271/271 → 271/271 | 0 → 0 | 178 → 178 | 1.144 → 1.125 | 19.89 → 19.82 | 0.9056 → 0.9088 | 38 | 29 (7 / 1 / 21) | |
| | holdout (845) | 845/845 → 845/845 | 0 → 0 | 438 → 438 | 1.611 → 1.593 | 13.17 → 13.09 | 0.8899 → 0.8940 | 124 | 102 (34 / 7 / 61) | |
| |
| **Shipped (with the admission guard, section 3; `evals/guard`):** |
|
|
| | set | hard checks | fallbacks | all text at floor | texts below floor per pair | smallest text, mean px | mean rule score | picks changed | to `learned` (smaller / larger / equal smallest text) | |
| |---|---|---|---|---|---|---|---|---| |
| | samples (46) | 46/46 → 46/46 | 0 → 0 | 35 → 35 | 0.913 → 0.913 | 22.91 → 22.91 | 0.9215 → 0.9228 | 3 | 3 (0 / 0 / 3) | |
| | synthetic (271) | 271/271 → 271/271 | 0 → 0 | 178 → 178 | 1.144 → 1.129 | 19.89 → 19.90 | 0.9056 → 0.9074 | 21 | 21 (2 / 1 / 18) | |
| | holdout (845) | 845/845 → 845/845 | 0 → 0 | 438 → 438 | 1.611 → 1.594 | 13.17 → 13.18 | 0.8899 → 0.8926 | 73 | 73 (13 / 6 / 54) | |
|
|
| Every pick that changes with the guard is a learned layout (the 8 side effects revert); the 15 with smaller smallest text lose 1 px each. |
| The table below and the rest of this section describe the family without the guard, as it was judged. |
|
|
| No hard-check, fallback or floor-count regression. Full run with the invariance checks (determinism, rename invariance, shuffle validity) and the self-check, family on: the same picks as the numbers above, 0 self-check warnings; determinism and shuffle validity hold in every pair; rename invariance fails in 2 pairs (05_wide 728x90, h27_panel 320x50), the same two with `CHAMELEON_LEARNED=0`: pre-existing on this branch, neither pick is a `learned` layout, not touched here. |
| Of the 136 picks that move to `learned`, 126 are the rule score's and tie-breaks' (text smaller in 38, larger in 8, equal in 80) and 10 are |
| the critic overriding to a learned finalist (smaller 5, equal 5). Adding the family also changes the critic's finalist set: 8 further |
| picks move to a different non-learned layout (3 with smaller text: 08_scrim 300x250 7 → 3 px, h37_split 300x250 4 → 3 px and 2480x3508 |
| 27 → 21 px), and 26 picks keep their layout and only change whether the critic or the rule chose it. Every one of the 136: `layout_picks.md`. |
|
|
| The picks are spread over every target class (the family is generated at every aspect change above 3%). Of the 43 picks with smaller |
| smallest text, 25 lose 1 px (9 to 14 px mostly), 4 lose 2 px, 7 lose 3 to 4 px (five of them on canvases where both layouts are under |
| 8 px: h01_stack 300x250 5 → 2, 06_story 160x600 7 → 3, 09_partial_panel 300x250 6 → 3, L1 160x600 13 → 10, h03_stack 1080x1350 14 → 11) and 7 |
| lose 5 to 12 px, all in large layouts where the old pick was a one-column layout (h03_stack 2480x3508 38 → 26, h35_text 2480x3508 37 → 27, |
| h39_stack 1080x1350 19 → 13, h15_text 1080x1080 20 → 14, h30_text 1080x1920 26 → 20, h28_left 2480x3508 27 → 21, 14_rotated 2480x3508 40 → 34). |
| The 8 larger ones: h08_text at three sizes (24 → 30, 26 → 33, 36 → 37 px), h30_text 1000x1500 16 → 20, 09_partial_panel 2480x3508 |
| 27 → 31, h26_stack 2480x3508 31 → 33, h30_text 1080x1080 14 → 16, h15_text 1080x1920 20 → 22. |
| |
| **Runtime.** `resize()` per task, three alternating runs each, best of three: 300 Crello pairs 19.3 s → 22.2 s (64 → 74 ms mean, +15%), the 252 |
| grid pairs of the samples and synthetic designs 30.0 s → 30.8 s (+3%); slowest task on those grids about 2 s either way. The machine was not |
| always quiet (the first run of each took 20 s and 32 s), so read it as roughly +3 to +15%: one or two more candidates are fitted and scored |
| per task. `designer_eval.py`'s summed engine time is dominated by machine state (the same `CHAMELEON_LEARNED=0` run took 2.6 min once and |
| 7.6 min later; interleaved on 800 pairs 1.8 min without the model, 2.2 min with). With `CHAMELEON_LEARNED=0` the new code reproduces |
| every candidate, score and pick of the original 3,000 Crello records. |
|
|
| ## 6. Visual review of changed picks (before the judge round) |
|
|
| Ten required, fifteen looked at, old pick (left) against new pick (right) rendered with the vendored renderer |
| (`tools/ai_judge.py:render_candidate`), by me, one reviewer, no judge model: |
|
|
| | design, size | old → new | what I see | |
| |---|---|---| |
| | L1 1080x1350 | lanes:1 (critic) → learned | same elements, centred stack, 37 px both; equal | |
| | 03_badge_1 1080x1080 | lanes:1 (critic) → learned (critic) | bottle smaller, copy equal, 20 → 18 px; equal, slightly less filled | |
| | L3 1080x1080 | prop → learned | near-identical, slightly more air between rows; equal | |
| | 18_collage 1280x720 | lanes:2 → learned | the split (bottle left, copy right) becomes a centred stack whose bottle base touches the headline; cramped, **worse** | |
| | 05_wide 1584x396 | lanes:3 → learned | larger headline and subhead (smallest text 10 → 9 px), tidy left copy, bottle centre, pills right; arguably **better** | |
| | 06_story 160x600 | lanes:1 → learned (critic) | 7 px → 3 px, legal text vanishes, smaller product; **worse** | |
| | 14_rotated 1080x1350 | anchor → learned | ribbon and rotated side label kept, a little more air; equal | |
| | h15_text 1080x1080 | lanes:1 → learned | the one-column stack that fills the canvas becomes a block in the top half with 14 px pills and an empty lower half; **worse** | |
| | h39_stack 1080x1350 | lanes:1 → learned | bigger bottle, content biased up, label 19 → 13 px; equal | |
| | h03_stack 1080x1350 | lanes:1 → learned | content moved up, 14 → 11 px, ribbon over a circle; equal to slightly **worse** | |
| | h10_stack 1080x1350 | lanes:1 → learned:0.85 | larger headline, fills a little better, 10 → 9 px; slightly **better** | |
| | h04_split 1500x500 | lanes:2 → learned | keeps the source arrangement (headline and copy top left, bottle right), 13 → 12 px; equal | |
| | h38_left 728x90 | lanes:1 → learned:0.55 (critic) | subhead 8 → 6 px under a two-line headline, rule score 0.832 → 0.782; **worse** | |
| | h01_stack 300x250 | lanes:1 → learned | both unreadable, legal text 5 → 2 px; **worse** | |
| | h23_split 1080x1920 | lanes:1+row2 (critic) → learned | bottle placed beside the copy block instead of floating low right; **better** | |
|
|
| Three better, six equal, six worse. The pattern behind the worse ones: (1) at very narrow or very short canvases the uniform scale gives |
| 2 to 3 px text and the lanes layouts it replaces reached 5 to 8 px; (2) the designers' habit of keeping relative positions leaves the |
| lower half of a taller canvas empty, where the one-column layouts the judge preferred fill it; (3) a collage whose parts overlap in the |
| source stays overlapped. The first and third are what the earlier judge rounds penalised. The rule score picks `learned` there because |
| its fidelity term (weight 0.70, tuned on these same designer pairs) is higher (fidelity 0.50 → 0.91 on h15_text 1080x1080, 0.60 → 0.96 |
| on 18_collage 1280x720), outweighing utilisation and readability. |
|
|
| **What to do with it.** A guard that a `learned` pick may not lower the smallest text (offline simulation on the Crello picks, reverting |
| to the old pick when the learned pick's smallest text is 1 px smaller or more) removes 47 of 290 learned picks on test families and keeps |
| +0.0123 of the +0.0169 pick distance gain (a 2 px guard: 24 removed, +0.0139); on our sets it would remove the 43 picks with smaller text. |
| That is a selector rule, which needs its own judge round (section 5 of `docs/crello-gains-2026-10-26.md` is the protocol: render old |
| against new for every changed pick, both letter orders). The round was run (6b) and the rule, with a utilization axis added, is the guard. |
|
|
| ## 6b. Judge round 6: old pick against new pick, and the admission guard (2026-10-27) |
|
|
| Protocol of rounds 4 and 5 (`docs/crello-gains-2026-10-26.md` §5): every pick that changes on the three sets with the family on and no |
| guard (144: 136 to a `learned` layout, 8 to another layout through the critic's finalist set) rendered old against new at 1:1 |
| (`tools/judge/build_change_set6.py`), two sheets per pair with the letters swapped, each sheet judged once by a strong vision model |
| (ten judges, the two orders of a pair by different judges, `tools/judge/make_lists6.py`; `JUDGE_PROMPT.md` unchanged); decided = the |
| same winner in both orders (`tools/judge/aggregate6.py`; raw answers with reasons in `tools/judge/labels6/`). |
|
|
| | set | comparisons | decided | new (learned family) wins | old wins | tie | inconsistent | mean confidence | |
| |---|---|---|---|---|---|---|---| |
| | samples | 5 | 2 | 1 | 1 | 0 | 3 | 2.7 | |
| | synthetic | 32 | 23 | 18 | 5 | 1 | 8 | 2.7 | |
| | holdout | 107 | 85 | 47 | 38 | 0 | 22 | 3.2 | |
| | all | 144 | 110 | 66 | 44 | 1 | 33 | 3.1 | |
|
|
| 66 to 44 (two-sided binomial p = 0.045) is a gain, far smaller than the critic's 67 to 13 (round 5) or the selection rules' 40 to 1 |
| (round 4), and it splits on the two things the visual review above had named before the round: |
|
|
| | subset of the decided comparisons | new wins | old wins | |
| |---|---|---| |
| | new pick is `learned` | 64 | 40 | |
| | … chosen by the rule score / by the critic | 58 / 6 | 36 / 4 | |
| | new pick is another layout (the critic's finalist set changed) | 2 | 4 | |
| | smallest text equal | 44 | 16 | |
| | smallest text larger | 3 | 3 | |
| | smallest text 1 px smaller | 14 | 9 | |
| | smallest text 2 px smaller | 2 | 2 | |
| | smallest text 3 px or more smaller | 1 | 10 | |
| | utilization score (S5) up by 0.05 or more | 38 | 6 | |
| | utilization within 0.05 | 18 | 13 | |
| | utilization down by 0.05 or more | 8 | 21 | |
| | … of which tall targets (height > width) | 1 | 15 | |
|
|
| The judges' reasons for the 44 old wins: top-heavy stacks that leave the lower half of a tall canvas empty (h15, h23, h24, h28, h30, h32, |
| h34, h36, h39), text that goes unreadable on narrow or short canvases (06_story and L1 at 160x600, h01 and h18 at 300x250, h22 at 468x60), |
| tag pills that touch or stagger (h07, h14, h15 1080x1920, h30), the legal line touching or crossing the CTA (h26 at four sizes: the source |
| draws them overlapping, so every family keeps the pair as intentional layering and only the one-column layouts separate them), and the |
| 16:9 collage that stays a portrait stack (18_collage twice). |
|
|
| **The admission guard** (`_learned_admissible`, `TUNABLES["learned"]["guard_px"] = 1`, `guard_util = 0.05`; section 3). On this round |
| it keeps 97 of the 144 changed picks, 71 of them decided at **54 to 17** (p = 1e-5), and reverts 47 that went 12 new, 27 old, 7 |
| inconsistent, 1 tie. Alternatives on the same labels: the text guard alone keeps 61 to 28; text or a 0.10 utilization drop 59 to 25; |
| 1 px or 0.10, 47 to 17. The two axes were named before the round (section 6: small text, top-heavy stacks) and the offline simulation |
| of the text guard was in the report before any label existed; the thresholds were chosen on the round, so 54 to 17 is in-sample and a |
| seventh round on designs neither the model nor the guard has seen is the honest check. What the guard costs on Crello is in section 4 |
| (+0.0169 → +0.0093) and what it does on the three sets in section 5. |
|
|
| ## 6c. Judge round 7: the guard out of sample (2026-10-28) |
|
|
| The check section 6b asked for. 90 new designs from the holdout generator with a new seed (`tools/holdout7/`, |
| `python tools/holdout7/build_holdout.py 90 20261028`: the same five style families as `tools/holdout/` with new copy, sizes and |
| element mixes; nobody looked at them before the labels were in) on the holdout grid of about 21 sizes per design, 1,900 pairs. |
| Three engine runs: family off (`CHAMELEON_LEARNED=0`), the shipped engine (guard on) and the family without its guard |
| (`CHAMELEON_LEARNED_GUARD=0`, an evaluation-only switch added for this round). Hard checks 1,900 of 1,900 and fallbacks 0 in all |
| three; all text at floor 996 → 1,001 (shipped) / 1,000 (no guard); mean smallest text 12.90 → 12.92 / 12.85 px; mean rule score |
| 0.8866 → 0.8889 / 0.8900. Without the guard the family changes 168 picks (8.8%), all to learned layouts; the guard keeps 117 (6.2%) |
| and reverts 51. Every one of the 168 was judged old against new, both letter orders, by twelve judges (the two orders of a pair by |
| different judges; `tools/judge/build_change_set7.py` and `build_change_set7b.py`, the second batch of 60 designs added when the |
| first 30 gave only 37 changed picks; `make_lists7*.py`; `aggregate7.py`; raw answers in `tools/judge/labels7/`). |
|
|
| Decision rule, written down before any label existed: merge if the kept picks win at least 60% of the decided comparisons and the |
| reverted picks win less often than the kept ones. |
|
|
| | | comparisons | decided | new wins | old wins | new share | |
| |---|---|---|---|---|---| |
| | all (family without the guard) | 168 | 141 | 86 | 55 | 61% (p = 0.011) | |
| | **kept by the guard (what ships)** | 117 | 98 | **71** | **27** | **72%** (p = 1e-5) | |
| | reverted by the guard | 51 | 43 | 15 | 28 | 35% | |
|
|
| Both conditions hold; the in-sample 76% of round 6 (54 to 17) becomes 72% out of sample. Kept picks by target class: small canvases |
| (300x250, 336x280, 300x600, 160x600, 320x50, 468x60) 31 to 3, strips 7 to 1, squares 3 to 0, vertical 23 to 15, other 7 to 8. |
| The breakdowns that produced the guard replicate on the utilization axis and only partly on the text axis: |
|
|
| | subset of the decided comparisons | new wins | old wins | |
| |---|---|---| |
| | utilization score up by 0.05 or more | 61 | 13 | |
| | within 0.05 | 17 | 20 | |
| | down by 0.05 or more | 8 | 22 | |
| | … of which tall targets | 3 | 13 | |
| | smallest text equal | 59 | 20 | |
| | smallest text larger | 6 | 9 | |
| | 1 px smaller | 9 | 13 | |
| | 2 px smaller | 8 | 3 | |
| | 3 px or more smaller | 4 | 10 | |
| | reverted for utilization alone | 3 | 15 | |
| | reverted for text alone | 7 | 6 | |
| | reverted for both | 5 | 7 | |
|
|
| The utilization guard does the work (3 to 15 where it alone reverts); the text guard alone is a wash out of sample (7 to 6) and costs |
| a few wins at exactly 2 px (8 to 3) while still removing the 3-px-and-more losses (4 to 10). It stays as judged: the shipped rule is |
| the one these labels measured, and loosening it to 2 px would be a new in-sample choice. |
|
|
| Caveats: the same generator as the old holdout (unseen instances, not unseen kinds of design); the judge is the model family whose |
| labels trained the critic (so this is an independent check on the family and the guard, not on the critic); local CoreText text |
| measurement. |
|
|
| ## 7. Tests |
|
|
| `tests/test_learned.py` (12 tests): features (shape, determinism, rename invariance, target enters only through aspect and size); |
| inert without weights (absent, corrupt, wrong names, wrong shapes, non-finite: no candidates, `resize()` byte-identical to |
| `CHAMELEON_LEARNED=0` on three samples at four sizes); the env switch and tunable; not generated within ±3%; a zero-output model |
| reproduces the stretch's boxes through the whole plumbing; with the shipped file, `learned` exists on L2 at 1600x400 and 1080x1920, |
| is complete, deterministic, on the canvas, unique, appended last with every older candidate unchanged, valid on L2 1080x1920 (at |
| 1600x400 only the last-resort rung exists and it fails H8); the file is ≤ 200 KB, loads without pickle and predicts finite boxes |
| on three samples at ten sizes; every learned candidate on that grid fits without errors and keeps every id; degenerate and |
| non-finite designs never raise; a failing model is skipped. Four older tests that pin the older families' behaviour or compare with |
| a models directory that is also missing `layout.npz` now set `CHAMELEON_LEARNED=0`; one (`test_proportional_candidates_exist...`) |
| ignores `learned` candidates. Three guard tests (`LearnedGuardTests`): the thresholds on stand-in picks; on the three samples at ten sizes the output with the |
| family differs from `CHAMELEON_LEARNED=0` only where the log says a learned layout shipped (and it ships at least once); with guards |
| that always revert, the two outputs are identical everywhere. |
| Results: all 66 tests pass (51 existing + 15 new; `tests/test_layout.py` and `python -m unittest discover -s tests -p "test_*.py"`, as CI runs them), and `python -m engine.main` on the three samples gives byte-identical JSON in two runs with no fallback or self-check warning. |
|
|
| ## 8. Caveats |
|
|
| * **The guard's thresholds were chosen on round 6 and checked out of sample in round 7** (section 6c: 90 fresh designs, kept picks |
| 71 to 27, reverted 15 to 28). The text axis alone did not replicate (7 to 6); the utilization axis did (3 to 15). The Crello data |
| says designers shrink text; the judge does not. |
| * **Crello is not the organizers' briefs.** Geometry of VistaCreate templates (grey boxes for art, synthetic words, Solstice fonts); |
| nothing here was checked on a brand banner that is not one of our 3 + 18 + 40 designs. |
| * **The critic had never seen these layouts**; of the 136 unguarded learned picks, 10 are its overrides (judged 6 to 4). The 8 other |
| picks its finalist set moved (judged 2 to 4) revert through the guard. |
| * **Selection, not generation, is the bottleneck for designer closeness**: oracle 0.1670 against pick 0.2352 on test families. |
| * **Local macOS only** (CoreText); the Docker (FreeType) run, which differs by up to ~8% in text width at small sizes (R15), was not |
| repeated, and `tools/evaluate.py` numbers in CI may differ by a few picks. |
| * **Trained on train families only** (915 of 1,316 families); retraining on all of them is one flag away but would remove the held-out |
| numbers. |
| * **Validity**: 95% of generated candidates pass every hard check (91% at aspect changes above 0.7); invalid ones are last-resort |
| layouts with text overlapping text and are never shipped. |
| * The sklearn MLP in the table is trained with squared loss because that is all `MLPRegressor` offers; it is the weakest row for a reason |
| that has nothing to do with network size. |
|
|
| ## 9. Commands |
|
|
| ```bash |
| .venv/bin/python tools/crello/layout_dataset.py |
| OMP_NUM_THREADS=4 /opt/anaconda3/bin/python3 tools/crello/train_layout.py --export engine/models/layout.npz --report output/crello/layout_train.md # add --skip-sklearn to save 2 min |
| CHAMELEON_LEARNED=0 .venv/bin/python tools/crello/designer_eval.py --workers 8 --out output/crello/eval_base |
| .venv/bin/python tools/crello/designer_eval.py --workers 8 --out output/crello/eval_learned |
| .venv/bin/python tools/crello/layout_eval.py --base output/crello/eval_base/candidates.jsonl --new output/crello/eval_learned/candidates.jsonl |
| CHAMELEON_LEARNED=0 .venv/bin/python tools/evaluate.py --set samples --no-render --no-invariance --summary-md /tmp/base_samples.md # likewise synthetic, holdout |
| .venv/bin/python tools/evaluate.py --set samples --no-render --no-invariance --summary-md /tmp/new_samples.md |
| .venv/bin/python tools/crello/layout_picks.py --base <dir of base summaries> --new <dir of new summaries> --out tools/crello/layout_picks.md |
| # the guard: CHAMELEON_LEARNED=0 vs the shipped engine is the "shipped" table of section 5; `eval_guard` in section 4 is designer_eval.py with the guard on |
| .venv/bin/python tests/test_layout.py && .venv/bin/python -m unittest discover -s tests -p "test_*.py" |
| ``` |
|
|
| Files: `engine/resize.py` (features, `_learned_size`, `_learned_candidates`, `_anchor_try(pred=)`, loader `_layout_model`, `_layout_predict`), |
| `engine/models/layout.npz`, `tools/crello/{layout_dataset,train_layout,layout_eval}.py`, `tests/test_learned.py`, this report, |
| `LAYOUT_MODEL_CARD.md`, `layout_picks.md`, `docs/engine.md` (B29, section 6), `tools/crello/README.md`. |
|
|
