Release evaluation artifacts batch 1

eval/category_winners.md

@@ -0,0 +1,33 @@
+# Category winners
+
+Generated by scripts/category_winners.py from eval/*/metrics.json, eval/hps3_summary.json and the judge reports. 24 prompts, 768 px, one seed each; the judge is a Claude model
+instance (+-0.2 noise per average); HPSv3 paired SE vs the teacher is ~0.2-0.3 (differences below ~0.5 mu are not distinguishable).
+
+| tag | GB | judge accept A | judge text T | text exact /6 | HPSv3 mu (diff to teacher +- SE) | DINOv2 | SSIM | latent cos | CLIP |
+|---|---|---|---|---|---|---|---|---|---|
+| mixH2_xm_f8 | 1.72 | 4.15 | 4.00 | 4 | 11.44 (-0.94 +- 0.20) | 0.851 | 0.594 | 0.656 | 36.61 |
+| mixI_x | 1.74 | - | - | - | 9.66 (-2.73 +- 0.29) | 0.745 | 0.541 | 0.572 | 35.60 |
+| mixI_tuned | 1.74 | 3.15 | 1.14 | 0 | 10.65 (-1.73 +- 0.31) | 0.788 | 0.554 | 0.607 | 35.93 |
+| mixH2_xm | 1.83 | - | - | - | 11.49 (-0.90 +- 0.22) | 0.856 | 0.594 | 0.659 | 36.68 |
+| mixH2_x_f8 | 1.85 | 4.21 (=mixH2_x) | 4.00 | 3 | 11.99 (-0.40 +- 0.18) | 0.877 | 0.641 | 0.720 | 36.39 |
+| mixH_x | 1.95 | - | - | - | 11.88 (-0.50 +- 0.22) | 0.867 | 0.627 | 0.700 | 36.64 |
+| mixH_tuned | 1.95 | 4.10 | 4.00 | 3 | 11.70 (-0.69 +- 0.30) | 0.860 | 0.615 | 0.680 | 36.11 |
+| mixH2_x | 1.95 | 4.21 | 4.00 | 3 | 11.92 (-0.47 +- 0.19) | 0.877 | 0.640 | 0.720 | 36.33 |
+| mixH2_tuned | 1.95 | 4.17 | 3.93 | 2 | 11.90 (-0.49 +- 0.23) | 0.864 | 0.622 | 0.691 | 36.26 |
+| mixG_x | 2.14 | 4.29 | 4.43 | 4 | 12.09 (-0.30 +- 0.19) | 0.884 | 0.656 | 0.733 | 36.31 |
+| mixG_tuned | 2.14 | 4.21 | 4.79 | 6 | 12.14 (-0.24 +- 0.29) | 0.901 | 0.666 | 0.750 | 36.35 |
+| mixF_ptq | 2.42 | 4.02 | 4.36 | - | 10.86 (-1.53 +- 0.23) | 0.802 | 0.438 | 0.572 | 36.07 |
+
+## Winners
+
+| category | winner | evidence | statistically tied with |
+|---|---|---|---|
+| best overall (judge 'would accept') | mixG_x (A 4.29) | judge reports | mixG_tuned, mixH_tuned, mixH2_x, mixH2_tuned, mixH2_xm_f8 |
+| best text rendering | mixG_tuned (T 4.79, 6/6 letter-exact) | judge reports | none (next best T 4.43) |
+| best aesthetics (HPSv3) | mixG_tuned (mu 12.14, teacher 12.38) | eval/hps3_summary.md | mixG_x, mixH_x, mixH_tuned, mixH2_x, mixH2_x_f8, mixH2_tuned |
+| closest to the teacher | mixG_tuned (DINOv2 0.901, SSIM 0.666, latent cos 0.750) | eval/*/metrics.json | none by DINOv2 (next 0.884) |
+| smallest acceptable (judge A >= 4.0) | mixH2_xm_f8 (1.72 GB, A 4.15, HPSv3 -0.94 vs teacher) | judge + size | - |
+| best size/quality value (A >= 4.0 and HPSv3 within 0.5 of the teacher) | mixH2_x_f8 (1.85 GB, HPSv3 -0.40) | judge + HPSv3 + size | - |
+
+Notes: mixH2_x_f8 was not judged separately (metrics identical to mixH2_x; same model within fp8 rounding of the LoRA factors). HPSv3 sees the int4 modulation layers (mixH2_xm*) as a real
+aesthetic cost (-0.94 vs -0.40 for fp8-LoRA only), which agrees with the judge's observation of a small cool/low-contrast drift. 1-bit-attention (mixI) and ternary/binary (tern_v1) are not acceptable.

eval/eval_prompts.json

@@ -0,0 +1,146 @@
+[
+  {
+    "id": 0,
+    "seed": 1001,
+    "category": "text",
+    "prompt": "Photograph of a small cheese shop storefront at dusk, a cream-painted facade with a navy hand-lettered sign reading \"LE PETIT FROMAGE\" above the door, a smaller line on the window glass \"Affineur depuis 1974\", warm lamplight falling on wheels of cheese, wooden crates and a ginger cat asleep on the step, a slate board on the pavement saying \"Tasting - Saturday 11 AM\", wet cobblestones reflecting the glow, shallow depth of field. The gold lettering is crisp and clearly readable, the slate writing is neat white chalk, and the rain-slick street shows glowing reflections of the shop windows."
+  },
+  {
+    "id": 1,
+    "seed": 1002,
+    "category": "text",
+    "prompt": "Clean flat-design infographic poster on an off-white background, the title \"WATER CYCLE\" in bold blue sans-serif capitals at the top, a simple illustration of a sun, a cloud, a mountain with a river and a sea, with neat arrows and four labels \"Evaporation\", \"Condensation\", \"Precipitation\", \"Collection\" placed beside each stage, rounded icons, tidy margins, soft pastel blues and greens. Each label sits in a clean rounded box beside its illustration, the lines and arrows are neatly aligned, all text is crisp and perfectly readable, and the layout leaves generous margins around the diagram."
+  },
+  {
+    "id": 2,
+    "seed": 1003,
+    "category": "text",
+    "prompt": "Vintage-style movie poster illustration of a lighthouse keeper's daughter rowing a small boat through storm waves toward a glowing lighthouse, the title \"BEYOND THE BREAKERS\" in large cream serif letters across the bottom, a tagline \"Some lights never go out\" in small italic text above it, painted look with textured brushwork, deep teal sea and warm yellow beam, slight print grain. A swirl of white spray crowns each wave, the girl wears a bright yellow raincoat, the beam of the lighthouse cuts through the storm clouds, and all lettering is sharp, evenly spaced and fully legible."
+  },
+  {
+    "id": 3,
+    "seed": 1004,
+    "category": "text",
+    "prompt": "Mobile app UI mockup shown on a modern phone screen: a music player with a square album cover of a purple sunset, the song title \"Neon Horizon\" in white bold text, the artist line \"Luna Vale\" in grey below, a progress bar at \"1:24 / 3:48\", round play, previous and next buttons, a dark gradient background, polished glassy interface, phone resting on a desk with soft light. Small rounded icons sit beneath the controls, the interface is neatly aligned with crisp white and grey typography, soft shadows give the cards depth, and the phone screen glows gently in the dim room."
+  },
+  {
+    "id": 4,
+    "seed": 1005,
+    "category": "text",
+    "prompt": "Chinese restaurant street scene at night: a large red illuminated signboard reading \"老王面馆\" in bold golden characters, a smaller line \"手工拉面 · 营业至凌晨两点\" underneath, paper lanterns hanging from the eaves, steam escaping from the kitchen door, a few customers seated at metal tables on the sidewalk, wet ground with colorful reflections, cinematic night photography. The golden characters are sharp and evenly spaced on the red board, the second line is smaller but clearly readable, and warm lantern light mixes with cool shadows across the wet pavement and the diners' faces."
+  },
+  {
+    "id": 5,
+    "seed": 1006,
+    "category": "text",
+    "prompt": "Photograph of a roadside fruit stand under a striped awning in the countryside, wooden crates of fruit with hand-painted signs reading \"PEACHES $3\", \"CHERRIES $5\" and \"HONEY $8\", white brush letters on dark green boards, a straw hat hanging on a post, wildflowers in the foreground, rolling hills and a bright blue sky behind, soft morning sun, crisp detail on the painted wood. A few daisies lean against a crate, the painted numbers are perfectly legible, and sunbeams scatter warm highlights across the grain of the weathered wood and the glossy fruit skins."
+  },
+  {
+    "id": 6,
+    "seed": 2001,
+    "category": "portrait",
+    "prompt": "Studio portrait of an elderly Himalayan monk with a shaved head and deeply lined face, wearing a maroon robe with a golden edge, a gentle half-smile and clear dark eyes looking just past the camera, soft window light from the left, a plain warm grey backdrop, fine skin pores, wrinkles and eyebrow hairs resolved, 85mm lens, shallow depth of field. His heavy-lidded eyes are warm and clear, tiny prayer beads loop around his wrist, the robe's wool weave is clearly visible, and the light gently models the bones of his skull and the silver stubble on his chin."
+  },
+  {
+    "id": 7,
+    "seed": 2002,
+    "category": "portrait",
+    "prompt": "Cinematic close-up portrait of a young woman with curly auburn hair and green eyes, wearing a navy turtleneck, standing in a rain-streaked train window at night, colored city lights as soft bokeh circles behind her, a faint reflection of the glass over her cheek, droplets on her eyelashes, teal and amber color grading, subtle film grain. A single red lamp glow on the glass mixes with the cool blue of the night, tiny droplets scatter across her skin, her lips are slightly parted, and the soft grain gives the image a quiet intimate mood."
+  },
+  {
+    "id": 8,
+    "seed": 2003,
+    "category": "portrait",
+    "prompt": "Anime-style portrait of a boy with messy black hair and golden eyes wearing a loose white shirt and a dark green scarf, standing in a field of blue flowers under a wide sky with drifting clouds, a gentle wind blowing his hair and scarf, soft cel shading, delicate detailed highlights in the eyes, pastel painterly background. Her fine hair strands catch the light, drifting petals cross the foreground, the white shirt billows with the breeze, the line art is clean and the eyes are rendered with glossy layered highlights."
+  },
+  {
+    "id": 9,
+    "seed": 2004,
+    "category": "portrait",
+    "prompt": "Environmental portrait of a middle-aged female ceramicist in a sunlit workshop, standing with her arms crossed and clay-smeared linen apron, short grey hair and round glasses, shelves of glazed bowls and vases behind her, warm morning light from a large window, a confident calm smile, natural skin texture, shallow depth of field. Clay crusts her forearms and her apron, her wrinkles around the eyes deepen with the smile, a kiln glows faintly in the background, and the light falling across the shelves is warm, honest and unretouched."
+  },
+  {
+    "id": 10,
+    "seed": 3001,
+    "category": "detail",
+    "prompt": "Extreme close-up photograph of a vintage mechanical typewriter, round chrome-rimmed glass keys with black letters, tiny typebars curled behind them, a sheet of cream paper with a few lines of typed text rolled in, worn enamel, scratches and dust in the crevices, soft side light, razor-thin depth of field, rich texture. Tiny serif letters on each key are readable, the carriage return lever is slightly bent, dust and ink smudges collect around the platen, and the warm light creates soft reflections along the metal rim."
+  },
+  {
+    "id": 11,
+    "seed": 3002,
+    "category": "detail",
+    "prompt": "Macro photograph of a peacock feather's eye, iridescent barbs in turquoise, emerald, gold and violet, each fine strand visible with a metallic sheen, tiny water droplets resting on the surface, dark soft background, studio lighting that shifts the colors, ultra sharp detail, shallow focus at the edges. The barbs overlap like fine silk threads, tiny water drops magnify the underlying texture, soft bokeh in the dark background has warm gold tints, and the photograph shows extreme sharpness at the center of the eye."
+  },
+  {
+    "id": 12,
+    "seed": 3003,
+    "category": "detail",
+    "prompt": "Photograph of an ancient oak tree trunk covered in thick moss, lichen, ferns and tiny mushrooms, deep bark furrows, snails on the surface, rain droplets on every leaf, soft green forest light with mist, fine texture visible all over, natural colors, shallow depth of field in the background. Droplets of water bead on tiny leaves of moss, bracket fungi in cream and orange cling to the bark, a ladybug crawls along a fern, and green light filters softly from above with delicate mist in the air."
+  },
+  {
+    "id": 13,
+    "seed": 3004,
+    "category": "detail",
+    "prompt": "Highly detailed photograph of a chainmail and leather medieval gauntlet resting on a wooden table, individual interlocking metal rings, rivets, stitching and scratches, a worn leather strap, a candle flame casting warm uneven light and long shadows, dust on the metal, dark background, macro lens detail. A single bright highlight runs along each steel ring, stitched seams show tiny dust and wax stains on the leather, the table's wood has deep scratches, and the candle's warm glow fades into complete darkness behind."
+  },
+  {
+    "id": 14,
+    "seed": 4001,
+    "category": "counting",
+    "prompt": "Photograph of exactly five colorful hot air balloons floating over a green valley at sunrise: a red one, an orange one, a yellow one, a blue one and a striped purple one, evenly spread across the sky with distance haze, misty hills, soft golden light, a tiny village below, wide shot. Each balloon has a distinct pattern and a small wicker basket with a tiny pilot, long morning shadows stretch across the fields, wisps of mist hang between hills, and the colors glow warmly in the low sun."
+  },
+  {
+    "id": 15,
+    "seed": 4002,
+    "category": "counting",
+    "prompt": "Studio photograph of exactly three ripe red apples, two yellow pears and one green pear arranged in a wooden bowl on a white linen cloth, each fruit clearly separated and recognizable, soft window light from the left, subtle shadows, a muted grey wall behind, fresh and clean still-life composition. The fruits have small natural blemishes and glossy highlights, the leaves of the pears are fresh, the linen fabric shows fine folds, and the soft shadow of the bowl falls toward the right edge of the table."
+  },
+  {
+    "id": 16,
+    "seed": 5001,
+    "category": "varied",
+    "prompt": "Epic cinematic landscape of a colossal ancient tree growing from the center of a mirror-like lake at twilight, its branches glowing with soft golden lanterns, roots reaching down into clear water, a small wooden boat with one traveler approaching, distant mountains under a deep violet sky with the first stars, painterly fantasy realism. Flowers glow in the water, the bridge between giant roots is lit by a handful of floating candles, a pale mist drifts across the lake, and the stars above are reflected in sharp points in the dark surface."
+  },
+  {
+    "id": 17,
+    "seed": 5002,
+    "category": "varied",
+    "prompt": "Photograph of a rainy neon-lit Seoul street at night, a woman in a clear umbrella and yellow trench coat walking past a row of glowing shop signs, steam from a food cart, reflections on the wet asphalt, taxis blurred in the distance, teal and magenta color palette, 50mm lens, cinematic atmosphere. Her coat is bright and slightly wet, signs in pink, blue and orange glow behind her in blurred bokeh, a scooter passes in the distance, and the street is full of mirrored lights and thin rain streaks."
+  },
+  {
+    "id": 18,
+    "seed": 5003,
+    "category": "varied",
+    "prompt": "Isometric 3D render of a cozy miniature bakery on a floating cube of ground, a pastel pink shop with a striped awning, tiny bread loaves in the window, a chimney puffing white smoke, a little tree, a cobblestone path and a tiny delivery bicycle, soft global illumination, clay-like materials, pastel blue backdrop. A tiny chimney, a delivery scooter, small round bread rolls and a hanging sign with a croissant icon create charming details, the soft shadows are smooth, and the pastel colors are clean and cheerful."
+  },
+  {
+    "id": 19,
+    "seed": 5004,
+    "category": "varied",
+    "prompt": "Wildlife photograph of a brown bear catching a salmon in a rushing river, the fish leaping mid-air with water droplets frozen around it, the bear's wet fur, open mouth and focused eyes, green forest and mist behind, bright overcast light, fast shutter speed, telephoto compression. The bear's fur is soaked and shining, water streams from the salmon's tail, rocks and logs are wet and mossy, the forest in the background is soft green, and the splashes sparkle in the light."
+  },
+  {
+    "id": 20,
+    "seed": 5005,
+    "category": "varied",
+    "prompt": "Watercolor and ink illustration of an old Venetian canal at sunrise, gondolas tied to striped poles, pastel-colored palazzos with arched windows reflected in the green water, a few pigeons on a stone bridge, loose washes with bleeding pigments, fine pen line details, visible paper texture. A small red flower pot hangs on one balcony, laundry lines cross above the canal, the reflections ripple in soft colors, a gondolier in a striped shirt rests his oar, and the paper shows a faint pencil underdrawing."
+  },
+  {
+    "id": 21,
+    "seed": 5006,
+    "category": "varied",
+    "prompt": "Dark fantasy digital painting of a lone knight in battered black armor kneeling before a giant glowing crystal in a ruined cathedral, shafts of cold blue light through broken stained-glass windows, floating dust and embers, torn banners on the pillars, heavy shadows, dramatic scale, richly detailed metal and stone. Ancient runes glow faintly in the stone floor, floating embers drift upward, his helmet lies at his side, a broken sword is planted in the floor, and the light of the crystal casts strong cold edge highlights on the armor."
+  },
+  {
+    "id": 22,
+    "seed": 5007,
+    "category": "varied",
+    "prompt": "High-key bright photograph of a minimalist Scandinavian kitchen with white cabinets, a light oak table set with a ceramic jug of tulips and two cups of coffee, morning sun streaming through a large window to create soft shadows, pale grey floor, a plant on the shelf, clean airy mood. Crumbs and a steaming coffee cup sit on the table, tulip petals cast delicate shadows, a pale wooden chair is pulled back, light bounces off the white cabinets, and the entire scene is bright and calm."
+  },
+  {
+    "id": 23,
+    "seed": 5008,
+    "category": "varied",
+    "prompt": "Low-light photograph of a street food stall in a rainy alley at night, a vendor in a hooded jacket frying dumplings in a large pan, warm yellow bulbs above, rising steam and smoke, a few customers' backs in the foreground, wet pavement with orange reflections, deep shadows, gritty documentary style. Orange oil sizzles in the pan, a pile of dumpling wrappers sits beside a bowl of filling, a customer's hand reaches out with a coin, drops of rain fall under the awning, and the atmosphere is gritty and warm."
+  }
+]

eval/ghost_rate_part1.md

@@ -0,0 +1,70 @@
+# Ghost-face rate, part 1 (gh_x_* folders, 4 text-encoder variants)
+
+Judge: a Claude model instance doing visual inspection with its own image viewing. Not a human, not a trained detector.
+Counts carry judgement noise of a few images. Each variant has only 36 images, so differences of 1-3 images are not meaningful.
+
+## Method
+
+- Data: `eval\gh_x_bf16`, `gh_x_int8`, `gh_x_gptq`, `gh_x_i4e8`; 36 images each (prompt ids 06, 07, 08, 09, 17, 22; six seeds each). 144 images total.
+- Blinding: one fixed random permutation of the four folders (`random.Random(20261030).shuffle`), mapping written to a temp file and not opened until all 144 were classified. Labels in composites: A, B, C, D.
+- For each (prompt id, seed) one 2x2 composite (640 px per variant, 1280x1280) was built and viewed at full size: 36 composites. Faces were then re-checked at 1:1 / zoomed crops (all 07 seeds, 06/08/09 face contact sheets, 17 face crops, 08 seeds 2203/2303, kitchen leg regions for 22).
+- Classes: clean / ghost (double or double-exposed face) / mushy (face smeared, no clear double) / other (heavy non-face artifact such as translucent or duplicated furniture).
+- Mapping revealed after judging: A = gh_x_bf16, B = gh_x_i4e8, C = gh_x_gptq, D = gh_x_int8.
+- Judging was blind to the mapping, but the images of one variant (later revealed as i4e8) often differed visibly in rendering style or composition from the other three, so I may have had an impression of "same variant" across composites. I did not know which encoder it was.
+- Tooling note: Pillow/numpy came from an existing venv (`[private metadata removed] no GPU, no process other than short CPU Python runs. Composites were written to the OS temp folder and deleted afterwards.
+
+## Total counts per variant (36 images each)
+
+| variant | clean | ghost | mushy | other | total |
+|---|---|---|---|---|---|
+| bf16 | 35 | 0 | 0 | 1 | 36 |
+| int8 | 35 | 0 | 0 | 1 | 36 |
+| gptq | 35 | 0 | 0 | 1 | 36 |
+| i4e8 | 29 | 2 | 0 | 5 | 36 |
+
+Of i4e8's 5 "other", 1 (08_seed2203) is mild (semi-transparent scarf end over the chest). Without it i4e8 would be 30 clean / 2 ghost / 4 other.
+The single "other" for bf16/int8/gptq is the same image in each (22_seed5207, translucent table legs in all four variants).
+
+## Per prompt id (clean / ghost / mushy / other; 6 images per cell)
+
+| prompt id | bf16 | int8 | gptq | i4e8 |
+|---|---|---|---|---|
+| 06 monk | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 |
+| 07 curly hair, rainy window | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 | 4/2/0/0 |
+| 08 anime boy | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 | 5/0/0/1 (mild) |
+| 09 pottery studio | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 |
+| 17 umbrella street | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 |
+| 22 white kitchen | 5/0/0/1 | 5/0/0/1 | 5/0/0/1 | 2/0/0/4 |
+| total | 35/0/0/1 | 35/0/0/1 | 35/0/0/1 | 29/2/0/5 |
+
+Ghost count per prompt id (face ghosts only): 07 has 0 / 0 / 0 / 2 for bf16 / int8 / gptq / i4e8; all other prompt ids have 0 ghost faces in every variant.
+
+## Flagged images
+
+| image | class | note |
+|---|---|---|
+| gh_x_i4e8/07_seed2002.png | ghost | second 3/4-view face smeared over the right cheek, second mouth, doubled eyes (double exposure) |
+| gh_x_i4e8/07_seed2202.png | ghost | two overlapping faces: doubled mouth and doubled eyes, mushy cheeks |
+| gh_x_i4e8/08_seed2203.png | other (mild) | scarf end semi-transparent over the open collar/chest; face itself clean |
+| gh_x_i4e8/22_seed5007.png | other | extra translucent table leg and pale ghost legs beside/under the chair |
+| gh_x_i4e8/22_seed5107.png | other | translucent ghost chair floating under the table |
+| gh_x_i4e8/22_seed5207.png | other | pale, doubled, see-through table legs on the left (strongest of the four for this seed) |
+| gh_x_i4e8/22_seed5407.png | other | translucent ghost chair at left under the table |
+| gh_x_bf16/22_seed5207.png | other | table legs washed out / see-through, almost white |
+| gh_x_gptq/22_seed5207.png | other | table legs washed out / see-through, almost white |
+| gh_x_int8/22_seed5207.png | other (milder) | one see-through table leg, the rest opaque |
+
+No image was classed as mushy.
+
+Borderline cases counted as clean (not in the tables above):
+- gh_x_bf16, gh_x_gptq, gh_x_int8 / 22_seed5507: faint pale far table legs against the white floor.
+- gh_x_int8 / 22_seed5307: a hairline-thin pale leg behind the table.
+- gh_x_i4e8 and gh_x_int8 / 07_seed2102: faint window-haze veil on the face, no second face.
+- gh_x_i4e8 / 08_seed2303: translucent scarf wisp beside the shoulder.
+
+## Observations
+
+- Face ghosting appears only in i4e8: 2 of 36 images, both in prompt 07 (seeds 2002 and 2202). bf16, int8 and gptq had no ghost faces and no mushy faces in any of the 108 images; the faces in 06, 08, 09 and 17 were clean in all four variants.
+- The ghost faces are seed-specific: 2 of the 6 prompt-07 seeds in i4e8, while the same seeds were clean in the other three variants, and the other four 07 seeds were clean in all four variants.
+- The "other" class is concentrated in the kitchen prompt 22: i4e8 has translucent/ghost furniture in 4 of 6 kitchen images (two clear ghost chairs on seeds 5107 and 5407), the other variants have it in 1 of 6. Seed 5207 shows see-through table legs in all four variants (same noise), so it does not separate the variants; seeds 5007, 5107 and 5407 flag i4e8 only.
+- Overall i4e8 has 7 flagged images against 1 for each of the other three (6 against 0 if the shared 5207 case is set aside). That gap is larger than the 1-3 image band, but with 36 images per variant, a single judge, and some borderline calls, it is a rough signal, not a measured rate. bf16, int8 and gptq cannot be told apart from each other on this data.

eval/ghost_rate_part2.md

@@ -0,0 +1,85 @@
+# Ghost-face rate, part 2 (gh_turbo_bf16, gh_x_bf16, gh_tuned_bf16)
+
+Judge: a Claude model instance doing visual inspection with its own image viewing. Not a human, not a trained detector.
+Counts carry judgement noise of a few images. Each variant has only 36 images, so differences of 1-3 images are not meaningful.
+
+## Method
+
+- Data: `eval\gh_turbo_bf16`, `eval\gh_x_bf16`, `eval\gh_tuned_bf16`; 36 images each (prompt ids 06, 07, 08, 09, 17, 22; six seeds each). 108 images total.
+- Blinding: one fixed random permutation of the three folders (PowerShell `Get-Random -SetSeed 918273`), mapping written to a temp file and not opened until all 108 were classified. Labels in composites: A, B, C.
+- For each (prompt id, seed) one side-by-side composite (3 x 640 px = 1920x640) was built and viewed: 36 composites. Where something looked doubled or translucent I re-checked 1:1 / zoomed crops from the original images (crops were built in the same blinded A/B/C order, recomputed in the script, mapping file still unread). Zoomed crops: 07 seeds 2002, 2102, 2202, 2402, 2502; 08 seeds 2103, 2203; 17 seeds 5202, 5302, 5402; 22 seeds 5007, 5107, 5307, 5507. All other images were judged from the composites only (06 and 09 faces looked sharp and single at that scale).
+- Classes: clean / ghost (double or double-exposed face) / mushy (face smeared, no clear double) / other (heavy non-face artifact such as translucent or duplicated furniture, see-through objects).
+- Mapping revealed after judging: A = gh_turbo_bf16, B = gh_x_bf16, C = gh_tuned_bf16.
+- Tooling note: Pillow/numpy were not installed in the default Python installs, so composites and crops were made with .NET System.Drawing from PowerShell (CPU only, no GPU, no other process). Composites, crops and helper scripts were written to the OS temp folder and deleted afterwards. No existing file was modified.
+- Rule used for "other": flagged when a major object (chair, table leg, umbrella) was visibly translucent or doubled at composite scale; "mild" marks cases that were visible at composite scale but small. Things only visible at zoom, or plausible stylistic effects (wind-blown translucent scarf in the anime prompt, kitchen towel, rain-window haze), were counted clean and listed under borderline.
+
+## Total counts per variant (36 images each)
+
+| variant | clean | ghost | mushy | other | total |
+|---|---|---|---|---|---|
+| turbo (gh_turbo_bf16) | 31 | 3 | 0 | 2 | 36 |
+| x (gh_x_bf16) | 34 | 0 | 0 | 2 | 36 |
+| tuned (gh_tuned_bf16) | 28 | 1 | 0 | 7 | 36 |
+
+If the "other (mild)" cases were counted clean instead (to show how much the borderline calls matter; the faint ghost 07_seed2102 in turbo is kept as ghost):
+turbo 33 clean / 3 ghost / 0 mushy / 0 other; x 35 / 0 / 0 / 1; tuned 29 / 1 / 0 / 6.
+
+## Per prompt id (clean / ghost / mushy / other; 6 images per cell)
+
+| prompt id | turbo | x | tuned |
+|---|---|---|---|
+| 06 monk | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 |
+| 07 curly hair, rainy window | 2/3/0/1 | 6/0/0/0 | 4/1/0/1 |
+| 08 anime boy | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 |
+| 09 pottery studio | 6/0/0/0 | 6/0/0/0 | 6/0/0/0 |
+| 17 umbrella street | 5/0/0/1 | 6/0/0/0 | 6/0/0/0 |
+| 22 white kitchen | 6/0/0/0 | 4/0/0/2 | 0/0/0/6 |
+| total | 31/3/0/2 | 34/0/0/2 | 28/1/0/7 |
+
+Ghost-face count per prompt id (face ghosts only): 07 has 3 / 0 / 1 for turbo / x / tuned; every other prompt id has 0 in all three variants.
+
+## Flagged images
+
+| image | class | note |
+|---|---|---|
+| gh_turbo_bf16/07_seed2102.png | ghost (faint) | second faint mouth/lips beside the face, behind the hair on the left |
+| gh_turbo_bf16/07_seed2202.png | ghost | faint second face (eye, cheek) to the left of the real face, behind the glass |
+| gh_turbo_bf16/07_seed2402.png | ghost | second faint face (eye and mouth) in the window area left of the real face |
+| gh_turbo_bf16/07_seed2502.png | other (mild) | cyan translucent smear across cheek/hair at left of the face |
+| gh_turbo_bf16/17_seed5202.png | other (mild) | second translucent umbrella overlapping behind the real umbrella |
+| gh_x_bf16/22_seed5207.png | other | table legs washed out / see-through, almost white |
+| gh_x_bf16/22_seed5507.png | other (mild) | faint pale vertical ghost legs behind the table |
+| gh_tuned_bf16/07_seed2002.png | ghost | second blurred face (nose, eye, mouth) double-exposed over the real face |
+| gh_tuned_bf16/07_seed2102.png | other (mild, borderline) | hard-edged translucent teal patch over the left eye/brow; no second face |
+| gh_tuned_bf16/22_seed5007.png | other | translucent extra chair and pale ghost legs under/left of the table |
+| gh_tuned_bf16/22_seed5107.png | other | translucent ghost chair floating under the table |
+| gh_tuned_bf16/22_seed5207.png | other | see-through doubled table legs plus a ghost chair |
+| gh_tuned_bf16/22_seed5307.png | other | chair back/seat see-through where it overlaps the table |
+| gh_tuned_bf16/22_seed5407.png | other | heavily doubled translucent chairs and legs |
+| gh_tuned_bf16/22_seed5507.png | other | several translucent table legs and chair, strong |
+
+No image was classed as mushy.
+
+Borderline cases counted as clean (not in the tables above):
+- turbo / 22_seed5107: translucent plaid towel on the chair back; turbo / 22_seed5307: table edge slightly see-through over the chair seat (zoom only).
+- x / 22_seed5107: small semi-transparent patches under the table edge (zoom only).
+- x and tuned / 07_seed2202: faint beige smudge above the left eyebrow; faces soft but features defined.
+- x / 07_seed2102: faint window-haze veil on the face, no second face.
+- x and tuned / 08 seeds 2003, 2103, 2203, 2303, 2403: semi-transparent scarf tails and layered scarf wisps; turbo had opaque scarves in those seeds. Faces clean in all three variants.
+
+## Observations
+
+- Face ghosting appears in prompt 07 only (woman behind a rainy window): turbo 3 of 6, tuned 1 of 6, x 0 of 6. The faces in 06, 08, 09 and 17 were clean in all 108 images, and no image was mushy. The turbo ghosts are faint second faces/mouths sitting in the window area beside the real face; the tuned ghost (seed 2002) is a stronger double exposure on the face itself.
+- The ghosts are seed-specific: turbo is ghosted on seeds 2102, 2202, 2402 (and has a mild cyan smear on 2502), tuned on seed 2002, which is clean in the other two variants. Only seed 2302 was clean for all three variants. Seed 2202 is ghosted in turbo and in part 1's i4e8, and 2002 is ghosted in tuned and in part 1's i4e8.
+- The kitchen prompt 22 separates the variants on furniture, not faces: tuned has translucent/ghost furniture in 6 of 6 seeds, x in 2 of 6 (5207 clear, 5507 mild), turbo in 0 of 6. Seed 5207 (see-through table legs) is flagged in x and tuned, but not in turbo.
+- Overall flagged images: tuned 8 (1 ghost + 7 other), turbo 5, x 2. With 36 images per variant, a single judge and several mild calls, the turbo-vs-x-vs-tuned difference in face ghosts (3 / 0 / 1) is within noise; the tuned kitchen result (6 of 6) is larger than the 1-3 image band.
+
+## Comparison with ghost_rate_part1.md
+
+Part 1 judged four variants that all use the quantized generator tmixG_x (the same as `gh_x_bf16` here): bf16 35/0/0/1, int8 35/0/0/1, gptq 35/0/0/1, i4e8 29/2/0/5 (clean/ghost/mushy/other).
+
+- `gh_x_bf16` is the shared reference. Part 1 gave it 35/0/0/1 (22_seed5207 only); I give 34/0/0/2: the same 22_seed5207 plus 22_seed5507, which part 1 listed as a borderline clean ("faint pale far table legs"). Both judgements agree on no face ghosts and no mushy in this folder.
+- Face ghosts: part 1 found them only in i4e8 (2, both prompt 07). Here turbo has 3 and tuned has 1, also only in prompt 07. So the x-based bf16/int8/gptq variants (0 ghost) and this x_bf16 sit at 0, while turbo (3), i4e8 (2) and tuned (1) show a few; at 36 images each, 0-3 face ghosts is a rough signal at best.
+- Furniture "other" in the kitchen: part 1 had 1 of 6 for bf16/int8/gptq and 4 of 6 for i4e8; here turbo has 0 of 6, x 2 of 6 (1 clear), tuned 6 of 6. Seed 5207 is see-through in all four part-1 variants and in tuned here, but not in turbo.
+- Same-seed overlap in prompt 07 ghosts: seed 2202 (turbo here, i4e8 in part 1) and seed 2002 (tuned here, i4e8 in part 1); seed 2102 is flagged in turbo (faint ghost), tuned (mild patch) and in part 1 only as a haze veil.
+- Caveats for the comparison: different judging passes (part 1 used 2x2 composites of four variants, this part 3-up composites), borderline calls were not identical (e.g. 22_seed5507, 07_seed2102), and both are single-judge counts with noise of a few images.

eval/hps3_enc_baseline.json

@@ -0,0 +1,45 @@
+{
+  "enc_int8_convrot": {
+    "n": 24,
+    "mean_mu": 10.945838987827301,
+    "sd_mu": 1.6878424869155904,
+    "ci95": [
+      10.261139473319053,
+      11.598907498021921
+    ],
+    "category_mean_mu": {
+      "text": 12.098817825317383,
+      "portrait": 11.843674421310425,
+      "detail": 9.474143862724304,
+      "counting": 10.993349552154541,
+      "varied": 10.356157064437866
+    },
+    "mu_by_prompt": {
+      "0": 11.561762809753418,
+      "1": 10.07781982421875,
+      "2": 13.57741928100586,
+      "3": 12.749553680419922,
+      "4": 11.740704536437988,
+      "5": 12.88564682006836,
+      "6": 11.89348030090332,
+      "7": 11.907999038696289,
+      "8": 11.463834762573242,
+      "9": 12.109383583068848,
+      "10": 8.401968002319336,
+      "11": 12.902873039245605,
+      "12": 9.782557487487793,
+      "13": 6.809176921844482,
+      "14": 10.684173583984375,
+      "15": 11.302525520324707,
+      "16": 10.40369701385498,
+      "17": 13.471558570861816,
+      "18": 9.411077499389648,
+      "19": 8.697176933288574,
+      "20": 10.233926773071289,
+      "21": 10.562956809997559,
+      "22": 10.744577407836914,
+      "23": 9.324285507202148
+    },
+    "size_gb": null
+  }
+}

eval/hps3_enc_ptq.json

@@ -0,0 +1,131 @@
+{
+  "enc_i4": {
+    "n": 24,
+    "mean_mu": 11.034659087657928,
+    "sd_mu": 1.8064966564013927,
+    "ci95": [
+      10.30820912073056,
+      11.735686944921811
+    ],
+    "category_mean_mu": {
+      "text": 12.530771573384603,
+      "portrait": 11.501980543136597,
+      "detail": 9.691685795783997,
+      "counting": 11.013746738433838,
+      "varied": 10.355628728866577
+    },
+    "mu_by_prompt": {
+      "0": 12.212133407592773,
+      "1": 10.685227394104004,
+      "2": 14.403738021850586,
+      "3": 12.342391014099121,
+      "4": 11.99782657623291,
+      "5": 13.543313026428223,
+      "6": 11.655359268188477,
+      "7": 11.481369972229004,
+      "8": 10.31234073638916,
+      "9": 12.558852195739746,
+      "10": 8.624855041503906,
+      "11": 13.177989959716797,
+      "12": 10.35336685180664,
+      "13": 6.610531330108643,
+      "14": 10.301910400390625,
+      "15": 11.72558307647705,
+      "16": 10.535961151123047,
+      "17": 12.761947631835938,
+      "18": 9.778426170349121,
+      "19": 8.138744354248047,
+      "20": 10.932673454284668,
+      "21": 10.173169136047363,
+      "22": 11.53548812866211,
+      "23": 8.988619804382324
+    },
+    "size_gb": 5.61
+  },
+  "enc_m3": {
+    "n": 24,
+    "mean_mu": 10.635515789190928,
+    "sd_mu": 1.8093861358357015,
+    "ci95": [
+      9.91959085514148,
+      11.339807620147864
+    ],
+    "category_mean_mu": {
+      "text": 11.88214635848999,
+      "portrait": 11.379156112670898,
+      "detail": 9.323559641838074,
+      "counting": 10.494130611419678,
+      "varied": 10.020047068595886
+    },
+    "mu_by_prompt": {
+      "0": 13.073763847351074,
+      "1": 8.502704620361328,
+      "2": 13.960399627685547,
+      "3": 11.80362319946289,
+      "4": 11.661916732788086,
+      "5": 12.290470123291016,
+      "6": 11.188579559326172,
+      "7": 11.254002571105957,
+      "8": 10.09174919128418,
+      "9": 12.982293128967285,
+      "10": 7.603389263153076,
+      "11": 13.012289047241211,
+      "12": 10.081485748291016,
+      "13": 6.597074508666992,
+      "14": 9.626181602478027,
+      "15": 11.362079620361328,
+      "16": 10.498995780944824,
+      "17": 11.699702262878418,
+      "18": 10.247276306152344,
+      "19": 8.36837100982666,
+      "20": 9.855698585510254,
+      "21": 9.963851928710938,
+      "22": 9.955434799194336,
+      "23": 9.571045875549316
+    },
+    "size_gb": 5.01
+  },
+  "enc_i3": {
+    "n": 24,
+    "mean_mu": 4.856175818790992,
+    "sd_mu": 3.3988304751516716,
+    "ci95": [
+      3.5031733943149446,
+      6.1502091315885385
+    ],
+    "category_mean_mu": {
+      "text": 5.54188072681427,
+      "portrait": 7.1700427532196045,
+      "detail": 2.088006906211376,
+      "counting": 5.593181848526001,
+      "varied": 4.384796619415283
+    },
+    "mu_by_prompt": {
+      "0": -1.0470497608184814,
+      "1": 7.336244583129883,
+      "2": 7.800258159637451,
+      "3": 5.490494728088379,
+      "4": 7.72861385345459,
+      "5": 5.942722797393799,
+      "6": 6.834472179412842,
+      "7": 7.286423683166504,
+      "8": 10.505233764648438,
+      "9": 4.054041385650635,
+      "10": -3.5405421257019043,
+      "11": 9.378467559814453,
+      "12": -0.1173417866230011,
+      "13": 2.631443977355957,
+      "14": 4.578416347503662,
+      "15": 6.60794734954834,
+      "16": 8.415070533752441,
+      "17": 2.7991318702697754,
+      "18": 4.980869770050049,
+      "19": 2.071200132369995,
+      "20": 6.5670952796936035,
+      "21": 5.090610027313232,
+      "22": 1.5781588554382324,
+      "23": 3.5762364864349365
+    },
+    "size_gb": 4.75
+  }
+}

eval/hps3_enc_ptq.md

@@ -0,0 +1,12 @@
+| tag | size GB | mean mu [bootstrap 95%] | text | portrait | detail | counting | varied | diff to teacher +- SE (wins/24) |
+|---|---|---|---|---|---|---|---|---|
+| enc_i4 | 5.61 | 11.03 [10.31, 11.74] | 12.53 | 11.50 | 9.69 | 11.01 | 10.36 | - |
+| enc_m3 | 5.01 | 10.64 [9.92, 11.34] | 11.88 | 11.38 | 9.32 | 10.49 | 10.02 | - |
+| enc_i3 | 4.75 | 4.86 [3.50, 6.15] | 5.54 | 7.17 | 2.09 | 5.59 | 4.38 | - |
+
+mu = HPSv3 raw score (higher = preferred), mean over 24 prompts, one seed per prompt, 768px images scored at ~448px (HPSv3 input).
+diff to teacher = mean of (student mu - teacher mu) per prompt, SE = sd of the 24 paired differences / sqrt(24); wins = prompts where student mu > teacher mu.
+
+Noise: only 24 prompts and ONE seed per prompt, so seed-to-seed variation of a model is not measured at all and the intervals only reflect which 24 prompts were drawn.
+- sd of mu across prompts is 1.81-3.40; bootstrap 95% half-width of a mean is 0.71-1.32.
+- paired SE between two candidate models (excluding teacher and tern_v1) is 0.14-0.67 (median 0.65); a difference below about 2 x that (1.30 mu) cannot be told apart from prompt-sampling noise.

eval/hps3_enc_ptq2.json

@@ -0,0 +1,131 @@
+{
+  "enc_i4e8": {
+    "n": 24,
+    "mean_mu": 11.091991861661276,
+    "sd_mu": 1.7556452830536504,
+    "ci95": [
+      10.388231111566226,
+      11.772854788104693
+    ],
+    "category_mean_mu": {
+      "text": 12.595286051432291,
+      "portrait": 11.365147113800049,
+      "detail": 9.674987316131592,
+      "counting": 11.102847576141357,
+      "varied": 10.533731937408447
+    },
+    "mu_by_prompt": {
+      "0": 12.308874130249023,
+      "1": 10.906404495239258,
+      "2": 14.414641380310059,
+      "3": 12.643335342407227,
+      "4": 11.775763511657715,
+      "5": 13.522697448730469,
+      "6": 11.209823608398438,
+      "7": 10.562433242797852,
+      "8": 11.092175483703613,
+      "9": 12.596156120300293,
+      "10": 8.60431957244873,
+      "11": 12.998363494873047,
+      "12": 10.34162425994873,
+      "13": 6.755641937255859,
+      "14": 10.533465385437012,
+      "15": 11.672229766845703,
+      "16": 10.680486679077148,
+      "17": 12.985466957092285,
+      "18": 10.035730361938477,
+      "19": 8.486477851867676,
+      "20": 10.956826210021973,
+      "21": 10.306949615478516,
+      "22": 11.811293601989746,
+      "23": 9.006624221801758
+    },
+    "size_gb": 4.6
+  },
+  "enc_l3": {
+    "n": 24,
+    "mean_mu": 9.768357813358307,
+    "sd_mu": 1.3258938818064758,
+    "ci95": [
+      9.239143864810467,
+      10.278334188461303
+    ],
+    "category_mean_mu": {
+      "text": 10.216746012369791,
+      "portrait": 10.868505001068115,
+      "detail": 8.685726881027222,
+      "counting": 10.24935531616211,
+      "varied": 9.303059160709381
+    },
+    "mu_by_prompt": {
+      "0": 10.018077850341797,
+      "1": 9.047893524169922,
+      "2": 10.448965072631836,
+      "3": 11.477543830871582,
+      "4": 10.774165153503418,
+      "5": 9.533830642700195,
+      "6": 10.359161376953125,
+      "7": 11.376570701599121,
+      "8": 10.234749794006348,
+      "9": 11.503538131713867,
+      "10": 7.847609519958496,
+      "11": 11.483872413635254,
+      "12": 8.430160522460938,
+      "13": 6.981265068054199,
+      "14": 9.427104949951172,
+      "15": 11.071605682373047,
+      "16": 10.225643157958984,
+      "17": 10.968631744384766,
+      "18": 8.514876365661621,
+      "19": 7.952888488769531,
+      "20": 9.436090469360352,
+      "21": 9.479061126708984,
+      "22": 10.167335510253906,
+      "23": 7.679946422576904
+    },
+    "size_gb": 4.4
+  },
+  "enc_a3": {
+    "n": 24,
+    "mean_mu": 11.020188371340433,
+    "sd_mu": 1.8679766509915257,
+    "ci95": [
+      10.28298435707887,
+      11.752896255254745
+    ],
+    "category_mean_mu": {
+      "text": 12.489479700724283,
+      "portrait": 11.711003541946411,
+      "detail": 9.484590530395508,
+      "counting": 10.592373847961426,
+      "varied": 10.447564840316772
+    },
+    "mu_by_prompt": {
+      "0": 12.082544326782227,
+      "1": 9.998886108398438,
+      "2": 14.684210777282715,
+      "3": 12.428792953491211,
+      "4": 12.565485000610352,
+      "5": 13.176959037780762,
+      "6": 11.885496139526367,
+      "7": 11.376730918884277,
+      "8": 11.167491912841797,
+      "9": 12.414295196533203,
+      "10": 8.093016624450684,
+      "11": 13.321942329406738,
+      "12": 10.130970001220703,
+      "13": 6.392433166503906,
+      "14": 10.154117584228516,
+      "15": 11.030630111694336,
+      "16": 9.916529655456543,
+      "17": 13.19386100769043,
+      "18": 9.735214233398438,
+      "19": 8.763481140136719,
+      "20": 11.289271354675293,
+      "21": 10.299274444580078,
+      "22": 11.002974510192871,
+      "23": 9.379912376403809
+    },
+    "size_gb": 4.54
+  }
+}

eval/hps3_enc_ptq2.md

@@ -0,0 +1,12 @@
+| tag | size GB | mean mu [bootstrap 95%] | text | portrait | detail | counting | varied | diff to teacher +- SE (wins/24) |
+|---|---|---|---|---|---|---|---|---|
+| enc_i4e8 | 4.6 | 11.09 [10.39, 11.77] | 12.60 | 11.37 | 9.67 | 11.10 | 10.53 | - |
+| enc_l3 | 4.4 | 9.77 [9.24, 10.28] | 10.22 | 10.87 | 8.69 | 10.25 | 9.30 | - |
+| enc_a3 | 4.54 | 11.02 [10.28, 11.75] | 12.49 | 11.71 | 9.48 | 10.59 | 10.45 | - |
+
+mu = HPSv3 raw score (higher = preferred), mean over 24 prompts, one seed per prompt, 768px images scored at ~448px (HPSv3 input).
+diff to teacher = mean of (student mu - teacher mu) per prompt, SE = sd of the 24 paired differences / sqrt(24); wins = prompts where student mu > teacher mu.
+
+Noise: only 24 prompts and ONE seed per prompt, so seed-to-seed variation of a model is not measured at all and the intervals only reflect which 24 prompts were drawn.
+- sd of mu across prompts is 1.33-1.87; bootstrap 95% half-width of a mean is 0.52-0.73.
+- paired SE between two candidate models (excluding teacher and tern_v1) is 0.10-0.23 (median 0.22); a difference below about 2 x that (0.44 mu) cannot be told apart from prompt-sampling noise.

eval/hps3_enc_w4a8_gptq.json

@@ -0,0 +1,45 @@
+{
+  "enc_w4a8_gptq": {
+    "n": 24,
+    "mean_mu": 10.98950481414795,
+    "sd_mu": 1.8582404056006003,
+    "ci95": [
+      10.239382770657539,
+      11.696771595875422
+    ],
+    "category_mean_mu": {
+      "text": 12.021718660990397,
+      "portrait": 11.957259893417358,
+      "detail": 9.272223472595215,
+      "counting": 10.894657135009766,
+      "varied": 10.613819479942322
+    },
+    "mu_by_prompt": {
+      "0": 10.913015365600586,
+      "1": 9.044050216674805,
+      "2": 13.881145477294922,
+      "3": 13.00432300567627,
+      "4": 12.235599517822266,
+      "5": 13.052178382873535,
+      "6": 11.801921844482422,
+      "7": 12.187568664550781,
+      "8": 11.914678573608398,
+      "9": 11.924870491027832,
+      "10": 8.108549118041992,
+      "11": 12.995185852050781,
+      "12": 9.44250774383545,
+      "13": 6.542651176452637,
+      "14": 10.373090744018555,
+      "15": 11.416223526000977,
+      "16": 10.093405723571777,
+      "17": 13.795523643493652,
+      "18": 9.823501586914062,
+      "19": 8.755511283874512,
+      "20": 11.18978214263916,
+      "21": 10.255514144897461,
+      "22": 11.602023124694824,
+      "23": 9.395294189453125
+    },
+    "size_gb": null
+  }
+}

eval/hps3_nd_gptq.json

@@ -0,0 +1,174 @@
+{
+  "mixG_tuned": {
+    "n": 24,
+    "mean_mu": 12.139749030272165,
+    "sd_mu": 1.9395526891947201,
+    "ci95": [
+      11.361157366633414,
+      12.872107101480166
+    ],
+    "category_mean_mu": {
+      "text": 13.390686511993408,
+      "portrait": 12.87514042854309,
+      "detail": 10.46267569065094,
+      "counting": 11.710683822631836,
+      "varied": 11.779653191566467
+    },
+    "mu_by_prompt": {
+      "0": 13.417725563049316,
+      "1": 12.467721939086914,
+      "2": 13.885729789733887,
+      "3": 11.765070915222168,
+      "4": 14.611907958984375,
+      "5": 14.195962905883789,
+      "6": 12.4324951171875,
+      "7": 12.14983081817627,
+      "8": 11.897542953491211,
+      "9": 15.020692825317383,
+      "10": 12.184309005737305,
+      "11": 14.310564041137695,
+      "12": 7.555306434631348,
+      "13": 7.800523281097412,
+      "14": 11.379426002502441,
+      "15": 12.04194164276123,
+      "16": 10.702643394470215,
+      "17": 13.159246444702148,
+      "18": 10.658286094665527,
+      "19": 10.344663619995117,
+      "20": 11.928624153137207,
+      "21": 11.974713325500488,
+      "22": 10.689987182617188,
+      "23": 14.779061317443848
+    },
+    "size_gb": 2.14
+  },
+  "nd_mixG_tuned_gptq": {
+    "n": 24,
+    "mean_mu": 12.163939317067465,
+    "sd_mu": 1.8067306763097963,
+    "ci95": [
+      11.441241880257925,
+      12.849267401297887
+    ],
+    "category_mean_mu": {
+      "text": 13.353586037953695,
+      "portrait": 13.092275142669678,
+      "detail": 10.582758665084839,
+      "counting": 11.811241149902344,
+      "varied": 11.686301231384277
+    },
+    "mu_by_prompt": {
+      "0": 13.440675735473633,
+      "1": 11.957268714904785,
+      "2": 13.895618438720703,
+      "3": 12.55988883972168,
+      "4": 14.599224090576172,
+      "5": 13.668840408325195,
+      "6": 12.092365264892578,
+      "7": 13.142611503601074,
+      "8": 12.048057556152344,
+      "9": 15.086066246032715,
+      "10": 12.212583541870117,
+      "11": 14.199559211730957,
+      "12": 8.30803394317627,
+      "13": 7.610857963562012,
+      "14": 11.580402374267578,
+      "15": 12.04207992553711,
+      "16": 10.661609649658203,
+      "17": 12.946452140808105,
+      "18": 11.319280624389648,
+      "19": 10.453644752502441,
+      "20": 11.457819938659668,
+      "21": 12.063944816589355,
+      "22": 10.652288436889648,
+      "23": 13.935369491577148
+    },
+    "size_gb": null
+  },
+  "mixH2_x_f8": {
+    "n": 24,
+    "mean_mu": 11.989253520965576,
+    "sd_mu": 1.5468661955186895,
+    "ci95": [
+      11.359737481673559,
+      12.58515132466952
+    ],
+    "category_mean_mu": {
+      "text": 12.905888080596924,
+      "portrait": 12.12691354751587,
+      "detail": 11.099243640899658,
+      "counting": 12.004260063171387,
+      "varied": 11.674200892448425
+    },
+    "mu_by_prompt": {
+      "0": 13.653209686279297,
+      "1": 11.297388076782227,
+      "2": 12.705009460449219,
+      "3": 11.617015838623047,
+      "4": 13.824302673339844,
+      "5": 14.33840274810791,
+      "6": 11.460399627685547,
+      "7": 12.014626502990723,
+      "8": 10.534360885620117,
+      "9": 14.49826717376709,
+      "10": 11.64794921875,
+      "11": 13.963139533996582,
+      "12": 11.265219688415527,
+      "13": 7.520666122436523,
+      "14": 11.920021057128906,
+      "15": 12.088499069213867,
+      "16": 11.032934188842773,
+      "17": 13.643148422241211,
+      "18": 11.172962188720703,
+      "19": 10.405658721923828,
+      "20": 11.057677268981934,
+      "21": 12.540094375610352,
+      "22": 11.082413673400879,
+      "23": 12.458718299865723
+    },
+    "size_gb": 1.85
+  },
+  "nd_mixH2_x_f8_gptq": {
+    "n": 24,
+    "mean_mu": 12.035317579905191,
+    "sd_mu": 1.5277030513915637,
+    "ci95": [
+      11.423713621497154,
+      12.625486142436662
+    ],
+    "category_mean_mu": {
+      "text": 12.712045351664225,
+      "portrait": 12.027806520462036,
+      "detail": 11.226006746292114,
+      "counting": 12.206240177154541,
+      "varied": 11.893452048301697
+    },
+    "mu_by_prompt": {
+      "0": 13.797195434570312,
+      "1": 11.58507251739502,
+      "2": 12.74012565612793,
+      "3": 11.010663986206055,
+      "4": 13.852313995361328,
+      "5": 13.286900520324707,
+      "6": 11.62837028503418,
+      "7": 11.955574035644531,
+      "8": 9.93774127960205,
+      "9": 14.589540481567383,
+      "10": 11.754337310791016,
+      "11": 14.261083602905273,
+      "12": 10.913103103637695,
+      "13": 7.975502967834473,
+      "14": 12.154131889343262,
+      "15": 12.25834846496582,
+      "16": 10.834696769714355,
+      "17": 13.356339454650879,
+      "18": 11.414605140686035,
+      "19": 10.90980339050293,
+      "20": 11.242483139038086,
+      "21": 12.390066146850586,
+      "22": 11.085796356201172,
+      "23": 13.913825988769531
+    },
+    "size_gb": null
+  }
+}

eval/hps3_nd_gptq.md

@@ -0,0 +1,13 @@
+| tag | size GB | mean mu [bootstrap 95%] | text | portrait | detail | counting | varied | diff to teacher +- SE (wins/24) |
+|---|---|---|---|---|---|---|---|---|
+| mixG_tuned | 2.14 | 12.14 [11.36, 12.87] | 13.39 | 12.88 | 10.46 | 11.71 | 11.78 | - |
+| nd_mixG_tuned_gptq | - | 12.16 [11.44, 12.85] | 13.35 | 13.09 | 10.58 | 11.81 | 11.69 | - |
+| mixH2_x_f8 | 1.85 | 11.99 [11.36, 12.59] | 12.91 | 12.13 | 11.10 | 12.00 | 11.67 | - |
+| nd_mixH2_x_f8_gptq | - | 12.04 [11.42, 12.63] | 12.71 | 12.03 | 11.23 | 12.21 | 11.89 | - |
+
+mu = HPSv3 raw score (higher = preferred), mean over 24 prompts, one seed per prompt, 768px images scored at ~448px (HPSv3 input).
+diff to teacher = mean of (student mu - teacher mu) per prompt, SE = sd of the 24 paired differences / sqrt(24); wins = prompts where student mu > teacher mu.
+
+Noise: only 24 prompts and ONE seed per prompt, so seed-to-seed variation of a model is not measured at all and the intervals only reflect which 24 prompts were drawn.
+- sd of mu across prompts is 1.53-1.94; bootstrap 95% half-width of a mean is 0.60-0.76.
+- paired SE between two candidate models (excluding teacher and tern_v1) is 0.09-0.22 (median 0.19); a difference below about 2 x that (0.38 mu) cannot be told apart from prompt-sampling noise.

eval/hps3_summary.json

@@ -0,0 +1,731 @@
+{
+  "teacher_bf16": {
+    "n": 24,
+    "mean_mu": 12.384581685066223,
+    "sd_mu": 1.6123343047925531,
+    "ci95": [
+      11.755278043945632,
+      13.006242449084917
+    ],
+    "category_mean_mu": {
+      "text": 13.332724253336588,
+      "portrait": 13.072895526885986,
+      "detail": 12.40628981590271,
+      "counting": 10.933318138122559,
+      "varied": 11.68127965927124
+    },
+    "mu_by_prompt": {
+      "0": 13.639022827148438,
+      "1": 11.825362205505371,
+      "2": 12.995671272277832,
+      "3": 12.526433944702148,
+      "4": 14.530313491821289,
+      "5": 14.479541778564453,
+      "6": 12.07623291015625,
+      "7": 13.430919647216797,
+      "8": 12.091690063476562,
+      "9": 14.692739486694336,
+      "10": 13.62648868560791,
+      "11": 14.686071395874023,
+      "12": 12.613874435424805,
+      "13": 8.698724746704102,
+      "14": 9.724932670593262,
+      "15": 12.141703605651855,
+      "16": 10.176751136779785,
+      "17": 13.937211036682129,
+      "18": 10.837376594543457,
+      "19": 11.01986312866211,
+      "20": 11.772106170654297,
+      "21": 12.926685333251953,
+      "22": 11.148569107055664,
+      "23": 11.631674766540527
+    },
+    "size_gb": 10.16
+  },
+  "mixA_x_tuned": {
+    "n": 24,
+    "mean_mu": 11.610886931419373,
+    "sd_mu": 1.7820989888716794,
+    "ci95": [
+      10.906349510947862,
+      12.311402688423792
+    ],
+    "category_mean_mu": {
+      "text": 11.760952949523926,
+      "portrait": 13.03148365020752,
+      "detail": 10.76393747329712,
+      "counting": 11.648460388183594,
+      "varied": 11.202120423316956
+    },
+    "mu_by_prompt": {
+      "0": 14.09498405456543,
+      "1": 11.100715637207031,
+      "2": 11.812602996826172,
+      "3": 10.70802116394043,
+      "4": 12.798552513122559,
+      "5": 10.050841331481934,
+      "6": 12.894301414489746,
+      "7": 13.858095169067383,
+      "8": 10.505816459655762,
+      "9": 14.867721557617188,
+      "10": 11.039589881896973,
+      "11": 14.058591842651367,
+      "12": 10.705621719360352,
+      "13": 7.251946449279785,
+      "14": 11.508694648742676,
+      "15": 11.788226127624512,
+      "16": 10.323859214782715,
+      "17": 13.585147857666016,
+      "18": 10.774235725402832,
+      "19": 9.016969680786133,
+      "20": 11.208295822143555,
+      "21": 11.186261177062988,
+      "22": 10.375581741333008,
+      "23": 13.146612167358398
+    },
+    "diff_to_teacher": {
+      "mean": -0.7736947536468506,
+      "se": 0.28107680515481503,
+      "sd": 1.3769895023219707,
+      "wins": 7
+    },
+    "size_gb": 1.67
+  },
+  "mixF_ptq": {
+    "n": 24,
+    "mean_mu": 10.857280095418295,
+    "sd_mu": 1.816638553281807,
+    "ci95": [
+      10.164406632383663,
+      11.594348962108294
+    ],
+    "category_mean_mu": {
+      "text": 12.170512040456137,
+      "portrait": 11.56724238395691,
+      "detail": 10.674527406692505,
+      "counting": 10.113127708435059,
+      "varied": 9.794789433479309
+    },
+    "mu_by_prompt": {
+      "0": 12.231085777282715,
+      "1": 12.160717964172363,
+      "2": 13.37264347076416,
+      "3": 11.19559097290039,
+      "4": 11.993592262268066,
+      "5": 12.069441795349121,
+      "6": 10.759611129760742,
+      "7": 10.656207084655762,
+      "8": 9.303426742553711,
+      "9": 15.549724578857422,
+      "10": 12.295470237731934,
+      "11": 13.153759002685547,
+      "12": 10.473901748657227,
+      "13": 6.7749786376953125,
+      "14": 10.282621383666992,
+      "15": 9.943634033203125,
+      "16": 8.922804832458496,
+      "17": 10.780318260192871,
+      "18": 9.721498489379883,
+      "19": 8.549283027648926,
+      "20": 10.071290016174316,
+      "21": 10.076916694641113,
+      "22": 10.67863655090332,
+      "23": 9.557567596435547
+    },
+    "diff_to_teacher": {
+      "mean": -1.527301589647929,
+      "se": 0.23380791654539382,
+      "sd": 1.145420186718895,
+      "wins": 4
+    },
+    "size_gb": 2.42
+  },
+  "mixG_x": {
+    "n": 24,
+    "mean_mu": 12.08710265159607,
+    "sd_mu": 1.7183072395722299,
+    "ci95": [
+      11.39186682899793,
+      12.736877583463986
+    ],
+    "category_mean_mu": {
+      "text": 13.42167854309082,
+      "portrait": 12.39479398727417,
+      "detail": 10.94544506072998,
+      "counting": 11.734143733978271,
+      "varied": 11.59139358997345
+    },
+    "mu_by_prompt": {
+      "0": 13.307634353637695,
+      "1": 11.922536849975586,
+      "2": 13.955156326293945,
+      "3": 12.162612915039062,
+      "4": 15.23203182220459,
+      "5": 13.950098991394043,
+      "6": 11.93759536743164,
+      "7": 12.26282787322998,
+      "8": 11.281140327453613,
+      "9": 14.097612380981445,
+      "10": 11.678287506103516,
+      "11": 14.476302146911621,
+      "12": 10.909077644348145,
+      "13": 6.718112945556641,
+      "14": 11.539464950561523,
+      "15": 11.92882251739502,
+      "16": 11.382481575012207,
+      "17": 13.015539169311523,
+      "18": 11.165454864501953,
+      "19": 10.521598815917969,
+      "20": 11.441516876220703,
+      "21": 12.8370361328125,
+      "22": 10.439844131469727,
+      "23": 11.927677154541016
+    },
+    "diff_to_teacher": {
+      "mean": -0.2974790334701538,
+      "se": 0.18837994685499737,
+      "sd": 0.9228694951347124,
+      "wins": 7
+    },
+    "size_gb": 2.14
+  },
+  "mixG_tuned": {
+    "n": 24,
+    "mean_mu": 12.139749030272165,
+    "sd_mu": 1.9395526891947201,
+    "ci95": [
+      11.361157366633414,
+      12.872107101480166
+    ],
+    "category_mean_mu": {
+      "text": 13.390686511993408,
+      "portrait": 12.87514042854309,
+      "detail": 10.46267569065094,
+      "counting": 11.710683822631836,
+      "varied": 11.779653191566467
+    },
+    "mu_by_prompt": {
+      "0": 13.417725563049316,
+      "1": 12.467721939086914,
+      "2": 13.885729789733887,
+      "3": 11.765070915222168,
+      "4": 14.611907958984375,
+      "5": 14.195962905883789,
+      "6": 12.4324951171875,
+      "7": 12.14983081817627,
+      "8": 11.897542953491211,
+      "9": 15.020692825317383,
+      "10": 12.184309005737305,
+      "11": 14.310564041137695,
+      "12": 7.555306434631348,
+      "13": 7.800523281097412,
+      "14": 11.379426002502441,
+      "15": 12.04194164276123,
+      "16": 10.702643394470215,
+      "17": 13.159246444702148,
+      "18": 10.658286094665527,
+      "19": 10.344663619995117,
+      "20": 11.928624153137207,
+      "21": 11.974713325500488,
+      "22": 10.689987182617188,
+      "23": 14.779061317443848
+    },
+    "diff_to_teacher": {
+      "mean": -0.24483265479405722,
+      "se": 0.28922744505696935,
+      "sd": 1.416919319996863,
+      "wins": 9
+    },
+    "size_gb": 2.14
+  },
+  "mixH_x": {
+    "n": 24,
+    "mean_mu": 11.880460023880005,
+    "sd_mu": 1.6990035552645357,
+    "ci95": [
+      11.192399097482364,
+      12.525516930222512
+    ],
+    "category_mean_mu": {
+      "text": 13.041773160298666,
+      "portrait": 12.490415334701538,
+      "detail": 10.68679404258728,
+      "counting": 11.49379014968872,
+      "varied": 11.397997975349426
+    },
+    "mu_by_prompt": {
+      "0": 13.99026870727539,
+      "1": 10.050540924072266,
+      "2": 13.606485366821289,
+      "3": 11.855009078979492,
+      "4": 14.160813331604004,
+      "5": 14.58752155303955,
+      "6": 11.589570999145508,
+      "7": 13.778044700622559,
+      "8": 10.534186363220215,
+      "9": 14.059859275817871,
+      "10": 11.494105339050293,
+      "11": 13.14444351196289,
+      "12": 10.829146385192871,
+      "13": 7.279480934143066,
+      "14": 11.513114929199219,
+      "15": 11.474465370178223,
+      "16": 10.609831809997559,
+      "17": 12.428728103637695,
+      "18": 10.733562469482422,
+      "19": 10.845394134521484,
+      "20": 10.504751205444336,
+      "21": 11.515116691589355,
+      "22": 11.304014205932617,
+      "23": 13.242585182189941
+    },
+    "diff_to_teacher": {
+      "mean": -0.5041216611862183,
+      "se": 0.2162678089390539,
+      "sd": 1.0594915593808092,
+      "wins": 8
+    },
+    "size_gb": 1.95
+  },
+  "mixH_tuned": {
+    "n": 24,
+    "mean_mu": 11.699441293875376,
+    "sd_mu": 2.029649702715248,
+    "ci95": [
+      10.869073146084943,
+      12.448687643309436
+    ],
+    "category_mean_mu": {
+      "text": 12.943698406219482,
+      "portrait": 12.36544394493103,
+      "detail": 10.056230425834656,
+      "counting": 11.240527629852295,
+      "varied": 11.3695809841156
+    },
+    "mu_by_prompt": {
+      "0": 14.136148452758789,
+      "1": 11.969319343566895,
+      "2": 12.627728462219238,
+      "3": 11.427092552185059,
+      "4": 14.123497009277344,
+      "5": 13.37840461730957,
+      "6": 12.329015731811523,
+      "7": 11.961216926574707,
+      "8": 10.635523796081543,
+      "9": 14.536019325256348,
+      "10": 12.294113159179688,
+      "11": 14.582603454589844,
+      "12": 6.157026290893555,
+      "13": 7.191178798675537,
+      "14": 10.864974975585938,
+      "15": 11.616080284118652,
+      "16": 10.46957778930664,
+      "17": 12.961014747619629,
+      "18": 10.876848220825195,
+      "19": 10.018270492553711,
+      "20": 11.60196304321289,
+      "21": 11.650176048278809,
+      "22": 10.586633682250977,
+      "23": 12.792163848876953
+    },
+    "diff_to_teacher": {
+      "mean": -0.6851403911908468,
+      "se": 0.2972779784018067,
+      "sd": 1.4563587177010893,
+      "wins": 7
+    },
+    "size_gb": 1.95
+  },
+  "mixH2_x": {
+    "n": 24,
+    "mean_mu": 11.917078256607056,
+    "sd_mu": 1.4976344977986062,
+    "ci95": [
+      11.3171836266915,
+      12.492527563373248
+    ],
+    "category_mean_mu": {
+      "text": 12.793494701385498,
+      "portrait": 12.09325361251831,
+      "detail": 10.962145805358887,
+      "counting": 11.970977306365967,
+      "varied": 11.635669708251953
+    },
+    "mu_by_prompt": {
+      "0": 13.581269264221191,
+      "1": 11.102155685424805,
+      "2": 12.775008201599121,
+      "3": 11.850469589233398,
+      "4": 13.954010963439941,
+      "5": 13.498054504394531,
+      "6": 11.450055122375488,
+      "7": 12.061519622802734,
+      "8": 10.337630271911621,
+      "9": 14.523809432983398,
+      "10": 11.705960273742676,
+      "11": 13.739737510681152,
+      "12": 10.581864356994629,
+      "13": 7.82102108001709,
+      "14": 11.763824462890625,
+      "15": 12.178130149841309,
+      "16": 11.282197952270508,
+      "17": 13.614313125610352,
+      "18": 11.304281234741211,
+      "19": 10.406351089477539,
+      "20": 10.849934577941895,
+      "21": 12.486143112182617,
+      "22": 10.684165000915527,
+      "23": 12.457971572875977
+    },
+    "diff_to_teacher": {
+      "mean": -0.4675034284591675,
+      "se": 0.19015638558118367,
+      "sd": 0.9315722320116647,
+      "wins": 5
+    },
+    "size_gb": 1.95
+  },
+  "mixH2_tuned": {
+    "n": 24,
+    "mean_mu": 11.895683626333872,
+    "sd_mu": 1.7010001135124344,
+    "ci95": [
+      11.21190027644237,
+      12.53292619784673
+    ],
+    "category_mean_mu": {
+      "text": 12.790374755859375,
+      "portrait": 12.12794017791748,
+      "detail": 10.82534396648407,
+      "counting": 11.896456718444824,
+      "varied": 11.643513560295105
+    },
+    "mu_by_prompt": {
+      "0": 13.311539649963379,
+      "1": 11.351049423217773,
+      "2": 12.961719512939453,
+      "3": 11.900381088256836,
+      "4": 13.324409484863281,
+      "5": 13.893149375915527,
+      "6": 12.187474250793457,
+      "7": 11.224508285522461,
+      "8": 10.200164794921875,
+      "9": 14.899613380432129,
+      "10": 11.416027069091797,
+      "11": 14.799453735351562,
+      "12": 10.560506820678711,
+      "13": 6.525388240814209,
+      "14": 11.732919692993164,
+      "15": 12.059993743896484,
+      "16": 10.965230941772461,
+      "17": 12.68959903717041,
+      "18": 11.342528343200684,
+      "19": 10.494524955749512,
+      "20": 11.686250686645508,
+      "21": 11.768655776977539,
+      "22": 11.264636039733887,
+      "23": 12.93668270111084
+    },
+    "diff_to_teacher": {
+      "mean": -0.48889805873235065,
+      "se": 0.22897005628476946,
+      "sd": 1.1217196085480594,
+      "wins": 8
+    },
+    "size_gb": 1.95
+  },
+  "mixH2_xm": {
+    "n": 24,
+    "mean_mu": 11.487676699956259,
+    "sd_mu": 1.6185009544025424,
+    "ci95": [
+      10.831634158889452,
+      12.11318797369798
+    ],
+    "category_mean_mu": {
+      "text": 12.731455167134603,
+      "portrait": 12.26175832748413,
+      "detail": 9.958102703094482,
+      "counting": 11.286506175994873,
+      "varied": 10.982881665229797
+    },
+    "mu_by_prompt": {
+      "0": 12.68850326538086,
+      "1": 11.272262573242188,
+      "2": 13.997017860412598,
+      "3": 11.66956901550293,
+      "4": 12.465785026550293,
+      "5": 14.29559326171875,
+      "6": 11.829322814941406,
+      "7": 12.416139602661133,
+      "8": 11.078559875488281,
+      "9": 13.723011016845703,
+      "10": 10.847691535949707,
+      "11": 12.856647491455078,
+      "12": 8.917652130126953,
+      "13": 7.210419654846191,
+      "14": 10.262415885925293,
+      "15": 12.310596466064453,
+      "16": 9.655908584594727,
+      "17": 12.390963554382324,
+      "18": 11.141194343566895,
+      "19": 10.1093111038208,
+      "20": 10.608089447021484,
+      "21": 11.77521800994873,
+      "22": 11.69253158569336,
+      "23": 10.489836692810059
+    },
+    "diff_to_teacher": {
+      "mean": -0.896904985109965,
+      "se": 0.2175312633346519,
+      "sd": 1.0656811965457924,
+      "wins": 5
+    },
+    "size_gb": 1.83
+  },
+  "mixH2_xm_f8": {
+    "n": 24,
+    "mean_mu": 11.442908763885498,
+    "sd_mu": 1.68771424350057,
+    "ci95": [
+      10.76411321957906,
+      12.09367621441682
+    ],
+    "category_mean_mu": {
+      "text": 12.668961842854818,
+      "portrait": 12.236000776290894,
+      "detail": 10.113464832305908,
+      "counting": 10.981910705566406,
+      "varied": 10.906794428825378
+    },
+    "mu_by_prompt": {
+      "0": 12.688547134399414,
+      "1": 10.991280555725098,
+      "2": 13.278096199035645,
+      "3": 11.797897338867188,
+      "4": 12.758256912231445,
+      "5": 14.499692916870117,
+      "6": 11.637022972106934,
+      "7": 12.572162628173828,
+      "8": 10.822105407714844,
+      "9": 13.912712097167969,
+      "10": 11.10090446472168,
+      "11": 13.388298034667969,
+      "12": 8.770405769348145,
+      "13": 7.19425106048584,
+      "14": 9.96194076538086,
+      "15": 12.001880645751953,
+      "16": 9.986132621765137,
+      "17": 12.722164154052734,
+      "18": 11.125619888305664,
+      "19": 10.118558883666992,
+      "20": 9.693978309631348,
+      "21": 11.55894660949707,
+      "22": 11.63875675201416,
+      "23": 10.410198211669922
+    },
+    "diff_to_teacher": {
+      "mean": -0.9416729211807251,
+      "se": 0.2028905801931563,
+      "sd": 0.9939567901809284,
+      "wins": 5
+    },
+    "size_gb": 1.72
+  },
+  "mixH2_x_f8": {
+    "n": 24,
+    "mean_mu": 11.989253520965576,
+    "sd_mu": 1.5468661955186895,
+    "ci95": [
+      11.359737481673559,
+      12.58515132466952
+    ],
+    "category_mean_mu": {
+      "text": 12.905888080596924,
+      "portrait": 12.12691354751587,
+      "detail": 11.099243640899658,
+      "counting": 12.004260063171387,
+      "varied": 11.674200892448425
+    },
+    "mu_by_prompt": {
+      "0": 13.653209686279297,
+      "1": 11.297388076782227,
+      "2": 12.705009460449219,
+      "3": 11.617015838623047,
+      "4": 13.824302673339844,
+      "5": 14.33840274810791,
+      "6": 11.460399627685547,
+      "7": 12.014626502990723,
+      "8": 10.534360885620117,
+      "9": 14.49826717376709,
+      "10": 11.64794921875,
+      "11": 13.963139533996582,
+      "12": 11.265219688415527,
+      "13": 7.520666122436523,
+      "14": 11.920021057128906,
+      "15": 12.088499069213867,
+      "16": 11.032934188842773,
+      "17": 13.643148422241211,
+      "18": 11.172962188720703,
+      "19": 10.405658721923828,
+      "20": 11.057677268981934,
+      "21": 12.540094375610352,
+      "22": 11.082413673400879,
+      "23": 12.458718299865723
+    },
+    "diff_to_teacher": {
+      "mean": -0.395328164100647,
+      "se": 0.1802226438840783,
+      "sd": 0.8829070352226305,
+      "wins": 5
+    },
+    "size_gb": 1.85
+  },
+  "mixI_x": {
+    "n": 24,
+    "mean_mu": 9.655051132043203,
+    "sd_mu": 1.679698239139186,
+    "ci95": [
+      8.966416811943056,
+      10.272336814800898
+    ],
+    "category_mean_mu": {
+      "text": 10.08133602142334,
+      "portrait": 10.162810564041138,
+      "detail": 8.78571367263794,
+      "counting": 10.202799320220947,
+      "varied": 9.379189431667328
+    },
+    "mu_by_prompt": {
+      "0": 10.26925277709961,
+      "1": 6.766984939575195,
+      "2": 10.589472770690918,
+      "3": 9.530107498168945,
+      "4": 12.589543342590332,
+      "5": 10.742654800415039,
+      "6": 10.981014251708984,
+      "7": 9.425098419189453,
+      "8": 9.351439476013184,
+      "9": 10.89369010925293,
+      "10": 11.534799575805664,
+      "11": 10.136062622070312,
+      "12": 8.623100280761719,
+      "13": 4.8488922119140625,
+      "14": 10.231926918029785,
+      "15": 10.17367172241211,
+      "16": 9.301717758178711,
+      "17": 11.269822120666504,
+      "18": 7.489152431488037,
+      "19": 8.805434226989746,
+      "20": 8.242559432983398,
+      "21": 8.819071769714355,
+      "22": 9.821023941040039,
+      "23": 11.284733772277832
+    },
+    "diff_to_teacher": {
+      "mean": -2.7295305530230203,
+      "se": 0.28547024864832266,
+      "sd": 1.3985128918676595,
+      "wins": 1
+    },
+    "size_gb": 1.74
+  },
+  "mixI_tuned": {
+    "n": 24,
+    "mean_mu": 10.650842626889547,
+    "sd_mu": 1.779264514185643,
+    "ci95": [
+      9.94127772450447,
+      11.3256166746219
+    ],
+    "category_mean_mu": {
+      "text": 11.08804210027059,
+      "portrait": 11.977301120758057,
+      "detail": 9.071760654449463,
+      "counting": 10.985382080078125,
+      "varied": 10.365619897842407
+    },
+    "mu_by_prompt": {
+      "0": 12.68350601196289,
+      "1": 8.426652908325195,
+      "2": 11.992571830749512,
+      "3": 9.905455589294434,
+      "4": 11.383698463439941,
+      "5": 12.136367797851562,
+      "6": 11.28021240234375,
+      "7": 12.629135131835938,
+      "8": 10.298370361328125,
+      "9": 13.701486587524414,
+      "10": 9.960355758666992,
+      "11": 12.58846664428711,
+      "12": 5.817440032958984,
+      "13": 7.920780181884766,
+      "14": 9.724404335021973,
+      "15": 12.246359825134277,
+      "16": 9.888011932373047,
+      "17": 11.552902221679688,
+      "18": 10.344181060791016,
+      "19": 8.561613082885742,
+      "20": 10.274347305297852,
+      "21": 10.694239616394043,
+      "22": 10.358518600463867,
+      "23": 11.251145362854004
+    },
+    "diff_to_teacher": {
+      "mean": -1.7337390581766765,
+      "se": 0.31209043163446204,
+      "sd": 1.5289246222187787,
+      "wins": 1
+    },
+    "size_gb": 1.74
+  },
+  "tern_v1": {
+    "n": 24,
+    "mean_mu": -8.961636026700338,
+    "sd_mu": 1.341661786296067,
+    "ci95": [
+      -9.442805964748064,
+      -8.39396297633648
+    ],
+    "category_mean_mu": {
+      "text": -8.419553756713867,
+      "portrait": -9.220449924468994,
+      "detail": -9.765491724014282,
+      "counting": -9.634195804595947,
+      "varied": -8.668722987174988
+    },
+    "mu_by_prompt": {
+      "0": -9.186239242553711,
+      "1": -5.7475481033325195,
+      "2": -7.011780738830566,
+      "3": -8.10580825805664,
+      "4": -10.226703643798828,
+      "5": -10.239242553710938,
+      "6": -9.475618362426758,
+      "7": -8.87272834777832,
+      "8": -8.483269691467285,
+      "9": -10.050183296203613,
+      "10": -10.196704864501953,
+      "11": -8.95408821105957,
+      "12": -10.226978302001953,
+      "13": -9.684195518493652,
+      "14": -9.198389053344727,
+      "15": -10.070002555847168,
+      "16": -9.747382164001465,
+      "17": -9.68046760559082,
+      "18": -5.4149274826049805,
+      "19": -9.003195762634277,
+      "20": -9.51219654083252,
+      "21": -7.509844779968262,
+      "22": -9.660802841186523,
+      "23": -8.820966720581055
+    },
+    "diff_to_teacher": {
+      "mean": -21.34621771176656,
+      "se": 0.46752777685313823,
+      "sd": 2.2904089877359692,
+      "wins": 0
+    },
+    "size_gb": 1.92
+  }
+}

eval/hps3_summary.md

@@ -0,0 +1,24 @@
+| tag | size GB | mean mu [bootstrap 95%] | text | portrait | detail | counting | varied | diff to teacher +- SE (wins/24) |
+|---|---|---|---|---|---|---|---|---|
+| teacher_bf16 | 10.16 | 12.38 [11.76, 13.01] | 13.33 | 13.07 | 12.41 | 10.93 | 11.68 | - |
+| mixA_x_tuned | 1.67 | 11.61 [10.91, 12.31] | 11.76 | 13.03 | 10.76 | 11.65 | 11.20 | -0.77 +- 0.28 (7/24) |
+| mixF_ptq | 2.42 | 10.86 [10.16, 11.59] | 12.17 | 11.57 | 10.67 | 10.11 | 9.79 | -1.53 +- 0.23 (4/24) |
+| mixG_x | 2.14 | 12.09 [11.39, 12.74] | 13.42 | 12.39 | 10.95 | 11.73 | 11.59 | -0.30 +- 0.19 (7/24) |
+| mixG_tuned | 2.14 | 12.14 [11.36, 12.87] | 13.39 | 12.88 | 10.46 | 11.71 | 11.78 | -0.24 +- 0.29 (9/24) |
+| mixH_x | 1.95 | 11.88 [11.19, 12.53] | 13.04 | 12.49 | 10.69 | 11.49 | 11.40 | -0.50 +- 0.22 (8/24) |
+| mixH_tuned | 1.95 | 11.70 [10.87, 12.45] | 12.94 | 12.37 | 10.06 | 11.24 | 11.37 | -0.69 +- 0.30 (7/24) |
+| mixH2_x | 1.95 | 11.92 [11.32, 12.49] | 12.79 | 12.09 | 10.96 | 11.97 | 11.64 | -0.47 +- 0.19 (5/24) |
+| mixH2_tuned | 1.95 | 11.90 [11.21, 12.53] | 12.79 | 12.13 | 10.83 | 11.90 | 11.64 | -0.49 +- 0.23 (8/24) |
+| mixH2_xm | 1.83 | 11.49 [10.83, 12.11] | 12.73 | 12.26 | 9.96 | 11.29 | 10.98 | -0.90 +- 0.22 (5/24) |
+| mixH2_xm_f8 | 1.72 | 11.44 [10.76, 12.09] | 12.67 | 12.24 | 10.11 | 10.98 | 10.91 | -0.94 +- 0.20 (5/24) |
+| mixH2_x_f8 | 1.85 | 11.99 [11.36, 12.59] | 12.91 | 12.13 | 11.10 | 12.00 | 11.67 | -0.40 +- 0.18 (5/24) |
+| mixI_x | 1.74 | 9.66 [8.97, 10.27] | 10.08 | 10.16 | 8.79 | 10.20 | 9.38 | -2.73 +- 0.29 (1/24) |
+| mixI_tuned | 1.74 | 10.65 [9.94, 11.33] | 11.09 | 11.98 | 9.07 | 10.99 | 10.37 | -1.73 +- 0.31 (1/24) |
+| tern_v1 | 1.92 | -8.96 [-9.44, -8.39] | -8.42 | -9.22 | -9.77 | -9.63 | -8.67 | -21.35 +- 0.47 (0/24) |
+
+mu = HPSv3 raw score (higher = preferred), mean over 24 prompts, one seed per prompt, 768px images scored at ~448px (HPSv3 input).
+diff to teacher = mean of (student mu - teacher mu) per prompt, SE = sd of the 24 paired differences / sqrt(24); wins = prompts where student mu > teacher mu.
+
+Noise: only 24 prompts and ONE seed per prompt, so seed-to-seed variation of a model is not measured at all and the intervals only reflect which 24 prompts were drawn.
+- sd of mu across prompts is 1.34-2.03; bootstrap 95% half-width of a mean is 0.52-0.79.
+- paired SE between two candidate models (excluding teacher and tern_v1) is 0.06-0.34 (median 0.23); a difference below about 2 x that (0.46 mu) cannot be told apart from prompt-sampling noise.

eval/hps3_tmixG.json

@@ -0,0 +1,88 @@
+{
+  "tmixG_x_1024": {
+    "n": 24,
+    "mean_mu": 11.093161900838217,
+    "sd_mu": 1.7534528688621807,
+    "ci95": [
+      10.383457724253336,
+      11.765245208144187
+    ],
+    "category_mean_mu": {
+      "text": 12.275560855865479,
+      "portrait": 12.011467218399048,
+      "detail": 9.47965931892395,
+      "counting": 10.9927339553833,
+      "varied": 10.579068303108215
+    },
+    "mu_by_prompt": {
+      "0": 12.029065132141113,
+      "1": 10.247049331665039,
+      "2": 13.635560035705566,
+      "3": 12.875860214233398,
+      "4": 11.864503860473633,
+      "5": 13.001326560974121,
+      "6": 12.741265296936035,
+      "7": 11.759806632995605,
+      "8": 11.334644317626953,
+      "9": 12.210152626037598,
+      "10": 8.151248931884766,
+      "11": 13.089639663696289,
+      "12": 9.873390197753906,
+      "13": 6.80435848236084,
+      "14": 10.72061538696289,
+      "15": 11.264852523803711,
+      "16": 10.157102584838867,
+      "17": 13.925820350646973,
+      "18": 9.778844833374023,
+      "19": 9.371624946594238,
+      "20": 10.581709861755371,
+      "21": 10.420363426208496,
+      "22": 11.24316120147705,
+      "23": 9.153919219970703
+    },
+    "size_gb": null
+  },
+  "tmixG_x_768": {
+    "n": 24,
+    "mean_mu": 11.060355603694916,
+    "sd_mu": 1.8899038827082177,
+    "ci95": [
+      10.284124094247817,
+      11.7712279672424
+    ],
+    "category_mean_mu": {
+      "text": 12.452021280924479,
+      "portrait": 12.091742038726807,
+      "detail": 9.59649384021759,
+      "counting": 10.613999366760254,
+      "varied": 10.344433069229126
+    },
+    "mu_by_prompt": {
+      "0": 12.194013595581055,
+      "1": 11.119743347167969,
+      "2": 14.199522972106934,
+      "3": 12.216362953186035,
+      "4": 12.408049583435059,
+      "5": 12.574435234069824,
+      "6": 12.863862991333008,
+      "7": 13.082260131835938,
+      "8": 10.617880821228027,
+      "9": 11.802964210510254,
+      "10": 10.203123092651367,
+      "11": 13.07459545135498,
+      "12": 9.950275421142578,
+      "13": 5.1579813957214355,
+      "14": 9.728994369506836,
+      "15": 11.499004364013672,
+      "16": 10.289899826049805,
+      "17": 12.423584938049316,
+      "18": 10.51021957397461,
+      "19": 8.268671035766602,
+      "20": 10.916498184204102,
+      "21": 11.067235946655273,
+      "22": 9.782889366149902,
+      "23": 9.496465682983398
+    },
+    "size_gb": null
+  }
+}

eval/hps3_tmixG.md

@@ -0,0 +1,11 @@
+| tag | size GB | mean mu [bootstrap 95%] | text | portrait | detail | counting | varied | diff to teacher +- SE (wins/24) |
+|---|---|---|---|---|---|---|---|---|
+| tmixG_x_1024 | - | 11.09 [10.38, 11.77] | 12.28 | 12.01 | 9.48 | 10.99 | 10.58 | - |
+| tmixG_x_768 | - | 11.06 [10.28, 11.77] | 12.45 | 12.09 | 9.60 | 10.61 | 10.34 | - |
+
+mu = HPSv3 raw score (higher = preferred), mean over 24 prompts, one seed per prompt, 768px images scored at ~448px (HPSv3 input).
+diff to teacher = mean of (student mu - teacher mu) per prompt, SE = sd of the 24 paired differences / sqrt(24); wins = prompts where student mu > teacher mu.
+
+Noise: only 24 prompts and ONE seed per prompt, so seed-to-seed variation of a model is not measured at all and the intervals only reflect which 24 prompts were drawn.
+- sd of mu across prompts is 1.75-1.89; bootstrap 95% half-width of a mean is 0.69-0.74.
+- paired SE between two candidate models (excluding teacher and tern_v1) is 0.19-0.19 (median 0.19); a difference below about 2 x that (0.37 mu) cannot be told apart from prompt-sampling noise.

eval/hps3_tmixG_traj.json

@@ -0,0 +1,88 @@
+{
+  "tmixG_traj_1024": {
+    "n": 24,
+    "mean_mu": 4.709925166020791,
+    "sd_mu": 3.983988371755315,
+    "ci95": [
+      3.075492799654603,
+      6.241217858375361
+    ],
+    "category_mean_mu": {
+      "text": 4.4555767228206,
+      "portrait": 6.16073477268219,
+      "detail": 2.9998376071453094,
+      "counting": 6.154835224151611,
+      "varied": 4.66909795999527
+    },
+    "mu_by_prompt": {
+      "0": 0.9237642288208008,
+      "1": 1.3860772848129272,
+      "2": 11.350065231323242,
+      "3": 7.726625919342041,
+      "4": 0.4157429039478302,
+      "5": 4.931184768676758,
+      "6": 6.6820149421691895,
+      "7": 5.373495578765869,
+      "8": 6.486237049102783,
+      "9": 6.101191520690918,
+      "10": -5.721309185028076,
+      "11": 10.541007995605469,
+      "12": 6.0127854347229,
+      "13": 1.1668661832809448,
+      "14": 6.27398681640625,
+      "15": 6.035683631896973,
+      "16": 8.108414649963379,
+      "17": 9.131568908691406,
+      "18": -1.9423153400421143,
+      "19": 6.569377899169922,
+      "20": 6.460616588592529,
+      "21": 5.1261210441589355,
+      "22": 2.049818277359009,
+      "23": 1.8491816520690918
+    },
+    "size_gb": null
+  },
+  "tmixG_traj_768": {
+    "n": 24,
+    "mean_mu": 4.852243738869826,
+    "sd_mu": 3.477296949103987,
+    "ci95": [
+      3.448883457357685,
+      6.230218659092982
+    ],
+    "category_mean_mu": {
+      "text": 4.018895824750264,
+      "portrait": 6.62620997428894,
+      "detail": 4.375384375452995,
+      "counting": 5.518620014190674,
+      "varied": 4.662107169628143
+    },
+    "mu_by_prompt": {
+      "0": 4.189118385314941,
+      "1": 1.9079973697662354,
+      "2": 9.796252250671387,
+      "3": 6.590702056884766,
+      "4": -1.5507299900054932,
+      "5": 3.180034875869751,
+      "6": 6.0802693367004395,
+      "7": 8.471721649169922,
+      "8": 5.010310173034668,
+      "9": 6.942538738250732,
+      "10": 0.11058086156845093,
+      "11": 10.911245346069336,
+      "12": 5.121137619018555,
+      "13": 1.3585736751556396,
+      "14": 4.785759925842285,
+      "15": 6.2514801025390625,
+      "16": 8.555427551269531,
+      "17": 8.209768295288086,
+      "18": 0.994794487953186,
+      "19": 6.622662544250488,
+      "20": 6.6200151443481445,
+      "21": 6.726078033447266,
+      "22": 1.2486697435379028,
+      "23": -1.680558443069458
+    },
+    "size_gb": null
+  }
+}

eval/hps3_tmixG_traj.md

@@ -0,0 +1,11 @@
+| tag | size GB | mean mu [bootstrap 95%] | text | portrait | detail | counting | varied | diff to teacher +- SE (wins/24) |
+|---|---|---|---|---|---|---|---|---|
+| tmixG_traj_1024 | - | 4.71 [3.08, 6.24] | 4.46 | 6.16 | 3.00 | 6.15 | 4.67 | - |
+| tmixG_traj_768 | - | 4.85 [3.45, 6.23] | 4.02 | 6.63 | 4.38 | 5.52 | 4.66 | - |
+
+mu = HPSv3 raw score (higher = preferred), mean over 24 prompts, one seed per prompt, 768px images scored at ~448px (HPSv3 input).
+diff to teacher = mean of (student mu - teacher mu) per prompt, SE = sd of the 24 paired differences / sqrt(24); wins = prompts where student mu > teacher mu.
+
+Noise: only 24 prompts and ONE seed per prompt, so seed-to-seed variation of a model is not measured at all and the intervals only reflect which 24 prompts were drawn.
+- sd of mu across prompts is 3.48-3.98; bootstrap 95% half-width of a mean is 1.39-1.58.
+- paired SE between two candidate models (excluding teacher and tern_v1) is 0.42-0.42 (median 0.42); a difference below about 2 x that (0.84 mu) cannot be told apart from prompt-sampling noise.