eval/eval.md

# PixelModel v1 — Evaluation

Benchmark results for the [Tiny-T2I-Leaderboard](https://huggingface.co/spaces/FlameF0X/Tiny-T2I-Leaderboard),
measured with the leaderboard's preferred tooling (torchmetrics). Everything
needed to reproduce these numbers ships in this repo — see `EVAL_REPRODUCTION.md`.

## Parameter count

23,747 parameters in 10 tensors (5 weight matrices + 5 bias vectors), all
stored in `model.png` (160×149 px, 16-bit/weight codec) and exported to
`model.safetensors` with the count in the header metadata:

| Tensor | Shape | Params | Role |
|---|---|---|---|
| T1 + b1 | 80×64 + 80 | 5,200 | prompt embedding → trunk hidden |
| T2 + b2 | 64×80 + 64 | 5,184 | trunk hidden → latent |
| D1 + bd1 | 80×82 + 80 | 6,640 | latent + fourier(x,y) → decoder hidden |
| D2 + bd2 | 80×80 + 80 | 6,480 | decoder hidden → decoder hidden |
| D3 + bd3 | 3×80 + 3 | 243 | decoder hidden → RGB |
| **Total** | | **23,747 (~0.024M)** | |

The prompt embedding is a hashed function (FNV-1a over character trigrams and
words) with zero learned parameters; there is no text encoder and no VAE
(`text_encoder_parameters: 0`, `vae_parameters: 0` in both `config.json` and
the safetensors metadata). Well under the 1.5B ceiling — and 8.5× smaller than
v0's 202,752.

## Method

- **Real set**: 5,000 MS-COCO val2014 (image, caption) pairs — rows 0–4,999 of
  the [`sayakpaul/coco-30-val-2014`](https://huggingface.co/datasets/sayakpaul/coco-30-val-2014)
  stream — images center-cropped to 256×256.
- **Generated set**: `model.png` run on those same 5,000 captions at the
  native 64×64 resolution (no post-upscaling).
- **FID**: `torchmetrics.image.fid.FrechetInceptionDistance` (`feature=2048`,
  InceptionV3 pool3; the metric resizes both sets to 299×299 internally).
- **CLIP Score**: `torchmetrics.multimodal.CLIPScore` with
  `openai/clip-vit-base-patch32` (the default), generated image vs the caption
  that produced it, averaged over all 5,000.
- **Train/eval hygiene**: the model was trained on 20,000 pairs from rows
  10,000+ of the same stream, content-hash-checked to be disjoint from the
  5,000 eval images (0 overlaps found; the 30K rows are 30K unique images).

n=5000 is far past the singular-covariance regime that made v0's n=40 FID
directional-only (a 2048×2048 covariance needs n ≫ 2048 to be non-singular;
5,000 clears it, 40 did not).

## Results

| Metric | v0 (n=40) | **v1 (n=5000)** | Context |
|---|---|---|---|
| FID ↓ | 566.84 | **439.46** | Competitive T2I models score ~10–30; v1's caption-conditioned color/layout statistics move it meaningfully toward the real-photo distribution, but 64×64 color washes remain far outside it. |
| CLIP Score ↑ | 18.60 | **20.02** (cosine 0.2002) | ~15–20 is the no-semantic-correspondence floor; ~28–32 is competitive. v1 sits measurably above the floor — captions genuinely steer palette and layout — with 8.5× fewer parameters than v0. |

Training loss context: the model's final MSE (0.0627) is 9% below the
"always output the mean COCO image" baseline (0.0691) — everything below that
baseline is caption-conditioning, since the mean image is the best possible
caption-blind predictor.

## Leaderboard entry

```js
{
  name:        "PixelModel-v1",
  org:         "bench-labs",
  params:      "0.024M",
  fid:         439.46,
  clipScore:   0.2002,
  resolution:  "64x64",
  releaseDate: "2026-07-17",
  links: {
    card: "https://huggingface.co/bench-labs/pixelmodel-v1"
  }
},
```

*(`clipScore` given as the 0–1 cosine value per the template's field
description; the 0–100 form of the same number is 20.02.)*