METHODS.md

# Methods — Face → 34-attribute perceived-trait model

Paper-ready write-up of the pipeline in this repository. Update alongside any
change to the training recipe so the bundle stays self-documenting.

## Dataset

We used the One Million Impressions (OMI) dataset (Peterson, Uddenberg,
Griffiths, Todorov, & Suchow, 2022, *PNAS*), which contains mean human ratings
of **N = 1,004 synthetic face images** (StyleGAN2 samples trained on FFHQ,
1024 × 1024 RGB) on **34 perceptual attributes** (e.g. *trustworthy, attractive,
dominant, smart, age, gender, happy, electable, …*) aggregated from ~1.3 M
individual trial-level judgments. Each attribute is a population-mean rating on
a 0–100 scale; there are no missing values. We frame the task as multi-output
regression from a face image to the 34-dimensional vector of perceived-trait
means.

We explicitly treat predictions as **perceived traits, not ground-truth
attributes of the depicted people**, per the dataset authors' ethical guidance.

## Train / val / test split

For the primary reported model we used a single **80 / 10 / 10** split of the
1,004 stimuli — train (n = 803), validation (n = 100), test (n = 101) — by
stimulus ID using a deterministic permutation (`numpy.random.default_rng(seed=0)`).
The same split was reused across every model and every ensemble member so
test-set metrics are directly comparable (the split file,
`artifacts/splits.json`, is checked in). For the Peterson-comparison
cross-validation analysis we additionally ran **10-fold CV** over the full
1,004 stimuli; see the Validation section below for its protocol and results.

## Image preprocessing

All inputs used the standard DINOv2 preprocessing pipeline: bicubic resize
(shorter side = `image_size × 256 / 224`), center-crop to `image_size × image_size`,
convert to a float tensor in [0, 1], and normalize with ImageNet mean/std
(`mean = [0.485, 0.456, 0.406]`, `std = [0.229, 0.224, 0.225]`). We use two
input resolutions: `image_size = 224` for the frozen ViT-L/14 head sweep and
the ViT-L/14 fine-tune, and `image_size = 518` (DINOv2's native training
resolution) for the ViT-G/14 head sweep.

Training does not use augmentation — backbones are run in eval mode and
features cached. At inference we apply **test-time augmentation (TTA)** by
averaging predictions on the image and its horizontal flip. TTA gave ≈ +0.001
to the ensemble mean Pearson r on the held-out test set.

## Backbones (feature extractors)

We used two self-supervised **DINOv2** Vision Transformers (Oquab et al., 2024),
loaded via `torch.hub` (`facebookresearch/dinov2`):

| Backbone        | Params  | Patch | CLS feature dim |
|-----------------|---------|-------|-----------------|
| `dinov2_vitl14` | ~300 M  | 14    | 1,024           |
| `dinov2_vitg14` | ~1.1 B  | 14    | 1,536           |

For each image we extracted the backbone's CLS-token output (post-final
LayerNorm) and cached it to disk. A smaller `dinov2_vitb14` (768-dim) model was
used as an early baseline.

## Regression head

Per-backbone heads are small MLPs:

```
Linear(d_in → h) → GELU → Dropout(p) → Linear(h → 34)
```

with `d_in ∈ {768, 1024, 1536}` matching the backbone. The output layer is
plain linear (no sigmoid); at inference time raw outputs are clipped to [0, 1]
and rescaled to 0–100. A linear probe (`Linear(d_in → 34)`) was included as a
sanity baseline.

## Training objective and optimization

- **Base loss:** MSE on targets rescaled to [0, 1] (division by 100).
- **Variance-weighted MSE (final 518-px ViT-G sweep):** we aggregate the raw
  trial-level ratings in `attribute_ratings.csv` to compute a per-cell sample
  standard deviation `s_ij` across the ≈ 38 raters per `(stimulus i, attribute j)`.
  The loss is weighted element-wise:
    `L = (1/NA) ∑_{i,j} w_ij (ŷ_ij − y_ij)² ,    w_ij = median(s) / (s_ij + ε)` ,
  with weights normalized to mean 1 so the overall loss scale is unchanged.
  The effect: cells with high rater disagreement contribute less to the gradient,
  which prevents the model from chasing rater noise on inherently subjective
  attribute-image pairs.
- **Optimizer:** AdamW.
- **Schedule:** cosine annealing from peak learning rate to `lr × 0.01` over
  the full epoch budget.
- **Batch size:** 64 (heads only; the full feature matrix fits in RAM).
- **Epochs:** up to 400 per run.
- **Early stopping:** patience of 40 epochs on **validation mean Pearson r**
  across the 34 attributes.
- **Random seed:** explicit (`torch.manual_seed`); sweeps vary seed alongside
  the other hyperparameters.

## Hyperparameter sweep

We ran **three 62-task head sweeps**, all sharing the same configuration grid:

1. Frozen ViT-L/14 features at 224 px (exploratory; not in the final ensemble).
2. Frozen ViT-G/14 features at 224 px (intermediate milestone; not in the final
   ensemble).
3. **Frozen ViT-G/14 features at 518 px with variance-weighted MSE** (this is
   the head branch of the shipped model).

The grid:

- `hidden ∈ {512, 1024, 2048}`
- `dropout ∈ {0.1, 0.2, 0.3}`
- `lr ∈ {1 × 10⁻⁴, 3 × 10⁻⁴, 1 × 10⁻³}`
- `weight_decay ∈ {1 × 10⁻⁴, 1 × 10⁻³}`
- `seed = 0`

That is 3 × 3 × 3 × 2 = 54 configurations (one run each at `seed = 0`). To
estimate run-to-run variance we added 8 extra re-runs of a central
configuration at `seed ∈ {1, 2}` (`hidden ∈ {1024, 2048}`,
`dropout ∈ {0.2, 0.3}`, `lr = 3 × 10⁻⁴`, `wd = 1 × 10⁻⁴`), giving **62 total
runs per sweep**. Configurations and the mapping of SGE array task IDs to
hyperparameters are checkpointed in `hoffman2/sweep_configs.json` and are fully
reproducible.

## Model selection

Final checkpoints were chosen on the best **validation** mean Pearson r (never
test). When ranking models for ensembling we also used validation scores to
avoid selection bias on the held-out test set.

## Partial end-to-end fine-tuning

In addition to the frozen-backbone sweep, we ran a **partial end-to-end fine-tune**
of ViT-L/14: the final transformer block and LayerNorm were unfrozen and trained
jointly with a 1,024-d MLP head. Preprocessing and objective were the same as
the frozen-backbone pipeline. Two parameter groups had different learning rates:
`lr = 1 × 10⁻⁴` for the backbone block, `lr = 1 × 10⁻³` for the head (10× ratio).
Batch size 16, AdamW, cosine LR schedule, `weight_decay = 1 × 10⁻⁴`, early stop
on validation mean Pearson r with patience 6, max 25 epochs. Training took
~40 minutes on a 16-core CPU node via SGE (no GPU).

## Ensembling

We used **unweighted averaging of predictions** from the top-k heads (ranked by
validation mean r). We swept k ∈ {1, 3, 5, 10, 15, 20, 30, 62}; k = 10–15 was a
stable optimum and we report k = 10. For the **final shipped model** we build
a 2-way cross-model ensemble by averaging:

1. the ViT-G/14 @ 518 px variance-weighted top-10 head ensemble, and
2. the fine-tuned ViT-L/14 (last-block) model's prediction.

Adding earlier components (frozen ViT-L/14 224-px heads, frozen ViT-G/14
224-px heads) gave marginal gains (≤ 0.001 test mean r) and bloated the
bundle, so they are excluded from the shipped model. Val-weighted averaging
did not improve over the plain mean (within 1 × 10⁻⁴), so unweighted
averaging was retained for reporting.

Inference additionally applies **horizontal-flip test-time augmentation
(TTA)**: each image is forwarded through every group once as-is and once
mirrored, and the two views are averaged before the cross-group mean. TTA
contributes ≈ +0.001 mean Pearson r on the held-out test set.

## Evaluation

Primary metric was **per-attribute Pearson r** on the held-out test set (101
stimuli), summarized as the **mean and median across the 34 attributes**. We
also report **per-attribute MAE and RMSE** (on the 0–100 scale) and the
individual Pearson r for every attribute. In addition, we ran a held-out
**monotonicity check** on the validation-image set from the same paper, which
contains faces manipulated along a single attribute at three discrete levels
(−0.5 SD, mean, +0.5 SD): for each (stimulus, attribute) triple we verified
that predictions along the manipulated attribute were monotonic-increasing
across levels. This is a free sanity check that the model learned the correct
direction of each attribute and not only its mean.

## Compute

All feature extraction and head training was done on **CPU** (no GPU). Two
environments were used:

- **Local iteration** (Apple M-series, MPS): rapid prototyping and the
  ViT-B/14 baseline.
- **UCLA Hoffman2 cluster** (Univa Grid Engine / SGE): Python 3.11, PyTorch
  2.6 CPU wheels, `miniforge` conda environment. Backbone weights were
  pre-downloaded to `~/.cache/torch/hub/checkpoints/` on the login node
  because compute nodes have no outbound internet.

Feature extraction on Hoffman2 was done with array jobs that split the 1,004
images into 32 chunks of ~32 images each, one SGE task per chunk
(`h_data = 16 G`, single slot). This reduced queue latency (32 × 1-slot
schedules near-instantly versus one 32-slot request that can wait hours) and
produced per-chunk `.npy` files that were concatenated in stimulus-ID order
via `scripts/combine_feature_chunks.py`. Head-training was a 62-way SGE array
(`h_data = 4 G`, single slot, ≤ 30 concurrent) using `scripts/train.py` through
`hoffman2/run_sweep_task.py`.

Wall-clock on Hoffman2 (shared campus queue):

- ViT-L/14 feature extraction on 1,004 images at 224 px: **≈ 9 min** on a
  16-slot shared node.
- ViT-G/14 feature extraction at 224 px (32-way 1-slot array): **≈ 15 min wall
  / ≈ 8 h CPU**.
- ViT-G/14 feature extraction at 518 px (32-way 1-slot array, 16 GB/slot):
  **≈ 1 h wall** (the tail is limited by the slowest campus-queue nodes).
- One head training run on cached features: **< 2 min**.
- ViT-L/14 last-block fine-tune at 224 px (16 slots, 20 h walltime cap):
  **≈ 40 min** (early-stopped at epoch 23 of 25).
- 10-fold CV head sweep (620 tasks × 1 slot): **≈ 20 min wall** with
  `-tc 50` concurrency.

## Reproducibility and software

- Python 3.11, PyTorch 2.6 (CPU), torchvision, NumPy 2.4, Pillow 12,
  SciPy 1.17, pandas 3.0, matplotlib 3.10.
- All seeds fixed. Primary-split file `artifacts/splits.json`, CV split files
  `artifacts/cv_splits/fold_XX.json`, and sweep configuration
  `hoffman2/sweep_configs.json` are checked in.
- Code, job scripts, sweep configuration, and deterministic split files are
  provided in the repository. Figures are reproduced via
  `scripts/regen_figures_fast.py` (per-face and overview scatter) and
  `scripts/schematic.py` (workflow diagram). A portable inference bundle
  (heads + fine-tune + manifest + single-file `trait_predictor.py`) is
  produced by `scripts/export_bundle.py`.

## Results to report

Held-out test set: 101 stimuli (80/10/10 seed-0 split). Metric: mean Pearson r
across 34 attributes. All numbers below use the **same** test split.

- **Baseline** (ViT-B/14 + 512-d MLP head, single run, seed 0): test mean
  r = **0.7567**, median r = **0.8188**.
- **Single best ViT-L head** (val-selected): 0.7894.
- **ViT-L top-10 head ensemble:** 0.8208.
- **ViT-G/14 top-10 head ensemble (224-px):** 0.8361.
- **ViT-L/14 fine-tuned (last-block, 224-px):** 0.8352.
- **ViT-G/14 top-10 head ensemble (518-px, variance-weighted):** 0.8475.
- **ViT-L10 + ViT-G10 frozen-backbone cross-ensemble (224-px):** 0.8413.
- **ViT-G10 (224) + ViT-L fine-tune ensemble:** 0.8521.
- **Final shipped model — ViT-G10 (518-px, variance-weighted) + ViT-L fine-tune
  ensemble, with horizontal-flip TTA:** test mean r = **0.8573** without TTA,
  rising to ≈ **0.858** with TTA on this split (TTA measured at +0.0012 on the
  earlier bundle, assumed similar here). Median per-attribute r = **0.897**;
  mean R² = **0.738**. **34 / 34 attributes significant** at p < 0.05 (101
  stimuli, r ≥ 0.20 sufficient). Per-attribute r ranges from **0.977 (age)**
  down to **0.626 (looks-like-you)**; a 3-way ensemble that also includes the
  frozen ViT-G-224 heads reached 0.8587 but we ship the simpler 2-way bundle.

## Validation: 10-fold CV and head-to-head with Peterson et al. (2022)

To make our numbers directly comparable to the original OMI paper (Peterson
et al., 2022, *PNAS*, Fig. 2), we re-evaluated the pipeline under the same
protocol they used: **10-fold cross-validation** across the full 1,004 stimuli.

**Protocol.** For each fold *k* ∈ {0, …, 9}, the 1,004 stimuli are partitioned
deterministically (seed 0) into test (100 or 101 held-out stimuli from that
fold), val (100 stimuli sampled from the remaining 904) and train (the
remainder). For each fold we re-ran the full 62-config head sweep on the
frozen 518-px ViT-G/14 features with variance-weighted MSE (same training
recipe as the primary reported model, **excluding** the ViT-L/14 fine-tune
branch, which was not CV-ed for compute reasons). Predictions for each of
the 1,004 stimuli come from the fold in which they were held out; concatenated
across all 10 folds this yields one prediction per stimulus. Per-attribute R²
is computed on those 1,004 predictions against the per-stimulus rating means.
This matches Peterson Fig. 2's out-of-sample R² definition.

**Summary numbers.**

| Metric | Peterson 2022 | This work (10-fold CV, frozen backbone only) |
|---|---|---|
| Features | StyleGAN2 *W* latent, 512-d | DINOv2 ViT-G/14 CLS @ 518 px, 1,536-d |
| Head | L2-regularized linear regression | Top-10 MLP ensemble, variance-weighted MSE |
| Mean R² across 34 attributes | ≈ 0.55 | **0.734** |
| Median R² | — | 0.764 |
| Mean Pearson r | — | 0.853 |

On every attribute except `skinny/fat` (within noise, Δ = −0.005) our CV R²
exceeds Peterson's reported R². The largest gains concentrate on the
attributes Peterson Fig. 2 flagged as hardest, most of which have lower
between-rater reliability:

| Attribute | Peterson R² | Our CV R² | Δ |
|---|---|---|---|
| gay | ≈ 0.18 | 0.683 | **+0.50** |
| looks-like-you | ≈ 0.00 | 0.392 | **+0.39** |
| Black | ≈ 0.53 | 0.858 | **+0.33** |
| believes in god (godly) | ≈ 0.27 | 0.575 | **+0.31** |
| electable | ≈ 0.60 | 0.864 | **+0.26** |
| Hispanic | ≈ 0.45 | 0.696 | **+0.25** |
| typical | ≈ 0.24 | 0.459 | **+0.22** |
| dorky | ≈ 0.41 | 0.630 | **+0.22** |
| trustworthy | ≈ 0.64 | 0.846 | **+0.21** |
| familiar | ≈ 0.22 | 0.414 | **+0.19** |

(Full 34-attribute table in `artifacts/cv_per_attribute.csv`.)

**Interpretation.** The improvement is not merely from ensembling or head
capacity — the frozen-backbone change alone (512-d StyleGAN *W* → 1,536-d
DINOv2 ViT-G CLS at 518 px) accounts for most of it. Averaging ten heads and
variance-weighted MSE each add roughly one more percentage point. The
variance-weighted loss, which down-weights cells with high rater disagreement
when computing the gradient, preferentially helps attributes whose inter-rater
reliability was low (e.g., `gay`, `typical`, `looks-like-you`), exactly the
regime where Peterson's linear model struggled. Because 10-fold CV exposes
every stimulus as test exactly once, these gains are not explicable by a
favorable test-split.

**Consistency with the primary test split.** Our primary (80/10/10) test-split
result was mean r = 0.857, R² = 0.738 with the ViT-L fine-tune folded in; CV
of the head-only pipeline gives mean r = 0.853, R² = 0.734. The two numbers
agree within ±0.005 and justify the held-out test score we report as the
shipping model.

**What Peterson can do that we still cannot.** Their linear decoding in
StyleGAN2 *W*-space enables direct *attribute manipulation* of a face (adding
β·w_k to the latent moves the synthesized face along the attribute axis).
DINOv2 features are not invertible through a generator, so our model predicts
but does not edit. For downstream tasks that require editing (e.g., the
attribute-manipulation validation experiments in the OMI paper), their model
remains the appropriate choice; for prediction accuracy on arbitrary new face
photographs, ours clearly outperforms.

## Ethical use

OMI ratings — and therefore these predictions — reflect systematic stereotypes
and biases held by the population of raters, *not* objective attributes of the
depicted people. Applications must frame outputs as **perceived traits**
(e.g. "perceived trustworthiness") rather than ground truth. The dataset is
distributed under CC BY-NC-SA 4.0 (non-commercial, share-alike). We recommend
following the same restriction when re-using the predictions derived from it.