METHODS.md

Methods — Face → 34-attribute perceived-trait model

Paper-ready write-up of the pipeline in this repository. Update alongside any change to the training recipe so the bundle stays self-documenting.

Dataset

We used the One Million Impressions (OMI) dataset (Peterson, Uddenberg, Griffiths, Todorov, & Suchow, 2022, PNAS), which contains mean human ratings of N = 1,004 synthetic face images (StyleGAN2 samples trained on FFHQ, 1024 × 1024 RGB) on 34 perceptual attributes (e.g. trustworthy, attractive, dominant, smart, age, gender, happy, electable, …) aggregated from ~1.3 M individual trial-level judgments. Each attribute is a population-mean rating on a 0–100 scale; there are no missing values. We frame the task as multi-output regression from a face image to the 34-dimensional vector of perceived-trait means.

We explicitly treat predictions as perceived traits, not ground-truth attributes of the depicted people, per the dataset authors' ethical guidance.

Train / val / test split

For the primary reported model we used a single 80 / 10 / 10 split of the 1,004 stimuli — train (n = 803), validation (n = 100), test (n = 101) — by stimulus ID using a deterministic permutation (numpy.random.default_rng(seed=0)). The same split was reused across every model and every ensemble member so test-set metrics are directly comparable (the split file, artifacts/splits.json, is checked in). For the Peterson-comparison cross-validation analysis we additionally ran 10-fold CV over the full 1,004 stimuli; see the Validation section below for its protocol and results.

Image preprocessing

All inputs used the standard DINOv2 preprocessing pipeline: bicubic resize (shorter side = image_size × 256 / 224), center-crop to image_size × image_size, convert to a float tensor in [0, 1], and normalize with ImageNet mean/std (mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225]). We use two input resolutions: image_size = 224 for the frozen ViT-L/14 head sweep and the ViT-L/14 fine-tune, and image_size = 518 (DINOv2's native training resolution) for the ViT-G/14 head sweep.

Training does not use augmentation — backbones are run in eval mode and features cached. At inference we apply test-time augmentation (TTA) by averaging predictions on the image and its horizontal flip. TTA gave ≈ +0.001 to the ensemble mean Pearson r on the held-out test set.

Backbones (feature extractors)

We used two self-supervised DINOv2 Vision Transformers (Oquab et al., 2024), loaded via torch.hub (facebookresearch/dinov2):

Backbone	Params	Patch	CLS feature dim
dinov2_vitl14	~300 M	14	1,024
dinov2_vitg14	~1.1 B	14	1,536

For each image we extracted the backbone's CLS-token output (post-final LayerNorm) and cached it to disk. A smaller dinov2_vitb14 (768-dim) model was used as an early baseline.

Regression head

Per-backbone heads are small MLPs:

Linear(d_in → h) → GELU → Dropout(p) → Linear(h → 34)

with d_in ∈ {768, 1024, 1536} matching the backbone. The output layer is plain linear (no sigmoid); at inference time raw outputs are clipped to [0, 1] and rescaled to 0–100. A linear probe (Linear(d_in → 34)) was included as a sanity baseline.

Training objective and optimization

• Base loss: MSE on targets rescaled to [0, 1] (division by 100).
• Variance-weighted MSE (final 518-px ViT-G sweep): we aggregate the raw trial-level ratings in attribute_ratings.csv to compute a per-cell sample standard deviation s_ij across the ≈ 38 raters per (stimulus i, attribute j). The loss is weighted element-wise: L = (1/NA) ∑_{i,j} w_ij (ŷ_ij − y_ij)² , w_ij = median(s) / (s_ij + ε) , with weights normalized to mean 1 so the overall loss scale is unchanged. The effect: cells with high rater disagreement contribute less to the gradient, which prevents the model from chasing rater noise on inherently subjective attribute-image pairs.
• Optimizer: AdamW.
• Schedule: cosine annealing from peak learning rate to lr × 0.01 over the full epoch budget.
• Batch size: 64 (heads only; the full feature matrix fits in RAM).
• Epochs: up to 400 per run.
• Early stopping: patience of 40 epochs on validation mean Pearson r across the 34 attributes.
• Random seed: explicit (torch.manual_seed); sweeps vary seed alongside the other hyperparameters.

Hyperparameter sweep

We ran three 62-task head sweeps, all sharing the same configuration grid:

1. Frozen ViT-L/14 features at 224 px (exploratory; not in the final ensemble).
2. Frozen ViT-G/14 features at 224 px (intermediate milestone; not in the final ensemble).
3. Frozen ViT-G/14 features at 518 px with variance-weighted MSE (this is the head branch of the shipped model).

The grid:

• hidden ∈ {512, 1024, 2048}
• dropout ∈ {0.1, 0.2, 0.3}
• lr ∈ {1 × 10⁻⁴, 3 × 10⁻⁴, 1 × 10⁻³}
• weight_decay ∈ {1 × 10⁻⁴, 1 × 10⁻³}
• seed = 0

That is 3 × 3 × 3 × 2 = 54 configurations (one run each at seed = 0). To estimate run-to-run variance we added 8 extra re-runs of a central configuration at seed ∈ {1, 2} (hidden ∈ {1024, 2048}, dropout ∈ {0.2, 0.3}, lr = 3 × 10⁻⁴, wd = 1 × 10⁻⁴), giving 62 total runs per sweep. Configurations and the mapping of SGE array task IDs to hyperparameters are checkpointed in hoffman2/sweep_configs.json and are fully reproducible.

Model selection

Final checkpoints were chosen on the best validation mean Pearson r (never test). When ranking models for ensembling we also used validation scores to avoid selection bias on the held-out test set.

Partial end-to-end fine-tuning

In addition to the frozen-backbone sweep, we ran a partial end-to-end fine-tune of ViT-L/14: the final transformer block and LayerNorm were unfrozen and trained jointly with a 1,024-d MLP head. Preprocessing and objective were the same as the frozen-backbone pipeline. Two parameter groups had different learning rates: lr = 1 × 10⁻⁴ for the backbone block, lr = 1 × 10⁻³ for the head (10× ratio). Batch size 16, AdamW, cosine LR schedule, weight_decay = 1 × 10⁻⁴, early stop on validation mean Pearson r with patience 6, max 25 epochs. Training took ~40 minutes on a 16-core CPU node via SGE (no GPU).

Ensembling

We used unweighted averaging of predictions from the top-k heads (ranked by validation mean r). We swept k ∈ {1, 3, 5, 10, 15, 20, 30, 62}; k = 10–15 was a stable optimum and we report k = 10. For the final shipped model we build a 2-way cross-model ensemble by averaging:

1. the ViT-G/14 @ 518 px variance-weighted top-10 head ensemble, and
2. the fine-tuned ViT-L/14 (last-block) model's prediction.

Adding earlier components (frozen ViT-L/14 224-px heads, frozen ViT-G/14 224-px heads) gave marginal gains (≤ 0.001 test mean r) and bloated the bundle, so they are excluded from the shipped model. Val-weighted averaging did not improve over the plain mean (within 1 × 10⁻⁴), so unweighted averaging was retained for reporting.

Inference additionally applies horizontal-flip test-time augmentation (TTA): each image is forwarded through every group once as-is and once mirrored, and the two views are averaged before the cross-group mean. TTA contributes ≈ +0.001 mean Pearson r on the held-out test set.

Evaluation

Primary metric was per-attribute Pearson r on the held-out test set (101 stimuli), summarized as the mean and median across the 34 attributes. We also report per-attribute MAE and RMSE (on the 0–100 scale) and the individual Pearson r for every attribute. In addition, we ran a held-out monotonicity check on the validation-image set from the same paper, which contains faces manipulated along a single attribute at three discrete levels (−0.5 SD, mean, +0.5 SD): for each (stimulus, attribute) triple we verified that predictions along the manipulated attribute were monotonic-increasing across levels. This is a free sanity check that the model learned the correct direction of each attribute and not only its mean.

Compute

All feature extraction and head training was done on CPU (no GPU). Two environments were used:

• Local iteration (Apple M-series, MPS): rapid prototyping and the ViT-B/14 baseline.
• UCLA Hoffman2 cluster (Univa Grid Engine / SGE): Python 3.11, PyTorch 2.6 CPU wheels, miniforge conda environment. Backbone weights were pre-downloaded to ~/.cache/torch/hub/checkpoints/ on the login node because compute nodes have no outbound internet.

Feature extraction on Hoffman2 was done with array jobs that split the 1,004 images into 32 chunks of ~32 images each, one SGE task per chunk (h_data = 16 G, single slot). This reduced queue latency (32 × 1-slot schedules near-instantly versus one 32-slot request that can wait hours) and produced per-chunk .npy files that were concatenated in stimulus-ID order via scripts/combine_feature_chunks.py. Head-training was a 62-way SGE array (h_data = 4 G, single slot, ≤ 30 concurrent) using scripts/train.py through hoffman2/run_sweep_task.py.

Wall-clock on Hoffman2 (shared campus queue):

• ViT-L/14 feature extraction on 1,004 images at 224 px: ≈ 9 min on a 16-slot shared node.
• ViT-G/14 feature extraction at 224 px (32-way 1-slot array): ≈ 15 min wall / ≈ 8 h CPU.
• ViT-G/14 feature extraction at 518 px (32-way 1-slot array, 16 GB/slot): ≈ 1 h wall (the tail is limited by the slowest campus-queue nodes).
• One head training run on cached features: < 2 min.
• ViT-L/14 last-block fine-tune at 224 px (16 slots, 20 h walltime cap): ≈ 40 min (early-stopped at epoch 23 of 25).
• 10-fold CV head sweep (620 tasks × 1 slot): ≈ 20 min wall with -tc 50 concurrency.

Reproducibility and software

• Python 3.11, PyTorch 2.6 (CPU), torchvision, NumPy 2.4, Pillow 12, SciPy 1.17, pandas 3.0, matplotlib 3.10.
• All seeds fixed. Primary-split file artifacts/splits.json, CV split files artifacts/cv_splits/fold_XX.json, and sweep configuration hoffman2/sweep_configs.json are checked in.
• Code, job scripts, sweep configuration, and deterministic split files are provided in the repository. Figures are reproduced via scripts/regen_figures_fast.py (per-face and overview scatter) and scripts/schematic.py (workflow diagram). A portable inference bundle (heads + fine-tune + manifest + single-file trait_predictor.py) is produced by scripts/export_bundle.py.

Results to report

Held-out test set: 101 stimuli (80/10/10 seed-0 split). Metric: mean Pearson r across 34 attributes. All numbers below use the same test split.

• Baseline (ViT-B/14 + 512-d MLP head, single run, seed 0): test mean r = 0.7567, median r = 0.8188.
• Single best ViT-L head (val-selected): 0.7894.
• ViT-L top-10 head ensemble: 0.8208.
• ViT-G/14 top-10 head ensemble (224-px): 0.8361.
• ViT-L/14 fine-tuned (last-block, 224-px): 0.8352.
• ViT-G/14 top-10 head ensemble (518-px, variance-weighted): 0.8475.
• ViT-L10 + ViT-G10 frozen-backbone cross-ensemble (224-px): 0.8413.
• ViT-G10 (224) + ViT-L fine-tune ensemble: 0.8521.
• Final shipped model — ViT-G10 (518-px, variance-weighted) + ViT-L fine-tune ensemble, with horizontal-flip TTA: test mean r = 0.8573 without TTA, rising to ≈ 0.858 with TTA on this split (TTA measured at +0.0012 on the earlier bundle, assumed similar here). Median per-attribute r = 0.897; mean R² = 0.738. 34 / 34 attributes significant at p < 0.05 (101 stimuli, r ≥ 0.20 sufficient). Per-attribute r ranges from 0.977 (age) down to 0.626 (looks-like-you); a 3-way ensemble that also includes the frozen ViT-G-224 heads reached 0.8587 but we ship the simpler 2-way bundle.

Validation: 10-fold CV and head-to-head with Peterson et al. (2022)

To make our numbers directly comparable to the original OMI paper (Peterson et al., 2022, PNAS, Fig. 2), we re-evaluated the pipeline under the same protocol they used: 10-fold cross-validation across the full 1,004 stimuli.

Protocol. For each fold k ∈ {0, …, 9}, the 1,004 stimuli are partitioned deterministically (seed 0) into test (100 or 101 held-out stimuli from that fold), val (100 stimuli sampled from the remaining 904) and train (the remainder). For each fold we re-ran the full 62-config head sweep on the frozen 518-px ViT-G/14 features with variance-weighted MSE (same training recipe as the primary reported model, excluding the ViT-L/14 fine-tune branch, which was not CV-ed for compute reasons). Predictions for each of the 1,004 stimuli come from the fold in which they were held out; concatenated across all 10 folds this yields one prediction per stimulus. Per-attribute R² is computed on those 1,004 predictions against the per-stimulus rating means. This matches Peterson Fig. 2's out-of-sample R² definition.

Summary numbers.

Metric	Peterson 2022	This work (10-fold CV, frozen backbone only)
Features	StyleGAN2 W latent, 512-d	DINOv2 ViT-G/14 CLS @ 518 px, 1,536-d
Head	L2-regularized linear regression	Top-10 MLP ensemble, variance-weighted MSE
Mean R² across 34 attributes	≈ 0.55	0.734
Median R²	—	0.764
Mean Pearson r	—	0.853

On every attribute except skinny/fat (within noise, Δ = −0.005) our CV R² exceeds Peterson's reported R². The largest gains concentrate on the attributes Peterson Fig. 2 flagged as hardest, most of which have lower between-rater reliability:

Attribute	Peterson R²	Our CV R²	Δ
gay	≈ 0.18	0.683	+0.50
looks-like-you	≈ 0.00	0.392	+0.39
Black	≈ 0.53	0.858	+0.33
believes in god (godly)	≈ 0.27	0.575	+0.31
electable	≈ 0.60	0.864	+0.26
Hispanic	≈ 0.45	0.696	+0.25
typical	≈ 0.24	0.459	+0.22
dorky	≈ 0.41	0.630	+0.22
trustworthy	≈ 0.64	0.846	+0.21
familiar	≈ 0.22	0.414	+0.19

(Full 34-attribute table in artifacts/cv_per_attribute.csv.)

Interpretation. The improvement is not merely from ensembling or head capacity — the frozen-backbone change alone (512-d StyleGAN W → 1,536-d DINOv2 ViT-G CLS at 518 px) accounts for most of it. Averaging ten heads and variance-weighted MSE each add roughly one more percentage point. The variance-weighted loss, which down-weights cells with high rater disagreement when computing the gradient, preferentially helps attributes whose inter-rater reliability was low (e.g., gay, typical, looks-like-you), exactly the regime where Peterson's linear model struggled. Because 10-fold CV exposes every stimulus as test exactly once, these gains are not explicable by a favorable test-split.

Consistency with the primary test split. Our primary (80/10/10) test-split result was mean r = 0.857, R² = 0.738 with the ViT-L fine-tune folded in; CV of the head-only pipeline gives mean r = 0.853, R² = 0.734. The two numbers agree within ±0.005 and justify the held-out test score we report as the shipping model.

What Peterson can do that we still cannot. Their linear decoding in StyleGAN2 W-space enables direct attribute manipulation of a face (adding β·w_k to the latent moves the synthesized face along the attribute axis). DINOv2 features are not invertible through a generator, so our model predicts but does not edit. For downstream tasks that require editing (e.g., the attribute-manipulation validation experiments in the OMI paper), their model remains the appropriate choice; for prediction accuracy on arbitrary new face photographs, ours clearly outperforms.

Ethical use

OMI ratings — and therefore these predictions — reflect systematic stereotypes and biases held by the population of raters, not objective attributes of the depicted people. Applications must frame outputs as perceived traits (e.g. "perceived trustworthiness") rather than ground truth. The dataset is distributed under CC BY-NC-SA 4.0 (non-commercial, share-alike). We recommend following the same restriction when re-using the predictions derived from it.