arkodeep/celeb-fbi-weight

Celeb-FBI body-weight from a single image — best config A(ViT)+C (7.04 kg MAE)

Research artifact — do not use for medical, fitness, or individual assessment. Trained on celebrity images; under-predicts heavier subjects.

Files: vit_b16_best.pt (best backbone), resnet50_best.pt (baseline), branch3d_calib.json (backbone-aware 3D-volume calibration).

Download & run the best model — A(ViT)+C

git clone https://github.com/arkodeepsen/weight-from-image && cd weight-from-image
pip install -r requirements.txt
huggingface-cli download arkodeep/celeb-fbi-weight vit_b16_best.pt --local-dir ckpts
# --height-cm enables the 3D volume refinement; predict.py auto-detects the ViT
# backbone and applies the matching A(ViT)+C meta-learner from reports/branch3d_calib.json
python -m src.predict --ckpt ckpts/vit_b16_best.pt --image photo.jpg --height-cm 175

Model Card — Human Weight Prediction from a Full-Body Image

Project: DeLTA26_041 (IIIT Allahabad research internship) · Task: single-image regression → body weight (kg) · Status: research prototype.

---

1. Model details

• Architecture. ResNet-50 (ImageNet-pretrained) trunk + regression head Linear(2048→256) → ReLU → Dropout(0.3) → Linear(256→1).
• Hybrid variants.
  • Feature fusion (FusionRegressor): CNN 2048-d penultimate features concatenated with a small MLP over a 7-d anthropometric vector → shared head.
  • Late fusion: per-branch predictions (CNN, XGBoost-on-anthro, volume prior) combined by a linear meta-learner → A+B, A+C, A+B+C.
• Branch C — 3D body volume (best config). Estimates a body volume from the pose landmarks (elliptical-frustum torso, cylinder limbs, ellipsoid head), anchors it to metres with the subject's height, and calibrates `w = 300.6·V
  • 43.2(the practical form ofw ≈ ρV). The only branch supplying *absolute* scale; an exact SMPL-mesh volume backend is scaffolded. On a ResNet-50 backbone A+C reaches 7.82 kg; on the strongest backbone (ViT-B/16) the same volume fusion — **A(ViT)+C`** — is the best model overall at 7.04 kg (§4). Inference auto-detects the checkpoint's backbone and applies the matching A+C meta-learner.
• Anthropometric features (7-d). Six scale-invariant pose features — shoulder-width/torso, hip-width/torso, shoulder/hip ratio, leg-length/torso, arm-length/torso, bbox width/height — plus a pose_valid flag. Extracted with MediaPipe PoseLandmarker (Tasks API, 33-point BlazePose). Normalising by torso length makes them invariant to camera distance (a single image has no absolute scale).
• Training. Huber (Smooth-L1) loss; AdamW with split LRs (head 1e-4, backbone 1e-5, weight-decay 1e-4); cosine schedule; AMP on CUDA; early stopping on validation MAE (patience 7); batch 64; ≤40 epochs; input 224×224 with ImageNet normalisation. Targets standardised (mean 65.8, std 16.0 kg); metrics inverted to kg.
• Frameworks. PyTorch / torchvision, MediaPipe, scikit-learn, XGBoost.

2. Intended use

• Intended. Research and education — studying how far appearance + body-shape cues can estimate weight from one image; coursework / IEEE write-up.
• Out of scope (do not use for). Any medical, clinical, insurance, fitness, employment, or individual assessment. Outputs are point estimates with large per-person error (see §5) and a known directional bias against heavier bodies.

3. Data

• Source. Celeb-FBI (alecccdd/celeb-fbi) — 7,208 full-body celebrity photos.
• Splits. Provided test split (721) held out untouched. Training pool split 90/10 into train 5,032 / val 559, BMI-stratified and seeded (seed 42); all ablation models share this identical split.
• Labels. weight (kg, target), height (cm, for BMI), gender, age; missing values use a -1 sentinel and are dropped (units confirmed kg/cm in EDA).
• Pose coverage. Landmarks detected on 6,480 / 6,487 train (99.9%) and 721 / 721 test images, so the anthropometric branch is real signal, not imputation.
• Leakage. Per-image id checked across splits — no id shared. Residual risk remains: a celebrity may appear under multiple ids, so subject-level leakage cannot be fully excluded (documented limitation).

4. Metrics (test set, 721 images)

Evidence ablation (backbone fixed at ResNet-50; isolates each evidence source):

Model	MAE (kg)	RMSE (kg)	Pearson r	R²
A — CNN only	8.11	12.51	0.635	0.402
B — Anthropometric only (XGBoost)	9.73	13.84	0.520	0.267
C — 3D volume prior (calibrated)	10.02	—	0.49	—
A+B — feature fusion	8.01	12.49	0.641	0.403
A+B — late fusion	8.35	12.22	0.657	0.429
A+C — CNN + volume	7.82	11.93	0.676	0.455
A+B+C — all three	8.08	11.95	0.675	0.454

Equal-footing backbone study (same pipeline, 100 epochs; only the backbone changes):

Backbone	Params (M)	MAE	RMSE	r	R²	BMI-acc
ViT-B/16	86.0	7.12	11.32	0.715	0.510	0.644
ResNet-50 (this work / Debnath)	24.0	8.09	12.49	0.636	0.403	0.624
EfficientNet-B0	4.3	8.15	12.58	0.633	0.395	0.595
MobileNetV3-L	3.2	8.42	13.01	0.600	0.352	0.611
ViT-B/16 + volume — A(ViT)+C (best, deployed)	86.0	7.04	10.96	0.736	0.540	0.618

Primary metric is MAE. Two orthogonal levers: the volume branch (evidence) improves every backbone (ResNet 8.11→7.82, ViT 7.12→7.04), and a stronger backbone (ViT-B/16) is worth ~1 kg on its own. Combining both, A(ViT)+C = 7.04 kg is the best model overall. 2D anthropometry (Branch B) stays redundant with a strong appearance model; BMI-band accuracy ≈0.62 across models.

5. Error analysis & limitations

Error by BMI band — CNN (A) vs the volume-fused A+C:

Band	A (CNN)	A+C	n
underweight	7.1	7.0	107
normal	6.2	6.1	381
overweight	13.9	12.3	66
obese	14.7	14.7	66

• Dominant failure: heavy subjects are under-predicted. The worst-20 CNN cases are large/muscular men (true 102–109 kg) predicted 70 kg — 30+ kg under. The model is accurate (6 kg) on the well-represented normal/underweight bands and collapses on the rare overweight/obese bands.
• The 3D volume branch is the fix — partially. A+C recovers the overweight band (13.9 → 12.3 kg) by injecting absolute size; but the obese band stays (14.7), because with only 66 obese images the volume map cannot calibrate the heavy tail either. So the bottleneck is both missing absolute scale (architectural, fixed by Branch C) and data coverage (which Branch C cannot fix).
• Why size matters. Weight fundamentally needs size, which a single image lacks; 2D scale-invariant ratios supply shape but not size (hence anthropometric-only is weakest, 9.7 kg). Branch C recovers absolute size by anchoring an estimated body volume to the subject's height — the only branch that does so.
• Other risks. Heavy clothing, unusual poses, occlusion, and extreme framing degrade both the CNN and pose features. Gender/age fairness beyond the gender/BMI breakdowns here was not exhaustively audited.

6. Ethical considerations

Inferring body weight from images is sensitive. Predictions could enable body-shaming, discrimination, or unwanted profiling. The model's directional bias means errors are not evenly distributed — it is least reliable exactly for the heavier individuals most vulnerable to such harms. This artifact must not be deployed against real people or used for any individual judgement. Any downstream use should obtain consent, disclose uncertainty, and re-validate on a representative, weight-balanced dataset.

7. Reproduce the evaluation

1. pip install -r requirements.txt
2. Obtain the best checkpoint (*_best.pt, written to the Drive cache during training) and the cached pose-feature parquets.
3. Run notebooks/97_colab_eval.ipynb (or the equivalent src.eval calls) — it regenerates the ablation table, BMI confusion matrices, error-by-group charts, and the worst-k grid into reports/.