progress/phase_2a_resnet_classifier.md

Phase 2A — ResNet50 multi-label damage-type classifier + XGBoost(A)

Status: ✅ Done (scaffold + verified end-to-end on MPS; full training is a sequence of CLI commands) Completed: 2026-05-13

Goal

Build the full Variant A stack: a ResNet50 multi-label classifier over the 6 CarDD damage types, a 2048-d feature extractor that streams every CarDD image into a parquet cache, a synthetic cost-target generator that joins iaai metadata distributions to a catalog-derived per-image cost, and an XGBoost(A) regressor on top. Wrap it in VariantAPipeline so a single image + optional metadata produces a complete prediction (damage types, parts, cost, tier, provenance).

Deliverables

• src/ccdp/data/damage_dataset.py — multi-hot label encoding, deterministic 80/10/10 splits, inverse-frequency pos_weight.
• src/ccdp/models/damage_classifier.py — ResNet50 + dropout/512/dropout/6 head, set_finetune_stage, extract_features.
• src/ccdp/train/train_damage_classifier.py — two-stage trainer, BCE+pos_weight, per-class P/R/F1, macro/micro F1, full resume.
• src/ccdp/train/extract_features.py — runs trained backbone over all CarDD images, writes parquet with image_id, split, damage_types, f_0..f_2047.
• src/ccdp/train/synthesize_cost.py — samples metadata from iaai pool, maps damage types → parts, computes catalog-driven cost with age + noise factors. Outputs cost_source = synthetic@<catalog_id> per row for traceability.
• src/ccdp/models/xgb_regressor.py — XGBRegressorBundle (feature schema + training catalog id) + make_feature_matrix for reproducible one-hot encoding at inference.
• src/ccdp/train/train_xgb.py — joins features+targets on image_id, trains XGBoost(A), reports RMSE/MAE/MAPE/R² on val and test, saves best.ubj + bundle.json.
• src/ccdp/infer/variant_a.py — VariantAPipeline (image → classifier → optional XGBoost → cost). Falls back gracefully to the three-tier catalog estimator when XGBoost or metadata is unavailable. Currency-converts via the FX module.
• CLI: ccdp train classifier, ccdp train extract-features, ccdp train synth-targets, ccdp train xgb, ccdp infer.
• Tests: tests/test_damage_classifier.py — 54 / 54 passing.
• End-to-end smoke verified on MPS.

Smoke-run results (MPS, M-series 16GB)

Classifier (1 + 1 epochs of 25 batches × 16):

metric	value
stage 1, ~9 s train + 6 s val on 400 samples	val macro-F1 = 0.61
stage 2, ~10 s train + 6 s val on 400 samples	val macro-F1 = 0.68, micro-F1 = 0.68
pos_weight (auto)	dent 1.3, scratch 1.0, crack 5.6, glass_shatter 5.1, lamp_broken 5.0, tire_flat 12.2
trainable params	stage 1: 1,052,166 • stage 2: 23,115,270

Feature extraction on 8 batches × 32 (768 images total): ~14 s. Full corpus (4000 images): ~75 s extrapolated.

XGBoost(A) on those 768 features:

split	RMSE	MAE	MAPE	R²
val	$392.99	$283.92	51.6%	0.50
test	$472.70	$323.25	55.6%	0.53

End-to-end ccdp infer on a real CarDD val image, user-supplied metadata --make honda --model-name civic --year 2018 --body-type sedan:

damage_types: ["dent", "scratch"]
cost_usd:     $642.31
tier:         "exact"
provenance:   "xgb_a(exact); training_catalog=2026-05-12T05-45-11_initial;
              calibrated to 2026-05-12T05-45-11_initial"
probabilities: {dent: 0.55, scratch: 0.72, crack: 0.41,
                glass_shatter: 0.08, lamp_broken: 0.31, tire_flat: 0.02}

How to verify

source .venv/bin/activate
pytest -q                                                # 54 / 54 passing

# One-time macOS env (torchvision weights + xgboost OMP)
export SSL_CERT_FILE=$(python -c "import certifi; print(certifi.where())")
export DYLD_LIBRARY_PATH=$(python -c "import torch, os; print(os.path.join(os.path.dirname(torch.__file__),'lib'))")

# Smoke (~30 s total)
ccdp train classifier --epochs-stage1 1 --epochs-stage2 1 --batch-size 16 \
    --num-workers 0 --smoke-batches 25 --tag smoke
CLS=$(ls checkpoints/classifier/run_*smoke*/best.pt | head -1)
ccdp train extract-features --checkpoint "$CLS" --batch-size 32 --num-workers 0 --smoke-batches 8
ccdp train synth-targets
ccdp train xgb --n-estimators 80 --max-depth 5 --tag xgb_a_smoke

# Inference
IMG=$(find data/raw/car-damage-detection -name '*.jpg' | head -1)
ccdp infer "$IMG" --make honda --model-name civic --year 2018 --body-type sedan

# Full training (~25 min wall-clock target):
ccdp train classifier --epochs-stage1 3 --epochs-stage2 12 --batch-size 32 --num-workers 4
ccdp registry promote <classifier_run_id> classifier
ccdp train extract-features
ccdp train synth-targets
ccdp train xgb --n-estimators 600 --max-depth 7 --learning-rate 0.05 --tag xgb_a_v1
ccdp registry promote <xgb_run_id> xgb_a

Honest disclosure (relevant for the report)

• The cost target is synthetic, derived from Catalog.estimate(parts_with_severity, segment) × age factor × ±10% Gaussian noise per row. There is no real per-image repair invoice data available publicly (see progress/phase_1_data_and_identification.md and PLAN.md §3).
• Tabular features (make/model/year/body_type) are sampled from the iaai metadata distribution, not derived from each CarDD image. They give XGBoost realistic feature correlations but they do not describe the specific car in the image.
• Each prediction records training_catalog_id and bundle_run_id in its provenance string. When the catalog is updated later, the Calibrator scales output by active.median / training.median so the trained XGBoost remains useful without retraining.
• Variant A produces no part localization — bbox-derived features arrive in Variant B (Phase 2B). The parts: [] field in inference responses is honest about this limitation; XGBoost still produces a calibrated cost.

Pending

None blocking. Two follow-ups deferred:

• Promotion auto-eval is still the simple symlink flip from Phase 1.5 — the A/B comparison harness lives with Phase 4.
• The training-time catalog is the only one ever seen by XGBoost. Until Phase 4, calibration coverage is small (one catalog). When a real second catalog lands, validate the scaling factor against a held-out test set before relying on it.

Notes for future phases

• Phase 2B (YOLOv8 detector) shares XGBRegressorBundle — just appends bbox-stat features (n_damage_regions, total_damaged_area_pct, per-part area dict) to the same image-feature row and trains a separate XGBoost(B). The Variant A → B comparison in Phase 3 reuses the metrics already produced by train_xgb.py.
• Phase 3 (comparison + serving) will read both registered runs (classifier + xgb_a) and run side-by-side eval on the held-out test split that damage_dataset.split_records(..., seed=42) produces — already deterministic so both variants compare on the same images.
• Phase 4 (promotion + continued training) will gate ccdp registry promote behind an A/B test against the current production model. The hook is already in place: production_target('classifier') and production_target('xgb_a') return the active symlink target.