01 End-to-end code flow
One photo flows through four models. A car-presence gate runs first; then a make/model identifier and a damage model run; their signals feed an XGBoost cost regressor that is calibrated to a swappable parts-price catalog and converted to the chosen currency. The dots below animate the data actually moving between stages.
02 How the data is parsed
Two datasets, two parsers, one canonical pipeline. Stanford Cars trains the identifier; CarDD (COCO format) trains the damage models. Every image is decoded, cropped, augmented, tensorised and normalised to ImageNet statistics before it touches a network.
Real samples from the data
Left: Stanford Cars (make/model/year). Right: CarDD (damage types). Thumbnails embedded from the local datasets.
Datasets at a glance
| Dataset | Role in ccdp | Size | Classes | Annotation | License |
|---|---|---|---|---|---|
| Stanford Cars Krause et al. 2013 |
Make/model/year identifier | 16,185 imgs | 196 (make·model·year) | bbox + class (.mat devkit) | research / academic |
| CarDD Wang et al. 2023 |
Damage classifier (A) + detector (B) | ~4,000 imgs | 6 damage types | COCO-format bbox + masks | academic (verify) |
These two are the only datasets trained on. CarDD's 6 damage types:
dent · scratch · crack · glass_shatter · lamp_broken · tire_flat. (CarDD ships its labels in
COCO format — that's the file format, not the COCO dataset.) Full attributions in
the Citations section.
Pretrained weights — not a training set: the car gate uses torchvision's
COCO-pretrained Mask R-CNN (weights only), run at inference to find car/truck/bus.
Nothing in this project downloads or trains on the COCO dataset.
Stanford Cars → identifier
Class names and per-image boxes live in MATLAB .mat devkit files, read with
scipy.io.loadmat. Each car is cropped to its annotation box, then transformed. A deterministic
stratified 90/10 split makes train/val.
CarDD → damage models
COCO JSON (instances_*2017.json) is parsed into canonical Records. Variant A
uses the whole image with a multi-label target; Variant B uses normalised YOLO boxes derived from the
COCO bboxes.
The transform pipeline (training)
03 Network architecture
The identifier and the Variant A damage classifier share the same skeleton: an ImageNet
ResNet-50 backbone (conv1 → layer1…layer4 → avgpool) with the final fully-connected layer
replaced by a small custom head. The two-stage fine-tune controls which blocks receive gradients.
Custom head — identifier (196 classes)
The 2048-d vector after avgpool is also reused as a frozen feature extractor for
the XGBoost cost regressors.
Custom head — damage classifier A (6 types)
Multi-label: each damage type fires independently, so the output is a sigmoid per class, not a softmax.
Variant B — YOLOv8 detector
A single-stage detector: a CSP-Darknet backbone + PAN neck + a decoupled head predicts boxes, objectness and the 6 damage classes in one pass. Trained on CarDD via Ultralytics.
The car gate — COCO Mask R-CNN (ResNet-50-FPN)
Same ResNet-50 family, wrapped in a Feature Pyramid Network + Region Proposal Network. Used
only at inference to find car/truck/bus — masks are ignored, the box is used to crop.
04 Activation functions
Four non-linearities appear across the models. Hover a plot to read its value.
ReLU hidden layers (ResNet + heads)
Sigmoid Variant A multi-label output
Softmax identifier output (196-way)
SiLU YOLOv8 detector (internal)
05 How the model is trained
The CNNs use a two-stage transfer-learning recipe; YOLOv8 trains end-to-end via Ultralytics. Below: the freeze schedule, optimisers, and the augmentation that fights overfitting on small datasets.
Optimisers
Identifier / classifier (A)
AdamW, lr 1e-3 → 1e-4, wd 1e-4, ReduceLROnPlateau
Detector (B)
Ultralytics SGD/AdamW auto, cosine LR, mosaic aug
Regularisation
Dropout (0.2–0.4), weight decay, label smoothing 0.1, gradient clipping (‖g‖ ≤ 5.0), early stopping on val metric.
Augmentation
RandAugment(2 ops, mag 9), MixUp(α=0.2) & CutMix(α=1.0) per-batch with probability 0.8 — targets become soft labels.
MixUp / CutMix — what the network actually sees
Two images and their labels are blended (MixUp) or one is pasted into the other (CutMix); the loss is computed against the mixed soft target — which is why training uses a soft cross-entropy.
06 Backpropagation logic
PyTorch autograd records the forward graph; loss.backward() walks it in reverse,
multiplying local gradients via the chain rule. Frozen blocks (Stage 1) get no gradient; gradients are clipped
and applied by the optimiser.
The chain rule, per layer
Stage 1 freezes layer1…layer4 (requires_grad=False) so the product stops at the head.
Stage 2 unfreezes layer3+layer4 and gradients flow deeper.
Stabilisers in the loop
| Step | Call |
|---|---|
| Zero grads | optimizer.zero_grad() |
| Backward | loss.backward() |
| Clip | clip_grad_norm_(…, 5.0) |
| Step | optimizer.step() |
| Schedule | scheduler.step(val_loss) |
07 Loss / error functions
Each task minimises a different objective. The plots show how loss grows as the prediction drifts from the truth.
Identifier — Cross-Entropy (+ label smoothing)
Damage classifier A — Binary Cross-Entropy
Detector B — YOLOv8 multi-part loss
Box regression uses Complete-IoU, classification uses BCE per class, and distribution focal loss sharpens box edges. Ultralytics sums them per image.
XGBoost cost — squared error
The target is a synthetic, catalog-derived repair cost; the booster fits residuals stage by stage with early stopping on validation RMSE.
08 How accuracy is measured
Different heads need different metrics. These are the production numbers reported in the repo.
| Model | Metric | Why this metric | Value |
|---|---|---|---|
| Make/model identifier | Top-1 accuracy | single correct class out of 196 | 77.0% |
| Damage classifier (A) | Macro-F1 | multi-label, class-imbalanced | 0.834 |
| Damage detector (B) | mAP@50 / @50-95 | localisation quality | 0.687 / 0.540 |
| Cost regressor XGBoost(A) | R² · MAPE | regression fit & % error | 0.642 · 32.9% |
| Cost regressor XGBoost(B) | R² · MAPE | + bbox features | 0.736 · 24.4% |
09 The final output & how the image is shown
The Gradio app returns an annotated image plus structured text. The original photo is redrawn with a green car-box (from the gate) and coloured damage boxes (Variant B); a JSON blob carries full provenance. The boxes below draw in sequence to mirror how the result is assembled.
Annotated image (animated)
Returned payload
If the identifier's confidence is below the 0.30 floor, make becomes
null and the cost falls back to body-type/segment pricing instead of a wrong label.
10 Citations & credits
This project is built entirely on others' datasets and tools. Full credit to the sources below;
the project's CITATIONS.md carries the complete list and license notes.