sidewalk-validator-ai-surfaceproblem

A binary image classifier that checks one crowdsourced Project Sidewalk SurfaceProblem label: given a crop of street-level imagery centred on the label, it predicts whether the label is correct (the image shows a sidewalk surface problem (cracks, heaving, uneven or broken surface)) or incorrect. It is facebook/dinov2-large fine-tuned with a two-class head (id2label = {0: correct, 1: incorrect}) by sidewalk-validator-ai.

Intended use

Automatic validation of new SurfaceProblem labels in Project Sidewalk. sidewalk-ai-api rebuilds the crop from the panorama and label point, runs this model, and looks the verdict's confidence up in confidence_precision_map.json. SidewalkWebpage records the verdict as an Agree/Disagree vote by the SidewalkAI account only when that estimated accuracy is at least 0.92 (ai-validation-min-accuracy); otherwise it is stored as Unsure. The model is one input to label quality, not a replacement for human validation.

Out of scope: finding features in whole panoramas (that is RampNet's job), other label types, and imagery that is not a label-centred crop made the same way as the training crops.

Training data

projectsidewalk/sidewalk-validator-ai-dataset-surfaceproblem (content hash TODO: dataset not staged with this model; copy the hash from the dataset card). TODO: the linked dataset repo holds an earlier selection; whether this model's training data is published, which includes deployments that are not public, is a policy decision (issues #2 and #7). Labels are taken as correct or incorrect from human validation votes only (AI votes excluded); see the dataset card for the exact thresholds. Split sizes: train 10,500, val 2,250, test 2,255, test_heldout 847 (tucson-az and 2 non-public deployments). The split is grouped by panorama (no panorama in two splits), stratified by class, seeded, with optional whole held-out cities (split_data.py).

How the input crop is made

The panorama is resized to 8192×4096. A 2048² perspective view with a 90° field of view is rendered centred on the label (sidewalk-panorama-tools), with rig-tilt correction from the label's camera pitch and roll (beta = 0.95). Depth Anything V2 (metric, VKITTI, ViT-B) estimates the depth at the view centre; the crop half-width is 6100 / depth_m pixels, clipped to the view, and the crop's long side is resized to 640 px (training crops are stored as WebP quality 95).

Deployment order. The model expects tilt-corrected crops, so the serving API must apply the same correction (sidewalk-ai-api#17).

Preprocessing and training

Inference uses this repo's preprocessor_config.json (BitImageProcessor: resize shortest edge 256, centre-crop 224, ImageNet mean/std); load it with use_fast=False to match the API's PIL path. Training adds RandomResizedCrop(224, scale=(0.7, 1.0)) and horizontal flips; validation uses the processor itself. Hugging Face Trainer, learning rate 1e-5, batch 8 × 4 gradient accumulation, 10% warm-up, 25 epochs, seed 42. The published weights are the checkpoint with the best validation macro_f1 (0.824; checkpoint-5593); trainer_state.json keeps the full log history.

from PIL import Image
from transformers import AutoImageProcessor, AutoModelForImageClassification

processor = AutoImageProcessor.from_pretrained("projectsidewalk/sidewalk-validator-ai-surfaceproblem", use_fast=False)
model = AutoModelForImageClassification.from_pretrained("projectsidewalk/sidewalk-validator-ai-surfaceproblem").eval()
probs = model(**processor(Image.open("crop.webp").convert("RGB"), return_tensors="pt")).logits.softmax(-1)[0]
verdict = model.config.id2label[int(probs.argmax())]   # "correct" or "incorrect"

Evaluation

From metrics.json (evaluate.py), on the best checkpoint. The verdict is the argmax with ties going to correct, as in the API.

slice	n (correct / incorrect)	accuracy	balanced accuracy	AUC	macro-F1
test (cities seen in training)	2,255 (1,256 / 999)	0.832	0.830	0.896	0.830
test_heldout (held-out cities)	847 (371 / 476)	0.658	0.653	0.710	0.653
hard (test + test_heldout rows the deployed model got wrong)	40 (11 / 29)	0.475	0.384	0.323	0.389

Per class:

slice	class	precision	recall	F1	support
test	correct	0.850	0.849	0.849	1,256
test	incorrect	0.810	0.812	0.811	999
test_heldout	correct	0.607	0.617	0.612	371
test_heldout	incorrect	0.698	0.689	0.693	476
hard	correct	0.143	0.182	0.160	11
hard	incorrect	0.654	0.586	0.618	29

Recall at precision on the test split (the score threshold that achieves it in brackets):

class	P ≥ 0.90	P ≥ 0.92	P ≥ 0.95	P ≥ 0.99
correct	0.756 (≥ 0.9996)	0.697 (≥ 1.0000)	0.000 (≥ n/a)	0.000 (≥ n/a)
incorrect	0.505 (≥ 1.0000)	0.402 (≥ 1.0000)	0.000 (≥ n/a)	0.000 (≥ n/a)

Calibration

confidence_precision_map.json maps the model's softmax confidence for its verdict to the estimated probability that the verdict is right. It is fitted by calibration/calibrate.py with isotonic regression, reweighted from the training class mix to the deployment class prior, and is a drop-in for sidewalk-ai-api/accuracy_mappings/surfaceproblem.json.

Lowest confidence at which the map reaches the 0.92 gate:

verdict	confidence at the gate
correct	0.999997
incorrect	never reaches the gate

Deployment prior: correct 0.680, incorrect 0.320 (from rawLabels of amsterdam, auckland, bayonne-fr, blackhawk-hills-il, burnaby, cdmx, chandigarh-india, chicago-il, cliffside-park-nj, clifton-nj, columbus-oh, cuenca, danville-il, detroit-mi, fort-wayne-in, gainesville-fl, hackensack-nj, houston-tx, knox-oh, la-ca, la-piedad, laurens-ia, madison-wi, maywood-nj, mendota-il, newberg-or, newport-ky, niagara-falls-ny, oradell-nj, paterson-nj, pittsburgh-pa, rancagua-chile, richmond-va, santiago-chile, sao-paulo-brazil, seattle-wa, spgg, st-louis-mo, teaneck-nj, tucson-az, vancouver-wa, virden-il, walla-walla-wa, waltham-ma, washington-dc, west-chester-pa, human votes only (and 8 non-public deployments), measured 2026-10-20). Fitted on 2,250 val rows; evaluated on 2,255 test rows, reweighted to the same prior.

Overall expected calibration error (ECE): 0.0586.

verdict	n	weighted share	precision	ECE	coverage at 0.92	precision at 0.92
correct	1,254	0.637	0.906	0.0440	0.676	0.957
incorrect	1,001	0.363	0.716	0.0920	0.000	n/a

Held back: the incorrect map is capped at 0.9199: every threshold whose fitted accuracy is higher serves 0.9199 instead, so that verdict never reaches the 0.92 gate and never becomes a vote. Reason: first ship of a retrained revision: the deployment prior is under review (issue #56) and the revision has no production outcomes to fit on yet. For incorrect the cap changes nothing: the uncapped map never reached the gate either.

The map is fitted on the validation split, which also selected the checkpoint, so it is mildly optimistic.

Known failure modes and limitations

• Trained and evaluated only on Google Street View panoramas from the cities listed in the dataset card; performance elsewhere, and on other imagery, is unmeasured beyond the held-out cities above.
• Test metrics are on the training class mix (about 55:45 correct:incorrect). Production traffic is mostly correct labels, so precision on incorrect verdicts is lower in deployment than the test table suggests; the calibrated map accounts for this, the raw metrics do not.
• The hard slice is small; read its numbers with wide error bars.
• Correct/incorrect comes from crowd votes, which are themselves noisy; only decisively voted labels are used, so borderline cases are under-represented.
• A crop made without the tilt correction, or with a different depth model, is out of distribution.
• TODO: per-label-type failure modes from a review of this model's errors (e.g. the hard-example list); add them as failure_modes in card_overrides.json.

Files

file	contents
model.safetensors, config.json	weights and Dinov2ForImageClassification config
preprocessor_config.json	the image processor used in validation, evaluation and serving
confidence_precision_map.json	confidence → estimated accuracy, per verdict (the API's format)
calibration_report.json	prior, ECE and gate coverage of that map (when present)
metrics.json	the evaluation above
trainer_state.json	trimmed: log history and best checkpoint only
training_run.json	seed, epochs, package versions, generation settings (when present)

Lineage

year	work	relation
2019	Weld et al., ASSETS'19 (doi)	First deep-learning auto-validation of Project Sidewalk labels on depth-sized crops.
2022	Duan et al., ASSETS'22 (doi)	Validated training data and cross-city transfer; the direct ancestor of this pipeline.
2023	Oquab et al., DINOv2 (arXiv)	The pretrained backbone.
2024	Yang et al., Depth Anything V2 (code)	Metric depth that sizes each crop.

Published from sidewalk-validator-ai at commit e313251c2571cb2a585835f812eaac85b851b7b4 with publish_to_hf.py.

Citation

No DOI has been minted for this model yet.

There is no paper for this model yet. Please cite Project Sidewalk and link the training repository:

@inproceedings{saha2019projectsidewalk,
  author    = {Saha, Manaswi and Saugstad, Michael and Maddali, Hanuma Teja and Zeng, Aileen and Holland, Ryan and Bower, Steven and Dash, Aditya and Chen, Sage and Li, Anthony and Hara, Kotaro and Froehlich, Jon},
  title     = {Project Sidewalk: A Web-based Crowdsourcing Tool for Collecting Sidewalk Accessibility Data at Scale},
  booktitle = {Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems},
  year      = {2019},
  doi       = {10.1145/3290605.3300292}
}