projectsidewalk/sidewalk-tagger-ai-obstacle

Project Sidewalk Tagger: Obstacle

A multi-label classifier that predicts which accessibility tags apply to a Obstacle label in a crop of a Google Street View image. Project Sidewalk serves it through sidewalk-ai-api to suggest tags for crowdsourced labels.

This is the DINOv2 model trained on the cleaned dataset in Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset (ASSETS '24). Training and evaluation code is in sidewalk-tagger-ai.

Provenance

validated-dino-cls-b-obstacle-tags-best.pth is a byte-for-byte copy of the file of the same name in projectsidewalk/sidewalk-tagger-ai-models at commit 65959dbc80b87e4f39385204c4b639cbcf58e1a8, which holds every model from the paper and remains available unchanged.

sha256: da4f7ae0bb21d1a5b13a785ebc8553dcd8746eb823183f5c53332fe078009b6d

Model

• Backbone: DINOv2 ViT-B/14 with 4 register tokens (dinov2_vitb14_reg4), fine-tuned.
• Head: Linear(768, 256) → ReLU → Linear(256, 20), with a sigmoid per class.
• Input: RGB crop resized to 256 px and padded to a multiple of 14, normalized with ImageNet mean and std.
• Format: a PyTorch state dict; it loads with torch.load(..., weights_only=True).

The architecture is custom rather than a transformers class. See DinoVisionTransformerClassifier and ImageTagger in sidewalk_ai_api/tagger.py for loading and preprocessing.

Classes

The output has 20 classes, in this order. NULL entries are unused placeholders that preserve the class indices from training.

Index	Tag
0	construction
1	fire hydrant
2	NULL
3	height difference
4	litter/garbage
5	narrow
6	NULL
7	parked bike
8	parked car
9	parked scooter/motorcycle
10	pole
11	sign
12	stairs
13	NULL
14	trash/recycling can
15	tree
16	vegetation
17	mailbox
18	seating
19	uneven/slanted

Thresholds

Sigmoid scores aren't calibrated probabilities. The per-tag thresholds Project Sidewalk uses to suggest that a tag is present or absent are in tag_thresholds.json in sidewalk-ai-api.

Evaluation

See the paper for per-tag results on the test split of sidewalk-tagger-ai-validated. REPRODUCE_RESULTS.md in sidewalk-tagger-ai reproduces them.

Citation

@inproceedings{10.1145/3663548.3688531,
  author = {Liu, Xinlei and Wu, Kevin and Kulkarni, Minchu and Saugstad, Michael and Rapo, Peyton Anton and Freiburger, Jeremy and Hosseini, Maryam and Li, Chu and Froehlich, Jon E.},
  title = {Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset},
  year = {2024},
  publisher = {Association for Computing Machinery},
  doi = {10.1145/3663548.3688531},
  booktitle = {Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility},
  articleno = {103},
  numpages = {12}
}