README.md

metadata

title: Street Scene Vision Toolkit
emoji: 🚦
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.49.1
python_version: '3.10'
app_file: app.py
pinned: false
license: mit
short_description: Street segmentation and detection with SegFormer + YOLO.

🚦 Street Scene Vision Toolkit

Pixel-level scene understanding and object detection in one browser app

Upload once, segment every pixel, detect individual objects, and export both forms of evidence.

🚀 Launch the live app · 🎨 SegFormer · 📦 YOLO26 · 💻 Source code

---

Live demo

The public Hugging Face Space runs entirely in the browser interface. Students can upload an image, choose the UBC example, inspect the results, and download the outputs without installing Python or providing an API key.

Featured classroom example: UBC campus street

Original image	Segmentation overlay	Color mask	Object detection

This scene provides a useful teaching example because it combines vegetation, sky, terrain, paving, buildings, poles, pedestrians, and a cyclist in one image. Students can compare the predicted area shares and inspect challenging boundaries between the tree canopy and sky, or between the paved route and grass.

The SegFormer result was generated at 2048 × 1536 px with an overlay opacity of 0.55. The largest predicted pixel classes were:

Predicted class	Image share
Vegetation	42.87%
Sky	20.86%
Road	20.73%
Terrain	11.50%
Building	1.76%
Sidewalk	1.07%

At a 0.25 confidence threshold, YOLO26-s detected five visible objects:

Detected class	Count	Average confidence	Maximum confidence
Person	4	0.633	0.839
Bicycle	1	0.335	0.335

What the project does

The app provides two complementary computer-vision methods on the same image:

• nvidia/segformer-b0-finetuned-cityscapes-1024-1024 assigns a Cityscapes class to every image pixel.
• YOLO26-s detects separate COCO objects and returns labeled bounding boxes with confidence scores.

For each image, the app produces:

Output	Purpose
Segmentation overlay	Keeps the original scene visible beneath the predicted classes
Cityscapes color mask	Shows a clean, standardized categorical map
Class coverage table	Reports class ID, pixel count, area share, and color
Raw class-ID PNG	Preserves the numeric prediction for later analysis
CSV summary	Supports statistics, plotting, and classroom exercises
Detection overlay	Locates individual people, vehicles, bicycles, and other COCO objects
Object summary	Reports counts and confidence by detected class
Street indicators	Summarizes visible people, active-mobility objects, and transport objects
Bounding-box CSV	Preserves each object's class, confidence, and coordinates

Model and dataset attribution

This repository provides the Gradio application, visualizations, derived indicators, and export utilities. It did not train the models or create their training datasets. External resources are credited below.

Borrowed resource	Creator and citation	How it is used
YOLO26-s	Glenn Jocher et al. (2026), distributed by Ultralytics	COCO object detection and bounding-box confidence
Ultralytics	Ultralytics	YOLO model loading and inference runtime
COCO dataset	Tsung-Yi Lin et al. (ECCV 2014)	80-category object vocabulary and pretrained detection data
SegFormer architecture	Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo (2021)	Transformer encoder and lightweight MLP decoder architecture
nvidia/segformer-b0-finetuned-cityscapes-1024-1024	NVIDIA checkpoint; model card maintained on Hugging Face	Public pretrained weights used for inference
Cityscapes dataset	Marius Cordts et al. (CVPR 2016)	Fine-tuning dataset, 19 evaluation classes, and standard class palette

The MIT license in this repository covers only this project's original code. It does not relicense YOLO, Ultralytics, COCO, SegFormer, or Cityscapes. Ultralytics code and models are offered under AGPL-3.0 and Enterprise licenses and require separate compliance. The Hugging Face SegFormer model card lists its license as other, and Cityscapes is subject to its official terms, including non-commercial-use and attribution conditions. Review those terms before reuse.

Output vocabularies

COCO object detection

YOLO26-s can detect 80 COCO categories. Street-relevant examples include person, bicycle, car, motorcycle, bus, truck, traffic light, stop sign, bench, bird, and dog. A missing box does not prove an object is absent; it may be too small, occluded, outside the trained vocabulary, or below the selected confidence threshold.

The 19 Cityscapes segmentation classes

Group	Classes	Example questions
Travel surfaces	road, sidewalk	How much of the view supports vehicle or pedestrian movement?
Built environment	building, wall, fence, pole, traffic light, traffic sign	Which fixed urban elements dominate the scene?
Nature and sky	vegetation, terrain, sky	How visually green or open is the street?
People	person, rider	Where are active street users visible?
Transport	car, truck, bus, train, motorcycle, bicycle	Which transport modes appear in the image?

The reported area share is the percentage of image pixels assigned to a class. It is a measure of visual coverage, not physical land area.

How it works

Street image
    ├── SegFormer-B0 (Cityscapes 19)
    │     ├── semantic overlay
    │     ├── Cityscapes color mask
    │     ├── raw class-ID image
    │     └── per-class pixel-share CSV
    │
    └── YOLO26-s (COCO 80)
          ├── labeled bounding-box overlay
          ├── class counts and confidence
          ├── transparent street indicators
          └── per-object coordinate CSV

Each model is downloaded on its first request and then reused from the container cache. CUDA is selected when available; local execution also works on CPU. Images larger than 2048 pixels on their longest side are resized while preserving aspect ratio. YOLO inference uses a 1024-pixel model input for small street objects.

Classroom and research examples

• Urban greenery: compare the visible share of vegetation across streets.
• Object counts: compare visible people, bicycles, cars, buses, or benches.
• Active mobility: examine transparent person + bicycle image counts while acknowledging that they are not traffic-flow measures.
• Street design: examine how roads, sidewalks, buildings, and terrain divide the view.
• Method comparison: discuss what boxes capture, what pixels capture, and what both methods miss.
• Model literacy: vary the confidence threshold and inspect false positives, missed objects, and segmentation boundary errors.
• Change over time: process repeat photographs of a location and compare their CSV summaries.

The UBC campus image above is the only built-in example and appears directly beneath the upload panel in the live app. It is hosted in the HNIVision example collection and loaded through GitHub Raw, following the same lightweight pattern as HNI-VLM.

Run locally

git clone https://github.com/LabMingzeChen/StreetSceneSegmentation.git
cd StreetSceneSegmentation
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py

Open the local URL printed by Gradio. The first prediction downloads the public model weights; no application secret is required.

Use the API

The deployed Space exposes separate segment and detect endpoints:

from gradio_client import Client, handle_file

client = Client("Mingze/StreetSceneSegmentation")
segmentation = client.predict(
    handle_file("street.jpg"),
    0.55,  # overlay opacity
    0.10,  # minimum table share (%)
    api_name="/segment",
)

detection = client.predict(
    handle_file("street.jpg"),
    0.25,  # minimum detection confidence
    api_name="/detect",
)

The Space's Use via API panel generates additional Python, JavaScript, and command-line examples for the current deployment.

Deploy your own Hugging Face Space

1. Create a new Gradio Space.
2. Select ZeroGPU when it is available for your account, or choose another supported hardware tier.
3. Clone this repository and add the Space as a second remote.
4. Push the same main branch to Hugging Face.

git remote add space https://huggingface.co/spaces/<USERNAME>/<SPACE_NAME>
git push -u space main

The YAML block at the top of this README is the Space configuration. Keep app_file: app.py and the declared Gradio version aligned with requirements.txt.

Project structure

StreetSceneSegmentation/
├── app.py                  Gradio interface and both inference workflows
├── segmentation_utils.py   Palette, pixel rendering, resizing, and CSV export
├── detection_utils.py      Bounding boxes, object summaries, and CSV export
├── tests/                   Lightweight post-processing tests
└── requirements.txt         Reproducible runtime dependencies

Limitations and responsible use

• YOLO is limited to COCO categories. Small, distant, occluded, or unfamiliar street objects may be missed or assigned an incorrect class.
• Detection-based indicators are visible-image counts, not estimates of people, trips, traffic flow, accessibility, exposure, or safety.
• SegFormer was trained for road-driving scenes and may perform poorly on unusual viewpoints, night images, severe weather, indoor scenes, or locations unlike the training data.
• Thin structures and object boundaries are common sources of pixel-level error.
• Results are estimates, not ground truth. Do not use them for navigation, safety-critical decisions, legal judgments, or identifying individuals.
• Uploaded images are processed by the selected Hugging Face Space runtime. Avoid submitting private or sensitive imagery to a public service.
• Model, software, and dataset terms remain governed by Ultralytics, COCO, the linked SegFormer model card, and Cityscapes.

Cite the models and datasets

If you use this application in research, cite YOLO26, COCO, SegFormer, and Cityscapes as applicable:

@misc{jocher2026ultralyticsyolo26unifiedrealtime,
  title         = {Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models},
  author        = {Jocher, Glenn and Qiu, Jing and Liu, Mengyu and Lyu, Shuai and Akyon, Fatih Cagatay and Kalfaoglu, Muhammet Esat},
  year          = {2026},
  eprint        = {2606.03748},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}

@inproceedings{lin2014microsoft,
  title     = {Microsoft COCO: Common Objects in Context},
  author    = {Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Bourdev, Lubomir and Girshick, Ross and Hays, James and Perona, Pietro and Ramanan, Deva and Zitnick, C. Lawrence and Dollár, Piotr},
  booktitle = {European Conference on Computer Vision},
  year      = {2014}
}

@article{xie2021segformer,
  title   = {SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers},
  author  = {Xie, Enze and Wang, Wenhai and Yu, Zhiding and Anandkumar, Anima and Alvarez, Jose M. and Luo, Ping},
  journal = {Advances in Neural Information Processing Systems},
  year    = {2021}
}

@inproceedings{Cordts2016Cityscapes,
  title     = {The Cityscapes Dataset for Semantic Urban Scene Understanding},
  author    = {Cordts, Marius and Omran, Mohamed and Ramos, Sebastian and Rehfeld, Timo and Enzweiler, Markus and Benenson, Rodrigo and Franke, Uwe and Roth, Stefan and Schiele, Bernt},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year      = {2016}
}

You may additionally cite this software:

@software{chen2026streetscenesegmentation,
  author = {Chen, Mingze},
  title  = {Street Scene Vision Toolkit},
  year   = {2026},
  url    = {https://github.com/LabMingzeChen/StreetSceneSegmentation}
}

License

Released under the MIT License.

---

If this project is useful, please star the GitHub repository and share the live Space with your students.