README.md

---
title: Street Scene Vision Toolkit
emoji: 🚦
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.49.1
python_version: "3.10"
app_file: app.py
pinned: false
license: mit
short_description: Street segmentation and detection with SegFormer + YOLO.
---

<div align="center">

# 🚦 Street Scene Vision Toolkit

### Pixel-level scene understanding and object detection in one browser app

[![Hugging Face Space](https://img.shields.io/badge/🤗_Hugging_Face-Live_Demo-FFD21E)](https://huggingface.co/spaces/Mingze/StreetSceneSegmentation)
[![Segmentation](https://img.shields.io/badge/Segmentation-SegFormer--B0-6366F1)](https://huggingface.co/nvidia/segformer-b0-finetuned-cityscapes-1024-1024)
[![Detection](https://img.shields.io/badge/Detection-YOLO26--s-2563EB)](https://docs.ultralytics.com/models/yolo26/)
[![Python](https://img.shields.io/badge/Python-3.10%2B-3776AB)](https://www.python.org/)
[![Gradio](https://img.shields.io/badge/UI-Gradio-FF7C00)](https://www.gradio.app/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**Upload once, segment every pixel, detect individual objects, and export both forms of evidence.**

[**🚀 Launch the live app**](https://huggingface.co/spaces/Mingze/StreetSceneSegmentation) ·
[**🎨 SegFormer**](https://huggingface.co/nvidia/segformer-b0-finetuned-cityscapes-1024-1024) ·
[**📦 YOLO26**](https://docs.ultralytics.com/models/yolo26/) ·
[**💻 Source code**](https://github.com/LabMingzeChen/StreetSceneSegmentation)

</div>

---

## Live demo

The public [Hugging Face Space](https://huggingface.co/spaces/Mingze/StreetSceneSegmentation)
runs entirely in the browser interface. Students can upload an image, choose the
UBC example, inspect the results, and download the outputs without installing
Python or providing an API key.

### Featured classroom example: UBC campus street

<table width="100%">
  <tr>
    <th width="25%">Original image</th>
    <th width="25%">Segmentation overlay</th>
    <th width="25%">Color mask</th>
    <th width="25%">Object detection</th>
  </tr>
  <tr>
    <td width="25%"><img src="https://raw.githubusercontent.com/LabMingzeChen/HNIVision/main/space/examples/ubc-campus-main-mall.jpeg" alt="UBC campus street input" width="100%"></td>
    <td width="25%"><img src="https://raw.githubusercontent.com/LabMingzeChen/HNIVision/main/space/examples/ubc-campus-segmentation-overlay.webp" alt="UBC campus segmentation overlay" width="100%"></td>
    <td width="25%"><img src="https://raw.githubusercontent.com/LabMingzeChen/HNIVision/main/space/examples/ubc-campus-segmentation-mask.webp" alt="UBC campus Cityscapes color mask" width="100%"></td>
    <td width="25%"><img src="https://raw.githubusercontent.com/LabMingzeChen/HNIVision/main/space/examples/ubc-campus-object-detection-overlay.webp" alt="UBC campus YOLO object detection" width="100%"></td>
  </tr>
</table>

This scene provides a useful teaching example because it combines vegetation,
sky, terrain, paving, buildings, poles, pedestrians, and a cyclist in one image.
Students can compare the predicted area shares and inspect challenging boundaries
between the tree canopy and sky, or between the paved route and grass.

The SegFormer result was generated at 2048 × 1536 px with an overlay opacity
of 0.55. The largest predicted pixel classes were:

| Predicted class | Image share |
|---|---:|
| Vegetation | 42.87% |
| Sky | 20.86% |
| Road | 20.73% |
| Terrain | 11.50% |
| Building | 1.76% |
| Sidewalk | 1.07% |

At a 0.25 confidence threshold, YOLO26-s detected five visible objects:

| Detected class | Count | Average confidence | Maximum confidence |
|---|---:|---:|---:|
| Person | 4 | 0.633 | 0.839 |
| Bicycle | 1 | 0.335 | 0.335 |

## What the project does

The app provides two complementary computer-vision methods on the same image:

- [`nvidia/segformer-b0-finetuned-cityscapes-1024-1024`](https://huggingface.co/nvidia/segformer-b0-finetuned-cityscapes-1024-1024)
  assigns a Cityscapes class to **every image pixel**.
- [YOLO26-s](https://docs.ultralytics.com/models/yolo26/) detects separate
  COCO objects and returns labeled bounding boxes with confidence scores.

For each image, the app produces:

| Output | Purpose |
|---|---|
| Segmentation overlay | Keeps the original scene visible beneath the predicted classes |
| Cityscapes color mask | Shows a clean, standardized categorical map |
| Class coverage table | Reports class ID, pixel count, area share, and color |
| Raw class-ID PNG | Preserves the numeric prediction for later analysis |
| CSV summary | Supports statistics, plotting, and classroom exercises |
| Detection overlay | Locates individual people, vehicles, bicycles, and other COCO objects |
| Object summary | Reports counts and confidence by detected class |
| Street indicators | Summarizes visible people, active-mobility objects, and transport objects |
| Bounding-box CSV | Preserves each object's class, confidence, and coordinates |

## Model and dataset attribution

This repository provides the Gradio application, visualizations, derived
indicators, and export utilities. It did not train the models or create their
training datasets. External resources are credited below.

| Borrowed resource | Creator and citation | How it is used |
|---|---|---|
| [YOLO26-s](https://docs.ultralytics.com/models/yolo26/) | Glenn Jocher et al. (2026), distributed by Ultralytics | COCO object detection and bounding-box confidence |
| [Ultralytics](https://github.com/ultralytics/ultralytics) | Ultralytics | YOLO model loading and inference runtime |
| [COCO dataset](https://cocodataset.org/) | Tsung-Yi Lin et al. (ECCV 2014) | 80-category object vocabulary and pretrained detection data |
| [SegFormer architecture](https://arxiv.org/abs/2105.15203) | Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo (2021) | Transformer encoder and lightweight MLP decoder architecture |
| [`nvidia/segformer-b0-finetuned-cityscapes-1024-1024`](https://huggingface.co/nvidia/segformer-b0-finetuned-cityscapes-1024-1024) | NVIDIA checkpoint; model card maintained on Hugging Face | Public pretrained weights used for inference |
| [Cityscapes dataset](https://www.cityscapes-dataset.com/) | Marius Cordts et al. (CVPR 2016) | Fine-tuning dataset, 19 evaluation classes, and standard class palette |

The MIT license in this repository covers **only this project's original code**.
It does not relicense YOLO, Ultralytics, COCO, SegFormer, or Cityscapes.
Ultralytics code and models are offered under
[AGPL-3.0 and Enterprise licenses](https://docs.ultralytics.com/) and require
separate compliance. The Hugging Face SegFormer model card lists its license as
`other`, and Cityscapes is subject to its
[official terms](https://www.cityscapes-dataset.com/license/), including
non-commercial-use and attribution conditions. Review those terms before reuse.

## Output vocabularies

### COCO object detection

YOLO26-s can detect 80 COCO categories. Street-relevant examples include
`person`, `bicycle`, `car`, `motorcycle`, `bus`, `truck`, `traffic light`,
`stop sign`, `bench`, `bird`, and `dog`. A missing box does not prove an object
is absent; it may be too small, occluded, outside the trained vocabulary, or
below the selected confidence threshold.

### The 19 Cityscapes segmentation classes

| Group | Classes | Example questions |
|---|---|---|
| Travel surfaces | `road`, `sidewalk` | How much of the view supports vehicle or pedestrian movement? |
| Built environment | `building`, `wall`, `fence`, `pole`, `traffic light`, `traffic sign` | Which fixed urban elements dominate the scene? |
| Nature and sky | `vegetation`, `terrain`, `sky` | How visually green or open is the street? |
| People | `person`, `rider` | Where are active street users visible? |
| Transport | `car`, `truck`, `bus`, `train`, `motorcycle`, `bicycle` | Which transport modes appear in the image? |

The reported area share is the percentage of **image pixels** assigned to a
class. It is a measure of visual coverage, not physical land area.

## How it works

```text
Street image
    ├── SegFormer-B0 (Cityscapes 19)
    │     ├── semantic overlay
    │     ├── Cityscapes color mask
    │     ├── raw class-ID image
    │     └── per-class pixel-share CSV
    │
    └── YOLO26-s (COCO 80)
          ├── labeled bounding-box overlay
          ├── class counts and confidence
          ├── transparent street indicators
          └── per-object coordinate CSV
```

Each model is downloaded on its first request and then reused from the container
cache. CUDA is selected when available; local execution also works on CPU.
Images larger than 2048 pixels on their longest side are resized while preserving
aspect ratio. YOLO inference uses a 1024-pixel model input for small street objects.

## Classroom and research examples

- **Urban greenery:** compare the visible share of vegetation across streets.
- **Object counts:** compare visible people, bicycles, cars, buses, or benches.
- **Active mobility:** examine transparent `person + bicycle` image counts while acknowledging that they are not traffic-flow measures.
- **Street design:** examine how roads, sidewalks, buildings, and terrain divide the view.
- **Method comparison:** discuss what boxes capture, what pixels capture, and what both methods miss.
- **Model literacy:** vary the confidence threshold and inspect false positives, missed objects, and segmentation boundary errors.
- **Change over time:** process repeat photographs of a location and compare their CSV summaries.

The UBC campus image above is the only built-in example and appears directly
beneath the upload panel in the live app. It is hosted in the
[HNIVision example collection](https://github.com/LabMingzeChen/HNIVision/tree/main/space/examples)
and loaded through GitHub Raw, following the same lightweight pattern as HNI-VLM.

## Run locally

```bash
git clone https://github.com/LabMingzeChen/StreetSceneSegmentation.git
cd StreetSceneSegmentation
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py
```

Open the local URL printed by Gradio. The first prediction downloads the public
model weights; no application secret is required.

## Use the API

The deployed Space exposes separate `segment` and `detect` endpoints:

```python
from gradio_client import Client, handle_file

client = Client("Mingze/StreetSceneSegmentation")
segmentation = client.predict(
    handle_file("street.jpg"),
    0.55,  # overlay opacity
    0.10,  # minimum table share (%)
    api_name="/segment",
)

detection = client.predict(
    handle_file("street.jpg"),
    0.25,  # minimum detection confidence
    api_name="/detect",
)
```

The Space's **Use via API** panel generates additional Python, JavaScript, and
command-line examples for the current deployment.

## Deploy your own Hugging Face Space

1. Create a new **Gradio Space**.
2. Select ZeroGPU when it is available for your account, or choose another supported hardware tier.
3. Clone this repository and add the Space as a second remote.
4. Push the same `main` branch to Hugging Face.

```bash
git remote add space https://huggingface.co/spaces/<USERNAME>/<SPACE_NAME>
git push -u space main
```

The YAML block at the top of this README is the Space configuration. Keep
`app_file: app.py` and the declared Gradio version aligned with
[`requirements.txt`](requirements.txt).

## Project structure

```text
StreetSceneSegmentation/
├── app.py                  Gradio interface and both inference workflows
├── segmentation_utils.py   Palette, pixel rendering, resizing, and CSV export
├── detection_utils.py      Bounding boxes, object summaries, and CSV export
├── tests/                   Lightweight post-processing tests
└── requirements.txt         Reproducible runtime dependencies
```

## Limitations and responsible use

- YOLO is limited to COCO categories. Small, distant, occluded, or unfamiliar
  street objects may be missed or assigned an incorrect class.
- Detection-based indicators are visible-image counts, not estimates of people,
  trips, traffic flow, accessibility, exposure, or safety.
- SegFormer was trained for road-driving scenes and may perform poorly on unusual
  viewpoints, night images, severe weather, indoor scenes, or locations unlike
  the training data.
- Thin structures and object boundaries are common sources of pixel-level error.
- Results are estimates, not ground truth. Do not use them for navigation,
  safety-critical decisions, legal judgments, or identifying individuals.
- Uploaded images are processed by the selected Hugging Face Space runtime.
  Avoid submitting private or sensitive imagery to a public service.
- Model, software, and dataset terms remain governed by Ultralytics, COCO, the
  linked SegFormer model card, and [Cityscapes](https://www.cityscapes-dataset.com/).

## Cite the models and datasets

If you use this application in research, cite YOLO26, COCO, SegFormer, and
Cityscapes as applicable:

```bibtex
@misc{jocher2026ultralyticsyolo26unifiedrealtime,
  title         = {Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models},
  author        = {Jocher, Glenn and Qiu, Jing and Liu, Mengyu and Lyu, Shuai and Akyon, Fatih Cagatay and Kalfaoglu, Muhammet Esat},
  year          = {2026},
  eprint        = {2606.03748},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}

@inproceedings{lin2014microsoft,
  title     = {Microsoft COCO: Common Objects in Context},
  author    = {Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Bourdev, Lubomir and Girshick, Ross and Hays, James and Perona, Pietro and Ramanan, Deva and Zitnick, C. Lawrence and Dollár, Piotr},
  booktitle = {European Conference on Computer Vision},
  year      = {2014}
}

@article{xie2021segformer,
  title   = {SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers},
  author  = {Xie, Enze and Wang, Wenhai and Yu, Zhiding and Anandkumar, Anima and Alvarez, Jose M. and Luo, Ping},
  journal = {Advances in Neural Information Processing Systems},
  year    = {2021}
}

@inproceedings{Cordts2016Cityscapes,
  title     = {The Cityscapes Dataset for Semantic Urban Scene Understanding},
  author    = {Cordts, Marius and Omran, Mohamed and Ramos, Sebastian and Rehfeld, Timo and Enzweiler, Markus and Benenson, Rodrigo and Franke, Uwe and Roth, Stefan and Schiele, Bernt},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year      = {2016}
}
```

You may additionally cite this software:

```bibtex
@software{chen2026streetscenesegmentation,
  author = {Chen, Mingze},
  title  = {Street Scene Vision Toolkit},
  year   = {2026},
  url    = {https://github.com/LabMingzeChen/StreetSceneSegmentation}
}
```

## License

Released under the [MIT License](LICENSE).

---

<div align="center">

**If this project is useful, please star the GitHub repository and share the live Space with your students.**

</div>