README.md

metadata

title: Street View Labeling Assistant
emoji: 🚀
colorFrom: red
colorTo: red
sdk: docker
app_port: 8501
tags:
  - streamlit
pinned: false
short_description: Streamlit template space

Street View Image Labeling Assistant

This application helps users draw 2D bounding boxes and label objects in images. Preloaded is a model trained on images from autonomous vehicles.

Dataset

A computer vision project for multiclass object classification using the NuImages dataset. For more information about the data processing and experimentation visit GitHub CV Project.

Features

Dual-Mode Workflow

• Manual Mode: Draw bounding boxes manually on canvas
• Assist+ Mode: Auto-detect objects with YOLO → AI classification pipeline
  • YOLOv11n detector with configurable hyperparameters (box, cls, dlf)
  • Editable/deletable detection boxes
  • Re-detect with Replace or Append modes

AI-Assisted Classification

• Top-3 predictions with confidence scores for each crop
• Accept, relabel, or ignore predictions workflow
• Visual feedback with quality badges (LOW-CONF, CLOSE-CALL, TINY)
• GradCAM heatmaps for model explainability

Triage Panel

• Filter problematic predictions by quality metrics:
  • Low confidence samples
  • Borderline classifications (close margins)
  • Small objects (tiny area fraction)
• User-defined thresholds for each metric
• Batch relabeling with visual inspection
• GradCAM analysis for difficult cases

Cloud Integration

• AWS S3: Browse and load images from S3 buckets
• Google Drive: OAuth2 authentication for Drive access
• Weight Resolution: Multi-source model loading (local → S3 → HuggingFace Hub)

Export System

• COCO, Pascal VOC, YOLO formats supported
• Session-based organization (timestamped folders)
• Original images + classified crops
• Structured export for training pipelines

Active Learning & New Classes

• Add/rename/remove session classes in the Class Manager; the dynamic taxonomy flows through detection, triage, and exports.
• Track readiness per new class with S3-backed bank totals and configurable promotion thresholds (min crops / unique images).
• Build training packages: download a local ZIP of crops/manifest and publish a JSON-only mirror to S3 under label-assistant/...; optionally include hard canonical cases.
• Base model snapshot (classifier + detector + env) is saved with each package for lineage.

Diagnostics & Model Resolution

• Diagnostics tab surfaces which classifier weights were resolved (local, Hugging Face, or cloud) and availability by source.
• Detector block shows weights path, device, and live detection params; system info summarizes Torch/Ultralytics/Streamlit versions and key env vars.
• Detector weights resolve via DET_WEIGHTS override or models/manifest.json (HF/S3/local); classifier weights honor CLOUD_WEIGHTS_MODE/CLOUD_WEIGHTS_ALLOW and optional WEIGHTS_KEY.

Safeguards & QoL

• Traffic-light detector/classifier mismatch guard with one-click sync suggestions.
• S3 sidebar includes a live log panel; Google Drive browser supports PKCE redirect and device-code login flows.

Tech Stack

• UI: Streamlit + drawable-canvas
• Detection: YOLOv11 (Ultralytics)
• Classification: PyTorch + torchvision
• Explainability: TorchCAM (GradCAM)
• Storage: boto3 (S3), Google API client
• Deployment: Docker

Usage

Quick Start

1. Select Mode: Choose Manual or Assist+ in the sidebar
2. Load Images: Upload files, or browse AWS S3 / Google Drive
3. Workflow Tab:
   • Manual Mode: Draw boxes → Classify → Review predictions
   • Assist+ Mode: Auto-detect → Edit boxes → Auto-classify
4. Triage Tab: Filter low-quality samples → Relabel → Apply GradCAM
5. Export: Generate COCO-style dataset with crops
6. Active Learning: Add/rename session classes, track S3 bank totals, and publish training packages (local ZIP + JSON-only to S3).
7. Diagnostics: Verify loaded weight sources and detector/system settings when debugging environment issues.

Detection Hyperparameters (Assist+ Mode)

• imgsz: divisible by 32 (configurable)
• conf: 0.25 (configurable confidence threshold)
• iou: 0.50 (NMS IoU threshold)
• max_det: 300 (maximum detections per image)
• FP16: Toggleable half-precision for CUDA

Supported Classes (NuImages Dataset)

The preloaded models support 10 street-view classes:

1. Car
2. Barrier
3. Truck
4. Pedestrian
5. Traffic cone
6. Motorcycle
7. Bicycle
8. Bus
9. Construction vehicle
10. Trailer

Custom Models

Deploy locally and add your own models:

• Upload classifier weights + class_map.json
• Upload YOLO detector weights
• Register in models/manifest.json
• Result: Your custom labeling assistant with detection + classification

Configuration Highlights

• Weights resolution: CLOUD_WEIGHTS_MODE (off|prefer_hf|prefer_cloud|auto) with optional CLOUD_WEIGHTS_ALLOW; pin a specific classifier checkpoint with WEIGHTS_KEY.
• Detector weights: DET_WEIGHTS override or models/manifest.json entries (HF/S3/local).
• Active Learning publish: requires S3_BUCKET/AWS_S3_BUCKET for JSON-only upload; uses label-assistant/ prefix by default.
• Cloud data sources: S3 sidebar uses AWS_* env vars; Google Drive supports OAuth (client id/secret + redirect URI) or service account JSON.