Hugging Face's logo

Spaces:
X2-0
/
AVIS
Sleeping

App Files Community
AVIS / docs /DESIGN.md
X2-0's picture
X2-0
HF Clean Deploy
1c0c94d
preview code
|
Raw
History Blame Contribute Delete
20.7 kB

AVIS — Automated Violation Intelligence System

Design & Concept Note · codename Gridlock

Thesis. Most teams will claim they detect all seven violations at high accuracy from a single photo. That claim is false, and judges who know vision will see through it. AVIS wins by being the system that is rigorous about evidence: it detects what a photograph can actually prove, attaches an explicit evidence-sufficiency level and confidence to every finding, uses a vision-language model to verify ambiguous cases and to abstain when the image doesn't support a verdict, and emits a court-ready, explainable evidence package. Honesty + explainability + human-in-the-loop is the differentiator.

This document is the single source of truth for the project's objective and design. It replaces the earlier draft (context.md), which over-engineered the infrastructure and over-claimed what is detectable from one image.

1. Problem & what makes it genuinely hard

Traffic cameras produce huge volumes of images; manual review is slow, inconsistent, and expensive. The task: automatically detect road users, identify and classify violations, recognise plates, and generate annotated evidence — robust to low light, rain, shadows, motion blur, density, and image quality.

The non-obvious difficulty — and the thing the original plan glossed over — is that violations differ enormously in how provable they are from a single still image:

  • Some are appearance facts visible in one frame (no helmet, three people on a bike).
  • Some are spatial facts that need to know the scene geometry (a vehicle past the stop line) — solvable from one image only with per-camera calibration.
  • Some are temporal/behavioural facts (running a red light, driving the wrong way, parking = staying put) that fundamentally require motion or duration. Real red-light cameras use video buffers and multiple angles and still misfire on cars that brake hard at the line. A single photo can produce a candidate, never proof.

A credible system must encode this distinction instead of pretending it doesn't exist.

2. Design philosophy (the objective, reframed)

  1. Evidence sufficiency first. Every violation type is tagged with what evidence a single image can provide. The system never auto-confirms a violation the photo can't prove — it produces a candidate and routes it for verification or human review.
  2. Deterministic CV for detection; AI for judgement. Fast, reliable open models do the detecting and measuring. A vision-language model (VLM) is used only to verify ambiguous candidates, to abstain when evidence is weak, and to write the human-readable justification. This keeps the pipeline cheap, fast, and explainable.
  3. Explainable & accountable by construction. Every output carries: the reason, the confidence, the evidence-sufficiency level, the legal reference, and a tamper-evident hash. No black-box verdicts; a human can always be put in the loop.
  4. Right-sized engineering. A clean modular monolith that runs on a laptop/CPU and free APIs — with a clearly documented path to horizontal scale. We spend our time on the CV and the demo, not on orchestration plumbing.
  5. Free and self-hostable. Open-source models + one free API tier (Gemini). No billing.

3. The Violation Detectability Matrix (the core intellectual contribution)

Each violation is assigned a tier that determines its evidence-sufficiency level and how the pipeline routes it. This table drives the Rule Engine and the routing policy.

Violation Tier What a single image proves Inputs needed Default route
Helmet non-compliance A — Appearance Reliable: rider head visible, helmet/no-helmet Detector + helmet classifier Auto-confirm if high conf, else VLM
Triple riding A — Appearance Reliable: count riders linked to one motorcycle Detector + rider↔bike association Auto-confirm if clean count, else VLM
Seatbelt non-compliance B — Hard appearance Weak: windshield glare/resolution make this unreliable Windshield crop classifier or VLM VLM-verify or human; mark low-confidence
Stop-line violation C — Spatial (needs calibration) Strong candidate: vehicle body past the stop-line zone Detector + per-camera stop-line polygon VLM-verify; human if no calibration
Red-light violation C/D — Spatial + temporal Candidate only: light=red (from image) AND vehicle past stop line; true running is temporal Detector + light-state + stop-line polygon Always VLM-verify + flag "confirm with sequence"
Illegal parking C/D — Spatial + temporal Candidate only: vehicle inside no-parking zone; "parked" = duration, unprovable from one frame Detector + no-parking polygon Candidate; flag "needs dwell-time confirmation"
Wrong-side driving D — Temporal Not provable: direction of travel needs motion; orientation-vs-lane is a weak proxy Detector + lane direction (+ orientation) Low-confidence candidate; recommend video
License-plate recognition (supporting) Reliable when plate is legible Plate detector + OCR + regex Always attempted; attach to any violation

Consequence for the build: Tier A is our headline, demo-grade capability. Tier B is best-effort with honest confidence. Tiers C/D are positioned as candidate generation + calibration-assisted evidence, never as "we solved red-light running from a JPEG." This framing is defensible and impressive; over-claiming is not.

4. System overview

A production-credible modular monolith: one FastAPI codebase running a linear pipeline of pure-ish stages, with the heavy CV work handed to a worker via a Redis queue so the API stays responsive. This is production-grade and demo-safe — real horizontal scalability (add workers) without splitting into many microservices. Everything is free/self-hosted and comes up with one docker compose up. Storage and the queue sit behind interfaces, so a developer can run a lightweight local mode (in-process tasks + filesystem) when they don't want to start the full stack.

  upload → FastAPI api ──enqueue──▶ Redis ──▶ Worker(s): pipeline stages 1–10
                                                  │
   1 Ingest+QualityGate → 2 Preprocess → 3 Detect → 4 SceneGraph
   → 5 Attributes(helmet/seatbelt/plate) → 6 RuleEngine → 7 Fuse+Route
   → 8 VLM Verify (Gemini, only when routed) → 9 Legal map → 10 Evidence Composer
                                                  │
        ┌──────────────────────────────────────┴─────────────────────────────┐
  Postgres (JSONB: metadata + evidence graph)                  MinIO (S3: images)
        └──────────────────────────────────────┬─────────────────────────────┘
                                                ▼
                                  React + Vite dashboard
                        (upload · review queue · analytics · search)
   scale: add workers · swap MinIO→S3, Postgres→managed · GPU detection workers

Stage responsibilities

  1. Ingest + Quality Gate — store the image + metadata (camera_id, timestamp, geo if given); run a blur/exposure/resolution check. If the image is unusable, mark undeterminable and stop — abstaining is a feature, not a failure.
  2. Preprocess — OpenCV: CLAHE (contrast), denoise, optional low-light enhancement (gamma / Retinex-style), light deblur. Produces a normalised image; keeps the original untouched for evidence.
  3. Detect — YOLO11/YOLOv8 (COCO) for car/truck/bus/motorcycle/bicycle/person/traffic light; a fine-tuned model adds helmet and plate classes. Output: boxes + class + conf.
  4. Scene Graph — associate people to vehicles by geometry (rider↔motorcycle, driver↔car), attach traffic lights and plates. This graph is the single source of truth for all downstream reasoning (§5).
  5. Attribute classifiers — helmet per rider (YOLO/classifier); seatbelt per driver (windshield crop classifier or VLM, Tier B); traffic-light state (HSV + classifier); plate text (fast-alpr OCR + Indian-plate regex correction).
  6. Rule Engine — pure, deterministic functions, one per violation, consuming the graph
    • optional per-camera calibration. Emits candidates with a rule-evidence score and the tier from §3. No ML, no I/O — fully unit-testable.
  7. Fuse + Route — combine detection / attribute / rule scores into a fused confidence; apply tier-aware routing (§7): auto-confirm, VLM-verify, human-review, or abstain.
  8. VLM Verify — only for routed candidates: send the cropped evidence + a strict prompt to Gemini Flash (free tier); get {verified, confidence, reason} or insufficient_evidence. Sparingly used to respect the 1,500/day, ~10/min free quota.
  9. Legal map — deterministic lookup table: violation → Motor Vehicles Act section → fine. (Correct, instant, free. Optional RAG is a bonus feature, not load-bearing — §10.)
  10. Evidence Composer — annotated image (boxes, labels, plate, reason) + structured JSON + SHA-256 hash + audit trail; persist to Postgres (JSONB) + MinIO.

5. The Evidence Graph (data model)

Detections become a typed graph, not a flat list. This is what makes the system queryable, explainable, and auditable.

Nodes:
  Vehicle  { id, type, bbox, conf, attrs:{orientation?, in_zones:[...]} }
  Person   { id, role: rider|driver|pedestrian, bbox, conf, attrs:{helmet?, seatbelt?} }
  Light    { id, state: red|amber|green|unknown, bbox, conf }
  Plate    { id, text, regex_ok, conf, bbox }
  Zone     { id, kind: stop_line|no_parking|lane, polygon }      # from camera calibration

Edges:
  rides(Person → Vehicle:motorcycle)      drives(Person → Vehicle:car)
  has_plate(Vehicle → Plate)              located_in(Vehicle → Zone)
  governed_by(Vehicle → Light)

A violation is then a small, explainable subgraph, e.g. triple riding = motorcycle_1 with three rides edges. "Why?" is answerable directly from the graph ("3 riders linked to 1 motorcycle"). Example output payload:

{
  "violation": "TRIPLE_RIDING",
  "tier": "A",
  "evidence_sufficiency": "sufficient",
  "subjects": ["motorcycle_1", "rider_1", "rider_2", "rider_3"],
  "scores": { "detection": 0.94, "rule": 0.90, "vlm": 0.93, "fused": 0.92 },
  "route": "auto_confirmed",
  "reason": "Three distinct riders are linked to a single motorcycle.",
  "plate": { "text": "UP32AB1234", "regex_ok": true, "conf": 0.88 },
  "legal": { "act": "Motor Vehicles Act, 1988", "section": "128", "fine": "₹1000" },
  "evidence_hash": "sha256:…",
  "timestamp": "2026-06-20T10:31:00Z"
}

6. Confidence fusion & routing (with abstention)

Per candidate, fuse available signals with tier-specific weights:

fused = w_det·detection_conf + w_attr·attribute_conf + w_rule·rule_evidence (+ w_vlm·vlm_conf)

Routing policy (thresholds are config, not hardcoded):

  • Tier A: fused ≥ 0.85 and unambiguous → auto-confirm; 0.55–0.85 → VLM-verify; < 0.55 → human review.
  • Tier B (seatbelt): never auto-confirm → VLM-verify, then human if VLM is unsure.
  • Tier C (stop-line / red-light / parking): never auto-confirm. With calibration → VLM-verify and label candidate; without calibration → human review.
  • Tier D (wrong-side): emit low-confidence candidate + "recommend video confirmation"; never auto-confirm.
  • Quality gate failed or VLM returns insufficient_evidence → abstain (undeterminable), not a false positive.

This is what makes the free Gemini quota workable: only the genuinely ambiguous minority hits the VLM, and clear Tier-A cases auto-confirm for free.

7. VLM verification layer (Gemini free tier)

  • Role: auditor, never detector. It receives a focused crop + the proposed violation
    • the graph evidence, and answers a strict JSON contract:
    {"verified": bool, "confidence": 0.0-1.0, "reason": "<one sentence>",
 "insufficient_evidence": bool}
  • Model: Gemini Flash (free tier) via a provider-agnostic client so it can swap to Groq / OpenRouter free models without touching call sites.
  • Budget discipline: free tier = ~1,500 req/day, ~10/min. Therefore: route sparingly, batch where possible, cache by evidence_hash (identical crops never re-billed), and degrade gracefully to "human review" if the quota is exhausted. For the live demo, pre-cache responses for the demo images.

8. Legal grounding

For seven fixed violation types, the violation→section→fine mapping is a small static table — correct, instant, and free. That table is the source of truth.

Optional wow-factor (only if time allows): a tiny RAG layer (ChromaDB + sentence-transformers over the Motor Vehicles Act text) that lets an officer ask free-form questions ("what's the penalty for X, and the appeal process?"). This is a bonus feature layered on top — it must never be on the critical path of issuing a verdict.

9. Evidence package (court-ready output)

Each confirmed/escalated violation produces:

  • Annotated image — original (untouched) + an overlay copy with boxes, labels, plate, and the natural-language reason.
  • Structured JSON — the payload in §5, including scores, tier, route, legal ref.
  • Integrity — SHA-256 hash of the original image + an append-only audit trail (who/what changed the verdict, when). This is the "admissible evidence" story.
  • e-challan-ready — plate + violation + section + fine in one record, exportable.

10. Robustness to varying conditions (explicitly required)

  • Preprocessing handles low light (CLAHE / gamma / Retinex), rain & noise (denoise), motion blur (mild deblur / sharpening).
  • Quality gate quantifies blur/exposure and abstains on images too poor to judge — preventing confident-but-wrong outputs, which is the real failure mode under bad conditions.
  • VLM second opinion adds robustness on ambiguous/degraded crops and can abstain.
  • Honest confidence means degraded inputs surface as lower confidence → review, not as silent false positives.

11. Why this wins (innovation summary)

  1. Evidence-sufficiency–aware detection — explicit tiers + abstention. Rigorous and rare; directly answers "robust and accurate" without over-claiming.
  2. Evidence Graph — relationship-first, explainable single source of truth.
  3. VLM-as-verifier-and-abstainer — cuts false positives, writes human-readable reasons, stays inside a free quota by only judging the hard cases.
  4. Confidence routing with human-in-the-loop — quantifiable "human-review reduction %".
  5. Court-ready evidence — legal grounding + tamper-evident hash + audit trail.
  6. Calibration-optional — works on arbitrary images for Tier A; uses optional per-camera zones for Tier C; clearly degrades elsewhere.

12. Evaluation plan (mapped to the required metrics)

  • Component metrics (P / R / F1 / mAP): on public datasets — helmet & triple-riding (Roboflow), plate detection + OCR character/whole-plate accuracy. Report per class.
  • End-to-end: a hand-curated ~60–100 image set spanning day/night/rain/density, with a per-violation confusion matrix and false-positive rate per image (the metric that actually governs reviewer workload).
  • Operational: mean latency/image on CPU, throughput (images/min), and the auto-confirm vs. VLM vs. human split → human-review-reduction %.
  • Ablation (great for judges): rule-only vs. rule+VLM, to quantify the false-positive reduction the VLM buys.
  • Honesty: report Tier A separately from C/D; never average them into one inflated "accuracy" number.

13. Tech stack (free) & scalability path

MVP (build this) — a production-credible monolith, all free/self-hosted:

  • Python 3.11 · FastAPI · Uvicorn · Redis-backed worker queue (Celery or RQ/ARQ) so the API stays responsive and scales by adding workers; in-process fallback for quick local dev.
  • Detection: Ultralytics YOLO11/YOLOv8 (pip, pretrained COCO + fine-tuned helmet/plate from Roboflow). (Open-vocab Grounding DINO / YOLO-World is an optional enhancement, not a dependency — it's slow and painful to install.)
  • Plates: fast-alpr (detection + fast-plate-ocr, ONNX, CPU-fast) + Indian-plate regex correction.
  • Seatbelt/ambiguous: Gemini VLM (Tier B/C).
  • VLM: Gemini Flash free tier via provider-agnostic core/llm/ client — a one-line swap to a paid model (Claude / GPT-4o) later, with zero code rework. Free is a demo-time choice, not a capability ceiling.
  • Storage: Postgres (JSONB) via SQLModel for metadata and the evidence graph (relational integrity + native JSON querying); MinIO (free, S3-compatible) for images. Both behind a core/storage/ abstraction.
  • Legal: static lookup table (+ optional ChromaDB RAG bonus).
  • Frontend: React + Vite + Chart.js.
  • Packaging: one docker compose up brings up api + worker + Postgres + Redis + MinIO.

Why not full microservices? This monolith already delivers the real scalability win — a true task queue and horizontally-scalable workers — in one maintainable codebase. Splitting the stages into separately-deployed services adds ops burden and demo risk for little extra scalability during a hackathon.

Scale-out path (a deployment change, not a rewrite): add more workers; swap MinIO → AWS S3 and Postgres → a managed instance; run the detection stage as dedicated GPU workers; and, if ever truly needed, peel stages into separate services behind the same storage/ and llm/ interfaces. Because everything already talks through those interfaces, none of this touches business logic.

14. Pitch & demo flow (3–4 min)

  1. Hook (20s): "Every team says they catch all 7 violations from a photo. They can't — and neither can a real red-light camera from one frame. We built the system that knows the difference."
  2. Idea (40s): hybrid CV + VLM, Evidence Graph, evidence-sufficiency tiers, routing.
  3. Live demo (90s): upload a triple-riding image → annotated result, graph panel ("motorcycle → 3 riders, 1 helmet"), VLM reason, plate OCR, legal section + fine; then a degraded night image → system abstains / routes to review (show this on purpose); then the analytics + review-queue dashboard.
  4. Proof (30s): metrics table (Tier A P/R/F1, ablation showing VLM cuts false positives, human-review-reduction %).
  5. Close (20s): free/self-hostable, calibration-optional, scale-out path, e-challan-ready.

15. Risks & honest limitations

Risk Mitigation
Seatbelt detection unreliable Tier B: best-effort + VLM/human; report honest confidence
Tier C/D need calibration we won't have for random images Position as candidate generation; demo with one calibrated camera config
Free Gemini quota exhausted Cache by hash, route sparingly, pre-cache demo, degrade to human review
Grounding DINO install eats hackathon time Default to Ultralytics YOLO; open-vocab is optional
Public model accuracy varies on Indian scenes Pick India-trained Roboflow models; report measured, not claimed, numbers

References