# AVIS — Automated Violation Intelligence System
### Design & Concept Note · codename *Gridlock*

> **Thesis.** Most teams will claim they detect all seven violations at high accuracy from a
> single photo. That claim is false, and judges who know vision will see through it. AVIS
> wins by being the system that is *rigorous about evidence*: it detects what a photograph
> can actually prove, attaches an explicit **evidence-sufficiency level** and confidence to
> every finding, uses a vision-language model to verify ambiguous cases and to **abstain**
> when the image doesn't support a verdict, and emits a court-ready, explainable evidence
> package. Honesty + explainability + human-in-the-loop is the differentiator.

This document is the single source of truth for the project's objective and design. It
replaces the earlier draft (`context.md`), which over-engineered the infrastructure and
over-claimed what is detectable from one image.

---

## 1. Problem & what makes it genuinely hard

Traffic cameras produce huge volumes of images; manual review is slow, inconsistent, and
expensive. The task: automatically detect road users, identify and classify violations,
recognise plates, and generate annotated evidence — robust to low light, rain, shadows,
motion blur, density, and image quality.

The non-obvious difficulty — and the thing the original plan glossed over — is that
**violations differ enormously in how provable they are from a single still image:**

- Some are **appearance facts** visible in one frame (no helmet, three people on a bike).
- Some are **spatial facts** that need to know the scene geometry (a vehicle past the
  stop line) — solvable from one image *only with per-camera calibration*.
- Some are **temporal/behavioural facts** (running a red light, driving the wrong way,
  *parking* = staying put) that fundamentally require motion or duration. Real red-light
  cameras use video buffers and multiple angles and *still* misfire on cars that brake
  hard at the line. A single photo can produce a *candidate*, never proof.

A credible system must encode this distinction instead of pretending it doesn't exist.

---

## 2. Design philosophy (the objective, reframed)

1. **Evidence sufficiency first.** Every violation type is tagged with what evidence a
   single image can provide. The system never auto-confirms a violation the photo can't
   prove — it produces a *candidate* and routes it for verification or human review.
2. **Deterministic CV for detection; AI for judgement.** Fast, reliable open models do the
   detecting and measuring. A vision-language model (VLM) is used **only** to verify
   ambiguous candidates, to abstain when evidence is weak, and to write the human-readable
   justification. This keeps the pipeline cheap, fast, and explainable.
3. **Explainable & accountable by construction.** Every output carries: the reason, the
   confidence, the evidence-sufficiency level, the legal reference, and a tamper-evident
   hash. No black-box verdicts; a human can always be put in the loop.
4. **Right-sized engineering.** A clean modular monolith that runs on a laptop/CPU and free
   APIs — with a clearly documented path to horizontal scale. We spend our time on the CV
   and the demo, not on orchestration plumbing.
5. **Free and self-hostable.** Open-source models + one free API tier (Gemini). No billing.

---

## 3. The Violation Detectability Matrix  *(the core intellectual contribution)*

Each violation is assigned a **tier** that determines its evidence-sufficiency level and
how the pipeline routes it. This table drives the Rule Engine and the routing policy.

| Violation | Tier | What a single image proves | Inputs needed | Default route |
|---|---|---|---|---|
| **Helmet non-compliance** | **A — Appearance** | Reliable: rider head visible, helmet/no-helmet | Detector + helmet classifier | Auto-confirm if high conf, else VLM |
| **Triple riding** | **A — Appearance** | Reliable: count riders linked to one motorcycle | Detector + rider↔bike association | Auto-confirm if clean count, else VLM |
| **Seatbelt non-compliance** | **B — Hard appearance** | Weak: windshield glare/resolution make this unreliable | Windshield crop classifier *or* VLM | VLM-verify or human; mark low-confidence |
| **Stop-line violation** | **C — Spatial (needs calibration)** | Strong *candidate*: vehicle body past the stop-line zone | Detector + per-camera stop-line polygon | VLM-verify; human if no calibration |
| **Red-light violation** | **C/D — Spatial + temporal** | Candidate only: light=red (from image) AND vehicle past stop line; true running is temporal | Detector + light-state + stop-line polygon | Always VLM-verify + flag "confirm with sequence" |
| **Illegal parking** | **C/D — Spatial + temporal** | Candidate only: vehicle inside no-parking zone; "parked" = duration, unprovable from one frame | Detector + no-parking polygon | Candidate; flag "needs dwell-time confirmation" |
| **Wrong-side driving** | **D — Temporal** | Not provable: direction of travel needs motion; orientation-vs-lane is a weak proxy | Detector + lane direction (+ orientation) | Low-confidence candidate; recommend video |
| **License-plate recognition** | *(supporting)* | Reliable when plate is legible | Plate detector + OCR + regex | Always attempted; attach to any violation |

**Consequence for the build:** Tier A is our headline, demo-grade capability. Tier B is
best-effort with honest confidence. Tiers C/D are positioned as **candidate generation +
calibration-assisted evidence**, never as "we solved red-light running from a JPEG." This
framing is defensible and impressive; over-claiming is not.

---

## 4. System overview

A **production-credible modular monolith**: one FastAPI codebase running a linear pipeline of
pure-ish stages, with the heavy CV work handed to a **worker via a Redis queue** so the API
stays responsive. This is production-grade *and* demo-safe — real horizontal scalability (add
workers) without splitting into many microservices. Everything is free/self-hosted and comes
up with one `docker compose up`. Storage and the queue sit behind interfaces, so a developer
can run a lightweight local mode (in-process tasks + filesystem) when they don't want to
start the full stack.

```
  upload → FastAPI api ──enqueue──▶ Redis ──▶ Worker(s): pipeline stages 1–10
                                                  │
   1 Ingest+QualityGate → 2 Preprocess → 3 Detect → 4 SceneGraph
   → 5 Attributes(helmet/seatbelt/plate) → 6 RuleEngine → 7 Fuse+Route
   → 8 VLM Verify (Gemini, only when routed) → 9 Legal map → 10 Evidence Composer
                                                  │
        ┌──────────────────────────────────────┴─────────────────────────────┐
  Postgres (JSONB: metadata + evidence graph)                  MinIO (S3: images)
        └──────────────────────────────────────┬─────────────────────────────┘
                                                ▼
                                  React + Vite dashboard
                        (upload · review queue · analytics · search)
   scale: add workers · swap MinIO→S3, Postgres→managed · GPU detection workers
```

### Stage responsibilities
1. **Ingest + Quality Gate** — store the image + metadata (camera_id, timestamp, geo if
   given); run a blur/exposure/resolution check. If the image is unusable, mark
   `undeterminable` and stop — *abstaining is a feature, not a failure.*
2. **Preprocess** — OpenCV: CLAHE (contrast), denoise, optional low-light enhancement
   (gamma / Retinex-style), light deblur. Produces a normalised image; keeps the original
   untouched for evidence.
3. **Detect** — YOLO11/YOLOv8 (COCO) for car/truck/bus/motorcycle/bicycle/person/traffic
   light; a fine-tuned model adds helmet and plate classes. Output: boxes + class + conf.
4. **Scene Graph** — associate people to vehicles by geometry (rider↔motorcycle,
   driver↔car), attach traffic lights and plates. This graph is the single source of truth
   for all downstream reasoning (§5).
5. **Attribute classifiers** — helmet per rider (YOLO/classifier); seatbelt per driver
   (windshield crop classifier or VLM, Tier B); traffic-light state (HSV + classifier);
   plate text (fast-alpr OCR + Indian-plate regex correction).
6. **Rule Engine** — pure, deterministic functions, one per violation, consuming the graph
   + optional per-camera calibration. Emits **candidates** with a rule-evidence score and
   the tier from §3. No ML, no I/O — fully unit-testable.
7. **Fuse + Route** — combine detection / attribute / rule scores into a fused confidence;
   apply tier-aware routing (§7): auto-confirm, VLM-verify, human-review, or abstain.
8. **VLM Verify** — only for routed candidates: send the cropped evidence + a strict prompt
   to **Gemini Flash (free tier)**; get `{verified, confidence, reason}` or
   `insufficient_evidence`. Sparingly used to respect the 1,500/day, ~10/min free quota.
9. **Legal map** — deterministic lookup table: violation → Motor Vehicles Act section →
   fine. (Correct, instant, free. Optional RAG is a bonus feature, not load-bearing — §10.)
10. **Evidence Composer** — annotated image (boxes, labels, plate, reason) + structured
    JSON + SHA-256 hash + audit trail; persist to Postgres (JSONB) + MinIO.

---

## 5. The Evidence Graph (data model)

Detections become a typed graph, not a flat list. This is what makes the system queryable,
explainable, and auditable.

```
Nodes:
  Vehicle  { id, type, bbox, conf, attrs:{orientation?, in_zones:[...]} }
  Person   { id, role: rider|driver|pedestrian, bbox, conf, attrs:{helmet?, seatbelt?} }
  Light    { id, state: red|amber|green|unknown, bbox, conf }
  Plate    { id, text, regex_ok, conf, bbox }
  Zone     { id, kind: stop_line|no_parking|lane, polygon }      # from camera calibration

Edges:
  rides(Person → Vehicle:motorcycle)      drives(Person → Vehicle:car)
  has_plate(Vehicle → Plate)              located_in(Vehicle → Zone)
  governed_by(Vehicle → Light)
```

A violation is then a small, explainable subgraph, e.g. *triple riding* =
`motorcycle_1` with three `rides` edges. "Why?" is answerable directly from the graph
(*"3 riders linked to 1 motorcycle"*). Example output payload:

```json
{
  "violation": "TRIPLE_RIDING",
  "tier": "A",
  "evidence_sufficiency": "sufficient",
  "subjects": ["motorcycle_1", "rider_1", "rider_2", "rider_3"],
  "scores": { "detection": 0.94, "rule": 0.90, "vlm": 0.93, "fused": 0.92 },
  "route": "auto_confirmed",
  "reason": "Three distinct riders are linked to a single motorcycle.",
  "plate": { "text": "UP32AB1234", "regex_ok": true, "conf": 0.88 },
  "legal": { "act": "Motor Vehicles Act, 1988", "section": "128", "fine": "₹1000" },
  "evidence_hash": "sha256:…",
  "timestamp": "2026-06-20T10:31:00Z"
}
```

---

## 6. Confidence fusion & routing (with abstention)

Per candidate, fuse available signals with tier-specific weights:

```
fused = w_det·detection_conf + w_attr·attribute_conf + w_rule·rule_evidence (+ w_vlm·vlm_conf)
```

Routing policy (thresholds are config, not hardcoded):

- **Tier A:** `fused ≥ 0.85` and unambiguous → **auto-confirm**; `0.55–0.85` → **VLM-verify**;
  `< 0.55` → **human review**.
- **Tier B (seatbelt):** never auto-confirm → **VLM-verify**, then human if VLM is unsure.
- **Tier C (stop-line / red-light / parking):** never auto-confirm. With calibration →
  **VLM-verify** and label `candidate`; without calibration → **human review**.
- **Tier D (wrong-side):** emit **low-confidence candidate** + "recommend video
  confirmation"; never auto-confirm.
- **Quality gate failed or VLM returns `insufficient_evidence`** → **abstain**
  (`undeterminable`), not a false positive.

This is what makes the free Gemini quota workable: only the genuinely ambiguous minority
hits the VLM, and clear Tier-A cases auto-confirm for free.

---

## 7. VLM verification layer (Gemini free tier)

- **Role:** auditor, never detector. It receives a *focused crop* + the proposed violation
  + the graph evidence, and answers a strict JSON contract:
  ```
  {"verified": bool, "confidence": 0.0-1.0, "reason": "<one sentence>",
   "insufficient_evidence": bool}
  ```
- **Model:** Gemini Flash (free tier) via a provider-agnostic client so it can swap to
  Groq / OpenRouter free models without touching call sites.
- **Budget discipline:** free tier = ~1,500 req/day, ~10/min. Therefore: route sparingly,
  batch where possible, **cache by `evidence_hash`** (identical crops never re-billed),
  and degrade gracefully to "human review" if the quota is exhausted. For the live demo,
  pre-cache responses for the demo images.

---

## 8. Legal grounding

For seven fixed violation types, the violation→section→fine mapping is a **small static
table** — correct, instant, and free. That table is the source of truth.

*Optional wow-factor (only if time allows):* a tiny RAG layer (ChromaDB +
sentence-transformers over the Motor Vehicles Act text) that lets an officer **ask
free-form questions** ("what's the penalty for X, and the appeal process?"). This is a
bonus feature layered on top — it must never be on the critical path of issuing a verdict.

---

## 9. Evidence package (court-ready output)

Each confirmed/escalated violation produces:
- **Annotated image** — original (untouched) + an overlay copy with boxes, labels,
  plate, and the natural-language reason.
- **Structured JSON** — the payload in §5, including scores, tier, route, legal ref.
- **Integrity** — SHA-256 hash of the original image + an append-only audit trail
  (who/what changed the verdict, when). This is the "admissible evidence" story.
- **e-challan-ready** — plate + violation + section + fine in one record, exportable.

---

## 10. Robustness to varying conditions (explicitly required)

- **Preprocessing** handles low light (CLAHE / gamma / Retinex), rain & noise (denoise),
  motion blur (mild deblur / sharpening).
- **Quality gate** quantifies blur/exposure and *abstains* on images too poor to judge —
  preventing confident-but-wrong outputs, which is the real failure mode under bad
  conditions.
- **VLM second opinion** adds robustness on ambiguous/degraded crops and can abstain.
- **Honest confidence** means degraded inputs surface as lower confidence → review, not as
  silent false positives.

---

## 11. Why this wins (innovation summary)

1. **Evidence-sufficiency–aware detection** — explicit tiers + abstention. Rigorous and
   rare; directly answers "robust and accurate" without over-claiming.
2. **Evidence Graph** — relationship-first, explainable single source of truth.
3. **VLM-as-verifier-and-abstainer** — cuts false positives, writes human-readable reasons,
   stays inside a free quota by only judging the hard cases.
4. **Confidence routing with human-in-the-loop** — quantifiable "human-review reduction %".
5. **Court-ready evidence** — legal grounding + tamper-evident hash + audit trail.
6. **Calibration-optional** — works on arbitrary images for Tier A; uses optional per-camera
   zones for Tier C; clearly degrades elsewhere.

---

## 12. Evaluation plan (mapped to the required metrics)

- **Component metrics (P / R / F1 / mAP):** on public datasets — helmet & triple-riding
  (Roboflow), plate detection + OCR character/whole-plate accuracy. Report per class.
- **End-to-end:** a hand-curated ~60–100 image set spanning day/night/rain/density, with a
  per-violation confusion matrix and false-positive rate **per image** (the metric that
  actually governs reviewer workload).
- **Operational:** mean latency/image on CPU, throughput (images/min), and the
  **auto-confirm vs. VLM vs. human** split → human-review-reduction %.
- **Ablation (great for judges):** rule-only vs. rule+VLM, to quantify the false-positive
  reduction the VLM buys.
- **Honesty:** report Tier A separately from C/D; never average them into one inflated
  "accuracy" number.

---

## 13. Tech stack (free) & scalability path

**MVP (build this) — a production-credible monolith, all free/self-hosted:**
- Python 3.11 · FastAPI · Uvicorn · **Redis-backed worker queue** (Celery or RQ/ARQ) so the
  API stays responsive and scales by adding workers; in-process fallback for quick local dev.
- **Detection:** Ultralytics **YOLO11/YOLOv8** (pip, pretrained COCO + fine-tuned
  helmet/plate from Roboflow). *(Open-vocab Grounding DINO / YOLO-World is an optional
  enhancement, not a dependency — it's slow and painful to install.)*
- **Plates:** **fast-alpr** (detection + `fast-plate-ocr`, ONNX, CPU-fast) + Indian-plate
  regex correction.
- **Seatbelt/ambiguous:** Gemini VLM (Tier B/C).
- **VLM:** **Gemini Flash free tier** via provider-agnostic `core/llm/` client — a one-line
  swap to a paid model (Claude / GPT-4o) later, with zero code rework. Free is a demo-time
  choice, not a capability ceiling.
- **Storage:** **Postgres (JSONB)** via SQLModel for metadata *and* the evidence graph
  (relational integrity + native JSON querying); **MinIO** (free, S3-compatible) for images.
  Both behind a `core/storage/` abstraction.
- **Legal:** static lookup table (+ optional ChromaDB RAG bonus).
- **Frontend:** React + Vite + Chart.js.
- **Packaging:** one `docker compose up` brings up api + worker + Postgres + Redis + MinIO.

**Why not full microservices?** This monolith already delivers the real scalability win — a
true task queue and horizontally-scalable workers — in one maintainable codebase. Splitting
the stages into separately-deployed services adds ops burden and demo risk for little extra
scalability during a hackathon.

**Scale-out path (a deployment change, not a rewrite):** add more workers; swap MinIO → AWS
S3 and Postgres → a managed instance; run the detection stage as dedicated GPU workers; and,
if ever truly needed, peel stages into separate services behind the same `storage/` and
`llm/` interfaces. Because everything already talks through those interfaces, none of this
touches business logic.

---

## 14. Pitch & demo flow (3–4 min)

1. **Hook (20s):** "Every team says they catch all 7 violations from a photo. They can't —
   and neither can a real red-light camera from one frame. We built the system that knows
   the difference."
2. **Idea (40s):** hybrid CV + VLM, Evidence Graph, evidence-sufficiency tiers, routing.
3. **Live demo (90s):** upload a triple-riding image → annotated result, graph panel
   ("motorcycle → 3 riders, 1 helmet"), VLM reason, plate OCR, legal section + fine; then a
   degraded night image → system **abstains / routes to review** (show this on purpose);
   then the analytics + review-queue dashboard.
4. **Proof (30s):** metrics table (Tier A P/R/F1, ablation showing VLM cuts false positives,
   human-review-reduction %).
5. **Close (20s):** free/self-hostable, calibration-optional, scale-out path, e-challan-ready.

---

## 15. Risks & honest limitations

| Risk | Mitigation |
|---|---|
| Seatbelt detection unreliable | Tier B: best-effort + VLM/human; report honest confidence |
| Tier C/D need calibration we won't have for random images | Position as candidate generation; demo with one calibrated camera config |
| Free Gemini quota exhausted | Cache by hash, route sparingly, pre-cache demo, degrade to human review |
| Grounding DINO install eats hackathon time | Default to Ultralytics YOLO; open-vocab is optional |
| Public model accuracy varies on Indian scenes | Pick India-trained Roboflow models; report measured, not claimed, numbers |

---

## References
- Gemini API free tier limits (2026): https://tokenmix.ai/blog/gemini-api-free-tier-limits
- fast-alpr / fast-plate-ocr: https://github.com/ankandrew/fast-alpr
- Triple-riding + helmet via YOLOv8 (CCTV): https://www.atlantis-press.com/proceedings/computatia-25/126010076
- Helmet violation detection, Indian smart-city (YOLOv8/TAO): https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2025.1582257/full
- Red-light running needs video (false positives from braking): https://www.researchgate.net/publication/3154744_An_effective_video_analysis_method_for_detecting_red_light_runners
- VLM on complex traffic events (GPT-4V study): https://arxiv.org/pdf/2402.02205