imp_docs/gridlock_final_doc.md

GridLoc

Parking-Induced Congestion Intelligence for Bengaluru

An Enforcement-Prioritisation Engine: Methodology & Technical Report

Problem Statement: AI-driven parking intelligence to detect illegal-parking hotspots and quantify their impact on traffic flow, enabling targeted enforcement.

Contents

1. Executive Summary

GridLoc turns a raw stream of parking-violation records into a ranked, time-aware enforcement plan for Bengaluru. Starting from roughly 298,000 ANPR-captured parking violations supplied by the Bengaluru Traffic Police (ASTraM) platform, it produces a single scored table covering every ~170-metre cell of the city, for each of four daily time blocks, labelled with one of four enforcement categories and an accompanying plain-English justification.

The central problem GridLoc solves is that a violation dataset records where enforcement looks, not where illegal parking actually happens. Every row exists because a camera or patrol caught it; areas with no recorded violations may be genuinely calm, or simply unwatched. Counting violations and calling the busy cells "hotspots" therefore re-draws the existing camera map-the naive output. GridLoc instead computes two independent estimates per cell-an Observed score from the recorded data and an Expected score from city conditions alone-and compares them. The disagreements are the insight: cells that should be busy but show little recorded activity are flagged as enforcement blind spots-the headline output that no count-based method can produce.

Key results at a glance

From 292,768 cleaned records across 2,534 hexagonal cells: 47 cells were certified as statistically significant hotspots (Getis-Ord Gi*, z > 1.96, up to z = 8.59). The final scored table spans 5,224 cell×time-block rows, classifying 95 confirmed hotspots, 661 blind spots, 57 enforcement quirks, and the remainder as calm. A sensitivity analysis perturbing every weight confirmed the prioritisation is robust-the observed ranking holds at Spearman ρ ≥ 0.998 and 97-100% of cells retain their category under ±20%+ weight changes.

2. Problem & Context

2.1 The operational challenge

On-street illegal parking and spillover near commercial areas, metro stations, and event venues choke carriageways and intersections across Bengaluru. Enforcement today is patrol-based and reactive: there is no heatmap relating parking violations to the congestion they cause, and it is difficult to decide which zones to prioritise. The problem statement asks how AI-driven parking intelligence can detect illegal-parking hotspots and quantify their impact on traffic flow, to enable targeted enforcement.

2.2 The partner systems

Bengaluru Traffic Police - ASTraM (Actionable Intelligence for Sustainable Traffic Management) is an AI big-data platform that ingests CCTV, ANPR, and open data. Critically, it runs a congestion-modelling module and an automated violation-detection system (~10,000 violations/day) as separate subsystems-nothing links a cluster of violations to the congestion that cluster causes. MapmyIndia / Mappls provides the mapping and location APIs (geocoding, reverse geocoding, snap-to-road, nearby search) used to enrich the data with real-world road context.

Where GridLoc fits

GridLoc is the missing join between ASTraM's two halves. It takes the violation-detection output and ranks it by congestion impact and enforcement priority-an "enter-prioritisation module" rather than another heatmap. That framing is the project's primary pitch.

2.3 The dataset

The supplied dataset is the output of ASTraM's ANPR / violation-detection pipeline. Each row is one captured violation, with spatial coordinates, vehicle and violation attributes, device and station identifiers, and a validation trail (the updated* and validation* columns record plate-read correction and human validation). The raw file holds 298,450 rows across 24 columns and spans 2023-11-09 to 2024-04-08.

Important data caveat

The dataset is synthetic / transformed by the organisers. Tell-tales: placeholder identifiers (FKID*, FKDEV*, FKN00GL*), and every timestamp's seconds field fixed at :46. This drives a key downstream decision on how timestamps are treated (Section 4.3).

3. The Core Idea: Observed vs. Expected

The whole pipeline rests on separating two questions that raw counts conflate: (1) where does illegal parking actually happen? (the ground truth we want) versus (2) where does it get recorded? (what the data shows). A low recorded count can mean a genuinely calm street or heavy uncaught parking with no camera pointed at it. The raw data cannot tell these apart; resolving that gap is the project.

Thesis. Estimate where illegal parking should occur from city conditions, compare it against where it was recorded, and recommend patrols for the confirmed hotspots and the blind spots-broken down by time of day.

This produces two independent scores per cell per time block. Observed (O) is built only from the violation data, weighted by congestion impact. Expected (E) is built only from city conditions-proximity to demand generators and road class-and never touches the violation data. That independence is what makes the comparison meaningful; if violation data leaked into E, the blind-spot finding would be circular.

The 2×2 category matrix

	Low Observed	High Observed
High Expected	🟣 BLIND SPOT - conditions predict heavy parking, little recorded. Likely coverage gap. Tell BTP to look here. (headline output)	🔴 CONFIRMED - busy in data and conditions agree. Enforce with high confidence.
Low Expected	⚪ CALM - quiet and expected to be quiet. Don't waste patrols.	🟠 QUIRK - high tickets, low demand. Possible camera trap or one-off event; footnote.

The two pitch pillars

Pillar 1 - the layer ASTraM lacks: the score and priority list IS the violation↔congestion join that ASTraM does not currently have. Pillar 2 - bias correction + blind spots: the data itself proves counts track enforcement resources more than conditions (Section 4.5), which is precisely why hunting blind spots is justified.

4. Exploratory Data Analysis

Before any modelling, the raw dataset was characterised in detail. The findings below are established facts that the pipeline relies on; several directly shaped locked decisions.

4.1 Shape, keys, and integrity

• 298,450 rows × 24 columns; the id column is a unique primary key.
• Coordinates are clean and in-range (lat 12.80-13.29, lng 77.44-77.77, all within Bengaluru; no (0,0) or swapped pairs).
• Three columns are 100% null and were dropped: description, closed_datetime, action_taken_timestamp.
• The offence_code and violation_type columns are strictly 1:1 (each code maps to exactly one violation tag and vice-versa).

4.2 Validation pipeline quality

The validation columns reflect a real human-in-the-loop correction process, and its error rates are low-evidence the underlying capture is reliable:

Metric	Value	Reading
Validation status breakdown	approved 115,400 (66.6%); rejected 49,754 (28.7%); created1 7,044; processing 678; duplicate 320	Two-thirds approved; ~42% never validated (null)
ANPR plate-read error rate	1.02% (1,774 / 173,196)	Plate reads highly reliable
Vehicle-type correction rate	3.56% (6,169 / 173,196)	Vehicle classification reliable
Overall rejection rate	16.67%	Baseline for device-level checks

4.3 The timestamp anomaly - and why no offset was applied

The hour-of-day distribution is implausible as real wall-clock time: it peaks at hour 5 (34,085 records) and bottoms at hour 14 (just 16 records)-roughly a 2,000× swing, with the trough in the early afternoon. Real parking-violation activity would peak during daytime commercial hours, not at 5 a.m.

Decision: use timestamps as given, apply no offset

Because the dataset is synthetic/transformed, the hour field almost certainly does not represent true IST wall-clock time. A tempting "forensic" fix (shifting the clock until the peak "looks right") was rejected as circular-you cannot validate an offset by shifting until it matches what you expect. The pipeline needs only the relative temporal pattern, which is invariant to any constant offset. We therefore use raw given hours and make no real-world claims about them (no "evening peak" labels).

4.4 Spatial concentration

• At H3 resolution 9, the data occupies 2,534 unique hexagonal cells (each ~170 m across).
• 10.77% of cells hold 80% of all records-activity is highly concentrated, so hotspots are real and findable.
• The busiest cell holds 12,123 records (near KR Market / city centre); the top cells each see 25-99 distinct devices, confirming genuine multi-device hotspots rather than single-camera artefacts.

4.5 Enforcement bias - the Pillar-2 evidence

A direct test of "does the data measure reality or effort?": the correlation between a cell's violation count and its number of distinct enforcement devices is 0.66. Recorded volume is materially influenced by enforcement presence (stated carefully-this is partly mechanical, since more violations naturally involve more passing devices). This is the quantitative justification for not trusting raw counts and for hunting blind spots.

4.6 De-prioritised findings

• Repeat offenders: weak signal. 74.26% of unique vehicles are needed to reach 80% of records (near-uniform, not heavy-tailed); the top offender has only 55 records. Not a core angle.
• Named junctions: ~50% coverage. "No Junction" accounts for 49.5% of rows; 169 named junctions exist. Junction labels cannot carry the Expected model alone and require independent geocoding.

5. Data Cleaning

Cleaning was deliberately conservative-the goal was to remove genuine artefacts without discarding signal. The steps, in order:

• Drop 3 fully-null columns (description, closed_datetime, action_taken_timestamp).
• Drop only validation_status == 'duplicate' (320 rows). All other statuses-including rejected and null-were kept (justified below).
• Drop 5 rows with null created_datetime.
• Parse created_datetime with pandas; no offset or timezone shift applied (Section 4.3).
• Remove 5,374 true duplicate events (identical vehicle_number + latitude + longitude + created_datetime to the second). The status flag had caught only 320 of these.
• Extract hour_of_day and assign a time bucket per the data-defined scheme (Section 6.1).
• Add h3_index at resolution 9 via h3.latlng_to_cell.
• Save as Parquet (not CSV)-Parquet preserves the datetime dtype, avoiding silent re-parsing of the sensitive timestamp on reload.

Why rejected rows were kept

A natural instinct is to drop rejected violations as "false positives." Four spatial checks showed this would discard signal, because rejected violations occur in essentially the same places as approved ones:

• Top-50-cell overlap between approved and rejected: 86% (43 of 50 cells appear in both lists).
• Per-cell correlation of approved-count vs rejected-count: Pearson 0.945.
• Per-device rejection rates vary (a few devices reject 70-87% vs a 16.67% baseline)-real device-level noise exists but does not overturn the spatial story.
• Top-20-cell rejection rates sit in a tight 10.6-27.1% band with no outlier cell.

Verdict: keep approved + rejected + created1 + processing + null; drop only the explicit duplicate status. The validation_status column is carried through for optional confidence-weighting rather than used as a hard filter.

Cleaned dataset - final state

292,768 rows × 29 columns. Zero nulls in the essential columns (latitude, longitude, created_datetime, h3_index, time_bucket, vehicle_type, violation_type). Zero true duplicates remaining. 2,534 unique cells-exactly matching the EDA count (consistency check passed).

6. The Modelling Pipeline

The pipeline runs in seven stages. Stages that need no external API (the Observed score, the statistical certification, and the final categorisation) run on the cleaned data alone; a single bounded block in the middle uses MapmyIndia for road context.

6.1 Time buckets - data-defined, not real-world

Because the hour field is not trustworthy as wall-clock time, the four time blocks were defined directly from the actual hour histogram, with neutral names that make no real-world claim:

Bucket	Hours	Records	Note
Block_B_00-07	00-07	189,875	Dominant block
Block_A_19-23	19-23	86,826	Second block
Block_C_08-09	08-09	11,479	Transition / taper
Block_D_10-18	10-18	4,588	Very thin - low confidence

Caveat carried forward: Block_D is sparse (4,588 of 292,768 rows across ~300 cells). Its per-cell scores are treated as lower-confidence, and the statistical certification gate (Stage 2, which runs on all-time totals) guards against over-reading any single thin-bucket cell.

6.2 Stage 1 - Observed Impact score, O(h, b)

The Observed score measures recorded congestion impact, not raw counts. Each violation is weighted by how much it obstructs traffic, aggregated per cell per bucket, dampened for one-off spikes, and then percentile-ranked.

Vehicle severity

Larger vehicles block more carriageway when parked illegally, so physical footprint is used as an impact proxy (three tiers):

• 1.0 - heavy: all BUS variants (BMTC/KSRTC, private, tourist, factory), HGV, LORRY/GOODS VEHICLE, TANKER, TEMPO, LGV, MINI LORRY, TRACTOR, SCHOOL VEHICLE.
• 0.6 - medium: CAR, MAXI-CAB, VAN, JEEP, PASSENGER AUTO, GOODS AUTO (and "OTHERS" / unknowns default here).
• 0.3 - light: SCOOTER, MOTOR CYCLE, MOPED.

Violation severity (max tag per record)

The violation_type field is a JSON list of tags. Each tag is scored by congestion impact and the maximum tag weight is taken per record. The tag set was audited against the live data so nothing scores silently by default:

Weight	Example tags	Rationale
1.0	Parking in a main road; Double parking; Parking opposite another parked vehicle; Against one-way / no-entry	Physically narrows / blocks the carriageway
0.8	Near road crossing; Near traffic light / zebra; Near bus-stop/school/hospital; Stopping on white/stop line	Crossing / intersection / critical-access obstruction
0.5	Wrong parking; No parking; Parking other than bus stop	Generic parking obstruction
0.3	On footpath; Obstructing driver; Carrying lengthy material	Minor / mild obstruction
0.1	Defective number plate; No helmet; No seatbelt; Mobile phone; Black film; Excess fare; etc.	Not congestion-related (moving / document offences bundled in)

Why the 0.1 tier matters: many records carry a parking tag plus an unrelated moving-violation tag. Scoring the unrelated tags low ensures the real parking tag wins the max, so non-congestion offences cannot inflate a cell.

Aggregation, persistence, and ranking

Per record, impact = vehicle severity × violation severity. These are summed per (cell, bucket) into a weighted count, then two adjustments are applied:

persistence(h) = distinct active days / total days in dataset (152 days)

The persistence multiplier (0-1, computed per cell across all time) dampens one-off spikes-a cell active on only a handful of days is discounted relative to a chronically active one. Finally the adjusted score is percentile-ranked within each time bucket:

O(h,b) = percentile_rank_within_bucket( weighted_count × persistence )

Percentile-ranking (rather than min-max) is deliberate: the count distribution has enormous outliers, which would wreck a min-max scale; ranks are robust to them and make every cell comparable. O ranges 0-1, where 1.0 is the worst cell in that bucket.

6.3 Stage 2 - Statistical certification (Getis-Ord Gi*)

High impact in one cell is not yet proof of a real spatial cluster-it could be a single busy camera. The Getis-Ord Gi* statistic tests whether a cell and its neighbours together form a concentration significantly above what random spatial scatter would produce. It returns a z-score per cell.

• Computed on all-time total impact per cell (not per-bucket-per-bucket is too sparse for reliable neighbour statistics).
• Spatial neighbours come from H3 adjacency (each cell plus its ring-1 neighbours; the cell includes itself, which is the "star" in Gi*).
• Implemented with the canonical esda / libpysal libraries for citability.

certified = ( gi_z > 1.96 ) → 95% confidence it is a real cluster, not noise

Certification result

47 of 2,534 cells certified (1.9%), with z-scores ranging −0.25 to 8.59. A small, selective set-exactly what Gi* should produce-picking out the genuine hot cores rather than rubber-stamping every high count. The top cells by z align with the top cells by impact (a consistency check).

6.4 Stages 3-5 - Road context via MapmyIndia

A single bounded block of the pipeline uses MapmyIndia's search API (authenticated by a static key on the search.mappls.com host). For each cell, the densest actual violation point-not the cell's geometric centroid, which can fall on the wrong road-is the representative location.

• Reverse geocoding of each cell's densest point → road name. All 2,534 cells resolved to real Bengaluru streets (e.g. Outer Ring Road, Mysore Road, Sahakar Nagar Road).
• Forward geocoding was attempted for the 169 named junctions to give the Expected model junction proximity even in cells with no recorded violations.

Engineering note: rate-limiting and a deliberate scope cut

Two practical findings shaped this block. (1) MapmyIndia throttles bursts-firing ~2,500 calls in 100s returned 401 errors across the board; pacing to ~4 calls/second with retry-and-backoff and on-disk caching resolved it cleanly (all 2,534 road names then succeeded). (2) The junction geocoder returns an address and a place code but no coordinates, so junction proximity could not be computed from it. Rather than chase a secondary lookup for the weakest, lowest-coverage feature, the junction component was dropped and the Expected score renormalised. The road-name signal-strong and complete across all cells-more than compensates. The Nearby (POI) API required an OAuth flow the static-key account could not issue, so demand generators were sourced from a curated list (Section 6.5).

6.5 Stage 6 - Expected Demand score, E(h)

The Expected score estimates how much illegal parking a cell should attract, using city conditions only and zero violation data. After the junction cut, it combines two components:

E(h) = 0.7 · POI_proximity(h) + 0.3 · road_hierarchy(h)

• POI proximity (0.7) - distance from the cell centroid to the nearest major demand generator (metro stations, wholesale markets, malls, tech parks, major bus stands), drawn from a curated list of Bengaluru's principal trip generators. Scored 1.0 within 200 m, fading linearly to 0 by ~1 km.
• Road hierarchy (0.3) - inferred from the geocoded road name: arterials and ring roads (Outer Ring Road, NH, "Main Road") score highest, generic "Road" mid, and residential "Cross" / "Layout" / "Nagar" lanes lowest.

POI proximity carries the larger weight because it is the broadest, most reliable demand signal; road hierarchy is a name-based heuristic and weighted accordingly. The resulting E ranges 0.12-1.0 (mean 0.29).

Time-awareness of the Expected score

Expected demand is time-varying where the geography supports it. Each demand generator carries a type-specific activity profile (tech parks and markets skew toward the busier blocks; metros and transit hubs draw steadily all day), and the profile is mapped onto the four blocks by the data's own activity ranking-not by assumed clock hours, which keeps the no-offset discipline of Section 4.3 intact. In practice the effect is concentrated: 284 cells show E varying across blocks (the tech-park- and market-adjacent corridors, e.g. KR Main Road and Hosur Road, swing by up to 0.56), while cells dominated by an all-day generator such as a metro correctly stay stable. This is deliberate-forcing uniform time-variation onto metro-adjacent areas would manufacture a pattern the geography does not support. The dominant time-awareness in the system therefore comes from the Observed side, which is the empirically grounded half; 36 cells change enforcement category across blocks, giving the time control a visible effect.

6.6 Stage 7 - Categorisation into the 2×2

With O, E, and the certification flag in hand, each (cell, bucket) row is placed in the matrix using clear thresholds:

• High Observed = certified (Gi* z > 1.96) and O above its within-bucket median.
• High Expected = E above the 60th percentile.

The four combinations map to Confirmed / Blind Spot / Quirk / Calm. A priority/intensity value drives the eventual heatmap (Confirmed → intensity O; Blind Spot → intensity E; Quirk/Calm → muted), and a plain-English reason string is assembled from the top contributing terms-e.g. "confirmed: certified z=8.6, 6th Main Road, O=1.00 E=0.74."

7. Results

The final scored table contains 5,224 (cell × time-block) rows. Each carries O, E, gi_z, certified, category, priority_score, road_name, snapped coordinates, and a reason string-everything a dashboard or patrol planner needs.

7.1 Category distribution

Category	Rows	Meaning
Calm	4,411	Quiet and expected to be quiet - no action
Blind spot	661	Should be busy, little recorded - investigate (headline)
Confirmed	95	Busy and conditions agree - enforce with confidence
Quirk	57	High tickets, low demand - possible camera trap / event

Per-bucket spread is healthy across all four blocks (~28 confirmed and 220-270 blind spots in the two dense blocks, scaling down appropriately in the thin ones). 36 cells change category across blocks, so the time control visibly re-categorises parts of the map shift to shift.

7.2 Top confirmed hotspots

The strongest confirmed hotspots are named arterial and market roads with very high Gi* significance-exactly the streets that genuinely choke in Bengaluru, which validates the model end-to-end:

Road	Block	O	E	Gi* z
6th Main Road	B (00-07)	1.00	0.74	8.59
3rd Main Road	A (19-23)	1.00	0.43	8.08
Mysore Road	D (10-18)	1.00	0.31	3.90
Hospital Cross Road	B (00-07)	1.00	0.51	5.20
Mysore Road	A (19-23)	1.00	0.62	3.90
Dispensary Road	B (00-07)	1.00	0.57	5.00

7.3 Top blind spots - the headline output

Blind spots are cells where conditions strongly predict heavy parking (E at or near the maximum) but recorded activity is comparatively low. Major demand corridors appear here-the actionable finding for BTP:

Road	Block	O	E
17th Main Road	B (00-07)	0.91	1.00
Outer Ring Road	B (00-07)	0.89	1.00
Outer Ring Road	A (19-23)	0.89	1.00
Whitefield Main Road	B (00-07)	0.54	1.00

Outer Ring Road and Whitefield surfacing as blind spots is a genuinely defensible result: high-demand corridors where recorded enforcement is lower than conditions warrant-"BTP, you may be under-enforcing here." This is the output no count-based method produces, because a count-based method has no independent yardstick to notice the gap.

8. Robustness & Sensitivity Analysis

To confirm the prioritisation is driven by the data structure rather than arbitrary weight choices, every major weight family was perturbed and the effect on the rankings measured. The metrics used are the right ones for this question: full-ranking Spearman correlation, and the share of cells that keep their category.

Perturbation	Observed ρ	Category retention
Vehicle weights - widen heavy/light gap	0.9992	100.0%
Vehicle weights - compress gap	0.9980	100.0%
Violation top tier +0.15	1.0000	100.0%
Violation top tier −0.15	1.0000	100.0%
Expected: POI weight → 0.5	0.9768	97.3%
Expected: POI weight → 0.9	0.9865	98.6%

Defensible robustness claim

Across perturbations of all severity and demand weights, the observed-impact ranking is near-invariant (Spearman ≥ 0.998), the expected-demand ranking holds at ρ ≥ 0.977 even under a ±40% swing in POI weighting, and 97-100% of cells retain their enforcement category. The prioritisation is driven by the data, not by specific weight choices.

Methodology note: an initial sensitivity test using top-20 overlap returned a suspicious, identical 11/20 across every scenario. This was correctly diagnosed as an artefact-percentile-ranking is scale-invariant (uniform weight scaling cannot move it) and the top-20 set was dominated by ties. The test was replaced with the Spearman + category-retention approach above, which measures the real question.

9. Honest Caveats & Limitations

These are stated plainly-they read as methodological rigour, not weakness, and each has a clear upgrade path.

• Impact is estimated, not measured. Congestion impact is inferred from vehicle and violation type, not from real road speeds (the free MapmyIndia tier provides no live/historical segment speeds). With traffic-API access this becomes directly measurable-a natural upgrade path.
• Blind spots are inferred risk. A blind spot means "high-demand location with sparse recorded enforcement, warranting investigation," not proven uncaught parking.
• Timestamps are relative only. Bucket labels are data-derived hour ranges, not real-world periods; no "morning/evening peak" meaning is claimed.
• The enforcement-bias correlation (0.66) is 'materially influenced by,' not 'pure artefact.' Part of it is mechanical (more violations naturally involve more passing devices).
• Block_D (10-18) is thin. Its 4,588 records make single-cell categories in that bucket lower-confidence; the all-time Gi* gate mitigates this.
• Expected-demand time-variation is concentrated. E varies by block only near time-skewed generators (tech parks, markets); cells near all-day hubs (metros) stay stable, by design. The system's primary time-awareness is therefore Observed-driven-honest, but worth stating so the time control's behaviour is not over-read.
• The POI list is curated. The Expected score reflects major known demand generators, not an exhaustive scan-transparent and reproducible, but not comprehensive for minor generators.
• Per-bucket slicing thins the data. Cells flagged on only 2-3 violations in a bucket should not be over-read; coarse buckets and the all-time certification gate guard against this.

10. Reproducing the Pipeline

The pipeline is structured so the no-API stages can be built and validated independently of any external service. End-to-end order:

• Clean the raw CSV → cleaned Parquet (Section 5).
• Stage 1 → Observed score O(h,b). No API.
• Stage 2 → Gi* certification (gi_z, certified). No API.
• Stages 3-5 → reverse-geocode road names (MapmyIndia, paced + cached).
• Stage 6 → Expected score E(h) from POIs + road hierarchy. No API.
• Stage 7 → categorise into the 2×2; write the final scored table.
• Validate → sensitivity analysis (Section 8) and a geographic sanity pass on the top cells.

Key artefacts: gridlocps1_cleaned.parquet (cleaned input) and gridloc_final_scored.parquet (the deliverable: per cell × time-block, with O, E, gi_z, certified, category, priority_score, road_name, reason). All MapmyIndia responses are cached to disk so re-runs make no repeat calls.

Remaining phase: the dashboard-a MapmyIndia map of Bengaluru with cells coloured by category, a time-block control that redraws scores per shift, and a side panel listing the top confirmed hotspots and blind spots for the selected block. The modelling track that this report describes is complete; the dashboard consumes the frozen scored table without further modelling work.

GridLoc - Parking-Induced Congestion Intelligence - Methodology & Technical Report