MODEL_REPORT.md

Gridlock — Model Report & Implementation Guide

A complete account of everything built for the Astram event dataset: the predicted variables, the models behind each, every metric on the chronological hold-out, which implementation is best for each task, and the exact single commands to reproduce the best models.

• Dataset: astram_events.csv — 8,173 raw → 8,057 clean Bengaluru traffic events, 9 Nov 2023 → 8 Apr 2024 (150-day span, 46 raw columns).
• Two independent implementations:
  1. Main pipeline (src/, run with python -m src.train) — three operational targets over one shared, leakage-safe feature pipeline.
  2. Standalone hotspot model (hotspot_model.py) — a fourth, newly-engineered forward-looking target, fully self-contained.
• Evaluation discipline (all tasks): chronological train→test split (train on the past, test on the future), skew-aware metrics, leakage-controlled features, probability calibration, and decision thresholds chosen on held-out data only.

---

0. TL;DR — the four tasks at a glance

#	Predicted variable	Type (positive rate)	Best implementation	Headline metric	Single command
T1	y_road_closure — will the event need a closure/diversion?	Binary (7.2%)	Main pipeline	PR-AUC 0.326 (4.5× base), MCC 0.40 @ balanced	python -m src.train
T2	y_high_priority — High vs Low operational priority	Binary (62.2%)	Main pipeline	PR-AUC 0.984, MCC 0.90	python -m src.train
T3	y_duration_min — minutes until cleared	Regression (heavy-tailed)	Main pipeline	log-R² 0.251, median AE 74 min	python -m src.train
T4	y_hotspot — will this ~110 m spot reoffend (≥2 events in 14 days)?	Binary (14.1%)	Standalone hotspot model	PR-AUC 0.441 (2.8× base), recall 0.88	python hotspot_model.py train

Get every best model in one line:

python -m src.train && python hotspot_model.py train

The first command trains and deploys the best T1/T2/T3 models; the second trains and deploys the best T4 model. Details in §6.

---

1. The two implementations

1A. Main pipeline — src/ (python -m src.train)

Three separately tuned models over one shared feature pipeline (multi-task by construction, not a single fragile multi-head net). Each model is a stacked ensemble:

Optuna-tuned LightGBM  ┐
Optuna-tuned XGBoost   ├─►  logistic meta-learner on out-of-fold preds
Optuna-tuned CatBoost  ┘     └─►  isotonic calibration  └─►  decision threshold

• Duration uses the same ensemble on a log1p target for the point estimate, plus p10/p50/p90 quantile models with a conformal correction for an honest 80% prediction interval.
• Shared features: temporal + cyclical encodings, spatial bins (geo_cluster KMeans), causal (past-only) hotspot/recurrence counts, empirical-Bayes causal target-rate encodings, multilingual sentence-transformer embeddings (per-task PCA width), a bilingual impact lexicon, and missingness flags.

1B. Standalone hotspot model — hotspot_model.py

A single self-contained file (no import from src/) with its own cleaning, target, features, training and prediction CLI. One calibrated LightGBM classifier on a strictly causal feature set, with an isotonic calibrator and a recall-favouring F2 threshold. Artifacts live under hotspot_artifacts/.

---

2. T1 — Road closure (y_road_closure)

What it predicts: at report time, will this event require a road closure or diversion? Drives barricading and diversion planning. Model: stacked LightGBM + XGBoost + CatBoost → logistic meta → isotonic calibration → F2 threshold. Embedding PCA width 96 (closure-specific). Test set: n = 1,611, positive rate 7.2% (≈116 real closures).

Metric	Value	Read as
Average precision (PR-AUC)	0.326	4.5× the 7.2% base rate
ROC-AUC	0.835	ranks closures well above non-closures
Recall	0.655	catches ~2/3 of real closures…
Precision	0.277	…at the recall-favoured deployed threshold (0.10)
F2 / F1	0.515 / 0.390	recall-weighted
MCC	0.360	at deployed threshold (0.40 at balanced point)
Balanced accuracy	0.761
Brier	0.057	well-calibrated probabilities
Confusion @ deploy	TP 76 · FP 198 · FN 40 · TN 1297	76/116 closures pre-flagged, 40 missed

Base learners (OOF AP): LightGBM 0.380 · XGBoost 0.392 · CatBoost 0.389 → stack 0.396.

The threshold is a policy choice — the same model, scored at different operating points (from reports/metrics.json):

Operating point	Threshold	Recall	Precision	F1	F2	MCC
MCC-optimal (balanced)	0.213	0.509	0.399	0.447	0.482	0.402
F2-optimal (recall-leaning, deployed)	0.097	0.655	0.277	0.390	0.515	0.360
Recall ≥ 0.8 (never miss)	0.057	0.828	0.151	0.255	0.436	0.247

Why it's hard (and honest): the strongest raw closure predictors (end-point coordinates / route_path) are consequences of the closure decision and were removed as leakage; closure is partly discretionary; and there is real temporal drift (OOF AP 0.40 → test 0.33). The PR-AUC of 0.326 is genuine signal at 4.5× the base rate.

---

3. T2 — Priority (y_high_priority)

What it predicts: High vs Low operational priority. Drives the manpower tier. Model: same stacked ensemble; corridor deliberately excluded (it makes the label a trivial one-field lookup). Embedding PCA width 32. Test set: n = 1,611, positive rate 62.2%.

Metric	Value
Average precision (PR-AUC)	0.984
ROC-AUC	0.980
F1	0.963
Precision / Recall	0.951 / 0.975
Balanced accuracy	0.946
MCC	0.901
Brier	0.040
Confusion @ deploy (thr 0.38)	TP 977 · FP 50 · FN 25 · TN 559

Base learners (OOF AP): LightGBM 0.991 · XGBoost 0.988 · CatBoost 0.986 → stack 0.989.

Note: even without corridor, priority is near-deterministic from location (it encodes "is this on a designated priority corridor?"), which is legitimately knowable at report time. The high score reflects a genuinely easy geofencing task, not leakage — verified via feature importance and per-junction purity. This is the "good" task with little headroom left.

---

4. T3 — Duration (y_duration_min)

What it predicts: minutes from start until the event is cleared (reconstructed by coalescing resolved → closed → end minus start, cleaned of non-positive and automated batch-closure rows). Drives the manpower estimate + clearance interval. Model: stacked ensemble on a log1p target (winsorised p99) for the point estimate, plus p10/p50/p90 quantile models with conformal correction for the interval. Embedding PCA width 32. Test set: n = 630 valid events (train n = 2,568).

Metric	Value	Note
R² (log scale)	0.251	honest fit on the heavy-tailed target
Median absolute error	74 min	typical incident error
MAE (log scale)	1.44	central-tendency error
MAE / RMSE (raw min)	2411 / 5581	inflated by the multi-week construction tail
R² (raw min)	0.092	misleading — dominated by 140-day outliers
MAPE	17.2
80% interval coverage	0.78	conformalized → ≈ nominal 80%
Median interval width	119 min	actionable uncertainty band

Why two R² numbers: raw-minute R² (0.09) is dominated by a handful of days-to-weeks construction events; the log-scale R² (0.251) is the honest measure of central-tendency fit, and the 74-min median error is what a control room actually experiences on normal incidents.

---

5. T4 — Chronic hotspot early warning (y_hotspot) — NEW

What it predicts (engineered from scratch): at the moment an event is reported at a location, will that same ~110 m spot generate ≥ 2 more events within the next 14 days? This is a recurring-hotspot early warning — instead of repeatedly firefighting the same junction/pothole/water-logging spot, the control room gets a flag to send a root-cause fix (drainage, resurfacing, a permanent marshal). Model: single calibrated LightGBM (isotonic + F2 threshold), built entirely on strictly past-only causal features. Test set: n = 1,383, base rate 15.6% (overall positive rate 14.1% across 6,914 labelable rows).

Metric	Value	Read as
Average precision (PR-AUC)	0.441	2.8× the 15.6% base rate
ROC-AUC	0.790
Brier	0.106
Recall (deployed F2, thr 0.071)	0.875	catches 189 of 216 emerging hotspots
Precision (deployed)	0.259	early-warning favours recall
F1 / MCC (deployed)	0.400 / 0.299
Confusion @ deploy	TP 189 · FP 541 · FN 27 · TN 626	only 27 emerging hotspots missed

Alternative balanced operating point (MCC-optimal, thr 0.138): recall 0.593, precision 0.414, MCC 0.381, F2 0.546.

Leakage safety: every feature uses only strictly-earlier events; the label uses only strictly-later events (disjoint time windows); right-censored rows (without a full 14-day future) are dropped from train/test. Top features (gain): junc_win30, zone_win30, ps_win30, police_station, lat/long, loc_days_since_first/last, area_to_loc_ratio, loc_rate_per_day.

Face validity: the highest-risk events are recurring construction at HAL Old Airport (metro work) and potholes at J.P. Nagar — exactly the chronic spots worth a permanent fix. Honest caveat: brand-new locations (no prior history) almost never turn chronic (6 positives in 383 first-ever sites), and the model correctly assigns them low risk rather than inventing signal.

---

6. How to get the best models

Best implementation per task

Task	Best implementation	Command	Why
T1 Road closure	Main pipeline (src/)	python -m src.train	Deploys the spw = 1 + PCA-96 config (PR-AUC 0.326), the best validated closure model.
T2 Priority	Main pipeline (src/)	python -m src.train	Only and best implementation (PR-AUC 0.984).
T3 Duration	Main pipeline (src/)	python -m src.train	Only and best implementation (log-R² 0.251 + calibrated interval).
T4 Chronic hotspot	Standalone (hotspot_model.py)	python hotspot_model.py train	Self-contained forward-looking model (PR-AUC 0.441).

Single command for the best models

# from the repo root, with the venv active:
source .venv/bin/activate

# Best T1 + T2 + T3 (Optuna-tuned) AND best T4, one line:
python -m src.train && python hotspot_model.py train

• python -m src.train → writes the 6 main model artifacts to models/, metrics to reports/metrics.json, and figures to reports/figures/.
• python hotspot_model.py train → writes hotspot_artifacts/hotspot_bundle.joblib, hotspot_history.parquet, and hotspot_metrics.json.

Inference (after training)

# T1/T2/T3 — closure prob, manpower tier, officers, barricading, diversion, duration interval:
python -m src.predict

# T4 — score events and list the top emerging hotspots:
python hotspot_model.py predict

Useful variants

GRIDLOCK_NO_TUNE=1 python -m src.train            # fast smoke run (skips Optuna)
GRIDLOCK_NO_TRANSFORMER=1 python -m src.train      # offline TF-IDF text fallback
python -m src.train_best                           # focused closure model + operating-point table
GRIDLOCK_CLOSURE_SPW=3 python -m src.train_best     # trade some PR-AUC for more closure recall

---

7. Leakage control (applies to all tasks)

• LEAKAGE_COLUMNS — anything known only after the event unfolds (status, all resolution timestamps, resolved_at_*, comment) is never a feature. The biggest trap: end-point coordinates / route_path are populated only when a closure/diversion segment is drawn, so has_end_point alone "predicts" closure at AP ≈ 0.98 — these are excluded as leakage.
• Causal features only — every history/recurrence/target-rate feature sees strictly earlier events (shift(1) / append-after accumulators).
• Train-only fitting — categorical vocabularies, embedding PCA, calibration and thresholds are learned on training rows; the chronological test set is untouched until final scoring.
• corridor excluded from T2 only — it makes priority a trivial lookup.

---

8. Artifact map

Path	Produced by	Contents
models/*.joblib (6)	python -m src.train	T1/T2/T3 models, preprocessors, calibrators, thresholds
models/closure_model_best.joblib	python -m src.train_best	focused closure model
models/geo_kmeans.joblib	python -m src.train	spatial bin assignment for inference
data/processed/history.parquet	python -m src.train	causal target-rate history for inference
reports/metrics.json	python -m src.train	full T1/T2/T3 metrics + operating points
reports/figures/*.png	python -m src.train	PR, calibration, SHAP
hotspot_artifacts/hotspot_bundle.joblib	python hotspot_model.py train	T4 model + isotonic + threshold + dtypes
hotspot_artifacts/hotspot_history.parquet	python hotspot_model.py train	event log for causal feature replay
hotspot_artifacts/hotspot_metrics.json	python hotspot_model.py train	full T4 metrics + operating points

---

9. Summary

• Four predicted variables, each engineered from a dataset that ships with no ready-made label, each forecastable strictly from report-time information.
• T2 (priority) is the saturated "easy" task (0.984); T1 (closure) and T3 (duration) are the genuinely hard operational tasks, improved to PR-AUC 0.326 and log-R² 0.251 through per-task embedding width and causal target-rate features; T4 (chronic hotspot) is a new forward-looking early-warning model at PR-AUC 0.441 with 0.88 recall.
• One command per implementation reproduces every best model: python -m src.train && python hotspot_model.py train.