| # YellowCab evaluation |
|
|
| ## Question |
|
|
| Can three chronological road images combined with past-and-current telemetry |
| forecast a GPS-derived taxi maneuver approximately 20 seconds ahead better than |
| tested single-modality and rule-based baselines? |
|
|
| ## Frozen evaluation contract |
|
|
| - Published model version: `v0.1-causal` |
| - Published Safetensors SHA-256: |
| `af1a98cd96bb68d642f7a0adeedfacac37b230917afc15f3c87230a99a15b8a2` |
| - Training manifest SHA-256: |
| `8b92ad563892776cc25148f39fec6bf207e1e442c618b58fb07af7e2b899364d` |
| - Split SHA-256: |
| `bfc5e960c8af32d0ea75a5e293bb95b3125111ace2cf8452eb42feb4bc2a1a5e` |
| - Seed: `20260726` |
| - Primary metric: macro F1 |
|
|
| The split is forward in time by complete route. The earliest 26 routes are used |
| for training, the next five for validation, and the newest five for testing. |
| Frames never cross route boundaries between splits. |
|
|
| ## Causal-input audit |
|
|
| The original dataset builder could use a future-bearing fallback when the |
| past-to-current displacement was too small to establish a reliable heading. The |
| causal manifest removes that fallback. When displacement is below four metres, |
| heading is represented as unknown using `(sin=0, cos=1)`. |
|
|
| The audit changed 5,974 of 74,680 rows: |
|
|
| - 4,552 training rows |
| - 819 validation rows |
| - 603 test rows |
|
|
| The released `v0.1-causal` checkpoint was trained on the corrected manifest. |
| Labels still use future trajectory by definition; model inputs do not. |
|
|
| ## Metrics |
|
|
| Macro F1 is primary because the test set is imbalanced. Accuracy, balanced |
| accuracy, weighted F1, log loss, Brier score, and 15-bin expected calibration |
| error are secondary. |
|
|
| The macro-F1 confidence interval is computed with 500 bootstrap resamples of |
| complete test routes. This preserves within-route dependence instead of treating |
| 8,594 sequential examples as independent observations. |
|
|
| ## Reproduction from published predictions |
|
|
| The repository publishes labels, predictions, probabilities, and generic route |
| groups for every test example in `eval_predictions.csv`. It excludes images, |
| capture identifiers, timestamps, grid cells, and geographic data. |
|
|
| ```bash |
| python -m pip install numpy pandas scikit-learn |
| python recompute_metrics.py |
| ``` |
|
|
| The script checks that probability rows sum to one and reproduces the core |
| point estimates and route-clustered macro-F1 interval. |
|
|
| ## Selective prediction |
|
|
| `selective_metrics.csv` evaluates the confidence-based abstention rule. At the |
| default 0.45 threshold, the model retains 5,971 of 8,594 examples (69.5% |
| coverage). Accuracy among retained examples is 0.6583 and macro F1 is 0.4327. |
|
|
| Selective metrics describe only the accepted subset. They do not mean the |
| abstained examples disappear operationally; a real system must route them to a |
| fallback or human review process. |
|
|
| ## Interpretation |
|
|
| YellowCab has the highest macro F1 among the tested models. It does not win every |
| metric: telemetry gradient boosting has higher balanced accuracy, and a majority |
| classifier has higher raw accuracy because most examples are `continue`. |
|
|
| This supports use as a ranking, indexing, and research model. It does not support |
| claims of safe actuation, universal geographic generalization, or global |
| state-of-the-art status. |
|
|
| ## Remaining evaluation work |
|
|
| A stronger future release should add: |
|
|
| - an independently human-labeled gold test set; |
| - a separate calibration split; |
| - evaluation across additional cities, camera mounts, weather, and daylight; |
| - selective-risk curves for the abstention threshold; |
| - an operational video-retrieval study at fixed event recall; and |
| - independent replication on a public, task-compatible benchmark. |
|
|
