YellowCab evaluation
Question
Can three chronological road images combined with past-and-current telemetry forecast a GPS-derived taxi maneuver approximately 20 seconds ahead better than tested single-modality and rule-based baselines?
Frozen evaluation contract
- Published model version:
v0.1-causal - Published Safetensors SHA-256:
af1a98cd96bb68d642f7a0adeedfacac37b230917afc15f3c87230a99a15b8a2 - Training manifest SHA-256:
8b92ad563892776cc25148f39fec6bf207e1e442c618b58fb07af7e2b899364d - Split SHA-256:
bfc5e960c8af32d0ea75a5e293bb95b3125111ace2cf8452eb42feb4bc2a1a5e - Seed:
20260726 - Primary metric: macro F1
The split is forward in time by complete route. The earliest 26 routes are used for training, the next five for validation, and the newest five for testing. Frames never cross route boundaries between splits.
Causal-input audit
The original dataset builder could use a future-bearing fallback when the
past-to-current displacement was too small to establish a reliable heading. The
causal manifest removes that fallback. When displacement is below four metres,
heading is represented as unknown using (sin=0, cos=1).
The audit changed 5,974 of 74,680 rows:
- 4,552 training rows
- 819 validation rows
- 603 test rows
The released v0.1-causal checkpoint was trained on the corrected manifest.
Labels still use future trajectory by definition; model inputs do not.
Metrics
Macro F1 is primary because the test set is imbalanced. Accuracy, balanced accuracy, weighted F1, log loss, Brier score, and 15-bin expected calibration error are secondary.
The macro-F1 confidence interval is computed with 500 bootstrap resamples of complete test routes. This preserves within-route dependence instead of treating 8,594 sequential examples as independent observations.
Reproduction from published predictions
The repository publishes labels, predictions, probabilities, and generic route
groups for every test example in eval_predictions.csv. It excludes images,
capture identifiers, timestamps, grid cells, and geographic data.
python -m pip install numpy pandas scikit-learn
python recompute_metrics.py
The script checks that probability rows sum to one and reproduces the core point estimates and route-clustered macro-F1 interval.
Selective prediction
selective_metrics.csv evaluates the confidence-based abstention rule. At the
default 0.45 threshold, the model retains 5,971 of 8,594 examples (69.5%
coverage). Accuracy among retained examples is 0.6583 and macro F1 is 0.4327.
Selective metrics describe only the accepted subset. They do not mean the abstained examples disappear operationally; a real system must route them to a fallback or human review process.
Interpretation
YellowCab has the highest macro F1 among the tested models. It does not win every
metric: telemetry gradient boosting has higher balanced accuracy, and a majority
classifier has higher raw accuracy because most examples are continue.
This supports use as a ranking, indexing, and research model. It does not support claims of safe actuation, universal geographic generalization, or global state-of-the-art status.
Remaining evaluation work
A stronger future release should add:
- an independently human-labeled gold test set;
- a separate calibration split;
- evaluation across additional cities, camera mounts, weather, and daylight;
- selective-risk curves for the abstention threshold;
- an operational video-retrieval study at fixed event recall; and
- independent replication on a public, task-compatible benchmark.