--- library_name: pytorch license: apache-2.0 tags: - driving - multimodal - computer-vision - time-series - maneuver-forecasting - custom-handler - research metrics: - f1 - accuracy model-index: - name: YellowCab v0.1-causal results: - task: type: image-classification name: Multimodal maneuver forecasting dataset: name: YellowCab private forward-route holdout type: private split: test metrics: - type: f1 name: Macro F1 value: 0.3915542658 - type: accuracy name: Balanced accuracy value: 0.3915633404 - type: accuracy name: Accuracy value: 0.5755178031 --- # YellowCab YellowCab is a compact multimodal model for forecasting a taxi's observed maneuver approximately 20 seconds ahead. It combines three chronological road images with twelve past-and-current telemetry signals and returns calibrated probabilities for: - `continue` - `slow` - `stop` - `turn_left` - `turn_right` The model has **5.29 million parameters**. It is intended for fleet-video indexing, maneuver analytics, multimodal research, and prototyping—not vehicle control. ## What is technically useful about it? YellowCab tests a practical hypothesis: recent visual context and vehicle motion history are more useful together than either signal alone. On the route-held-out silver benchmark, the causal temporal-fusion checkpoint reaches **0.3916 macro F1**, compared with **0.3414** for the strongest tested past-only telemetry baseline and **0.3050** for the tested current-frame vision baseline. This supports uses such as: - assigning likely maneuver tags to fleet footage; - retrieving candidate stop and turn events for human review; - prioritizing uncertain sequences for data collection or annotation; - studying disagreement between visual and motion signals; and - prototyping calibrated, abstaining multimodal classifiers. These results do not establish a global rank. YellowCab uses a private, task-specific benchmark and has not been evaluated under a directly comparable public autonomous-driving leaderboard protocol. ## Architecture - Frozen ImageNet-pretrained EfficientNet-B0 visual encoder - Three frames at nominal offsets `t-20 s`, `t-10 s`, and `t` - Layer normalization followed by a one-layer unidirectional GRU - Telemetry MLP over twelve normalized causal features - Fusion classifier over five maneuver classes - Temperature scaling and optional confidence-based abstention The repository contains the exact architecture, configuration, integrity-checked Safetensors checkpoint, and custom Hugging Face endpoint handler. ## Evaluation ### Protocol - **74,680** examples across **36** routes - **55,711** training examples from 26 routes - **10,375** validation examples from 5 routes - **8,594** test examples from the newest 5 completely held-out routes - Complete forward-in-time route separation - No frame from a validation or test route enters training - Model inputs use only information available at or before the newest frame - Labels are GPS-derived maneuver proxies approximately 20 seconds ahead - Primary metric: macro F1 - 95% confidence interval: 500 route-clustered bootstrap samples ### Headline results | Metric | YellowCab | | --- | ---: | | Macro F1 | **0.3916** | | Route-bootstrap 95% CI | **0.3476–0.4281** | | Balanced accuracy | **0.3916** | | Accuracy | **0.5755** | | Weighted F1 | **0.5807** | | Log loss | **1.0841** | | Brier score | **0.5563** | | ECE, 15 bins | **0.0118** | Raw accuracy is not the primary metric because `continue` is the majority class. A majority classifier reaches 0.6232 accuracy while scoring only 0.1536 macro F1. ### Baselines | Model | Macro F1 | Balanced accuracy | Accuracy | | --- | ---: | ---: | ---: | | **YellowCab: temporal vision + telemetry** | **0.3916** | 0.3916 | 0.5755 | | Telemetry gradient boosting | 0.3414 | **0.4217** | 0.4147 | | Current-frame vision linear model | 0.3050 | 0.2968 | 0.5752 | | Past-motion rule | 0.2235 | 0.2238 | 0.4550 | | Telemetry logistic regression | 0.1887 | 0.2872 | 0.1963 | | Majority class | 0.1536 | 0.2000 | **0.6232** | YellowCab improves macro F1 over every tested baseline, but the gradient-boosted telemetry baseline has higher balanced accuracy. We report both rather than claiming uniform dominance. ### Per-class results | Class | Precision | Recall | F1 | Support | | --- | ---: | ---: | ---: | ---: | | Continue | 0.7595 | 0.7394 | **0.7493** | 5,356 | | Slow | 0.4031 | 0.3544 | **0.3772** | 951 | | Stop | 0.3754 | 0.3423 | **0.3581** | 634 | | Turn left | 0.2447 | 0.3157 | **0.2757** | 833 | | Turn right | 0.1897 | 0.2061 | **0.1975** | 820 | Turn performance is the clearest weakness. YellowCab should be used to produce ranked candidates or analytics, not treated as a reliable maneuver oracle. ### Confidence-based abstention The default confidence threshold is 0.45. On this held-out set it retains **69.5%** of examples, with **0.6583 selective accuracy** and **0.4327 selective macro F1** among retained examples. Raising the threshold trades coverage for higher accuracy: | Threshold | Coverage | Selective accuracy | Selective macro F1 | | --- | ---: | ---: | ---: | | 0.00 | 100.0% | 0.5755 | 0.3916 | | 0.35 | 91.0% | 0.6007 | 0.4061 | | **0.45** | **69.5%** | **0.6583** | **0.4327** | | 0.55 | 47.2% | 0.7324 | 0.4702 | | 0.65 | 30.2% | 0.8092 | 0.5275 | This makes the model more useful for candidate retrieval and review queues, but these thresholds must be recalibrated after domain shift. Full results are in [`selective_metrics.csv`](selective_metrics.csv). ![Benchmark comparison](benchmark.png) ![Held-out confusion matrix](confusion_matrix.png) The complete checkpoint-specific report is in [`evaluation.json`](evaluation.json). Recomputable, privacy-sanitized held-out probabilities are in [`eval_predictions.csv`](eval_predictions.csv), and the metric script is in [`recompute_metrics.py`](recompute_metrics.py). Additional methodology and limitations are documented in [`EVALUATION.md`](EVALUATION.md). ## Checkpoint identity - Version: `v0.1-causal` - Published checkpoint SHA-256: `af1a98cd96bb68d642f7a0adeedfacac37b230917afc15f3c87230a99a15b8a2` - Training manifest SHA-256: `8b92ad563892776cc25148f39fec6bf207e1e442c618b58fb07af7e2b899364d` - Split SHA-256: `bfc5e960c8af32d0ea75a5e293bb95b3125111ace2cf8452eb42feb4bc2a1a5e` - Training seed: `20260726` The causal-manifest audit replaced unavailable current headings with the neutral encoding `(sin=0, cos=1)` rather than deriving them from future motion. The published checkpoint was trained and evaluated on that corrected manifest. ## Input Requests require exactly three base64-encoded JPEG or PNG frames ordered oldest to newest. Optional frame timestamps must be strictly increasing. All telemetry fields are required: `current_speed_mps`, `past_speed_mps`, `acceleration_mps2`, `past_turn_degrees`, `past_yaw_rate_deg_s`, `heading_sin`, `heading_cos`, `gps_accuracy_m`, `hour_sin`, `hour_cos`, `dow_sin`, and `dow_cos`. ```json { "inputs": { "frames": ["", "", ""], "frame_timestamps": [1721990000.0, 1721990010.2, 1721990020.1], "telemetry": { "current_speed_mps": 7.2, "past_speed_mps": 8.1, "acceleration_mps2": -0.045, "past_turn_degrees": 3.1, "past_yaw_rate_deg_s": 0.15, "heading_sin": 0.5, "heading_cos": 0.8660254038, "gps_accuracy_m": 4.0, "hour_sin": -0.7071067812, "hour_cos": 0.7071067812, "dow_sin": 0.7818314825, "dow_cos": 0.6234898019 }, "abstention_threshold": 0.45 } } ``` The custom `EndpointHandler` validates the request and returns the predicted label, ordered class probabilities, confidence, and abstention state. This custom multimodal architecture is not compatible with the standard image-classification `pipeline()` or Hub widget. ## Local use ```bash python -m venv .venv # Activate the environment for your shell. python -m pip install --requirement requirements.txt ``` ```python from handler import EndpointHandler predict = EndpointHandler(".") result = predict(request_body) ``` ## Training data and privacy The model was trained on privacy-redacted imagery collected from the operator's taxi fleet under contributor agreements authorizing model training. No source imagery, exact GPS records, route manifests, or personal identifiers are distributed in this repository. The published evaluation predictions use sequential example identifiers and generic held-out route groups. Capture identifiers, timestamps, grid cells, and locations have been removed. ## Intended uses - Offline fleet-video indexing and search - Maneuver analytics and aggregate research - Candidate-event retrieval followed by human review - Multimodal classification and calibration research - Prototyping and collection-planning experiments ## Out-of-scope uses Do not use YellowCab: - as the sole input to steering, braking, throttle, or dispatch decisions; - as a safety-certified driver-assistance component; - to identify drivers, passengers, pedestrians, or locations; - to infer intent, fault, liability, or legal compliance; or - outside a target domain without measuring performance and calibration again. ## Limitations - Ground truth is GPS-derived silver labeling, not independent human annotation. - The evaluation covers five held-out routes from one operating domain. - Geography, weather, camera placement, traffic, and telemetry quality can shift performance. - Rare classes are substantially weaker than `continue`. - Temperature scaling used the validation split rather than a separate calibration split. - No membership-inference, model-inversion, or differential-privacy assessment has been completed. - Benchmark results are author-reported and have not been independently replicated on a public dataset. ## License Unless otherwise noted, the model weights and original code, configuration, and documentation are licensed under the Apache License 2.0. Third-party components remain subject to their respective terms; see [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md). ## Citation ```bibtex @software{yellowcab_v0_1_causal_2026, author = {{The General Data Corporation}}, title = {YellowCab: Multimodal Taxi Maneuver Forecasting}, year = {2026}, version = {v0.1-causal}, url = {https://huggingface.co/generaldata/YellowCab} } ```