README.md

---
library_name: pytorch
license: apache-2.0
tags:
  - driving
  - multimodal
  - computer-vision
  - time-series
  - maneuver-forecasting
  - custom-handler
  - research
metrics:
  - f1
  - accuracy
model-index:
  - name: YellowCab v0.1-causal
    results:
      - task:
          type: image-classification
          name: Multimodal maneuver forecasting
        dataset:
          name: YellowCab private forward-route holdout
          type: private
          split: test
        metrics:
          - type: f1
            name: Macro F1
            value: 0.3915542658
          - type: accuracy
            name: Balanced accuracy
            value: 0.3915633404
          - type: accuracy
            name: Accuracy
            value: 0.5755178031
---

# YellowCab

YellowCab is a compact multimodal model for forecasting a taxi's observed
maneuver approximately 20 seconds ahead. It combines three chronological road
images with twelve past-and-current telemetry signals and returns calibrated
probabilities for:

- `continue`
- `slow`
- `stop`
- `turn_left`
- `turn_right`

The model has **5.29 million parameters**. It is intended for fleet-video
indexing, maneuver analytics, multimodal research, and prototyping—not vehicle
control.

## What is technically useful about it?

YellowCab tests a practical hypothesis: recent visual context and vehicle motion
history are more useful together than either signal alone.

On the route-held-out silver benchmark, the causal temporal-fusion checkpoint
reaches **0.3916 macro F1**, compared with **0.3414** for the strongest tested
past-only telemetry baseline and **0.3050** for the tested current-frame
vision baseline. This supports uses such as:

- assigning likely maneuver tags to fleet footage;
- retrieving candidate stop and turn events for human review;
- prioritizing uncertain sequences for data collection or annotation;
- studying disagreement between visual and motion signals; and
- prototyping calibrated, abstaining multimodal classifiers.

These results do not establish a global rank. YellowCab uses a private,
task-specific benchmark and has not been evaluated under a directly comparable
public autonomous-driving leaderboard protocol.

## Architecture

- Frozen ImageNet-pretrained EfficientNet-B0 visual encoder
- Three frames at nominal offsets `t-20 s`, `t-10 s`, and `t`
- Layer normalization followed by a one-layer unidirectional GRU
- Telemetry MLP over twelve normalized causal features
- Fusion classifier over five maneuver classes
- Temperature scaling and optional confidence-based abstention

The repository contains the exact architecture, configuration, integrity-checked
Safetensors checkpoint, and custom Hugging Face endpoint handler.

## Evaluation

### Protocol

- **74,680** examples across **36** routes
- **55,711** training examples from 26 routes
- **10,375** validation examples from 5 routes
- **8,594** test examples from the newest 5 completely held-out routes
- Complete forward-in-time route separation
- No frame from a validation or test route enters training
- Model inputs use only information available at or before the newest frame
- Labels are GPS-derived maneuver proxies approximately 20 seconds ahead
- Primary metric: macro F1
- 95% confidence interval: 500 route-clustered bootstrap samples

### Headline results

| Metric | YellowCab |
| --- | ---: |
| Macro F1 | **0.3916** |
| Route-bootstrap 95% CI | **0.3476–0.4281** |
| Balanced accuracy | **0.3916** |
| Accuracy | **0.5755** |
| Weighted F1 | **0.5807** |
| Log loss | **1.0841** |
| Brier score | **0.5563** |
| ECE, 15 bins | **0.0118** |

Raw accuracy is not the primary metric because `continue` is the majority
class. A majority classifier reaches 0.6232 accuracy while scoring only 0.1536
macro F1.

### Baselines

| Model | Macro F1 | Balanced accuracy | Accuracy |
| --- | ---: | ---: | ---: |
| **YellowCab: temporal vision + telemetry** | **0.3916** | 0.3916 | 0.5755 |
| Telemetry gradient boosting | 0.3414 | **0.4217** | 0.4147 |
| Current-frame vision linear model | 0.3050 | 0.2968 | 0.5752 |
| Past-motion rule | 0.2235 | 0.2238 | 0.4550 |
| Telemetry logistic regression | 0.1887 | 0.2872 | 0.1963 |
| Majority class | 0.1536 | 0.2000 | **0.6232** |

YellowCab improves macro F1 over every tested baseline, but the gradient-boosted
telemetry baseline has higher balanced accuracy. We report both rather than
claiming uniform dominance.

### Per-class results

| Class | Precision | Recall | F1 | Support |
| --- | ---: | ---: | ---: | ---: |
| Continue | 0.7595 | 0.7394 | **0.7493** | 5,356 |
| Slow | 0.4031 | 0.3544 | **0.3772** | 951 |
| Stop | 0.3754 | 0.3423 | **0.3581** | 634 |
| Turn left | 0.2447 | 0.3157 | **0.2757** | 833 |
| Turn right | 0.1897 | 0.2061 | **0.1975** | 820 |

Turn performance is the clearest weakness. YellowCab should be used to produce
ranked candidates or analytics, not treated as a reliable maneuver oracle.

### Confidence-based abstention

The default confidence threshold is 0.45. On this held-out set it retains
**69.5%** of examples, with **0.6583 selective accuracy** and **0.4327 selective
macro F1** among retained examples. Raising the threshold trades coverage for
higher accuracy:

| Threshold | Coverage | Selective accuracy | Selective macro F1 |
| --- | ---: | ---: | ---: |
| 0.00 | 100.0% | 0.5755 | 0.3916 |
| 0.35 | 91.0% | 0.6007 | 0.4061 |
| **0.45** | **69.5%** | **0.6583** | **0.4327** |
| 0.55 | 47.2% | 0.7324 | 0.4702 |
| 0.65 | 30.2% | 0.8092 | 0.5275 |

This makes the model more useful for candidate retrieval and review queues, but
these thresholds must be recalibrated after domain shift. Full results are in
[`selective_metrics.csv`](selective_metrics.csv).

![Benchmark comparison](benchmark.png)

![Held-out confusion matrix](confusion_matrix.png)

The complete checkpoint-specific report is in
[`evaluation.json`](evaluation.json). Recomputable, privacy-sanitized held-out
probabilities are in [`eval_predictions.csv`](eval_predictions.csv), and the
metric script is in [`recompute_metrics.py`](recompute_metrics.py). Additional
methodology and limitations are documented in
[`EVALUATION.md`](EVALUATION.md).

## Checkpoint identity

- Version: `v0.1-causal`
- Published checkpoint SHA-256:
  `af1a98cd96bb68d642f7a0adeedfacac37b230917afc15f3c87230a99a15b8a2`
- Training manifest SHA-256:
  `8b92ad563892776cc25148f39fec6bf207e1e442c618b58fb07af7e2b899364d`
- Split SHA-256:
  `bfc5e960c8af32d0ea75a5e293bb95b3125111ace2cf8452eb42feb4bc2a1a5e`
- Training seed: `20260726`

The causal-manifest audit replaced unavailable current headings with the neutral
encoding `(sin=0, cos=1)` rather than deriving them from future motion. The
published checkpoint was trained and evaluated on that corrected manifest.

## Input

Requests require exactly three base64-encoded JPEG or PNG frames ordered oldest
to newest. Optional frame timestamps must be strictly increasing.

All telemetry fields are required:

`current_speed_mps`, `past_speed_mps`, `acceleration_mps2`,
`past_turn_degrees`, `past_yaw_rate_deg_s`, `heading_sin`, `heading_cos`,
`gps_accuracy_m`, `hour_sin`, `hour_cos`, `dow_sin`, and `dow_cos`.

```json
{
  "inputs": {
    "frames": ["<base64 t-20>", "<base64 t-10>", "<base64 now>"],
    "frame_timestamps": [1721990000.0, 1721990010.2, 1721990020.1],
    "telemetry": {
      "current_speed_mps": 7.2,
      "past_speed_mps": 8.1,
      "acceleration_mps2": -0.045,
      "past_turn_degrees": 3.1,
      "past_yaw_rate_deg_s": 0.15,
      "heading_sin": 0.5,
      "heading_cos": 0.8660254038,
      "gps_accuracy_m": 4.0,
      "hour_sin": -0.7071067812,
      "hour_cos": 0.7071067812,
      "dow_sin": 0.7818314825,
      "dow_cos": 0.6234898019
    },
    "abstention_threshold": 0.45
  }
}
```

The custom `EndpointHandler` validates the request and returns the predicted
label, ordered class probabilities, confidence, and abstention state. This
custom multimodal architecture is not compatible with the standard
image-classification `pipeline()` or Hub widget.

## Local use

```bash
python -m venv .venv
# Activate the environment for your shell.
python -m pip install --requirement requirements.txt
```

```python
from handler import EndpointHandler

predict = EndpointHandler(".")
result = predict(request_body)
```

## Training data and privacy

The model was trained on privacy-redacted imagery collected from the operator's
taxi fleet under contributor agreements authorizing model training. No source
imagery, exact GPS records, route manifests, or personal identifiers are
distributed in this repository.

The published evaluation predictions use sequential example identifiers and
generic held-out route groups. Capture identifiers, timestamps, grid cells, and
locations have been removed.

## Intended uses

- Offline fleet-video indexing and search
- Maneuver analytics and aggregate research
- Candidate-event retrieval followed by human review
- Multimodal classification and calibration research
- Prototyping and collection-planning experiments

## Out-of-scope uses

Do not use YellowCab:

- as the sole input to steering, braking, throttle, or dispatch decisions;
- as a safety-certified driver-assistance component;
- to identify drivers, passengers, pedestrians, or locations;
- to infer intent, fault, liability, or legal compliance; or
- outside a target domain without measuring performance and calibration again.

## Limitations

- Ground truth is GPS-derived silver labeling, not independent human annotation.
- The evaluation covers five held-out routes from one operating domain.
- Geography, weather, camera placement, traffic, and telemetry quality can shift
  performance.
- Rare classes are substantially weaker than `continue`.
- Temperature scaling used the validation split rather than a separate
  calibration split.
- No membership-inference, model-inversion, or differential-privacy assessment
  has been completed.
- Benchmark results are author-reported and have not been independently
  replicated on a public dataset.

## License

Unless otherwise noted, the model weights and original code, configuration, and
documentation are licensed under the Apache License 2.0. Third-party components
remain subject to their respective terms; see
[`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md).

## Citation

```bibtex
@software{yellowcab_v0_1_causal_2026,
  author = {{The General Data Corporation}},
  title = {YellowCab: Multimodal Taxi Maneuver Forecasting},
  year = {2026},
  version = {v0.1-causal},
  url = {https://huggingface.co/generaldata/YellowCab}
}
```