reference/code/Fast-dLLM/fast_ddrive/data/README.md

Waymo End-to-End Dataset Layout (Fast-dDrive)

Fast-dDrive trains and evaluates on the Waymo End-to-End Driving (WOD-E2E) dataset. We do not redistribute Waymo data — follow the official process and preprocess it into the JSON layout below.

1. Obtain the raw data

1. Apply for access to Waymo Open Dataset: https://waymo.com/open/
2. Download the WOD-E2E train and val tfrecord shards.
3. Optional: pre-compute the GT pickle for evaluation (see below).

2. Convert to Fast-dDrive JSON

Each training / val sample is a JSON object roughly of the form:

{
  "id": "scene_uuid",
  "image": [
    "waymo_train/scene_uuid/frame_0.jpg",
    "waymo_train/scene_uuid/frame_1.jpg",
    "..."
  ],
  "conversations": [
    {"from": "human", "value": "<image>...<image>Drive task prompt..."},
    {"from": "gpt",   "value": "{\"critical_objects\": {...}, \"future_meta_behavior\": {...}, \"trajectory\": \"[[+003.30,-000.01], ...]\", \"explanation\": \"...\"}"}
  ]
}

Notes:

• The model output is a JSON-shaped string; trajectory waypoints use the +XXX.XX,-XXX.XX zero-padded format (parsed by eval/batch_inference.py).

A reference preprocessing script will be added under fast_ddrive/data/. For now, point the training scripts at your own conversion via env vars:

export DATASET_JSON=/path/to/waymo_train.json
export IMAGE_FOLDER=/path/to/image/root      # JSON image paths are relative to this
export EVAL_JSON=/path/to/waymo_val.json
export IMAGE_ROOT=/path/to/image/root        # same root used for eval

3. Official Waymo ADE / RFS evaluation

eval/evaluate_waymo_metrics.py computes ADE@3s, ADE@5s and Rater Feedback Score (RFS). It requires tensorflow + waymo-open-dataset-tf-2-12-0, which conflict with the inference stack — install in a separate conda env (we call ours autovla):

conda create -n autovla python=3.10
conda activate autovla
pip install tensorflow==2.12.0 waymo-open-dataset-tf-2-12-0 numpy tqdm
python eval/evaluate_waymo_metrics.py \
    --pred_json /path/to/predictions.json \
    --gt_tfrecords "/path/to/val*.tfrecord*" \
    --output_dir results/

For faster eval on compute nodes, cache the GT dict once:

python eval/evaluate_waymo_metrics.py \
    --pred_json /path/to/predictions.json \
    --save_gt_dict_pkl gt_dict_val.pkl \
    --gt_tfrecords "/path/to/val*.tfrecord*"
# subsequent runs:
python eval/evaluate_waymo_metrics.py \
    --pred_json /path/to/predictions.json \
    --gt_dict_pkl gt_dict_val.pkl