add: unidrive_vla_nusc_base_evaluation

PLANS/agile-discovering-sunbeam.md

@@ -0,0 +1,51 @@
+# Plan: unidrive_vla_nusc_base_quantize_all.py
+
+## Goal
+量化 `owl10/UniDriveVLA_Nusc_Base_Stage1` (Qwen3-VL-2B, bf16) 到 INT8 和 NVFP4，使用 Nvidia ModelOpt PyTorch 原生 API，并在 DriveLM 上评测。
+
+## Architecture
+单文件脚本，分为 3 个阶段：
+1. **量化阶段** — 使用 `modelopt.torch.quantization` 对 PyTorch 模型做 PTQ
+2. **评测阶段** — 复用 `unidrive_vla_nusc_base_evaluation.py` 的评测逻辑
+3. **汇总阶段** — 打印 bf16/INT8/NVFP4 的对比结果
+
+## Key Decisions
+
+### ModelOpt API
+- `modelopt.torch.quantization.quantize()` + `QConfig`
+- INT8: `percentile` calibration (对 VLM 比 max 更稳健)
+- NVFP4: `awq` (weight-only, 适合大模型)
+
+### Calibration Data
+- 来源: DriveLM `v1_1_train_nus.json` 前 128 个样本
+- 每个样本生成 multi-view 输入 (6 相机 + 文本) 送入模型 forward pass
+- 用 `forward_loop` 参数传给 `mtq.quantize()`
+
+### Model Loading
+- `Qwen3VLForConditionalGeneration.from_pretrained()` bf16
+- 量化在 bf16 基础上做 (ModelOpt 0.44+ 支持 bf16 基模型)
+
+### Evaluation
+- 完全复用 `unidrive_vla_nusc_base_evaluation.py` 的:
+  - `load_drivelm_dataset()` — 加载数据
+  - `run_inference()` — 模型推理
+  - `compute_accuracy()` / `compute_language_metrics()` / `compute_gpt_score()` / `compute_match_score()` / `compute_final_score()` — 指标
+- 在 val split 上评测，subset=200 加速
+
+## Script Structure
+
+```
+unidrive_vla_nusc_base_quantize_all.py
+├── Imports (复用 evaluation.py 的 metric 函数和 dataset 加载)
+├── Config constants
+├── load_calibration_data() — 从 DriveLM 取 128 样本构造 forward_loop
+├── quantize_int8()  — mtq.quantize(model, INT8_CFG, forward_loop)
+├── quantize_nvfp4() — mtq.quantize(model, NVFP4_CFG, forward_loop)
+├── evaluate_quantized() — 在 DriveLM val 上跑全套指标
+├── main() — 依次: bf16 baseline → INT8 → NVFP4, 打印汇总
+└── CLI args: --mode {all,int8,nvfp4} --subset --split --skip_baseline_eval
+```
+
+## Files
+- **Create**: `unidrive_vla_nusc_base_quantize_all.py` (~500 lines)
+- **Reference** (不修改): `unidrive_vla_nusc_base_evaluation.py`

README.md

@@ -8,7 +8,7 @@ pinned: false
 license: apache-2.0
 ---
 
-
+QAT
 
 脚本 hgnetv2_b2_qat.py 已完成并验证。以下是摘要：
 
@@ -31,25 +31,16 @@ license: apache-2.0
 
   CLI 标志
 
-  ┌─────────────────┬────────┬───────────────────────────────┐
-  │      标志       │ 默认值 │              描述               │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --epochs        │ 3      │ QAT 微调周期数                  │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --lr            │ 1e-5   │ 学习率                          │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --batch-size    │ 32     │ 训练/评估的批处理大小           │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --calib-samples │ 1000   │ PTQ 校准图像数量                │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --train-samples │ 0      │ 限制 QAT 训练样本（0=全部）     │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --device        │ cpu    │ 训练设备（如可用则使用 cuda）   │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --subset        │ 0      │ 评估前 N 张图像（0=所有 5万张） │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --skip-eval     │ -      │ 跳过中间 PTQ/QAT PyTorch 评估   │
-  ├─────────────────┼────────┼─────────────────────────────────┤
+  |      标志       | 默认值 |              描述               |
+  | --- | ---- | ------------------------------ |
+  | --epochs        | 3      | QAT 微调周期数                  |
+  | --lr            | 1e-5   | 学习率                          |
+  | --batch-size    | 32     | 训练/评估的批处理大小           |
+  | --calib-samples | 1000   | PTQ 校准图像数量                |
+  | --train-samples | 0      | 限制 QAT 训练样本（0=全部）     |
+  | --device        | cpu    | 训练设备（如可用则使用 cuda）   |
+  | --subset        | 0      | 评估前 N 张图像（0=所有 5万张） |
+  | --skip-eval     | -      | 跳过中间 PTQ/QAT PyTorch 评估   |
 
   - 默认使用 CPU，因为 modelopt 的 CUDA 假量化扩展在此系统上无法编译 (nvcc 不支持 c++20)。请使用 --device cuda，如果您的 CUDA 扩展正常工作。
   - 使用 modelopt PyTorch QAT (modelopt.torch.quantization) 而非已弃用的 torch.ao.quantization，因为 modelopt 已安装并支持 INT8_DEFAULT_CFG
@@ -58,98 +49,60 @@ license: apache-2.0
 
   CLI 标志
 
-  ┌─────────────────┬────────┬─────────────────────────────────┐
-  │      标志       │ 默认值 │              描述               │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --epochs        │ 3      │ QAT 微调周期数                  │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --lr            │ 1e-5   │ 学习率                          │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --batch-size    │ 32     │ 训练/评估的批处理大小           │
-  ├─────────────────┼────────┼───��─────────────────────────────┤
-  │ --calib-samples │ 1000   │ PTQ 校准图像数量                │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --train-samples │ 0      │ 限制 QAT 训练样本（0=全部）     │
-  ├─────────────────┼────────┼─────────────────────────────────┤
+  |      标志       | 默认值 |              描述               |
+  | --- | ---- | ------------------------------ |
+  | --epochs        | 3      | QAT 微调周期数                  |
+  | --lr            | 1e-5   | 学习率                          |
+  | --calib-samples | 1000   | PTQ 校准图像数量                |
+  | --train-samples | 0      | 限制 QAT 训练样本（0=全部）     |
   下的 CNN 模型。
   - ONNX 导出使用 dynamo=False，因为 modelopt 中的数据相关控制流（if min_amax < 0）与 torch.export 不兼容。
 
   CLI 标志
 
-  ┌─────────────────┬────────┬─────────────────────────────────┐
-  │      标志       │ 默认值 │              描述               │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --epochs        │ 3      │ QAT 微调周期数                  │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --lr            │ 1e-5   │ 学习率                          │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --batch-size    │ 32     │ 训练/评估的批处理大小           │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --calib-samples │ 1000   │ PTQ 校准图像数量                │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --train-samples │ 0      │ 限制 QAT 训练样本（0=全部）     │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --device        │ cpu    │ 训练设备（如可用则使用 cuda）   │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --subset        │ 0      │ 评估前 N 张图像（0=所有 5万张） │
-  │ --lr            │ 1e-5   │ 学习率                          │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --batch-size    │ 32     │ 训练/评估的批处理大小           │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --calib-samples │ 1000   │ PTQ 校准图像数量                │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --train-samples │ 0      │ 限制 QAT 训练样本（0=全部）     │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --device        │ cpu    │ 训练设备（如可用则使用 cuda）   │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --subset        │ 0      │ 评估前 N 张图像（0=所有 5万张） │
-  │ --batch-size    │ 32     │ 训练/评估的批处理大小           │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --calib-samples │ 1000   │ PTQ 校准图像数量                │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --train-samples │ 0      │ 限制 QAT 训练样本（0=全部）     │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --device        │ cpu    │ 训练设备（如可用则使用 cuda）   │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --subset        │ 0      │ 评估前 N 张图像（0=所有 5万张） │
-  │ --train-samples │ 0      │ 限制 QAT 训练样本（0=全部）     │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --device        │ cpu    │ 训练设备（如可用则使用 cuda）   │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --subset        │ 0      │ 评估前 N 张图像（0=所有 5万张） │
-  │ --device        │ cpu    │ 训练设备（如可用则使用 cuda）   │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --subset        │ 0      │ 评估前 N 张图像（0=所有 5万张） │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --subset        │ 0      │ 评估前 N 张图像（0=所有 5万张） │
+  |      标志       | 默认值 |              描述               |
+  | --- | ---- | ------------------------------ |
+  | --epochs        | 3      | QAT 微调周期数                  |
+  | --lr            | 1e-5   | 学习率                          |
+  | --batch-size    | 32     | 训练/评估的批处理大小           |
+  | --calib-samples | 1000   | PTQ 校准图像数量                |
+  | --train-samples | 0      | 限制 QAT 训练样本（0=全部）     |
+  | --device        | cpu    | 训练设备（如可用则使用 cuda）   |
+  | --subset        | 0      | 评估前 N 张图像（0=所有 5万张） |
+  | --lr            | 1e-5   | 学习率                          |
+  | --batch-size    | 32     | 训练/评估的批处理大小           |
+  | --calib-samples | 1000   | PTQ 校准图像数量                |
+  | --train-samples | 0      | 限制 QAT 训练样本（0=全部）     |
+  | --device        | cpu    | 训练设备（如可用则使用 cuda）   |
+  | --subset        | 0      | 评估前 N 张图像（0=所有 5万张） |
+  | --batch-size    | 32     | 训练/评估的批处理大小           |
+  | --calib-samples | 1000   | PTQ 校准图像数量                |
+  | --train-samples | 0      | 限制 QAT 训练样本（0=全部）     |
+  | --device        | cpu    | 训练设备（如可用则使用 cuda）   |
+  | --subset        | 0      | 评估前 N 张图像（0=所有 5万张） |
+  | --train-samples | 0      | 限制 QAT 训练样本（0=全部）     |
+  | --device        | cpu    | 训练设备（如可用则使用 cuda）   |
+  | --subset        | 0      | 评估前 N 张图像（0=所有 5万张） |
+  | --device        | cpu    | 训练设备（如可用则使用 cuda）   |
+  | --subset        | 0      | 评估前 N 张图像（0=所有 5万张） |
+  | --subset        | 0      | 评估前 N 张图像（0=所有 5万张） |
   下的 CNN 模型。
   - ONNX 导出使用 dynamo=False，因为 modelopt 中的数据相关控制流（if min_amax < 0）与 torch.export 不兼容。
 
   CLI 标志
 
-  ┌─────────────────┬────────┬─────────────────────────────────┐
-  │      标志       │ 默认值 │              描述               │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --epochs        │ 3      │ QAT 微调周期数                  │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --lr            │ 1e-5   │ 学习率                          │
-  ├────��────────────┼────────┼─────────────────────────────────┤
-  │ --batch-size    │ 32     │ 训练/评估的批处理大小           │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --calib-samples │ 1000   │ PTQ 校准图像数量                │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --train-samples │ 0      │ 限制 QAT 训练样本（0=全部）     │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --device        │ cpu    │ 训练设备（如可用则使用 cuda）   │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --subset        │ 0      │ 评估前 N 张图像（0=所有 5万张） │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --skip-eval     │ -      │ 跳过中间 PTQ/QAT PyTorch 评估   │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --calib-only    │ -      │ 在 PTQ 校准后停止               │
-  ├─────────────────┼────────┼─────────────────────────────────┤
-  │ --eval-only     │ -      │ 只评估现有 ONNX 模型            │
-  └─────────────────┴────────┴─────────────────────────────────┘
+  |      标志       | 默认值 |              描述               |
+  | --- | ---- | ------------------------------ |
+  | --epochs        | 3      | QAT 微调周期数                  |
+  | --lr            | 1e-5   | 学习率                          |
+  | --batch-size    | 32     | 训练/评估的批处理大小           |
+  | --calib-samples | 1000   | PTQ 校准图像数量                |
+  | --train-samples | 0      | 限制 QAT 训练样本（0=全部）     |
+  | --device        | cpu    | 训练设备（如可用则使用 cuda）   |
+  | --subset        | 0      | 评估前 N 张图像（0=所有 5万张） |
+  | --skip-eval     | -      | 跳过中间 PTQ/QAT PyTorch 评估   |
+  | --calib-only    | -      | 在 PTQ 校准后停止               |
+  | --eval-only     | -      | 只评估现有 ONNX 模型            |
 
   输出文件
 
@@ -159,46 +112,33 @@ license: apache-2.0
   └── hgnetv2_b2_int8_qat_calib.pth   45.3 MB   # PTQ 校准检查点
 
 
-  ---
-  
+---
+
+  脚本已创建完成：prepare_env.sh
   
   ✅ 全部完成
   
   1. 项目传输
 
-  ┌────────────────────────────────┬────────┬─────────────────────────────────────────────────────────────────────┬──────┐
-  │              项目              │  大小  │                              目标路径                               │ 状态 │
-  ├────────────────────────────────┼────────┼─────────────────────────────────────────────────────────────────────┼──────┤
-  │ MODULES_PLAY                   │ 7.8 GB │ /mnt/vepfs/share/GW00387266/MODULES_PLAY/                           │ ✅   │
-  ├────────────────────────────────┼────────┼─────────────────────────────────────────────────────────────────────┼──────┤
-  │ ImageNet arrow shards (14个)   │ 6.3 GB │ ~/.cache/huggingface/datasets/Tsomaros___imagenet-1k_validation/... │ ✅   │
-  ├────────────────────────────────┼────────┼─────��───────────────────────────────────────────────────────────────┼──────┤
-  │ HF model cache (ViT + hgnetv2) │ 2.4 GB │ ~/.cache/huggingface/hub/                                           │ ✅   │
-  └────────────────────────────────┴────────┴─────────────────────────────────────────────────────────────────────┴──────┘
+  |              项目              |  大小  |                              目标路径                               | 状态 |
+  | --- | ------ | ---------------------------------------------------------- | ---- |
+  | MODULES_PLAY                   | 7.8 GB | /mnt/vepfs/share/GW00387266/MODULES_PLAY/                           | ✅   |
+  | ImageNet arrow shards (14个)   | 6.3 GB | ~/.cache/huggingface/datasets/Tsomaros___imagenet-1k_validation/... | ✅   |
+  | HF model cache (ViT + hgnetv2) | 2.4 GB | ~/.cache/huggingface/hub/                                           | ✅   |
 
   2. 远程环境配置
 
-  ┌──────────────────┬────────────────────────┬──────┐
-  │       组件       │          版本          │ 状态 │
-  ├──────────────────┼────────────────────────┼──────┤
-  │ GPU              │ NVIDIA H20 48GB        │ ✅   │
-  ├──────────────────┼────────────────────────┼──────┤
-  │ CUDA             │ 12.6                   │ ✅   │
-  ├──────────────────┼────────────────────────┼──────┤
-  │ PyTorch          │ 2.6.0+cu126            │ ✅   │
-  ├──────────────────┼────────────────────────┼──────┤
-  │ ONNX Runtime GPU │ 1.21.0 (CUDA+TensorRT) │ ✅   │
-  ├──────────────────┼────────────────────────┼──────┤
-  │ timm             │ 1.0.27 (升级自0.9.2)   │ ✅   │
-  ├──────────────────┼────────────────────────┼──────┤
-  │ transformers     │ 4.57.6                 │ ✅   │
-  ├──────────────────┼────────────────────────┼──────┤
-  │ nvidia-modelopt  │ 0.43.0.dev99           │ ✅   │
-  ├──────────────────┼────────────────────────┼──────┤
-  │ numpy            │ 1.26.4 (升级自1.23.0)  │ ✅   │
-  ├──────────────────┼────────────────────────┼──────┤
-  │ HF_HUB_OFFLINE   │ 1 (已写入.bashrc)      │ ✅   │
-  └──────────────────┴────────────────────────┴──────┘
+  |       组件       |          版本          | 状态 |
+  | --- | ---------------------- | ---- |
+  | GPU              | NVIDIA H20 48GB        | ✅   |
+  | CUDA             | 12.6                   | ✅   |
+  | PyTorch          | 2.6.0+cu126            | ✅   |
+  | ONNX Runtime GPU | 1.21.0 (CUDA+TensorRT) | ✅   |
+  | timm             | 1.0.27 (升级自0.9.2)   | ✅   |
+  | transformers     | 4.57.6                 | ✅   |
+  | nvidia-modelopt  | 0.43.0.dev99           | ✅   |
+  | numpy            | 1.26.4 (升级自1.23.0)  | ✅   |
+  | HF_HUB_OFFLINE   | 1 (已写入.bashrc)      | ✅   |
 
   3. prepare_env.sh
 
@@ -218,4 +158,111 @@ license: apache-2.0
 
   5. 代码修复
 
-  - hgnetv2_b2_eval_quantized.py: pretrained=True → pretrained=False（获取 transform 不需要下载权重）
+  - hgnetv2_b2_eval_quantized.py: pretrained=True → pretrained=False（获取 transform 不需要下载权重）
+
+
+---
+
+脚本已创建完成：unidrive_vla_nusc_base_evaluation.py
+
+  功能概览
+  
+  脚本参考 vit_large_patch16_224_evaluate.py 的结构，实现了以下功能：
+  
+  模型加载
+  
+  - 下载并加载 owl10/UniDriveVLA_Nusc_Base_Stage1（Qwen3VLForConditionalGeneration，~2.1B 参数，BF16 精度）
+  - 使用 AutoProcessor + qwen_vl_utils.process_vision_info 处理多视角图像输入
+  - 6 个 nuScenes 摄像头视角通过特殊 token 映射：<FRONT_VIEW>, <FRONT_LEFT_VIEW> 等
+
+  数据集
+
+  - 从 HuggingFace 下载 OpenDriveLab/DriveLM 的 v1.1 nuScenes JSON（ gated，需先申请访问权限）
+  - 自动解析 scene → key_frame → QA 的层级结构
+  - 支持 4 类任务：perception / prediction / planning / behavior
+  - 图像路径自动解析到 nuScenes samples/ 目录
+
+  评测指标（遵循 DriveLM Challenge 规范）
+
+  | Tag | 指标 | 适用问题类型 | 实现 |
+  | --- | ---- | ---------- | ---- |
+  | 0 | Accuracy | 多选/是否/behavior | 精确匹配 |
+  | 1 | GPT-Score | 开放式 planning | GPT-3.5 打分 (0-100) |
+  | 2 | Language | 描述性 perception | BLEU-1/2/3/4, ROUGE-L, CIDEr |
+  | 3 | Match | 坐标引用 prediction | F1(16px L1阈值) + GPT打分 |
+
+  Final Score = 0.4×GPT + 0.2×Language + 0.2×Match + 0.2×Accuracy
+
+  使用方法
+
+  # 基本评测（不含 GPT-Score）
+  python unidrive_vla_nusc_base_evaluation.py \
+      --data_dir /path/to/DriveLM \
+      --nuscenes_dir /path/to/nuscenes
+
+  # 完整评测（含 GPT-Score，需 OpenAI API Key）
+  python unidrive_vla_nusc_base_evaluation.py \
+      --data_dir /path/to/DriveLM \
+      --nuscenes_dir /path/to/nuscenes \
+      --gpt_eval --openai_api_key sk-xxx
+
+  # 快速测试（子集）
+  python unidrive_vla_nusc_base_evaluation.py \
+      --subset 50 --data_dir /path/to/DriveLM
+
+  # 保存/加载预测结果（避免重复推理）
+  python unidrive_vla_nusc_base_evaluation.py \
+      --save_predictions results.json ...
+  python unidrive_vla_nusc_base_evaluation.py \
+      --load_predictions results.json --gpt_eval
+
+  依赖
+
+  - transformers (需 Qwen3-VL 支持)
+  - qwen-vl-utils
+  - nltk, rouge-score, pycocoevalcap（Language 指标）
+  - openai（GPT-Score，可选）
+  - huggingface_hub（数据集下载）
+
+
+unidrive_vla_nusc_base_quantize_all.py 已创建完成 (586 行)。
+
+脚本结构
+
+量化 (ModelOpt PyTorch 原生 API)
+
+┌───────┬────────────────────┬────────────────────────────────────────────┐
+│ 模式  │        方法        │                    说明                    │
+├───────┼────────────────────┼────────────────────────────────────────────┤
+│ INT8  │ percentile         │ 激活+权重均量化，percentile 对 VLM 比 max  │
+│       │ calibration        │ 更稳健                                     │
+├───────┼────────────────────┼────────────────────────────────────────────┤
+│ NVFP4 │ awq_lite           │ NVIDIA FP4 格式，仅量化权重，AWQ           │
+│       │ weight-only        │ 保留重要权重                               │
+└───────┴────────────────────┴────────────────────────────────────────────┘
+
+校准数据
+
+- 从 DriveLM 训练集取 128 个样本的多视角图像+问题
+- 通过 forward_loop 传入 mtq.quantize() 收集激活统计
+
+评测
+
+- 完全复用 unidrive_vla_nusc_base_evaluation.py 的全套指标：
+  - Accuracy (tag 0)、GPT-Score (tag 1)、Language Score (tag 2)、Match Score (tag 3)
+  - Final Score = 0.4×GPT + 0.2×Language + 0.2×Match + 0.2×Accuracy
+
+执行流程
+
+bf16 baseline → INT8 量化+评测 → NVFP4 量化+评测 → 汇总对比表 → JSON 保存
+
+CLI 用法
+
+python unidrive_vla_nusc_base_quantize_all.py                     # 全部模式
+python unidrive_vla_nusc_base_quantize_all.py --mode int8         # 仅 INT8
+python unidrive_vla_nusc_base_quantize_all.py --mode nvfp4        # 仅 NVFP4
+python unidrive_vla_nusc_base_quantize_all.py --skip_baseline_eval  # 跳过 bf16
+python unidrive_vla_nusc_base_quantize_all.py --gpt_eval --openai_api_key KEY  # GPT评分
+python unidrive_vla_nusc_base_quantize_all.py --subset 100        # 评测前100样本
+
+▎ 注意: 实际运行时 ModelOpt 的 NVFP4 配置可能需要根据安装版本微调 (nf4 vs fp4 格式名、block_size 参数等)，建议先跑 --mode int8 验证基础流程，再调试 NVFP4。

unidrive_vla_nusc_base_evaluation.py

@@ -0,0 +1,915 @@
+"""
+Evaluate owl10/UniDriveVLA_Nusc_Base_Stage1 on the DriveLM benchmark.
+
+UniDriveVLA is a Vision-Language-Action (VLA) model for autonomous driving built on
+Qwen3-VL-2B-Instruct. The Stage-1 checkpoint is the VLM pretraining stage that
+produces a driving-aware VLM backbone.
+
+We evaluate on the DriveLM (OpenDriveLab/DriveLM) Graph-VQA benchmark, which
+provides perception / prediction / planning / behavior QA pairs over nuScenes
+multi-view camera frames.
+
+Evaluation metrics (following the DriveLM challenge):
+  - Accuracy      (tag 0): exact match for multi-choice / yes-no questions
+  - GPT-Score     (tag 1): GPT-rated similarity for open-ended planning Qs
+  - Language Score (tag 2): BLEU-1/2/3/4, ROUGE-L, CIDEr for descriptive Qs
+  - Match Score   (tag 3): F1 coordinate matching + GPT score for object-ref Qs
+  - Final Score = 0.4*GPT + 0.2*Language + 0.2*Match + 0.2*Accuracy
+
+Dataset:  OpenDriveLab/DriveLM  (v1.1 nuScenes, gated — request access first)
+Model:    owl10/UniDriveVLA_Nusc_Base_Stage1  (Qwen3VLForConditionalGeneration)
+
+Usage:
+    python unidrive_vla_nusc_base_evaluation.py \
+        [--batch_size 1] [--num_workers 4] [--subset 0] \
+        [--split train] [--gpt_eval] [--openai_api_key YOUR_KEY]
+
+Notes:
+    - This script uses the Stage-1 VLM checkpoint only (no perception/planning
+      experts from Stages 2-3). Results will be VQA-only, not end-to-end driving.
+    - DriveLM is gated; run `huggingface-cli login` and request access at
+      https://huggingface.co/datasets/OpenDriveLab/DriveLM before running.
+    - nuScenes images must be downloaded separately or via the DriveLM image zip.
+    - GPT-Score requires an OpenAI API key (--openai_api_key or OPENAI_API_KEY env).
+"""
+
+import argparse
+import json
+import os
+import re
+import time
+from collections import defaultdict
+
+# ---------------------------------------------------------------------------
+# Patch: disable MLX on non-Apple platforms
+# ---------------------------------------------------------------------------
+# The `mlx` pip package may be installed on Linux but its native shared
+# library (libmlx.so) is Apple-Silicon-only.  Transformers checks
+# `_is_mlx_available` at import time and later calls `import mlx.core`
+# inside `is_mlx_array()` during ModelOutput construction, which crashes
+# with "libmlx.so: cannot open shared object file".  We pre-emptively
+# set the flag to False so that code path is never taken.
+# ---------------------------------------------------------------------------
+import transformers.utils.generic as _tug
+_tug._is_mlx_available = False
+
+import numpy as np
+import torch
+from datasets import load_dataset
+from huggingface_hub import hf_hub_download
+from PIL import Image
+from tqdm import tqdm
+
+# ---------------------------------------------------------------------------
+# Metric helpers
+# ---------------------------------------------------------------------------
+
+# View token mapping: nuScenes camera name -> UniDriveVLA special token
+CAMERA_VIEW_TOKENS = {
+    "CAM_FRONT": "<FRONT_VIEW>",
+    "CAM_FRONT_LEFT": "<FRONT_LEFT_VIEW>",
+    "CAM_FRONT_RIGHT": "<FRONT_RIGHT_VIEW>",
+    "CAM_BACK": "<BACK_VIEW>",
+    "CAM_BACK_LEFT": "<BACK_LEFT_VIEW>",
+    "CAM_BACK_RIGHT": "<BACK_RIGHT_VIEW>",
+}
+
+CAMERAS = list(CAMERA_VIEW_TOKENS.keys())
+
+# QA type -> evaluation tag
+#   [0] accuracy  — multi-choice, yes/no, behavior
+#   [1] gpt-score — open-ended planning questions
+#   [2] language  — descriptive / free-text answers
+#   [3] match     — coordinate-referencing prediction questions
+QA_TAG_MAP = {
+    "perception": 2,     # descriptive ("important objects")
+    "prediction": 3,     # coordinate-matching
+    "planning": 1,       # open-ended action reasoning
+    "behavior": 0,       # multi-choice
+}
+
+
+def classify_question_tag(task_type: str, question: str, answer: str) -> int:
+    """Assign evaluation tag based on task type and question/answer content.
+
+    Heuristics follow the DriveLM extract_data.py tagging logic:
+      - Behavior questions are always tag 0 (multi-choice)
+      - Perception: tag 2 (language) for descriptive, tag 0 if multi-choice
+      - Prediction: tag 3 (match) if has c-tags, tag 0 if yes/no, tag 2 otherwise
+      - Planning: tag 1 (gpt-score) for action questions, tag 0 if multi-choice
+    """
+    has_options = "select the correct answer" in question.lower()
+    is_yes_no = answer.strip().lower() in ("yes.", "no.", "yes", "no")
+    has_ctag = bool(re.search(r"<c\d+,", question)) or bool(re.search(r"<c\d+,", answer))
+
+    if task_type == "behavior":
+        return 0
+    elif task_type == "perception":
+        if has_options:
+            return 0
+        return 2
+    elif task_type == "prediction":
+        if has_options:
+            return 0
+        if is_yes_no:
+            return 0
+        if has_ctag:
+            return 3
+        return 2
+    elif task_type == "planning":
+        if has_options:
+            return 0
+        return 1
+    else:
+        return 2
+
+
+# ---------------------------------------------------------------------------
+# Accuracy metric (tag 0)
+# ---------------------------------------------------------------------------
+
+def compute_accuracy(predictions: list[str], references: list[str]) -> dict:
+    """Exact-match accuracy for multi-choice / yes-no questions."""
+    correct = 0
+    total = len(predictions)
+    for pred, ref in zip(predictions, references):
+        pred_clean = pred.strip().lower()
+        ref_clean = ref.strip().lower()
+        if pred_clean == ref_clean:
+            correct += 1
+        # Also check first character / word for option-style answers
+        elif ref_clean in ("a", "b", "c", "d") and pred_clean.startswith(ref_clean):
+            correct += 1
+    acc = correct / total if total > 0 else 0.0
+    return {"accuracy": acc, "correct": correct, "total": total}
+
+
+# ---------------------------------------------------------------------------
+# Language metric (tag 2): BLEU, ROUGE-L, CIDEr
+# ---------------------------------------------------------------------------
+
+def compute_language_metrics(predictions: list[str], references: list[str]) -> dict:
+    """Compute BLEU-1/2/3/4, ROUGE-L, and CIDEr using standard NLP libraries."""
+    try:
+        from nltk.translate.bleu_score import corpus_bleu, SmoothingFunction
+    except ImportError:
+        print("  [WARN] nltk not installed; BLEU scores will be 0. Install: pip install nltk")
+        corpus_bleu = None
+
+    try:
+        from rouge_score import rouge_scorer as _rouge_scorer
+    except ImportError:
+        print("  [WARN] rouge_score not installed; ROUGE-L will be 0. Install: pip install rouge-score")
+        _rouge_scorer = None
+
+    try:
+        from pycocoevalcap.cider.cider import Cider
+    except ImportError:
+        print("  [WARN] pycocoevalcap not installed; CIDEr will be 0. Install: pip install pycocoevalcap")
+        Cider = None
+
+    results = {
+        "Bleu_1": 0.0, "Bleu_2": 0.0, "Bleu_3": 0.0, "Bleu_4": 0.0,
+        "ROUGE_L": 0.0, "CIDEr": 0.0,
+    }
+
+    if not predictions:
+        return results
+
+    # Tokenize
+    def _tokenize(s):
+        return s.lower().strip().split()
+
+    ref_tokens = [[_tokenize(r)] for r in references]  # list of list of token lists
+    pred_tokens = [_tokenize(p) for p in predictions]
+
+    # BLEU
+    if corpus_bleu is not None:
+        smooth = SmoothingFunction().method1
+        for n in range(1, 5):
+            weights = [1.0 / n] * n + [0.0] * (4 - n)
+            score = corpus_bleu(ref_tokens, pred_tokens, weights=weights, smoothing_function=smooth)
+            results[f"Bleu_{n}"] = score
+
+    # ROUGE-L
+    if _rouge_scorer is not None:
+        scorer = _rouge_scorer.RougeScorer(["rougeL"], use_stemmer=True)
+        rouge_scores = []
+        for pred, ref in zip(predictions, references):
+            s = scorer.score(ref, pred)
+            rouge_scores.append(s["rougeL"].fmeasure)
+        results["ROUGE_L"] = np.mean(rouge_scores)
+
+    # CIDEr
+    if Cider is not None:
+        cider = Cider()
+        # CIDEr expects dict format: {id: [reference], id: [hypothesis]}
+        gts = {i: [r] for i, r in enumerate(references)}
+        res = {i: [p] for i, p in enumerate(predictions)}
+        try:
+            score, _ = cider.compute_score(gts, res)
+            results["CIDEr"] = score
+        except Exception as e:
+            print(f"  [WARN] CIDEr computation failed: {e}")
+
+    # Normalized language score (following DriveLM challenge)
+    # (Bleu_1 + Bleu_4) / 3 + ROUGE_L / 3 + CIDEr / 10 / 3
+    norm = (
+        (results["Bleu_1"] + results["Bleu_4"]) / 3.0
+        + results["ROUGE_L"] / 3.0
+        + results["CIDEr"] / 10.0 / 3.0
+    )
+    results["language_score_normalized"] = norm
+
+    return results
+
+
+# ---------------------------------------------------------------------------
+# GPT-Score metric (tag 1)
+# ---------------------------------------------------------------------------
+
+def gpt_score_single(question: str, prediction: str, reference: str, client) -> float:
+    """Score a single prediction against reference using GPT-3.5/4."""
+    prompt = (
+        "Rate my answer based on the correct answer out of 100, with higher "
+        "scores indicating that the answer is closer to the correct answer, "
+        "and you should be accurate to single digits like 62, 78, 41, etc. "
+        "Output the number only.\n\n"
+        f"Question: {question}\n"
+        f"Correct answer: {reference}\n"
+        f"My answer: {prediction}"
+    )
+    try:
+        resp = client.chat.completions.create(
+            model="gpt-3.5-turbo",
+            messages=[{"role": "user", "content": prompt}],
+            max_tokens=10,
+            temperature=0.0,
+        )
+        text = resp.choices[0].message.content.strip()
+        score = float(re.search(r"\d+", text).group())
+        return min(max(score, 0.0), 100.0)
+    except Exception as e:
+        print(f"  [WARN] GPT scoring failed: {e}")
+        return 0.0
+
+
+def compute_gpt_score(
+    questions: list[str],
+    predictions: list[str],
+    references: list[str],
+    api_key: str | None = None,
+) -> dict:
+    """Compute GPT-score for open-ended QA pairs."""
+    key = api_key or os.environ.get("OPENAI_API_KEY")
+    if not key:
+        print("  [WARN] No OpenAI API key; GPT-Score will be 0. "
+              "Set --openai_api_key or OPENAI_API_KEY env var.")
+        return {"gpt_score": 0.0, "total": len(predictions), "scored": 0}
+
+    from openai import OpenAI
+    client = OpenAI(api_key=key)
+
+    scores = []
+    for q, pred, ref in tqdm(
+        zip(questions, predictions, references),
+        total=len(questions),
+        desc="  GPT scoring",
+        disable=None,
+    ):
+        s = gpt_score_single(q, pred, ref, client)
+        scores.append(s)
+        # Small delay to avoid rate limits
+        time.sleep(0.1)
+
+    avg = np.mean(scores) if scores else 0.0
+    return {"gpt_score": avg, "total": len(scores), "scored": len(scores)}
+
+
+# ---------------------------------------------------------------------------
+# Match metric (tag 3): coordinate F1 + GPT score
+# ---------------------------------------------------------------------------
+
+def extract_coordinates(text: str) -> list[tuple[str, float, float]]:
+    """Extract <cN,CAM,x,y> coordinate references from text."""
+    pattern = r"<c(\d+),([^,]+),([\d.]+),([\d.]+)>"
+    matches = re.findall(pattern, text)
+    return [(f"c{m[0]}", float(m[2]), float(m[3])) for m in matches]
+
+
+def compute_match_score(
+    questions: list[str],
+    predictions: list[str],
+    references: list[str],
+    api_key: str | None = None,
+    coord_threshold: float = 16.0,
+) -> dict:
+    """Compute match score = (F1 * 100 + GPT_score) / 2.
+
+    F1 is based on coordinate matching: predicted 2D coords matched to GT
+    coords if L1 distance < threshold pixels.
+    """
+    all_prec, all_rec, all_f1 = [], [], []
+
+    for pred, ref in zip(predictions, references):
+        pred_coords = extract_coordinates(pred)
+        ref_coords = extract_coordinates(ref)
+
+        if not ref_coords and not pred_coords:
+            all_f1.append(1.0)
+            all_prec.append(1.0)
+            all_rec.append(1.0)
+            continue
+        if not ref_coords:
+            all_f1.append(0.0)
+            all_prec.append(0.0)
+            all_rec.append(0.0)
+            continue
+        if not pred_coords:
+            all_f1.append(0.0)
+            all_prec.append(0.0)
+            all_rec.append(0.0)
+            continue
+
+        # Match predicted coords to reference coords by L1 distance
+        matched_pred = set()
+        matched_ref = set()
+
+        for ri, (_, rx, ry) in enumerate(ref_coords):
+            best_dist = float("inf")
+            best_pi = -1
+            for pi, (_, px, py) in enumerate(pred_coords):
+                if pi in matched_pred:
+                    continue
+                dist = abs(px - rx) + abs(py - ry)
+                if dist < best_dist:
+                    best_dist = dist
+                    best_pi = pi
+            if best_dist < coord_threshold and best_pi >= 0:
+                matched_pred.add(best_pi)
+                matched_ref.add(ri)
+
+        tp = len(matched_ref)
+        fp = len(pred_coords) - tp
+        fn = len(ref_coords) - tp
+
+        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
+        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
+        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
+
+        all_prec.append(prec)
+        all_rec.append(rec)
+        all_f1.append(f1)
+
+    avg_f1 = np.mean(all_f1) if all_f1 else 0.0
+
+    # GPT component of match score
+    gpt_result = compute_gpt_score(questions, predictions, references, api_key)
+    gpt_avg = gpt_result["gpt_score"]
+
+    match_score = (avg_f1 * 100 + gpt_avg) / 2.0
+
+    return {
+        "match_score": match_score,
+        "coord_f1": avg_f1,
+        "coord_precision": np.mean(all_prec) if all_prec else 0.0,
+        "coord_recall": np.mean(all_rec) if all_rec else 0.0,
+        "gpt_component": gpt_avg,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Final weighted score
+# ---------------------------------------------------------------------------
+
+def compute_final_score(
+    accuracy: float,
+    gpt_score: float,
+    language_score: float,
+    match_score: float,
+) -> float:
+    """DriveLM weighted final score.
+
+    Weights: GPT=0.4, Language=0.2, Match=0.2, Accuracy=0.2.
+    All scores are in [0, 1] (GPT/match/language normalized).
+    """
+    weights = [0.4, 0.2, 0.2, 0.2]
+    components = [gpt_score / 100.0, language_score, match_score / 100.0, accuracy]
+    return sum(w * c for w, c in zip(weights, components))
+
+
+# ---------------------------------------------------------------------------
+# DriveLM dataset loader
+# ---------------------------------------------------------------------------
+
+def load_drivelm_dataset(
+    split: str = "train",
+    data_dir: str | None = None,
+    nuscenes_dir: str | None = None,
+) -> list[dict]:
+    """Load DriveLM QA data and return a list of evaluation samples.
+
+    Each sample dict has:
+        - scene_token, frame_token
+        - scene_description
+        - image_paths: dict of {CAM_NAME: absolute_path}
+        - task_type: perception | prediction | planning | behavior
+        - question: str
+        - answer: str (ground truth)
+        - tag: int (0-3 evaluation tag)
+
+    Args:
+        split: 'train' or 'val'
+        data_dir: directory containing the DriveLM JSON and nuscenes images.
+                  If None, attempts to download from HuggingFace.
+        nuscenes_dir: directory containing nuScenes 'samples/' folder.
+    """
+    filename = f"v1_1_{split}_nus.json"
+    if split == "val":
+        filename = "v1_1_val_nus_q_only.json"
+
+    json_path = None
+
+    if data_dir:
+        candidate = os.path.join(data_dir, filename)
+        if os.path.exists(candidate):
+            json_path = candidate
+
+    if json_path is None:
+        print(f"  Downloading {filename} from HuggingFace ...")
+        try:
+            json_path = hf_hub_download(
+                repo_id="OpenDriveLab/DriveLM",
+                filename=filename,
+                repo_type="dataset",
+            )
+        except Exception as e:
+            print(f"  ERROR: Failed to download {filename}: {e}")
+            print("  Please request access at https://huggingface.co/datasets/OpenDriveLab/DriveLM")
+            return []
+
+    print(f"  Loading {json_path} ...")
+    with open(json_path, "r") as f:
+        raw_data = json.load(f)
+
+    # Determine image base directory
+    img_base = nuscenes_dir or data_dir or os.path.dirname(json_path)
+
+    samples = []
+    for scene_token, scene_data in raw_data.items():
+        scene_desc = scene_data.get("scene_description", "")
+        key_frames = scene_data.get("key_frames", {})
+
+        for frame_token, frame_data in key_frames.items():
+            image_paths_raw = frame_data.get("image_paths", {})
+            key_objects = frame_data.get("key_object_infos", {})
+            qa = frame_data.get("QA", {})
+
+            # Resolve absolute image paths
+            abs_image_paths = {}
+            for cam, rel_path in image_paths_raw.items():
+                # Try multiple possible base directories
+                candidates = [
+                    os.path.join(img_base, rel_path),
+                    os.path.join(img_base, "nuscenes", rel_path),
+                    os.path.join(img_base, "samples", os.path.basename(rel_path)),
+                    os.path.join(img_base, "nuscenes", "samples",
+                                 cam, os.path.basename(rel_path)),
+                ]
+                for c in candidates:
+                    if os.path.exists(c):
+                        abs_image_paths[cam] = c
+                        break
+                else:
+                    # Keep relative path; will be checked at inference time
+                    abs_image_paths[cam] = rel_path
+
+            for task_type in ["perception", "prediction", "planning", "behavior"]:
+                qa_list = qa.get(task_type, [])
+                for idx, qa_item in enumerate(qa_list):
+                    question = qa_item.get("Q", "")
+                    answer = qa_item.get("A", "")
+                    if not question:
+                        continue
+
+                    tag = classify_question_tag(task_type, question, answer)
+
+                    samples.append({
+                        "id": f"{scene_token}_{frame_token}_{task_type}_{idx}",
+                        "scene_token": scene_token,
+                        "frame_token": frame_token,
+                        "scene_description": scene_desc,
+                        "image_paths": abs_image_paths,
+                        "key_objects": key_objects,
+                        "task_type": task_type,
+                        "question": question,
+                        "answer": answer,
+                        "tag": tag,
+                    })
+
+    return samples
+
+
+# ---------------------------------------------------------------------------
+# Model inference
+# ---------------------------------------------------------------------------
+
+SYSTEM_PROMPT = (
+    "You are an advanced autonomous driving assistant. You analyze multi-view "
+    "camera images from a vehicle and answer questions about the driving scene, "
+    "including perception, prediction, planning, and behavior. "
+    "Provide concise and accurate answers."
+)
+
+
+@torch.no_grad()
+def run_inference(
+    model,
+    processor,
+    sample: dict,
+    device: torch.device,
+    max_new_tokens: int = 256,
+) -> str:
+    """Run inference on a single DriveLM sample.
+
+    Constructs a multi-view chat message using UniDriveVLA's special view tokens,
+    processes the inputs, and generates a text response.
+    """
+    # Build multi-view image content
+    image_content = []
+    image_files = []
+    for cam in CAMERAS:
+        cam_path = sample["image_paths"].get(cam)
+        if cam_path and os.path.exists(cam_path):
+            view_token = CAMERA_VIEW_TOKENS[cam]
+            image_content.append({"type": "text", "text": f"{view_token}"})
+            image_content.append({"type": "image", "image": f"file://{cam_path}"})
+            image_files.append(cam_path)
+
+    # Build messages in Qwen3-VL chat format
+    user_content = image_content + [
+        {"type": "text", "text": sample["question"]},
+    ]
+
+    messages = [
+        {"role": "system", "content": SYSTEM_PROMPT},
+        {"role": "user", "content": user_content},
+    ]
+
+    # Process with Qwen3-VL processor
+    try:
+        from qwen_vl_utils import process_vision_info
+    except ImportError:
+        print("  [ERROR] qwen_vl_utils not installed. Install: pip install qwen-vl-utils")
+        return ""
+
+    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    image_inputs, video_inputs = process_vision_info(messages)
+    inputs = processor(
+        text=[text],
+        images=image_inputs,
+        videos=video_inputs,
+        padding=True,
+        return_tensors="pt",
+    ).to(device)
+
+    output_ids = model.generate(
+        **inputs,
+        max_new_tokens=max_new_tokens,
+        do_sample=False,
+        temperature=1.0,  # greedy when do_sample=False
+    )
+
+    # Decode only the generated tokens (skip input)
+    generated_ids = output_ids[:, inputs.input_ids.shape[1]:]
+    response = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
+    return response.strip()
+
+
+# ---------------------------------------------------------------------------
+# Main evaluation loop
+# ---------------------------------------------------------------------------
+
+def evaluate_model(
+    model,
+    processor,
+    samples: list[dict],
+    device: torch.device,
+    max_new_tokens: int = 256,
+    print_every: int = 100,
+) -> dict:
+    """Run evaluation on all samples and collect predictions."""
+    model.eval()
+    predictions = {}  # tag -> list of (question, prediction, reference)
+    total = len(samples)
+    total_inference_time = 0.0
+    errors = 0
+
+    start = time.time()
+    for i, sample in enumerate(tqdm(samples, desc="Evaluating", disable=None)):
+        t0 = time.perf_counter()
+        try:
+            pred = run_inference(model, processor, sample, device, max_new_tokens)
+        except Exception as e:
+            print(f"  [ERROR] Sample {sample['id']}: {e}")
+            pred = ""
+            errors += 1
+        t1 = time.perf_counter()
+        total_inference_time += (t1 - t0)
+
+        tag = sample["tag"]
+        if tag not in predictions:
+            predictions[tag] = {"questions": [], "predictions": [], "references": []}
+        predictions[tag]["questions"].append(sample["question"])
+        predictions[tag]["predictions"].append(pred)
+        predictions[tag]["references"].append(sample["answer"])
+
+        if print_every and (i + 1) % print_every == 0:
+            elapsed = time.time() - start
+            speed = (i + 1) / elapsed
+            print(f"  [{i + 1:>6d}/{total}]  {speed:.2f} samples/s  "
+                  f"avg_inference={(t1-t0)*1000:.0f}ms")
+
+    elapsed = time.time() - start
+    return {
+        "predictions": predictions,
+        "total_samples": total,
+        "total_errors": errors,
+        "elapsed": elapsed,
+        "total_inference_time": total_inference_time,
+        "avg_process_ms": elapsed / total * 1000 if total > 0 else 0.0,
+        "avg_inference_ms": total_inference_time / total * 1000 if total > 0 else 0.0,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Evaluate owl10/UniDriveVLA_Nusc_Base_Stage1 on DriveLM"
+    )
+    parser.add_argument("--batch_size", type=int, default=1,
+                        help="Batch size (VLM inference is typically 1)")
+    parser.add_argument("--num_workers", type=int, default=4,
+                        help="DataLoader workers (unused for VLM, kept for compat)")
+    parser.add_argument("--subset", type=int, default=0,
+                        help="Evaluate on first N samples only (0 = all)")
+    parser.add_argument("--split", type=str, default="train",
+                        choices=["train", "val"],
+                        help="DriveLM split to evaluate on")
+    parser.add_argument("--data_dir", type=str, default=None,
+                        help="Local directory with DriveLM JSON + nuScenes images")
+    parser.add_argument("--nuscenes_dir", type=str, default=None,
+                        help="Directory containing nuScenes samples/ folder")
+    parser.add_argument("--max_new_tokens", type=int, default=256,
+                        help="Max tokens to generate per sample")
+    parser.add_argument("--gpt_eval", action="store_true",
+                        help="Enable GPT-Score evaluation (requires OpenAI API key)")
+    parser.add_argument("--openai_api_key", type=str, default=None,
+                        help="OpenAI API key for GPT-Score (or set OPENAI_API_KEY)")
+    parser.add_argument("--save_predictions", type=str, default=None,
+                        help="Save predictions to this JSON file")
+    parser.add_argument("--load_predictions", type=str, default=None,
+                        help="Load predictions from JSON file (skip inference)")
+    args = parser.parse_args()
+
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    print(f"Device: {device}")
+    if torch.cuda.is_available():
+        print(f"  GPU: {torch.cuda.get_device_name(0)}")
+        print(f"  VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
+
+    # ------------------------------------------------------------------
+    # Load model & processor
+    # ------------------------------------------------------------------
+    model_name = "owl10/UniDriveVLA_Nusc_Base_Stage1"
+    print(f"\nLoading {model_name} ...")
+
+    from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
+
+    processor = AutoProcessor.from_pretrained(model_name)
+    model = Qwen3VLForConditionalGeneration.from_pretrained(
+        model_name,
+        torch_dtype=torch.bfloat16,
+        device_map="auto",
+    )
+    model.eval()
+
+    # Print model info
+    param_count = sum(p.numel() for p in model.parameters()) / 1e9
+    print(f"  Architecture: Qwen3VLForConditionalGeneration")
+    print(f"  Parameters:   {param_count:.2f}B")
+    print(f"  Dtype:        bfloat16")
+
+    # ------------------------------------------------------------------
+    # Load dataset
+    # ------------------------------------------------------------------
+    print(f"\nLoading DriveLM dataset (split={args.split}) ...")
+    samples = load_drivelm_dataset(
+        split=args.split,
+        data_dir=args.data_dir,
+        nuscenes_dir=args.nuscenes_dir,
+    )
+
+    if not samples:
+        print("ERROR: No samples loaded. Check dataset access and paths.")
+        return
+
+    # Tag distribution
+    tag_counts = defaultdict(int)
+    task_counts = defaultdict(int)
+    for s in samples:
+        tag_counts[s["tag"]] += 1
+        task_counts[s["task_type"]] += 1
+
+    print(f"  Total QA pairs: {len(samples)}")
+    print(f"  Task distribution: {dict(task_counts)}")
+    print(f"  Tag distribution:  {dict(tag_counts)}")
+
+    if args.subset > 0:
+        samples = samples[:args.subset]
+        print(f"  Using subset: {len(samples)} samples")
+
+    # ------------------------------------------------------------------
+    # Run evaluation (or load cached predictions)
+    # ------------------------------------------------------------------
+    if args.load_predictions:
+        print(f"\nLoading predictions from {args.load_predictions} ...")
+        with open(args.load_predictions, "r") as f:
+            cached = json.load(f)
+        eval_result = cached.get("eval_result", {})
+        predictions = {}
+        for tag_str, data in cached.get("predictions", {}).items():
+            predictions[int(tag_str)] = data
+    else:
+        print(f"\n{'='*60}")
+        print(f"Running DriveLM evaluation on {model_name}")
+        print(f"{'='*60}\n")
+
+        eval_result = evaluate_model(
+            model, processor, samples, device,
+            max_new_tokens=args.max_new_tokens,
+        )
+        predictions = eval_result["predictions"]
+
+    # ------------------------------------------------------------------
+    # Compute metrics per tag
+    # ------------------------------------------------------------------
+    print(f"\n{'='*60}")
+    print(f"Evaluation Results — {model_name}")
+    print(f"{'='*60}")
+
+    tag_metrics = {}
+
+    # --- Tag 0: Accuracy ---
+    if 0 in predictions and predictions[0]["predictions"]:
+        m = compute_accuracy(predictions[0]["predictions"], predictions[0]["references"])
+        tag_metrics[0] = m
+        print(f"\n  [Tag 0] Accuracy (multi-choice / yes-no):")
+        print(f"    Accuracy:  {m['accuracy']*100:.2f}%")
+        print(f"    Correct:   {m['correct']}/{m['total']}")
+    else:
+        tag_metrics[0] = {"accuracy": 0.0, "correct": 0, "total": 0}
+
+    # --- Tag 2: Language Score ---
+    if 2 in predictions and predictions[2]["predictions"]:
+        m = compute_language_metrics(predictions[2]["predictions"], predictions[2]["references"])
+        tag_metrics[2] = m
+        print(f"\n  [Tag 2] Language Score (descriptive answers):")
+        print(f"    BLEU-1:    {m['Bleu_1']:.4f}")
+        print(f"    BLEU-2:    {m['Bleu_2']:.4f}")
+        print(f"    BLEU-3:    {m['Bleu_3']:.4f}")
+        print(f"    BLEU-4:    {m['Bleu_4']:.4f}")
+        print(f"    ROUGE-L:   {m['ROUGE_L']:.4f}")
+        print(f"    CIDEr:     {m['CIDEr']:.4f}")
+        print(f"    Norm Lang: {m['language_score_normalized']:.4f}")
+    else:
+        tag_metrics[2] = {
+            "Bleu_1": 0.0, "Bleu_2": 0.0, "Bleu_3": 0.0, "Bleu_4": 0.0,
+            "ROUGE_L": 0.0, "CIDEr": 0.0, "language_score_normalized": 0.0,
+        }
+
+    # --- Tag 1: GPT Score ---
+    if 1 in predictions and predictions[1]["predictions"]:
+        if args.gpt_eval:
+            m = compute_gpt_score(
+                predictions[1]["questions"],
+                predictions[1]["predictions"],
+                predictions[1]["references"],
+                args.openai_api_key,
+            )
+            tag_metrics[1] = m
+            print(f"\n  [Tag 1] GPT-Score (open-ended planning):")
+            print(f"    GPT-Score: {m['gpt_score']:.2f}/100")
+            print(f"    Scored:    {m['scored']}/{m['total']}")
+        else:
+            tag_metrics[1] = {"gpt_score": 0.0, "total": len(predictions[1]["predictions"]), "scored": 0}
+            print(f"\n  [Tag 1] GPT-Score: SKIPPED (use --gpt_eval to enable)")
+    else:
+        tag_metrics[1] = {"gpt_score": 0.0, "total": 0, "scored": 0}
+
+    # --- Tag 3: Match Score ---
+    if 3 in predictions and predictions[3]["predictions"]:
+        if args.gpt_eval:
+            m = compute_match_score(
+                predictions[3]["questions"],
+                predictions[3]["predictions"],
+                predictions[3]["references"],
+                args.openai_api_key,
+            )
+            tag_metrics[3] = m
+            print(f"\n  [Tag 3] Match Score (coordinate + GPT):")
+            print(f"    Match Score:    {m['match_score']:.2f}")
+            print(f"    Coord F1:       {m['coord_f1']:.4f}")
+            print(f"    Coord Prec:     {m['coord_precision']:.4f}")
+            print(f"    Coord Rec:      {m['coord_recall']:.4f}")
+            print(f"    GPT Component:  {m['gpt_component']:.2f}")
+        else:
+            # Compute coordinate F1 without GPT component
+            m_no_gpt = compute_match_score(
+                predictions[3]["questions"],
+                predictions[3]["predictions"],
+                predictions[3]["references"],
+                api_key=None,  # skip GPT
+            )
+            tag_metrics[3] = m_no_gpt
+            print(f"\n  [Tag 3] Match Score (coordinate F1 only, GPT skipped):")
+            print(f"    Coord F1:    {m_no_gpt['coord_f1']:.4f}")
+            print(f"    Coord Prec:  {m_no_gpt['coord_precision']:.4f}")
+            print(f"    Coord Rec:   {m_no_gpt['coord_recall']:.4f}")
+    else:
+        tag_metrics[3] = {"match_score": 0.0, "coord_f1": 0.0}
+
+    # ------------------------------------------------------------------
+    # Final weighted score
+    # ------------------------------------------------------------------
+    accuracy = tag_metrics[0].get("accuracy", 0.0)
+    gpt_score = tag_metrics[1].get("gpt_score", 0.0)
+    language_score = tag_metrics[2].get("language_score_normalized", 0.0)
+    match_score = tag_metrics[3].get("match_score", 0.0)
+
+    final = compute_final_score(accuracy, gpt_score, language_score, match_score)
+
+    print(f"\n{'='*60}")
+    print(f"  Final Score (DriveLM weighting)")
+    print(f"    Accuracy:      {accuracy*100:.2f}%  (weight=0.2)")
+    print(f"    GPT-Score:     {gpt_score:.2f}/100  (weight=0.4)")
+    print(f"    Language:      {language_score:.4f}  (weight=0.2)")
+    print(f"    Match:         {match_score:.2f}  (weight=0.2)")
+    print(f"    ------------------------------------------")
+    print(f"    FINAL SCORE:   {final:.4f}")
+    print(f"{'='*60}")
+
+    # Timing info
+    if "elapsed" in eval_result:
+        print(f"\n  Total Samples:        {eval_result.get('total_samples', 'N/A')}")
+        print(f"  Errors:               {eval_result.get('total_errors', 0)}")
+        print(f"  Total Time:           {eval_result.get('elapsed', 0):.1f}s")
+        print(f"  Avg Process Time:     {eval_result.get('avg_process_ms', 0):.0f}ms/sample")
+        print(f"  Avg Inference Time:   {eval_result.get('avg_inference_ms', 0):.0f}ms/sample")
+
+    # Reference baseline
+    print(f"\n{'='*60}")
+    print("Reference (DriveLM zero-shot baseline from challenge):")
+    print("  Accuracy:  0.00%  |  GPT-Score: 67.75  |  Match: 18.83")
+    print("  Language:  BLEU-1=0.238, BLEU-4=0.011, ROUGE-L=0.199, CIDEr=0.007")
+    print("  Final Score: 0.328")
+    print(f"{'='*60}")
+
+    # ------------------------------------------------------------------
+    # Save predictions
+    # ------------------------------------------------------------------
+    if args.save_predictions:
+        # Convert predictions to serializable format
+        serializable_predictions = {}
+        for tag, data in predictions.items():
+            serializable_predictions[str(tag)] = data
+
+        output = {
+            "model": model_name,
+            "dataset": "OpenDriveLab/DriveLM",
+            "split": args.split,
+            "subset": args.subset,
+            "predictions": serializable_predictions,
+            "metrics": {
+                "accuracy": accuracy,
+                "gpt_score": gpt_score,
+                "language_score": language_score,
+                "match_score": match_score,
+                "final_score": final,
+            },
+            "eval_result": {
+                k: v for k, v in eval_result.items()
+                if k != "predictions" and not isinstance(v, torch.Tensor)
+            },
+        }
+
+        with open(args.save_predictions, "w") as f:
+            json.dump(output, f, indent=2, ensure_ascii=False)
+        print(f"\n  Predictions saved to {args.save_predictions}")
+
+
+if __name__ == "__main__":
+    main()

unidrive_vla_nusc_base_quantize_all.py

@@ -0,0 +1,873 @@
+"""
+Quantize owl10/UniDriveVLA_Nusc_Base_Stage1 using Nvidia ModelOpt.
+
+Quantization modes:
+  - INT8:        percentile calibration (activation + weight, PyTorch native)
+  - NVFP4:       AWQ weight-only quantization (4-bit floating point, PyTorch native)
+  - ONNX_NVFP4:  PyTorch NVFP4 quantization → ONNX export → NVFP4QuantExporter
+                 post-processing (FLOAT4E2M1 weights + 2×DQ dequant chains)
+
+Calibration data: First 128 samples from DriveLM training set (multi-view images + QA)
+Evaluation: DriveLM benchmark metrics (Accuracy, GPT-Score, Language, Match, Final Score)
+
+Usage:
+    python unidrive_vla_nusc_base_quantize_all.py                    # run all modes
+    python unidrive_vla_nusc_base_quantize_all.py --mode int8        # INT8 only
+    python unidrive_vla_nusc_base_quantize_all.py --mode nvfp4       # NVFP4 only
+    python unidrive_vla_nusc_base_quantize_all.py --mode onnx_nvfp4  # ONNX NVFP4 only
+    python unidrive_vla_nusc_base_quantize_all.py --skip_baseline_eval  # skip bf16 eval
+
+Note on onnx_nvfp4:
+    modelopt.onnx.quantization.quantize() does NOT support quantize_mode="nvfp4"
+    (only int8/fp8/int4 are wired in the dispatch). The ONNX NVFP4 pipeline is:
+      1. PyTorch-level: mtq.quantize(model, NVFP4_AWQ_LITE_CFG, forward_loop)
+      2. ONNX export:   torch.onnx.export() with QuantizedLinear ops
+      3. Post-process:  NVFP4QuantExporter replaces TRT_FP4QDQ custom ops
+                        with chained DequantizeLinear (FP4→FP8→FP32)
+    This is the canonical ModelOpt 0.44.0 ONNX NVFP4 path.
+"""
+
+import argparse
+import gc
+import json
+import os
+import re
+import time
+from collections import defaultdict
+
+# ---------------------------------------------------------------------------
+# Patch: disable MLX on non-Apple platforms (same as evaluation script)
+# ---------------------------------------------------------------------------
+import transformers.utils.generic as _tug
+_tug._is_mlx_available = False
+
+import numpy as np
+import torch
+from huggingface_hub import hf_hub_download
+from tqdm import tqdm
+
+# Import evaluation utilities from the evaluation script
+from unidrive_vla_nusc_base_evaluation import (
+    CAMERA_VIEW_TOKENS,
+    CAMERAS,
+    QA_TAG_MAP,
+    SYSTEM_PROMPT,
+    classify_question_tag,
+    compute_accuracy,
+    compute_final_score,
+    compute_gpt_score,
+    compute_language_metrics,
+    compute_match_score,
+    evaluate_model,
+    load_drivelm_dataset,
+    run_inference,
+)
+
+
+# ---------------------------------------------------------------------------
+# Configuration
+# ---------------------------------------------------------------------------
+MODEL_NAME = "owl10/UniDriveVLA_Nusc_Base_Stage1"
+NUM_CALIBRATION_SAMPLES = 128
+ALL_MODES = ["int8", "nvfp4", "onnx_nvfp4"]
+
+
+# ---------------------------------------------------------------------------
+# Calibration data preparation
+# ---------------------------------------------------------------------------
+
+def prepare_calibration_data(
+    samples: list[dict],
+    processor,
+    device: torch.device,
+    num_samples: int = NUM_CALIBRATION_SAMPLES,
+) -> list:
+    """Prepare calibration inputs from DriveLM samples.
+
+    Each calibration sample is a dict of model inputs ready for forward().
+    We use multi-view images + question text from the first N samples.
+    """
+    from qwen_vl_utils import process_vision_info
+
+    calibration_inputs = []
+    for i, sample in enumerate(samples[:num_samples]):
+        # Build multi-view image content
+        image_content = []
+        for cam in CAMERAS:
+            cam_path = sample["image_paths"].get(cam)
+            if cam_path and os.path.exists(cam_path):
+                view_token = CAMERA_VIEW_TOKENS[cam]
+                image_content.append({"type": "text", "text": f"{view_token}"})
+                image_content.append({"type": "image", "image": f"file://{cam_path}"})
+
+        # Build messages
+        user_content = image_content + [
+            {"type": "text", "text": sample["question"]},
+        ]
+        messages = [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": user_content},
+        ]
+
+        # Process with Qwen3-VL processor
+        try:
+            text = processor.apply_chat_template(
+                messages, tokenize=False, add_generation_prompt=True
+            )
+            image_inputs, video_inputs = process_vision_info(messages)
+            inputs = processor(
+                text=[text],
+                images=image_inputs,
+                videos=video_inputs,
+                padding=True,
+                return_tensors="pt",
+            ).to(device)
+
+            calibration_inputs.append(inputs)
+        except Exception as e:
+            print(f"  [WARN] Failed to prepare sample {i}: {e}")
+            continue
+
+    print(f"  Prepared {len(calibration_inputs)}/{num_samples} calibration samples")
+    return calibration_inputs
+
+
+def make_forward_loop(calibration_inputs: list):
+    """Create a forward_loop function for ModelOpt calibration.
+
+    ModelOpt calls forward_loop(model) and expects the function to run
+    forward passes on calibration data to collect activation statistics.
+    """
+    def forward_loop(model):
+        with torch.no_grad():
+            for inputs in tqdm(
+                calibration_inputs,
+                desc="Calibration forward",
+                disable=None,
+            ):
+                try:
+                    model(**inputs)
+                except Exception as e:
+                    print(f"  [WARN] Forward pass failed: {e}")
+    return forward_loop
+
+
+# ---------------------------------------------------------------------------
+# INT8 Quantization
+# ---------------------------------------------------------------------------
+
+def quantize_int8(model, calibration_inputs, output_dir="int8"):
+    """Quantize model to INT8 using percentile calibration.
+
+    Uses ModelOpt's PyTorch-native quantization API.
+    """
+    import modelopt.torch.quantization as mtq
+
+    print("\n" + "=" * 70)
+    print("INT8 Quantization (percentile calibration)")
+    print("=" * 70)
+
+    # INT8 config: activation + weight quantization
+    # Percentile calibration is more robust for VLMs than max calibration
+    INT8_CFG = {
+        "quant_cfg": {
+            "*weight": {"num_bits": 8, "axis": 0},
+            "*input_quantizer": {"num_bits": 8, "axis": None, "calib_method": "percentile"},
+        },
+        "algorithm": "percentile",
+    }
+
+    forward_loop = make_forward_loop(calibration_inputs)
+
+    t0 = time.time()
+    quantized_model = mtq.quantize(
+        model,
+        quant_cfg=INT8_CFG,
+        forward_loop=forward_loop,
+    )
+    elapsed = time.time() - t0
+
+    # Save quantized model
+    if not os.path.exists(output_dir):
+        os.makedirs(output_dir)
+    output_path = os.path.join(output_dir, "unidrive_vla_int8")
+    quantized_model.save_pretrained(output_path)
+    print(f"  Saved: {output_path} in {elapsed:.1f}s")
+
+    return quantized_model, output_path
+
+
+# ---------------------------------------------------------------------------
+# NVFP4 Quantization
+# ---------------------------------------------------------------------------
+
+def quantize_nvfp4(model, calibration_inputs, output_dir="nvfp4"):
+    """Quantize model to NVFP4 using AWQ weight-only quantization.
+
+    NVFP4 is NVIDIA's 4-bit floating point format optimized for inference
+    on Blackwell and newer architectures.
+    """
+    import modelopt.torch.quantization as mtq
+
+    print("\n" + "=" * 70)
+    print("NVFP4 Quantization (AWQ weight-only)")
+    print("=" * 70)
+
+    # NVFP4 config: weight-only quantization with AWQ
+    # AWQ (Activation-aware Weight Quantization) preserves important weights
+    NVFP4_CFG = {
+        "quant_cfg": {
+            "*weight": {"num_bits": "nf4", "axis": 0, "block_size": 128},
+            "*input_quantizer": {"enable": False},  # weight-only
+        },
+        "algorithm": "awq_lite",
+    }
+
+    forward_loop = make_forward_loop(calibration_inputs)
+
+    t0 = time.time()
+    quantized_model = mtq.quantize(
+        model,
+        quant_cfg=NVFP4_CFG,
+        forward_loop=forward_loop,
+    )
+    elapsed = time.time() - t0
+
+    # Save quantized model
+    if not os.path.exists(output_dir):
+        os.makedirs(output_dir)
+    output_path = os.path.join(output_dir, "unidrive_vla_nvfp4")
+    quantized_model.save_pretrained(output_path)
+    print(f"  Saved: {output_path} in {elapsed:.1f}s")
+
+    return quantized_model, output_path
+
+
+# ---------------------------------------------------------------------------
+# ONNX NVFP4 Quantization
+# ---------------------------------------------------------------------------
+
+def _export_quantized_model_to_onnx(
+    model,
+    calibration_inputs: list,
+    output_path: str,
+    opset: int = 23,
+):
+    """Export a quantized PyTorch model to ONNX.
+
+    Uses the first calibration sample as the dummy input for torch.onnx.export().
+    Opset 23+ is required for FLOAT4E2M1 support.
+    """
+    import torch.onnx
+
+    # Use the first calibration sample as dummy input
+    sample_input = calibration_inputs[0]
+
+    # Build a wrapper that accepts the sample input kwargs and returns forward output
+    class _ONNXWrapper(torch.nn.Module):
+        def __init__(self, model, sample_input):
+            super().__init__()
+            self.model = model
+            self._input_names = list(sample_input.keys())
+
+        def forward(self, input_ids, attention_mask, pixel_values=None,
+                    image_grid_thw=None, **kwargs):
+            inputs = {"input_ids": input_ids, "attention_mask": attention_mask}
+            if pixel_values is not None:
+                inputs["pixel_values"] = pixel_values
+            if image_grid_thw is not None:
+                inputs["image_grid_thw"] = image_grid_thw
+            out = self.model(**inputs, use_cache=False)
+            return out.logits
+
+    wrapper = _ONNXWrapper(model, sample_input)
+
+    # Prepare dynamic axes — batch dim and sequence dim are dynamic
+    dynamic_axes = {
+        "input_ids": {0: "batch", 1: "seq"},
+        "attention_mask": {0: "batch", 1: "seq"},
+        "logits": {0: "batch", 1: "seq"},
+    }
+
+    # Build args tuple from sample input
+    args = (
+        sample_input["input_ids"],
+        sample_input["attention_mask"],
+    )
+    kwargs_onnx = {}
+    if "pixel_values" in sample_input:
+        args = args + (sample_input["pixel_values"],)
+        dynamic_axes["pixel_values"] = {0: "batch", 1: "patches"}
+    else:
+        args = args + (None,)
+    if "image_grid_thw" in sample_input:
+        args = args + (sample_input["image_grid_thw"],)
+        dynamic_axes["image_grid_thw"] = {0: "batch"}
+    else:
+        args = args + (None,)
+
+    print(f"  Exporting to ONNX (opset={opset}) ...")
+    os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)
+
+    torch.onnx.export(
+        wrapper,
+        args,
+        output_path,
+        opset_version=opset,
+        input_names=[k for k in ["input_ids", "attention_mask",
+                                  "pixel_values", "image_grid_thw"]],
+        output_names=["logits"],
+        dynamic_axes=dynamic_axes,
+        use_external_data_format=True,
+        do_constant_folding=False,
+    )
+
+    size_mb = os.path.getsize(output_path) / 1e6
+    data_path = output_path + ".data"
+    if os.path.exists(data_path):
+        size_mb += os.path.getsize(data_path) / 1e6
+    print(f"  Exported: {output_path} ({size_mb:.1f} MB)")
+    return output_path
+
+
+def quantize_onnx_nvfp4(model, calibration_inputs, output_dir="onnx_nvfp4"):
+    """Quantize model to NVFP4 and export to ONNX with FLOAT4E2M1 weights.
+
+    This is the canonical ModelOpt 0.44.0 ONNX NVFP4 pipeline:
+
+    Step 1: PyTorch NVFP4 quantization via mtq.quantize() with NVFP4_AWQ_LITE_CFG.
+            Inserts TRT_FP4QDQ custom ops into the model graph.
+
+    Step 2: ONNX export via torch.onnx.export().  The TRT custom ops are
+            serialized as-is into the ONNX graph.
+
+    Step 3: Post-processing via NVFP4QuantExporter.  Replaces each TRT_FP4QDQ
+            node with a chain of two DequantizeLinear nodes:
+              FLOAT4E2M1 weights → (DQ, FP8 block scales) → FP8 →
+              (DQ, FP32 per-tensor scale) → FP32 output
+            This is the standard 2×DQ dequantization pattern for NVFP4.
+
+    Note: modelopt.onnx.quantization.quantize() does NOT support
+    quantize_mode="nvfp4" — only int8/fp8/int4 are wired in the dispatch.
+    The NVFP4QuantExporter path is the only supported way to produce
+    FLOAT4E2M1 ONNX models in ModelOpt 0.44.0.
+    """
+    import modelopt.torch.quantization as mtq
+
+    print("\n" + "=" * 70)
+    print("ONNX NVFP4 Quantization (PyTorch NVFP4 → ONNX → NVFP4QuantExporter)")
+    print("=" * 70)
+
+    if not os.path.exists(output_dir):
+        os.makedirs(output_dir)
+
+    # ------------------------------------------------------------------
+    # Step 1: PyTorch NVFP4 quantization
+    # ------------------------------------------------------------------
+    print("\n  Step 1/3: PyTorch NVFP4 quantization ...")
+    try:
+        NVFP4_CFG = mtq.NVFP4_AWQ_LITE_CFG
+    except AttributeError:
+        # Fallback if built-in config not available
+        NVFP4_CFG = {
+            "quant_cfg": {
+                "*weight": {"num_bits": (2, 1), "block_sizes": {-1: 16},
+                            "type": "dynamic", "scale_bits": (4, 3)},
+                "*input_quantizer": {"enable": False},
+            },
+            "algorithm": "awq_lite",
+        }
+
+    forward_loop = make_forward_loop(calibration_inputs)
+
+    t0 = time.time()
+    quantized_model = mtq.quantize(
+        model,
+        quant_cfg=NVFP4_CFG,
+        forward_loop=forward_loop,
+    )
+    elapsed = time.time() - t0
+    print(f"  PyTorch NVFP4 quantization done in {elapsed:.1f}s")
+
+    # Save the quantized PyTorch model as intermediate
+    torch_save_path = os.path.join(output_dir, "unidrive_vla_nvfp4_torch")
+    quantized_model.save_pretrained(torch_save_path)
+    print(f"  Saved PyTorch checkpoint: {torch_save_path}")
+
+    # ------------------------------------------------------------------
+    # Step 2: Export to ONNX
+    # ------------------------------------------------------------------
+    print("\n  Step 2/3: ONNX export ...")
+    onnx_raw_path = os.path.join(output_dir, "unidrive_vla_nvfp4_raw.onnx")
+
+    try:
+        _export_quantized_model_to_onnx(
+            quantized_model, calibration_inputs,
+            output_path=onnx_raw_path,
+            opset=23,  # opset 23+ required for FLOAT4E2M1
+        )
+    except Exception as e:
+        print(f"  [WARN] Standard ONNX export failed: {e}")
+        print(f"  Trying with opset=21 and do_constant_folding=True ...")
+        try:
+            _export_quantized_model_to_onnx(
+                quantized_model, calibration_inputs,
+                output_path=onnx_raw_path,
+                opset=21,
+            )
+        except Exception as e2:
+            print(f"  [ERROR] ONNX export failed entirely: {e2}")
+            print(f"  The quantized PyTorch model is still saved at: {torch_save_path}")
+            print(f"  You can use modelopt.torch.export.export_hf_checkpoint() "
+                  f"for TRT-LLM deployment instead.")
+            raise
+
+    # ------------------------------------------------------------------
+    # Step 3: NVFP4QuantExporter post-processing
+    # ------------------------------------------------------------------
+    print("\n  Step 3/3: NVFP4QuantExporter post-processing ...")
+    onnx_final_path = os.path.join(output_dir, "unidrive_vla_nvfp4.onnx")
+
+    try:
+        from modelopt.onnx.export import NVFP4QuantExporter
+
+        exporter = NVFP4QuantExporter()
+        exporter.process(onnx_raw_path, onnx_final_path)
+
+        size_mb = os.path.getsize(onnx_final_path) / 1e6
+        data_path = onnx_final_path + ".data"
+        if os.path.exists(data_path):
+            size_mb += os.path.getsize(data_path) / 1e6
+        print(f"  Post-processed ONNX: {onnx_final_path} ({size_mb:.1f} MB)")
+
+        # Clean up raw ONNX
+        if os.path.exists(onnx_raw_path):
+            os.remove(onnx_raw_path)
+            raw_data = onnx_raw_path + ".data"
+            if os.path.exists(raw_data):
+                os.remove(raw_data)
+            print(f"  Cleaned up raw ONNX: {onnx_raw_path}")
+
+    except ImportError:
+        print(f"  [WARN] NVFP4QuantExporter not available in this modelopt version.")
+        print(f"  Using raw ONNX export as final output.")
+        if os.path.exists(onnx_raw_path):
+            os.rename(onnx_raw_path, onnx_final_path)
+            raw_data = onnx_raw_path + ".data"
+            if os.path.exists(raw_data):
+                os.rename(raw_data, onnx_final_path + ".data")
+        print(f"  Final ONNX: {onnx_final_path}")
+
+    except Exception as e:
+        print(f"  [WARN] NVFP4QuantExporter post-processing failed: {e}")
+        print(f"  Using raw ONNX export as final output.")
+        if os.path.exists(onnx_raw_path):
+            import shutil
+            shutil.copy2(onnx_raw_path, onnx_final_path)
+            raw_data = onnx_raw_path + ".data"
+            if os.path.exists(raw_data):
+                shutil.copy2(raw_data, onnx_final_path + ".data")
+        print(f"  Final ONNX (raw): {onnx_final_path}")
+
+    return quantized_model, onnx_final_path
+
+
+# ---------------------------------------------------------------------------
+# Evaluation wrapper
+# ---------------------------------------------------------------------------
+
+def evaluate_on_drivelm(
+    model,
+    processor,
+    samples: list[dict],
+    device: torch.device,
+    label: str,
+    max_new_tokens: int = 256,
+    subset: int = 0,
+    gpt_eval: bool = False,
+    openai_api_key: str | None = None,
+) -> dict:
+    """Evaluate a model on DriveLM and compute all metrics."""
+    print(f"\n{'='*70}")
+    print(f"Evaluating: {label}")
+    print(f"{'='*70}")
+
+    eval_samples = samples[:subset] if subset > 0 else samples
+    print(f"  Samples: {len(eval_samples)}")
+
+    model.eval()
+    eval_result = evaluate_model(
+        model, processor, eval_samples, device,
+        max_new_tokens=max_new_tokens,
+    )
+    predictions = eval_result["predictions"]
+
+    # Compute metrics per tag
+    tag_metrics = {}
+
+    # Tag 0: Accuracy
+    if 0 in predictions and predictions[0]["predictions"]:
+        m = compute_accuracy(predictions[0]["predictions"], predictions[0]["references"])
+        tag_metrics[0] = m
+        print(f"  [Tag 0] Accuracy: {m['accuracy']*100:.2f}% ({m['correct']}/{m['total']})")
+    else:
+        tag_metrics[0] = {"accuracy": 0.0, "correct": 0, "total": 0}
+
+    # Tag 2: Language Score
+    if 2 in predictions and predictions[2]["predictions"]:
+        m = compute_language_metrics(predictions[2]["predictions"], predictions[2]["references"])
+        tag_metrics[2] = m
+        print(f"  [Tag 2] Language: BLEU-1={m['Bleu_1']:.4f}, ROUGE-L={m['ROUGE_L']:.4f}, "
+              f"Norm={m['language_score_normalized']:.4f}")
+    else:
+        tag_metrics[2] = {"language_score_normalized": 0.0}
+
+    # Tag 1: GPT Score
+    if 1 in predictions and predictions[1]["predictions"]:
+        if gpt_eval:
+            m = compute_gpt_score(
+                predictions[1]["questions"],
+                predictions[1]["predictions"],
+                predictions[1]["references"],
+                openai_api_key,
+            )
+            tag_metrics[1] = m
+            print(f"  [Tag 1] GPT-Score: {m['gpt_score']:.2f}/100")
+        else:
+            tag_metrics[1] = {"gpt_score": 0.0}
+            print(f"  [Tag 1] GPT-Score: SKIPPED")
+    else:
+        tag_metrics[1] = {"gpt_score": 0.0}
+
+    # Tag 3: Match Score
+    if 3 in predictions and predictions[3]["predictions"]:
+        if gpt_eval:
+            m = compute_match_score(
+                predictions[3]["questions"],
+                predictions[3]["predictions"],
+                predictions[3]["references"],
+                openai_api_key,
+            )
+            tag_metrics[3] = m
+            print(f"  [Tag 3] Match: {m['match_score']:.2f} (F1={m['coord_f1']:.4f})")
+        else:
+            m = compute_match_score(
+                predictions[3]["questions"],
+                predictions[3]["predictions"],
+                predictions[3]["references"],
+                api_key=None,
+            )
+            tag_metrics[3] = m
+            print(f"  [Tag 3] Match: F1={m['coord_f1']:.4f} (GPT skipped)")
+    else:
+        tag_metrics[3] = {"match_score": 0.0}
+
+    # Final score
+    accuracy = tag_metrics[0].get("accuracy", 0.0)
+    gpt_score = tag_metrics[1].get("gpt_score", 0.0)
+    language_score = tag_metrics[2].get("language_score_normalized", 0.0)
+    match_score = tag_metrics[3].get("match_score", 0.0)
+
+    final = compute_final_score(accuracy, gpt_score, language_score, match_score)
+    print(f"  Final Score: {final:.4f}")
+
+    return {
+        "label": label,
+        "accuracy": accuracy,
+        "gpt_score": gpt_score,
+        "language_score": language_score,
+        "match_score": match_score,
+        "final_score": final,
+        "tag_metrics": tag_metrics,
+        "eval_result": eval_result,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Summary printer
+# ---------------------------------------------------------------------------
+
+def print_summary(results: list[dict]):
+    """Print a comparison table of all quantization modes."""
+    print("\n" + "=" * 70)
+    print("Quantization & Evaluation Summary")
+    print("=" * 70)
+    print(f"{'Mode':<12} {'Accuracy':>10} {'GPT':>10} {'Language':>10} {'Match':>10} {'Final':>10}")
+    print("-" * 70)
+    for r in results:
+        print(f"{r['label']:<12} "
+              f"{r['accuracy']*100:>9.2f}% "
+              f"{r['gpt_score']:>9.2f} "
+              f"{r['language_score']:>10.4f} "
+              f"{r['match_score']:>9.2f} "
+              f"{r['final_score']:>10.4f}")
+    print("=" * 70)
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Quantize UniDriveVLA and evaluate on DriveLM"
+    )
+    parser.add_argument(
+        "--mode", type=str, nargs="*", default=ALL_MODES,
+        choices=ALL_MODES,
+        help=f"Quantization mode(s) to run (default: all). Choices: {ALL_MODES}",
+    )
+    parser.add_argument(
+        "--skip_baseline_eval", action="store_true",
+        help="Skip bf16 baseline evaluation",
+    )
+    parser.add_argument(
+        "--subset", type=int, default=200,
+        help="Evaluate on first N samples only (0 = all, default: 200)",
+    )
+    parser.add_argument(
+        "--split", type=str, default="train",
+        choices=["train", "val"],
+        help="DriveLM split for evaluation",
+    )
+    parser.add_argument(
+        "--calibration_samples", type=int, default=NUM_CALIBRATION_SAMPLES,
+        help=f"Number of calibration samples (default: {NUM_CALIBRATION_SAMPLES})",
+    )
+    parser.add_argument(
+        "--max_new_tokens", type=int, default=256,
+        help="Max tokens to generate per sample",
+    )
+    parser.add_argument(
+        "--gpt_eval", action="store_true",
+        help="Enable GPT-Score evaluation (requires OpenAI API key)",
+    )
+    parser.add_argument(
+        "--openai_api_key", type=str, default=None,
+        help="OpenAI API key for GPT-Score (or set OPENAI_API_KEY)",
+    )
+    parser.add_argument(
+        "--data_dir", type=str, default=None,
+        help="Local directory with DriveLM JSON + nuScenes images",
+    )
+    parser.add_argument(
+        "--nuscenes_dir", type=str, default=None,
+        help="Directory containing nuScenes samples/ folder",
+    )
+    args = parser.parse_args()
+
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    print(f"Device: {device}")
+    if torch.cuda.is_available():
+        print(f"  GPU: {torch.cuda.get_device_name(0)}")
+        print(f"  VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
+
+    # ------------------------------------------------------------------
+    # Load model & processor (bf16 baseline)
+    # ------------------------------------------------------------------
+    print(f"\nLoading {MODEL_NAME} (bf16) ...")
+
+    from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
+
+    processor = AutoProcessor.from_pretrained(MODEL_NAME)
+    model = Qwen3VLForConditionalGeneration.from_pretrained(
+        MODEL_NAME,
+        torch_dtype=torch.bfloat16,
+        device_map="auto",
+    )
+    model.eval()
+
+    param_count = sum(p.numel() for p in model.parameters()) / 1e9
+    print(f"  Architecture: Qwen3VLForConditionalGeneration")
+    print(f"  Parameters:   {param_count:.2f}B")
+    print(f"  Dtype:        bfloat16")
+
+    # ------------------------------------------------------------------
+    # Load dataset
+    # ------------------------------------------------------------------
+    print(f"\nLoading DriveLM dataset (split={args.split}) ...")
+    samples = load_drivelm_dataset(
+        split=args.split,
+        data_dir=args.data_dir,
+        nuscenes_dir=args.nuscenes_dir,
+    )
+
+    if not samples:
+        print("ERROR: No samples loaded. Check dataset access and paths.")
+        return
+
+    print(f"  Total QA pairs: {len(samples)}")
+
+    # ------------------------------------------------------------------
+    # Prepare calibration data
+    # ------------------------------------------------------------------
+    print(f"\nPreparing calibration data ({args.calibration_samples} samples) ...")
+    calibration_inputs = prepare_calibration_data(
+        samples, processor, device, num_samples=args.calibration_samples
+    )
+
+    if not calibration_inputs:
+        print("ERROR: No calibration data prepared.")
+        return
+
+    # ------------------------------------------------------------------
+    # Collect results
+    # ------------------------------------------------------------------
+    all_results = []
+
+    # --- bf16 Baseline ---
+    if not args.skip_baseline_eval:
+        print("\n" + "=" * 70)
+        print("Phase 1: bf16 Baseline Evaluation")
+        print("=" * 70)
+
+        baseline_result = evaluate_on_drivelm(
+            model, processor, samples, device,
+            label="bf16",
+            max_new_tokens=args.max_new_tokens,
+            subset=args.subset,
+            gpt_eval=args.gpt_eval,
+            openai_api_key=args.openai_api_key,
+        )
+        all_results.append(baseline_result)
+    else:
+        print("\n  Skipping bf16 baseline evaluation")
+
+    # --- INT8 ---
+    if "int8" in args.mode:
+        print("\n" + "=" * 70)
+        print("Phase 2: INT8 Quantization")
+        print("=" * 70)
+
+        try:
+            quantized_model, output_path = quantize_int8(
+                model, calibration_inputs, output_dir="int8"
+            )
+
+            int8_result = evaluate_on_drivelm(
+                quantized_model, processor, samples, device,
+                label="int8",
+                max_new_tokens=args.max_new_tokens,
+                subset=args.subset,
+                gpt_eval=args.gpt_eval,
+                openai_api_key=args.openai_api_key,
+            )
+            all_results.append(int8_result)
+
+            # Free memory
+            del quantized_model
+            gc.collect()
+            torch.cuda.empty_cache()
+        except Exception as e:
+            print(f"  INT8 FAILED: {e}")
+            import traceback
+            traceback.print_exc()
+
+    # --- NVFP4 ---
+    if "nvfp4" in args.mode:
+        print("\n" + "=" * 70)
+        print("Phase 3: NVFP4 Quantization")
+        print("=" * 70)
+
+        # Reload bf16 model if needed
+        if "int8" in args.mode:
+            print("  Reloading bf16 model ...")
+            model = Qwen3VLForConditionalGeneration.from_pretrained(
+                MODEL_NAME,
+                torch_dtype=torch.bfloat16,
+                device_map="auto",
+            )
+            model.eval()
+
+        try:
+            quantized_model, output_path = quantize_nvfp4(
+                model, calibration_inputs, output_dir="nvfp4"
+            )
+
+            nvfp4_result = evaluate_on_drivelm(
+                quantized_model, processor, samples, device,
+                label="nvfp4",
+                max_new_tokens=args.max_new_tokens,
+                subset=args.subset,
+                gpt_eval=args.gpt_eval,
+                openai_api_key=args.openai_api_key,
+            )
+            all_results.append(nvfp4_result)
+
+            # Free memory
+            del quantized_model
+            gc.collect()
+            torch.cuda.empty_cache()
+        except Exception as e:
+            print(f"  NVFP4 FAILED: {e}")
+            import traceback
+            traceback.print_exc()
+
+    # --- ONNX NVFP4 ---
+    if "onnx_nvfp4" in args.mode:
+        print("\n" + "=" * 70)
+        print("Phase 4: ONNX NVFP4 Quantization")
+        print("=" * 70)
+
+        # Reload bf16 model if needed (prior quantization modifies model in-place)
+        if "int8" in args.mode or "nvfp4" in args.mode:
+            print("  Reloading bf16 model ...")
+            del model
+            gc.collect()
+            torch.cuda.empty_cache()
+            model = Qwen3VLForConditionalGeneration.from_pretrained(
+                MODEL_NAME,
+                torch_dtype=torch.bfloat16,
+                device_map="auto",
+            )
+            model.eval()
+
+        try:
+            quantized_model, output_path = quantize_onnx_nvfp4(
+                model, calibration_inputs, output_dir="onnx_nvfp4"
+            )
+
+            # Evaluate the quantized PyTorch model (not ONNX — inference uses
+            # the torch model; ONNX is for TensorRT deployment)
+            onnx_nvfp4_result = evaluate_on_drivelm(
+                quantized_model, processor, samples, device,
+                label="onnx_nvfp4",
+                max_new_tokens=args.max_new_tokens,
+                subset=args.subset,
+                gpt_eval=args.gpt_eval,
+                openai_api_key=args.openai_api_key,
+            )
+            onnx_nvfp4_result["onnx_path"] = output_path
+            all_results.append(onnx_nvfp4_result)
+
+            # Free memory
+            del quantized_model
+            gc.collect()
+            torch.cuda.empty_cache()
+        except Exception as e:
+            print(f"  ONNX NVFP4 FAILED: {e}")
+            import traceback
+            traceback.print_exc()
+
+    # ------------------------------------------------------------------
+    # Summary
+    # ------------------------------------------------------------------
+    if all_results:
+        print_summary(all_results)
+
+        # Save results to JSON
+        output_file = "unidrive_vla_quantization_results.json"
+        serializable_results = []
+        for r in all_results:
+            sr = {k: v for k, v in r.items() if k != "eval_result"}
+            serializable_results.append(sr)
+
+        with open(output_file, "w") as f:
+            json.dump(serializable_results, f, indent=2)
+        print(f"\nResults saved to {output_file}")
+    else:
+        print("\nNo results to display.")
+
+
+if __name__ == "__main__":
+    main()