README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-VL-4B-Instruct
library_name: peft
pipeline_tag: video-classification
tags:
- driving
- safety
- accident-anticipation
- driver-alert
- vision-language-model
- lora
datasets:
- HenryYHW/VLAlert
---

# VLAlert — Observe Before You Alert: Belief-State Driver Alerting with Vision–Language Models (CoRL 2026)

This repository holds the model weights and code behind the CoRL 2026 paper
*Observe Before You Alert: Belief-State Driver Alerting with Vision–Language Models*
(Yuhang Wang, Lingyao Li, Hao Zhou). The benchmark, annotations, and all result files live in the
companion dataset repository: https://huggingface.co/datasets/HenryYHW/VLAlert

arXiv version of the paper: https://arxiv.org/abs/2609.08130

VLAlert casts dashcam driver alerting as a tri-action policy over **SILENT / OBSERVE / ALERT**.
A LoRA-adapted Qwen3-VL-4B generates a structured safety-evidence span
`<|BELIEF|> ... </|BELIEF|>`; the hidden states inside that span are pooled into a compact belief
vector that drives a DangerHead (risk) and a PolicyHeadV2 (three-way action).

## Contents

| Path | What it is |
|---|---|
| `sft_x_v3_lora/` | Stage-1 LoRA adapter for `Qwen/Qwen3-VL-4B-Instruct` (PEFT, r=32, α=64, targets q/k/v/o/gate/up/down), tokenizer files with the added `<|BELIEF|>` / `</|BELIEF|>` tokens (`added_tokens.json`, `belief_tokens.json`), processor config, and `train_args.json`. |
| `heads/armB/danger_c1last_seed{1..5}/best.pt` | DangerHead checkpoints (5 seeds) trained on the source-balanced 11,640-tick subset; these heads produce the Table 2 / test-split numbers in the paper. |
| `heads/armB/policy/best.pt` | PolicyHeadV2 (three-way SILENT/OBSERVE/ALERT) trained on the same subset. |
| `heads/armB/danger_full_seed1/best.pt` | Full-size DangerHead variant used together with the policy head for the qualitative demo videos. |
| `code/lkalert/` | Model components under `models/` (`belief_vlm.py` span pooling, `danger_head.py`, `policy_head_v2.py`, `adaptive_window.py`) and benchmark label rules under `labels/`. |
| `code/training/SFT/` | Stage-1 belief-format supervised fine-tuning (LoRA). |
| `code/training/Policy/`, `code/training/danger/` | Belief caching, head training, policy evaluation and post-processing. |
| `code/tools/` | Evaluation and paper scripts, e.g. `build_paper_final_v3.py` (Table 2 protocol, DAUS), `compute_daus_v1_val.py`, `eval_badas_v1_val.py`, `score_v1_val_gemini.py`, `score_v1_val_baselines.py`, `demo_score_v3_new.py`, `render_demo_videos.py`. |
| `code/colab/train_heads_colab.py` | Head training script used for the camera-ready runs (5 seeds, validation-only selection). |
| `code/requirements.txt`, `code/setup.py`, `code/configs/` | Environment and configs. |

Scripts reference local paths of the original experiments (e.g. `/home/henry/Desktop/LKAlert/...`);
point them at your own copies of the benchmark files from the dataset repository.

## Pipeline

1. **Belief generation.** For each 1 Hz tick the VLM receives the last 8 frames and emits
   `<|BELIEF|> evidence text </|BELIEF|>` followed by an action token.
2. **Belief pooling.** Hidden states between the two delimiters are mean-pooled per frame and the last
   four transformer layers are concatenated, giving a 10,240-d belief vector per frame; the hidden state
   at the closing delimiter is kept as a decision register.
3. **Heads.** DangerHead scores visual risk from the 8 frame beliefs; PolicyHeadV2 outputs the
   SILENT/OBSERVE/ALERT distribution. P(ALERT) is thresholded at **0.3394**, selected on the validation
   split by maximizing balanced accuracy and then frozen for the test split.

## Results (VLAlert-Bench, full denominators)

| | AUROC | AP_tick | Recall_v | F1_t | BalAcc | DAUS |
|---|---|---|---|---|---|---|
| Validation (11,220 ticks) | 0.680 | 0.229 | 0.914 | 0.321 | 0.638 | 0.5001 |
| Test (23,661 ticks, threshold frozen on validation) | 0.679 | 0.254 | 0.952 | 0.349 | 0.634 | — |
| BADAS-Open re-scored on identical frames, test | 0.519 | 0.158 | 0.935 | 0.266 | 0.523 | — |

Paired bootstrap over test videos: ΔAUROC = +0.159 [+0.146, +0.172]. On 221 ADAS-TO-Critic takeover
clips VLAlert reaches R@5s 88.7% vs. 74.2% for BADAS-Open. Per-tick score files that reproduce every
number are in the dataset repository under `results/`.

## Quickstart (sketch)

```python
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
import torch

base = "Qwen/Qwen3-VL-4B-Instruct"
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="cuda")
processor = AutoProcessor.from_pretrained("HenryYHW/VLAlert", subfolder="sft_x_v3_lora")  # tokenizer with BELIEF tokens
model.resize_token_embeddings(len(processor.tokenizer))
model = PeftModel.from_pretrained(model, "HenryYHW/VLAlert", subfolder="sft_x_v3_lora")

# 1) generate the BELIEF span for 8 frames (see code/training/Policy/make_belief_cache_v2.py),
# 2) pool hidden states between <|BELIEF|> and </|BELIEF|> over layers 20/24/28/32 (code/lkalert/models/belief_vlm.py),
# 3) run heads/armB/danger_c1last_seed1/best.pt and heads/armB/policy/best.pt (code/lkalert/models/danger_head.py, policy_head_v2.py),
# 4) alert when P(ALERT) >= 0.3394.
```

`code/tools/demo_score_v3_new.py` runs the complete streaming pipeline on an mp4 and writes per-tick
action probabilities; `code/tools/render_demo_videos.py` renders the side-by-side demo videos.

## License

Adapter weights, head checkpoints and code: Apache-2.0. The base model Qwen3-VL-4B-Instruct is
released by Alibaba under Apache-2.0. Benchmark data and annotations are in the dataset repository
under CC-BY-4.0; third-party source videos are not redistributed.

## Citation

CoRL 2026 camera-ready:

```bibtex
@inproceedings{wang2026vlalert,
  title     = {Observe Before You Alert: Belief-State Driver Alerting with Vision--Language Models},
  author    = {Wang, Yuhang and Li, Lingyao and Zhou, Hao},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}
```

arXiv preprint (submitted under the earlier title):

```bibtex
@misc{wang2026observe_arxiv,
  title         = {Observe Before You Alert: Adaptive Driver Alerting with Vision-Language Models},
  author        = {Wang, Yuhang and Li, Lingyao and Zhou, Hao},
  year          = {2026},
  eprint        = {2609.08130},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2609.08130}
}
```