license: apache-2.0
base_model: Qwen/Qwen3-VL-4B-Instruct
library_name: peft
pipeline_tag: video-classification
tags:
- driving
- safety
- accident-anticipation
- driver-alert
- vision-language-model
- lora
datasets:
- HenryYHW/VLAlert
VLAlert — Observe Before You Alert: Belief-State Driver Alerting with Vision–Language Models (CoRL 2026)
This repository holds the model weights and code behind the CoRL 2026 paper Observe Before You Alert: Belief-State Driver Alerting with Vision–Language Models (Yuhang Wang, Lingyao Li, Hao Zhou). The benchmark, annotations, and all result files live in the companion dataset repository: https://huggingface.co/datasets/HenryYHW/VLAlert
arXiv version of the paper: https://arxiv.org/abs/2609.08130
VLAlert casts dashcam driver alerting as a tri-action policy over SILENT / OBSERVE / ALERT.
A LoRA-adapted Qwen3-VL-4B generates a structured safety-evidence span
<|BELIEF|> ... </|BELIEF|>; the hidden states inside that span are pooled into a compact belief
vector that drives a DangerHead (risk) and a PolicyHeadV2 (three-way action).
Contents
| Path | What it is |
|---|---|
sft_x_v3_lora/ |
Stage-1 LoRA adapter for Qwen/Qwen3-VL-4B-Instruct (PEFT, r=32, α=64, targets q/k/v/o/gate/up/down), tokenizer files with the added `< |
heads/armB/danger_c1last_seed{1..5}/best.pt |
DangerHead checkpoints (5 seeds) trained on the source-balanced 11,640-tick subset; these heads produce the Table 2 / test-split numbers in the paper. |
heads/armB/policy/best.pt |
PolicyHeadV2 (three-way SILENT/OBSERVE/ALERT) trained on the same subset. |
heads/armB/danger_full_seed1/best.pt |
Full-size DangerHead variant used together with the policy head for the qualitative demo videos. |
code/lkalert/ |
Model components under models/ (belief_vlm.py span pooling, danger_head.py, policy_head_v2.py, adaptive_window.py) and benchmark label rules under labels/. |
code/training/SFT/ |
Stage-1 belief-format supervised fine-tuning (LoRA). |
code/training/Policy/, code/training/danger/ |
Belief caching, head training, policy evaluation and post-processing. |
code/tools/ |
Evaluation and paper scripts, e.g. build_paper_final_v3.py (Table 2 protocol, DAUS), compute_daus_v1_val.py, eval_badas_v1_val.py, score_v1_val_gemini.py, score_v1_val_baselines.py, demo_score_v3_new.py, render_demo_videos.py. |
code/colab/train_heads_colab.py |
Head training script used for the camera-ready runs (5 seeds, validation-only selection). |
code/requirements.txt, code/setup.py, code/configs/ |
Environment and configs. |
Scripts reference local paths of the original experiments (e.g. /home/henry/Desktop/LKAlert/...);
point them at your own copies of the benchmark files from the dataset repository.
Pipeline
- Belief generation. For each 1 Hz tick the VLM receives the last 8 frames and emits
<|BELIEF|> evidence text </|BELIEF|>followed by an action token. - Belief pooling. Hidden states between the two delimiters are mean-pooled per frame and the last four transformer layers are concatenated, giving a 10,240-d belief vector per frame; the hidden state at the closing delimiter is kept as a decision register.
- Heads. DangerHead scores visual risk from the 8 frame beliefs; PolicyHeadV2 outputs the SILENT/OBSERVE/ALERT distribution. P(ALERT) is thresholded at 0.3394, selected on the validation split by maximizing balanced accuracy and then frozen for the test split.
Results (VLAlert-Bench, full denominators)
| AUROC | AP_tick | Recall_v | F1_t | BalAcc | DAUS | |
|---|---|---|---|---|---|---|
| Validation (11,220 ticks) | 0.680 | 0.229 | 0.914 | 0.321 | 0.638 | 0.5001 |
| Test (23,661 ticks, threshold frozen on validation) | 0.679 | 0.254 | 0.952 | 0.349 | 0.634 | — |
| BADAS-Open re-scored on identical frames, test | 0.519 | 0.158 | 0.935 | 0.266 | 0.523 | — |
Paired bootstrap over test videos: ΔAUROC = +0.159 [+0.146, +0.172]. On 221 ADAS-TO-Critic takeover
clips VLAlert reaches R@5s 88.7% vs. 74.2% for BADAS-Open. Per-tick score files that reproduce every
number are in the dataset repository under results/.
Quickstart (sketch)
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
import torch
base = "Qwen/Qwen3-VL-4B-Instruct"
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="cuda")
processor = AutoProcessor.from_pretrained("HenryYHW/VLAlert", subfolder="sft_x_v3_lora") # tokenizer with BELIEF tokens
model.resize_token_embeddings(len(processor.tokenizer))
model = PeftModel.from_pretrained(model, "HenryYHW/VLAlert", subfolder="sft_x_v3_lora")
# 1) generate the BELIEF span for 8 frames (see code/training/Policy/make_belief_cache_v2.py),
# 2) pool hidden states between <|BELIEF|> and </|BELIEF|> over layers 20/24/28/32 (code/lkalert/models/belief_vlm.py),
# 3) run heads/armB/danger_c1last_seed1/best.pt and heads/armB/policy/best.pt (code/lkalert/models/danger_head.py, policy_head_v2.py),
# 4) alert when P(ALERT) >= 0.3394.
code/tools/demo_score_v3_new.py runs the complete streaming pipeline on an mp4 and writes per-tick
action probabilities; code/tools/render_demo_videos.py renders the side-by-side demo videos.
License
Adapter weights, head checkpoints and code: Apache-2.0. The base model Qwen3-VL-4B-Instruct is released by Alibaba under Apache-2.0. Benchmark data and annotations are in the dataset repository under CC-BY-4.0; third-party source videos are not redistributed.
Citation
CoRL 2026 camera-ready:
@inproceedings{wang2026vlalert,
title = {Observe Before You Alert: Belief-State Driver Alerting with Vision--Language Models},
author = {Wang, Yuhang and Li, Lingyao and Zhou, Hao},
booktitle = {Conference on Robot Learning (CoRL)},
year = {2026}
}
arXiv preprint (submitted under the earlier title):
@misc{wang2026observe_arxiv,
title = {Observe Before You Alert: Adaptive Driver Alerting with Vision-Language Models},
author = {Wang, Yuhang and Li, Lingyao and Zhou, Hao},
year = {2026},
eprint = {2609.08130},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2609.08130}
}