File size: 2,312 Bytes
59e93c8 0eb2912 59e93c8 0eb2912 59e93c8 0eb2912 59e93c8 13fd990 59e93c8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | ---
license: mit
tags:
- traffic-safety
- video-question-answering
- captioning
- ai-city-challenge
- v-jepa
- qwen3-vl
base_model:
- Qwen/Qwen3-VL-8B-Instruct
---
# Traffic-JEPA — checkpoints
Checkpoints for **Traffic-JEPA**, the 1st place solution to the 10th AI City Challenge (2026),
Track 2: Transportation Safety Understanding and Captioning (team **Latent Painter – UTE**,
S2 60.0853, VQA accuracy 87.09 %).
Code: https://github.com/ThuongBuiRVC/Traffic-JEPA
## Files
| File | Stage | What |
|---|---|---|
| `model_best.pt` | VQA | the trained predictor head (5.83 M params) on top of frozen V-JEPA 2.1, Llama-3.2-1B and EmbeddingGemma |
| `caption_lora/` | Captioning, **default** | LoRA (rank 16, alpha 32) for `Qwen/Qwen3-VL-8B-Instruct`, trained text-only on SynWTS: facts -> caption. This is the configuration the paper reports. |
| `caption_lora_mm/` | Captioning, variant | the same LoRA shape trained **multimodally** — SynWTS facts *and* simulation frames -> caption. Used by the grounded variant, which also shows real frames at inference. |
## Usage
```bash
git clone https://github.com/ThuongBuiRVC/Traffic-JEPA
cd Traffic-JEPA
hf download ThuongBuiRVC/Traffic-JEPA --local-dir checkpoints/
bash scripts/05_submit_test.sh # VQA -> submissions/submission_final.json
bash scripts/06_caption.sh # caption -> submissions/caption_submission.json
bash scripts/06_caption.sh mm # grounded variant -> submissions/caption_submission_mm.json
```
The VQA model is rebuilt from `run_args.json`, which `scripts/05_submit_test.sh` writes next to
`model_best.pt` from `configs/train_args.json`. A mismatched config changes the scores without
raising anything, so do not swap it by hand.
## Training data
The VQA predictor is trained on SynWTS simulation questions; the decoder tables are estimated from
simulation counts. The caption LoRA is trained on SynWTS captions only. Neither uses test ground truth.
## Citation
```bibtex
@inproceedings{bui2026trafficjepa,
author = {Nguyen Hoai Thuong Bui and Thanh Nguyen Vo and Trinh Tra Giang Nguyen and Ha Duc Bui},
title = {Sim-to-Real Traffic Scene Understanding by Decoupling Semantics from Caption Generation with V-JEPA},
booktitle = {AI City Challenge Workshop},
year = {2026}
}
```
|