---
license: mit
tags:
  - traffic-safety
  - video-question-answering
  - captioning
  - ai-city-challenge
  - v-jepa
  - qwen3-vl
base_model:
  - Qwen/Qwen3-VL-8B-Instruct
---

# Traffic-JEPA — checkpoints

Checkpoints for **Traffic-JEPA**, the 1st place solution to the 10th AI City Challenge (2026),
Track 2: Transportation Safety Understanding and Captioning (team **Latent Painter – UTE**,
S2 60.0853, VQA accuracy 87.09 %).

Code: https://github.com/ThuongBuiRVC/Traffic-JEPA

## Files

| File | Stage | What |
|---|---|---|
| `model_best.pt` | VQA | the trained predictor head (5.83 M params) on top of frozen V-JEPA 2.1, Llama-3.2-1B and EmbeddingGemma |
| `caption_lora/` | Captioning, **default** | LoRA (rank 16, alpha 32) for `Qwen/Qwen3-VL-8B-Instruct`, trained text-only on SynWTS: facts -> caption. This is the configuration the paper reports. |
| `caption_lora_mm/` | Captioning, variant | the same LoRA shape trained **multimodally** — SynWTS facts *and* simulation frames -> caption. Used by the grounded variant, which also shows real frames at inference. |

## Usage

```bash
git clone https://github.com/ThuongBuiRVC/Traffic-JEPA
cd Traffic-JEPA
hf download ThuongBuiRVC/Traffic-JEPA --local-dir checkpoints/

bash scripts/05_submit_test.sh     # VQA     -> submissions/submission_final.json
bash scripts/06_caption.sh         # caption -> submissions/caption_submission.json
bash scripts/06_caption.sh mm      # grounded variant -> submissions/caption_submission_mm.json
```

The VQA model is rebuilt from `run_args.json`, which `scripts/05_submit_test.sh` writes next to
`model_best.pt` from `configs/train_args.json`. A mismatched config changes the scores without
raising anything, so do not swap it by hand.

## Training data

The VQA predictor is trained on SynWTS simulation questions; the decoder tables are estimated from
simulation counts. The caption LoRA is trained on SynWTS captions only. Neither uses test ground truth.

## Citation

```bibtex
@inproceedings{bui2026trafficjepa,
  author    = {Nguyen Hoai Thuong Bui and Thanh Nguyen Vo and Trinh Tra Giang Nguyen and Ha Duc Bui},
  title     = {Sim-to-Real Traffic Scene Understanding by Decoupling Semantics from Caption Generation with V-JEPA},
  booktitle = {AI City Challenge Workshop},
  year      = {2026}
}
```