| --- |
| license: mit |
| tags: |
| - traffic-safety |
| - video-question-answering |
| - captioning |
| - ai-city-challenge |
| - v-jepa |
| - qwen3-vl |
| base_model: |
| - Qwen/Qwen3-VL-8B-Instruct |
| --- |
| |
| # Traffic-JEPA — checkpoints |
|
|
| Checkpoints for **Traffic-JEPA**, the 1st place solution to the 10th AI City Challenge (2026), |
| Track 2: Transportation Safety Understanding and Captioning (team **Latent Painter – UTE**, |
| S2 60.0853, VQA accuracy 87.09 %). |
|
|
| Code: https://github.com/ThuongBuiRVC/Traffic-JEPA |
|
|
| ## Files |
|
|
| | File | Stage | What | |
| |---|---|---| |
| | `model_best.pt` | VQA | the trained predictor head (5.83 M params) on top of frozen V-JEPA 2.1, Llama-3.2-1B and EmbeddingGemma | |
| | `caption_lora/` | Captioning, **default** | LoRA (rank 16, alpha 32) for `Qwen/Qwen3-VL-8B-Instruct`, trained text-only on SynWTS: facts -> caption. This is the configuration the paper reports. | |
| | `caption_lora_mm/` | Captioning, variant | the same LoRA shape trained **multimodally** — SynWTS facts *and* simulation frames -> caption. Used by the grounded variant, which also shows real frames at inference. | |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/ThuongBuiRVC/Traffic-JEPA |
| cd Traffic-JEPA |
| hf download ThuongBuiRVC/Traffic-JEPA --local-dir checkpoints/ |
| |
| bash scripts/05_submit_test.sh # VQA -> submissions/submission_final.json |
| bash scripts/06_caption.sh # caption -> submissions/caption_submission.json |
| bash scripts/06_caption.sh mm # grounded variant -> submissions/caption_submission_mm.json |
| ``` |
|
|
| The VQA model is rebuilt from `run_args.json`, which `scripts/05_submit_test.sh` writes next to |
| `model_best.pt` from `configs/train_args.json`. A mismatched config changes the scores without |
| raising anything, so do not swap it by hand. |
|
|
| ## Training data |
|
|
| The VQA predictor is trained on SynWTS simulation questions; the decoder tables are estimated from |
| simulation counts. The caption LoRA is trained on SynWTS captions only. Neither uses test ground truth. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{bui2026trafficjepa, |
| author = {Nguyen Hoai Thuong Bui and Thanh Nguyen Vo and Trinh Tra Giang Nguyen and Ha Duc Bui}, |
| title = {Sim-to-Real Traffic Scene Understanding by Decoupling Semantics from Caption Generation with V-JEPA}, |
| booktitle = {AI City Challenge Workshop}, |
| year = {2026} |
| } |
| ``` |
|
|
