Add model card

5a3a92b verified about 1 month ago

4.67 kB

license: apache-2.0
base_model:
  - Qwen/Qwen2.5-VL-3B-Instruct
  - Qwen/Qwen2.5-VL-7B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
tags:
  - lora
  - peft
  - vlm
  - mapdr
  - traffic-sign
  - sign-to-lane
  - autonomous-driving

SignVLM — LoRA adapters

LoRA adapters for SignVLM: vision-faithful sign-to-lane rule binding on MapDR, built on Qwen2.5-VL-{3B,7B}-Instruct.

Code & full reproduction recipe: https://github.com/ray90100/SignVLM-public

Each subfolder is one adapter. SFT subfolders hold a single final/-style adapter; DPO and GRPO subfolders each hold a 4-checkpoint trajectory (step_60/, step_100/, step_140/, step_200/) so you can reproduce the full training-curve plot from the paper.

Quick start

Install the repo per its README §2 (Python 3.10 + the pinned requirements.txt), download the matching base model, then:

from peft import PeftModel
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

base = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    torch_dtype="bfloat16",
    device_map="cuda",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

# Paper main adapter (Qwen2.5-VL-7B + LoRA SFT + CAVP, seed 42).
model = PeftModel.from_pretrained(
    base,
    "ray90100/SignVLM-public",
    subfolder="sft-7B-CAVP-p0.3-s42",
)
model.eval()

For a DPO / GRPO trajectory checkpoint, point subfolder at the specific step, e.g. subfolder="dpo-7B-tokenmask-beta5.0/step_140".

Download just one adapter to disk

huggingface-cli download ray90100/SignVLM-public \
    --include "sft-7B-CAVP-p0.3-s42/*" \
    --local-dir ./ckpts/

Then pass the local path to the eval / training scripts in the GitHub repo, e.g.

python scripts/eval_sft.py --adapter ckpts/sft-7B-CAVP-p0.3-s42
python scripts/eval_canonical.py runs/sft/.../eval_<...> --rule-version v3

Adapter index

All adapters use LoRA rank 64, max_image_pixels = 802816, bf16, AdamW, trained on 6× RTX 4090 (24 GB).

Stage 1 — SFT

Subfolder	Base	Training	Paper reference
`sft-7B-CAVP-p0.3-s42`	Qwen2.5-VL-7B	`PERTURB_MODE=conflict PERTURB_PROB=0.3`, seed 42	Main result. Tab 1 / Tab 3 / Tab 4 SignVLM-CAVP row; the headline Overall F1 ≈ 0.80 figure cited in the GitHub README §4.3
`sft-7B-CAVP-p0.3-s43`	Qwen2.5-VL-7B	same, seed 43	Tab 1 dual-seed
`sft-7B-no_CAVP-s42`	Qwen2.5-VL-7B	`PERTURB_MODE=none`, seed 42	Tab 1 "SFT no-CAVP" row (illustrates the −61 pt collapse under panel-conflict input)
`sft-7B-noise0.3-s42`	Qwen2.5-VL-7B	`PERTURB_MODE=noise PERTURB_PROB=0.3`, seed 42	Legacy noise ablation (input-fidelity training, not vision-prior robustness)
`sft-3B-CAVP-p0.3-s42`	Qwen2.5-VL-3B	`PERTURB_MODE=conflict PERTURB_PROB=0.3`, seed 42	Tab 1 / Tab 3 capacity-conditional finding (most stable 3B run)
`sft-3B-CAVP-p0.5-s42`	Qwen2.5-VL-3B	`PERTURB_MODE=conflict PERTURB_PROB=0.5`, seed 42	Tab 3 — at high conflict intensity matches 7B-p0.3 within noise floor

Stage 2 — DPO (experimental, results withheld)

Token-mask DPO (--token-mask-dpo) with KL anchor + adapter swap (β=5.0, lr=5e-6, 2 epochs), initialised from the matching SFT-CAVP-p0.3 adapter. Each subfolder contains step_60/, step_100/, step_140/, step_200/.

Subfolder	Init adapter
`dpo-7B-tokenmask-beta5.0`	`sft-7B-CAVP-p0.3-s42`
`dpo-3B-tokenmask-beta5.0`	`sft-3B-CAVP-p0.3-s42`

Stage 2 — GRPO (experimental, results withheld)

GRPO with per-field-graded reward, initialised from sft-3B-CAVP-p0.3-s42. Same 4-step trajectory layout.

Subfolder	Reward weighting
`grpo-3B-uniform`	uniform across conflict types
`grpo-3B-B-variant`	direction-weighted `(0.15, 0.55, 0.15, 0.15)`

Limitations

All adapters are trained on MapDR (urban / expressway dash-cam, roadside pillar signs in mainland China). Behaviour on other regions or on overhead gantry signs is not characterised.
Stage 2 (DPO / GRPO) checkpoints are released for transparency; the corresponding paper numbers are withheld pending review.
The base Qwen2.5-VL checkpoints are not included here — download them from their official Hugging Face repos.

License

Adapters are released under Apache-2.0. The underlying MapDR dataset is CC BY-NC-SA 4.0 (non-commercial). Cite the MapDR paper if you publish results using these adapters.

Citation

Paper under review. Placeholder in the GitHub repo's README §10 will be updated once accepted.