README.md

---
license: apache-2.0
base_model:
  - Qwen/Qwen2.5-VL-3B-Instruct
  - Qwen/Qwen2.5-VL-7B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
tags:
  - lora
  - peft
  - vlm
  - mapdr
  - traffic-sign
  - sign-to-lane
  - autonomous-driving
---

# SignVLM — LoRA adapters

LoRA adapters for **SignVLM**: vision-faithful sign-to-lane rule binding on
MapDR, built on `Qwen2.5-VL-{3B,7B}-Instruct`.

**Code & full reproduction recipe**: <https://github.com/ray90100/SignVLM-public>

Each subfolder is one adapter. SFT subfolders hold a single `final/`-style
adapter; DPO and GRPO subfolders each hold a 4-checkpoint trajectory
(`step_60/`, `step_100/`, `step_140/`, `step_200/`) so you can reproduce
the full training-curve plot from the paper.

## Quick start

Install the repo per its README §2 (Python 3.10 + the pinned
`requirements.txt`), download the matching base model, then:

```python
from peft import PeftModel
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

base = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    torch_dtype="bfloat16",
    device_map="cuda",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

# Paper main adapter (Qwen2.5-VL-7B + LoRA SFT + CAVP, seed 42).
model = PeftModel.from_pretrained(
    base,
    "ray90100/SignVLM-public",
    subfolder="sft-7B-CAVP-p0.3-s42",
)
model.eval()
```

For a DPO / GRPO trajectory checkpoint, point `subfolder` at the specific
step, e.g. `subfolder="dpo-7B-tokenmask-beta5.0/step_140"`.

### Download just one adapter to disk

```bash
huggingface-cli download ray90100/SignVLM-public \
    --include "sft-7B-CAVP-p0.3-s42/*" \
    --local-dir ./ckpts/
```

Then pass the local path to the eval / training scripts in the GitHub
repo, e.g.

```bash
python scripts/eval_sft.py --adapter ckpts/sft-7B-CAVP-p0.3-s42
python scripts/eval_canonical.py runs/sft/.../eval_<...> --rule-version v3
```

## Adapter index

All adapters use LoRA rank 64, `max_image_pixels = 802816`, bf16, AdamW,
trained on 6× RTX 4090 (24 GB).

### Stage 1 — SFT

| Subfolder | Base | Training | Paper reference |
|---|---|---|---|
| `sft-7B-CAVP-p0.3-s42` | Qwen2.5-VL-7B | `PERTURB_MODE=conflict PERTURB_PROB=0.3`, seed 42 | **Main result.** Tab 1 / Tab 3 / Tab 4 SignVLM-CAVP row; the headline Overall F1 ≈ 0.80 figure cited in the GitHub README §4.3 |
| `sft-7B-CAVP-p0.3-s43` | Qwen2.5-VL-7B | same, seed 43 | Tab 1 dual-seed |
| `sft-7B-no_CAVP-s42` | Qwen2.5-VL-7B | `PERTURB_MODE=none`, seed 42 | Tab 1 "SFT no-CAVP" row (illustrates the −61 pt collapse under panel-conflict input) |
| `sft-7B-noise0.3-s42` | Qwen2.5-VL-7B | `PERTURB_MODE=noise PERTURB_PROB=0.3`, seed 42 | Legacy noise ablation (input-fidelity training, not vision-prior robustness) |
| `sft-3B-CAVP-p0.3-s42` | Qwen2.5-VL-3B | `PERTURB_MODE=conflict PERTURB_PROB=0.3`, seed 42 | Tab 1 / Tab 3 capacity-conditional finding (most stable 3B run) |
| `sft-3B-CAVP-p0.5-s42` | Qwen2.5-VL-3B | `PERTURB_MODE=conflict PERTURB_PROB=0.5`, seed 42 | Tab 3 — at high conflict intensity matches 7B-p0.3 within noise floor |

### Stage 2 — DPO (experimental, results withheld)

Token-mask DPO (`--token-mask-dpo`) with KL anchor + adapter swap
(`β=5.0`, `lr=5e-6`, 2 epochs), initialised from the matching SFT-CAVP-p0.3
adapter. Each subfolder contains `step_60/`, `step_100/`, `step_140/`,
`step_200/`.

| Subfolder | Init adapter |
|---|---|
| `dpo-7B-tokenmask-beta5.0` | `sft-7B-CAVP-p0.3-s42` |
| `dpo-3B-tokenmask-beta5.0` | `sft-3B-CAVP-p0.3-s42` |

### Stage 2 — GRPO (experimental, results withheld)

GRPO with per-field-graded reward, initialised from
`sft-3B-CAVP-p0.3-s42`. Same 4-step trajectory layout.

| Subfolder | Reward weighting |
|---|---|
| `grpo-3B-uniform` | uniform across conflict types |
| `grpo-3B-B-variant` | direction-weighted `(0.15, 0.55, 0.15, 0.15)` |

## Limitations

- All adapters are trained on **MapDR** (urban / expressway dash-cam,
  roadside pillar signs in mainland China). Behaviour on other regions or
  on overhead gantry signs is not characterised.
- Stage 2 (DPO / GRPO) checkpoints are released for transparency; the
  corresponding paper numbers are withheld pending review.
- The base Qwen2.5-VL checkpoints are **not** included here — download
  them from their official Hugging Face repos.

## License

Adapters are released under Apache-2.0. The underlying MapDR dataset is
CC BY-NC-SA 4.0 (non-commercial). Cite the MapDR paper if you publish
results using these adapters.

## Citation

Paper under review. Placeholder in the GitHub repo's README §10 will be
updated once accepted.