Brusnicki/SAVANT-anomaly-classifier-lora

SAVANT Direct Anomaly Classifier (LoRA Adapter)

This repository contains the LoRA adapter for the direct anomaly classification model described in the paper Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning.

Project Page: https://TUM-AVS.github.io/SAVANT/

This repository is provided for peer-review purposes only. After the review process, the model will be made publicly available through the authors' main account.

Model Description

LoRA adapter for Qwen/Qwen2.5-VL-7B-Instruct, fine-tuned for binary anomaly detection in autonomous driving scenes. Given a front-camera image, the model classifies the scene as anomaly or normal.

This is the direct classification model of the SAVANT framework — a single-shot approach that achieves the best accuracy among the evaluated configurations.

Performance

Evaluated on a balanced test set of 1,020 driving scene images:

Metric	Value
Accuracy	93.8%
Precision	96.7%
Recall	90.8%
F1-Score	93.6%

Training Details

• Base model: Qwen/Qwen2.5-VL-7B-Instruct
• Method: LoRA (Low-Rank Adaptation)
• Dataset: 4,055 balanced samples (anomaly/normal driving scenes)
• Epochs: 3
• Learning rate: 1e-4 (cosine schedule)
• Batch size: 8 per device
• Hardware: 4x NVIDIA RTX 4090
• Precision: bfloat16 with Flash Attention 2

LoRA Configuration

Parameter	Value
Rank (r)	16
Alpha	32
Dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, fc1, fc2, qkv, mlp.0, mlp.2

Usage

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
import torch

# Load base model
base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "u94fmn391j/SAVANT-anomaly-classifier-lora")

# Load processor
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

Limitations

• Trained on the CODA dataset; generalization to other driving domains not evaluated
• Single-frame analysis only (no temporal context)

Citation

@article{brusnicki2025can,
  title={Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning},
  author={Brusnicki, Roberto and Pop, David and Gao, Yuan and Piccinini, Mattia and Betz, Johannes},
  journal={arXiv preprint arXiv:2510.18034},
  year={2025}
}