abhifdsdf/Qwen2.5-VL-FUNSD-Adversarial

Qwen2.5-VL-3B — Adversarially Robust FUNSD Document Entity Extraction

Executive Summary

This repository contains LoRA adapters for Qwen/Qwen2.5-VL-3B-Instruct, supervised fine-tuned on the FUNSD form-understanding dataset for structured key/value entity extraction ({Header, Question, Answer}), with explicit training and evaluation under simulated scanner/photo degradation (blur, noise, contrast shift, rotation).

Architecture & Training Setup

• Quantization: 4-bit NF4, double quantization, bfloat16 compute dtype.
• PEFT/LoRA: r=16, alpha=32, dropout=0.05, target modules = q/k/v/o/gate/up/down projections.
• Optimizer: paged_adamw_8bit, LR=0.0002, scheduler=cosine, warmup_ratio=0.05, grad_accum_steps=4, max_grad_norm=1.0.
• Curriculum adversarial sampling: noise probability ramps 0.2 → 0.5 across 5 epochs.
• Best checkpoint selected by validation loss; early stopping patience = 2.

Adversarial Robustness Results

Metric	Clean Test Set	Adversarial Test Set (100% degraded)
Exact match rate	0.0000	0.0000
Char-level similarity	0.2098	0.1565
Entity precision	1.0000	1.0000
Entity recall	1.0000	1.0000
Entity F1	1.0000	1.0000

Artifact Index

• adapters/ — final LoRA adapter weights + processor config; load with PeftModel.from_pretrained.
• graphs/training_loss_curve.png — training vs. validation loss over training.
• graphs/clean_vs_adversarial_metrics.png — bar chart of all eval metrics, clean vs. adversarial.
• graphs/parameter_efficiency_pie.png — trainable vs. frozen parameter share.
• predictions/sample_clean_inference.png — qualitative clean-image predictions vs. ground truth.
• predictions/sample_adversarial_inference.png — qualitative degraded-image predictions vs. ground truth.
• metrics.json — full numerical results breakdown.

Inference Code Snippet

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-VL-3B-Instruct"
adapter_repo = "abhifdsdf/Qwen2.5-VL-FUNSD-Adversarial"

processor = AutoProcessor.from_pretrained(base_model_id, trust_remote_code=True)
base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base_model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, adapter_repo)

image = Image.open("your_document.png").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": "Analyze this document image and extract all structured key-value pairs in standard text format."},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
print(processor.batch_decode(output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Comparison: Fine-Tuned Model vs. Zero-Shot Baselines

This table shows the exact same clean/adversarial evaluation harness run against:

• The fine-tuned adapter produced by this notebook.
• The same base model (Qwen/Qwen2.5-VL-3B-Instruct) with no fine-tuning at all (zero-shot).
• A different off-the-shelf VLM that never saw FUNSD or this prompt format.

Model	Split	Exact Match	Char Similarity	Entity Precision	Entity Recall	Entity F1
Fine-tuned (ours)	clean	0.0000	0.2098	1.0000	1.0000	1.0000
Fine-tuned (ours)	adversarial	0.0000	0.1565	1.0000	1.0000	1.0000

Comparison: Fine-Tuned Model vs. Zero-Shot Baselines

This table shows the exact same clean/adversarial evaluation harness run against:

• The fine-tuned adapter produced by this notebook.
• The same base model (Qwen/Qwen2.5-VL-3B-Instruct) with no fine-tuning at all (zero-shot).
• A different off-the-shelf VLM that never saw FUNSD or this prompt format.

Model	Split	Exact Match	Char Similarity	Entity Precision	Entity Recall	Entity F1
Fine-tuned (ours)	clean	0.0000	0.2098	1.0000	1.0000	1.0000
Fine-tuned (ours)	adversarial	0.0000	0.1565	1.0000	1.0000	1.0000
Qwen2.5-VL-3B-Instruct (zero-shot, no fine-tuning)	clean	0.0000	0.2120	0.3283	0.2335	0.2531
Qwen2.5-VL-3B-Instruct (zero-shot, no fine-tuning)	adversarial	0.0000	0.1651	0.2439	0.1847	0.1987
Qwen2-VL-2B-Instruct (different model, zero-shot)	clean	0.0000	0.0618	0.1440	0.1440	0.1440
Qwen2-VL-2B-Instruct (different model, zero-shot)	adversarial	0.0000	0.0353	0.1422	0.1433	0.1427