README.md

---
license: apache-2.0
base_model: Qwen/Qwen2-VL-7B-Instruct
tags:
- vision
- image-to-text
- document-ai
- acord-forms
- qlora
---

# Qwen2-VL-7B: Specialized ACORD Insurance Form Extractor

## Engineering Overview
This model is a state-of-the-art Vision-Language Intelligence (V-LIE) engine, fine-tuned from **Qwen2-VL-7B-Instruct**. It is architected to eliminate traditional multi-stage OCR pipelines by mapping pixels directly to structured semantic JSON for complex, multi-column insurance documents.

### Technical Specifications
- **Base Architecture:** Qwen2-VL (ViT + Qwen2 LLM) utilizing Multi-Modal Rotary Positional Embeddings (M-RoPE) for spatial reasoning.
- **Fine-tuning Method:** Parameter-Efficient Fine-Tuning (PEFT) via QLoRA (4-bit NormalFloat quantization). 
- **Adaptation Strategy:** LoRA target modules included `q_proj`, `k_proj`, `v_proj`, `o_proj`, and MLP layers (`gate_proj`, `up_proj`, `down_proj`). The vision encoder remained frozen to preserve generalized OCR capabilities while the language head was adapted for schema-strict JSON generation.
- **Resolution Handling:** Supports dynamic resolution (256-1280 pixels) to maintain aspect ratio integrity, crucial for identifying dense ACORD form checkboxes and fine-print labels.
- **Training Objective:** Supervised Fine-Tuning (SFT) using a custom data collator to mask prompt/image tokens, focusing the cross-entropy loss calculation exclusively on the assistant's JSON output tokens.

### Optimized Inference Pipeline

```python
import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
from PIL import Image

# Model weights are merged into BF16 for inference stability
model_name = "solvrays/scribegene-llm-v1.2"
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_name, 
    torch_dtype=torch.bfloat16, 
    device_map="auto",
    attn_implementation="flash_attention_2"
)
processor = AutoProcessor.from_pretrained(model_name)

def extract_acord(image_path):
    image = Image.open(image_path).convert("RGB")
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image, "min_pixels": 256*28*28, "max_pixels": 1280*28*28},
                {"type": "text", "text": "Extract the structured field JSON for this ACORD form page."},
            ],
        }
    ]

    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    image_inputs, _ = process_vision_info(messages)
    inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)

    # Use greedy decoding for deterministic JSON output
    output_ids = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
    trimmed = output_ids[:, inputs["input_ids"].shape[1]:]
    return processor.batch_decode(trimmed, skip_special_tokens=True)[0]

print(extract_acord("sample_acord_125.png"))
```

## Key Advantages vs. LayoutLMv3
1. **Unified Architecture:** No need for external OCR (Tesseract/Textract) or bounding box pre-processing.
2. **Generative Flexibility:** Handles non-standard field layouts and handwritten text variations that often break token-classification models.
3. **Multi-Page Context:** The native image-text interleaving allows for potential extension into multi-page document reasoning in a single context window.