solvrays/scribegene-llm-v1.2

Qwen2-VL-7B: Specialized ACORD Insurance Form Extractor

Engineering Overview

This model is a state-of-the-art Vision-Language Intelligence (V-LIE) engine, fine-tuned from Qwen2-VL-7B-Instruct. It is architected to eliminate traditional multi-stage OCR pipelines by mapping pixels directly to structured semantic JSON for complex, multi-column insurance documents.

Technical Specifications

• Base Architecture: Qwen2-VL (ViT + Qwen2 LLM) utilizing Multi-Modal Rotary Positional Embeddings (M-RoPE) for spatial reasoning.
• Fine-tuning Method: Parameter-Efficient Fine-Tuning (PEFT) via QLoRA (4-bit NormalFloat quantization).
• Adaptation Strategy: LoRA target modules included q_proj, k_proj, v_proj, o_proj, and MLP layers (gate_proj, up_proj, down_proj). The vision encoder remained frozen to preserve generalized OCR capabilities while the language head was adapted for schema-strict JSON generation.
• Resolution Handling: Supports dynamic resolution (256-1280 pixels) to maintain aspect ratio integrity, crucial for identifying dense ACORD form checkboxes and fine-print labels.
• Training Objective: Supervised Fine-Tuning (SFT) using a custom data collator to mask prompt/image tokens, focusing the cross-entropy loss calculation exclusively on the assistant's JSON output tokens.

Optimized Inference Pipeline

import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
from PIL import Image

# Model weights are merged into BF16 for inference stability
model_name = "solvrays/scribegene-llm-v1.2"
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_name, 
    torch_dtype=torch.bfloat16, 
    device_map="auto",
    attn_implementation="flash_attention_2"
)
processor = AutoProcessor.from_pretrained(model_name)

def extract_acord(image_path):
    image = Image.open(image_path).convert("RGB")
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image, "min_pixels": 256*28*28, "max_pixels": 1280*28*28},
                {"type": "text", "text": "Extract the structured field JSON for this ACORD form page."},
            ],
        }
    ]

    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    image_inputs, _ = process_vision_info(messages)
    inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)

    # Use greedy decoding for deterministic JSON output
    output_ids = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
    trimmed = output_ids[:, inputs["input_ids"].shape[1]:]
    return processor.batch_decode(trimmed, skip_special_tokens=True)[0]

print(extract_acord("sample_acord_125.png"))

Key Advantages vs. LayoutLMv3

1. Unified Architecture: No need for external OCR (Tesseract/Textract) or bounding box pre-processing.
2. Generative Flexibility: Handles non-standard field layouts and handwritten text variations that often break token-classification models.
3. Multi-Page Context: The native image-text interleaving allows for potential extension into multi-page document reasoning in a single context window.