KielLens-ocr

KielLens-ocr is a fine-tuned Vision-Language Model (VLM) designed for high-precision document parsing, multi-lingual OCR, table extraction, and complex visual reasoning. It is built by fine-tuning the powerhouse vision architecture Qwen/Qwen2.5-VL-3B-Instruct using low-rank adapters (LoRA) via Hugging Face's TRL framework.

Model Details

• Developed by: kielTech
• Model Type: Vision-Language Model (OCR & Document AI)
• Base Model: Qwen/Qwen2.5-VL-3B-Instruct
• Language(s): Multi-lingual (English, supported Qwen languages)
• License: Apache 2.0
• Fine-tuning Method: Parameter-Efficient Fine-Tuning (PEFT) / LoRA

Intended Uses & Limitations

Intended Uses

• Automated text extraction from structured and unstructured documents (PDF screenshots, receipts, forms).
• Handling and reading messy or handwritten text.
• Table understanding and structural key-value pair extractions.

Limitations

• The model inherits the native constraints of the Qwen2.5-VL architecture.
• Performance on highly dense technical schematics or low-resolution text depends heavily on the input resolution configured during inference.

Training Hyperparameters (TRL Configuration)

The following configurations were leveraged during the parameter-efficient SFT training phase:

• Quantization: 4-bit NormalFloat (NF4) with double quantization active.
• Learning Rate: 2e-4
• Optimizer: paged_adamw_32bit
• Target Modules: Vision attention layers (q_proj, v_proj)
• Precision: Mixed precision (fp16/bf16 depending on hardware fallback)

How to Load and Use

You can load the adapters directly on top of the base model using the transformers library:

import torch
from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "Qwen/Qwen2.5-VL-3B-Instruct"
adapter_id = "kiel2/KielLens-ocr"

# Load the base model with the custom fine-tuned adapters
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=None, # Update with BitsAndBytesConfig if running on limited VRAM
    device_map="auto",
    torch_dtype=torch.float16
)
model.load_adapter(adapter_id)

processor = AutoProcessor.from_pretrained(model_id)