zeeshan141/doc-decision

Document decision model (run 2, step 2,806)

A fine-tune of Qwen/Qwen3-VL-4B-Instruct that picks one answer from a list of options about a document image. It was trained with LoRA (r=16, alpha=32) and then merged into the base model, so it loads as a complete model.

Benchmark results

Scored on the English records of SoMarkAI/doc-decision-bench (2,617 questions), using Docision's evaluator with --protocol none.

Model	Overall	Document decisions	Industry classification
This model	86.70%	83.82%	93.63%
SoMarkAI/Docision-1.0	94.15%	92.64%	97.79%
Qwen/Qwen3-VL-4B-Instruct	83.57%	78.68%	95.32%

This model doesn't beat Docision-1.0, and its classification score is below the base model's. The benchmark wasn't used for training, but it was used to choose between saved models, so these scores may be slightly optimistic. Chinese wasn't tested.

Training data

Generated multiple-choice questions built from DocVQA, ChartQA and RVL-CDIP. RVL-CDIP has known label noise of about 8%, and some ChartQA labels are wrong.

Inference

Needs a GPU and a version of transformers that supports Qwen3-VL. MODEL can also be a local folder.

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

MODEL = "zeeshan141/doc-decision"

processor = AutoProcessor.from_pretrained(MODEL)
# The same image size limits the benchmark scores were measured with.
processor.image_processor.size = {"shortest_edge": 3136, "longest_edge": 501760}
model = Qwen3VLForConditionalGeneration.from_pretrained(MODEL, dtype=torch.bfloat16, device_map="auto").eval()


def choose(image_path, question, options, context="Inspect the supplied media carefully."):
    """Returns the chosen option and each option's probability."""
    labels = [chr(ord("A") + i) for i in range(len(options))]
    listed = "".join(f"[{label}] {option}\n" for label, option in zip(labels, options))
    text = f"{context}\nQuestion: {question}\nChoose exactly one candidate and answer with its label.\nCandidates:\n{listed}"
    with Image.open(image_path) as page:
        image = page.convert("RGB")
    messages = [{"role": "user", "content": [{"type": "image", "image": image}, {"type": "text", "text": text}]}]
    prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) + "Answer:\n"
    inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
    with torch.no_grad():
        logits = model(**inputs).logits[0, -1]
    label_ids = [processor.tokenizer.encode(label, add_special_tokens=False)[0] for label in labels]
    probabilities = logits[label_ids].float().softmax(-1).tolist()
    best = max(range(len(options)), key=probabilities.__getitem__)
    return options[best], dict(zip(options, probabilities))


answer, probabilities = choose("invoice.png", "Is the invoice total above $500?", ["Yes", "No"])
print(answer, probabilities)

The model reads the answer from the probabilities of the option labels, so it doesn't need generate(). Options must be different from each other, and the list can hold up to 26 with this code.