PanoVLD-500M

Typed, calibrated decisions about an image plus optional text, in one forward pass with no text generation. Give it one or more pictures, some context and a set of questions; it answers each one as a multiple choice (choice), a yes / no probability (noul) or a level on a rubric you write (score).

The question format follows Laya Vision (thaitea/laya-vision). The backbone is PanoVLM: a FastViT-HD vision encoder and a PanoLM linear-attention language model.

How it works

482.5M parameters: FastViT-HD vision encoder 122.7M, linear projector 3.1M, PanoLM language model 355.6M, decision head 1.1M.

• Backbone: PanoVLM without its language-model output layer: it returns the final hidden states. PanoLM has 24 layers: 20 linear-attention (KDA) and 4 full-attention (MLA).

• Input: one row per question, built by the processor:

  <s> ...chat template... USER: <image tokens> {"context": ...}
  <type> question: <instructions> ASSISTANT: Options:
   - <option 0>
   - <option 1>
   ...
  

  Images are pad-resized to 768 x 768 pixels, which gives 144 image tokens each. A row holds at most 1,024 tokens, and each option at most 48.

• Decision head: a question-type embedding added to every hidden state, then a small scorer that turns the hidden state at the newline closing each option into one logit. The number of options is not fixed.

• Calibration: one temperature per question type divides the logits before the softmax (choice 2.27, score 1.02, noul 1.83). It never changes the answer, only its probability.

• Option order: the language model reads left to right, so an option's logit has seen the image, the question and that option, but not the options listed after it.

Usage

import requests
from PIL import Image
from transformers import AutoModel, AutoProcessor

repo_id = "PanocularAI/PanoVLD-500M"
model = AutoModel.from_pretrained(
    repo_id,
    trust_remote_code=True,
).cuda()  # fla's kernels run on CUDA tensors only.
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)

url = "https://llava-vl.github.io/static/images/view.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# A state is an image plus optional text; each question is typed.
answers = model.predict(
    processor,
    {"image": image, "context": "Photo taken by a hiker."},
    {
        "boat": {"type": "noul", "instructions": "Is there a boat in the image?"},
        "place": {
            "type": "choice",
            "instructions": "Where was this photo taken?",
            "criteria": ["a lake", "a city street", "a desert", "an office"],
        },
        "weather": {
            "type": "score",
            "instructions": "How good is the weather?",
            "criteria": ["stormy", "overcast", "partly cloudy", "clear and sunny"],
        },
    },
)
print(answers["boat"]["noul"])  # P(yes)
print(answers["place"]["choice"], answers["place"]["probabilities"])
print(answers["weather"]["score"])  # expected level, 0 to 3

# The same in three steps: one row per question, one forward pass, answers.
questions = {"boat": {"type": "noul", "instructions": "Is there a boat in the image?"}}
inputs = processor(state=image, questions=questions, return_tensors="pt")
out = model(**inputs.to(model.device))
print(processor.post_process(out.probabilities.cpu(), questions))

predict returns, per question:

type	fields
choice	choice (the most probable option), probabilities, confidence
score	score (the expected level), probabilities, legend, confidence
noul	noul (the probability of yes), confidence

confidence is 1 minus the normalized entropy of the probabilities (for noul, the larger of the two probabilities), as in Laya Vision. predict_probabilities returns the calibrated probabilities alone, one list per question in option order.

Loading prints a report listing the vision encoder's BatchNorm num_batches_tracked counters as missing: they are step counters, unused at inference; every weight is in the checkpoint.

Fast inference (CUDA graphs)

The forward pass of one question launches about 1,500 small GPU kernels, so in PyTorch eager mode the GPU mostly waits for Python. enable_fast_inference keeps the same weights and runs predict through CUDA graphs:

model = AutoModel.from_pretrained(
    repo_id, trust_remote_code=True, dtype=torch.bfloat16
).cuda()
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
model.enable_fast_inference(processor)
answers = model.predict(processor, state, questions)  # as above

• The images are resized on the GPU, bit-exactly as the processor does on the CPU; the vision encoder is compiled (torch.compile).
• One CUDA graph is captured per input shape: number of images, row length (in steps of 32 tokens), number of questions and of options (rounded up to a power of two). The first call of each shape is slow (the very first one also compiles the vision encoder); later calls replay the graph.
• With several questions about the same state, the part of the row they share (chat template and images) runs once and its language-model state is reused for every question.
• The model must be in bfloat16 on a CUDA GPU. States without an image run eagerly.

Latency from a decoded image to the answers, the median of 30 calls after warm-up, on one A100 80GB:

	1 question	15 questions, one call	15 questions, one call each
eager (bf16)	94 ms	289 ms	1414 ms
enable_fast_inference	18 ms	41 ms	276 ms

Questions asked one at a time through the fast path get the same answer as in our reference evaluation (float32 weights, bf16 autocast) for 99.0% (75,981 of 76,764) of our evaluation questions, the sets below included; the probabilities differ by bf16 rounding.

Requirements

Tested with:

torch 2.15 (nightly, CUDA 13.2)
transformers==5.8.1
flash-linear-attention==0.5.2
timm==1.0.25

Results

Seven benchmarks: Laya Vision's three sets (validation splits) and Visual-Jev's four, built with its released scripts (Visual-Jev). Every option order as written.

set	questions	accuracy
A-OKVQA val	1,138	68.1%
ScienceQA val (image subset)	1,546	66.0%
VQAv2 yes/no val	5,000	75.8%
GQA-Choice	4,402	82.5%
SNLI-VE	1,368	76.8%
TextVQA-Choice	1,742	91.7%
TallyQA-Choice	338	46.2%
mean of the seven		72.4%

ScienceQA leaves out 551 of its 2,097 image questions, which also appear, with the same image, question and answer, in the ScienceQA subset of the Cauldron.

Rating sets (score questions), on each set's test questions; accuracy is the exact level:

set	questions	accuracy	ECE (calibrated)	NLL (calibrated)
VLFeedback	2,000	54.3%	0.035	1.15
RichHF-18K	1,990	55.5%	0.071	0.98
AVA	1,000	84.7%	0.323	0.76
CrisisMMD	529	73.7%	0.110	0.68