PANORAMA-4B

PANORAMA is a model for panoptic grounded captioning: it writes a full-scene caption and grounds every mentioned entity to a pixel-level mask. Grounding is formulated as mask proposal selection: for each phrase the VLM produces a concept vector that conditions a promptable segmenter to generate candidate masks, and a learned scorer selects the relevant subset, so a phrase can ground to one region, several instances, or none.

This repository hosts the 4B variant, built on Qwen3-VL-4B-Instruct and SAM 3. It is a generalist model (see Training data). The package is self-contained: from_pretrained(..., trust_remote_code=True) needs no external code.

Quick start

pip install -r requirements.txt    # inference deps only; skip if you use the repo's environment

python predict_panocaps.py --model . --image examples/robot_table.png --out ./out   # caption + masks
python visualize.py       --image examples/robot_table.png --pred ./out             # overlay + per-phrase panels

For the full evaluation — PanoCaps and the other benchmarks reported in the paper (grounded conversation generation, referring expression segmentation, GroundingSuite) — use the code repository and its environment; the metric stack is not part of this package.

Or directly:

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor, AutoTokenizer

path = 'Panorama-grounding/PANORAMA-4B'
model = AutoModel.from_pretrained(path, dtype=torch.bfloat16,
                                  low_cpu_mem_usage=True, trust_remote_code=True).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(path, trust_remote_code=True)

image = Image.open('photo.jpg').convert('RGB')
with torch.no_grad():
    out = model.predict_forward(
        image=image,
        text='<image>\nWrite a caption covering all objects and background regions in the image, '
             'inserting the matching segmentation mask after each phrase.',
        tokenizer=tokenizer, processor=processor)
caption = out['prediction'].replace('<|im_end|>', '').strip()   # "... <p> red mug </p> [SEG] ..."
masks   = out['prediction_masks']   # one (1, H, W) bool mask per [SEG] (union); per-instance: see predict_panocaps.py

Use the prompt verbatim — the model was trained with a deliberate split between this phrasing (dense panoptic coverage) and "describe the image"-style prompts (sparser output), so paraphrasing quietly thins the coverage.

Output format

The caption marks every grounded phrase as <p> phrase </p> [SEG]; the i-th [SEG] maps to the i-th mask. A phrase may own several instance masks (masks_per_phrase in the predict_panocaps.py json gives the split), and background regions ("stuff") are grounded like objects.

Model family

Model	Base VLM	Proposal model
PANORAMA-2B	Qwen3-VL-2B-Instruct	SAM 3
PANORAMA-4B	Qwen3-VL-4B-Instruct	SAM 3

Training data

Trained on the 957K-sample data mixture described in the paper, including our released PanoCaps, COCONut-PanCap-Recaptioned and MRSeg-Referring-Expressions datasets.

License

For research and non-commercial use only. The package contains weights derived from SAM 3 (Meta, under its own license) and Qwen3-VL-4B-Instruct (Apache 2.0); their terms apply to the corresponding components.

Citation

@article{pieri2026panorama,
  title   = {PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection},
  author  = {Pieri, Sara and Kazakos, Evangelos and Chen, Shizhe and Sivic, Josef and Schmid, Cordelia},
  journal = {arXiv preprint arXiv:2609.19143},
  year    = {2026}
}