README.md

metadata

title: PerceptionDLM Region Captioning
emoji: 🎯
colorFrom: indigo
colorTo: yellow
sdk: gradio
sdk_version: 5.42.0
app_file: app.py
short_description: Parallel region captioning with multimodal diffusion LLM
python_version: '3.12'
startup_duration_timeout: 1h

PerceptionDLM Region Captioning

A Gradio demo for MSALab/PerceptionDLM, a 9.2B parameter multimodal diffusion language model for parallel region captioning.

How it works

Upload an image, then choose regions to caption. By default you generate masks with SAM 3 by clicking point(s) on the image; SAM 3 turns each click set into a binary mask. You can also switch to the secondary mode and upload pre-existing binary mask files. The model then generates descriptions for all provided regions simultaneously in a single denoising process — avoiding the linear latency growth of autoregressive region captioners.

The decoding animation replays each diffusion step so you can watch captions emerge token by token.

SAM 3 (facebook/sam3) is used only as a mask-generation tool; the region captioning itself is always done by PerceptionDLM.

Model details

• Base: LLaDA-8B (diffusion language model) + SigLIP2 vision encoder
• Precision: bfloat16
• Region prompts: up to 6 per image
• Default inference: 32 diffusion steps, generation length 32 per mask
• Paper: arXiv:2606.19534
• Code: GitHub