README.md

---
title: PerceptionDLM Region Captioning
emoji: 🎯
colorFrom: indigo
colorTo: yellow
sdk: gradio
sdk_version: 5.42.0
app_file: app.py
short_description: Parallel region captioning with multimodal diffusion LLM
python_version: "3.12"
startup_duration_timeout: 1h
---

# PerceptionDLM Region Captioning

A Gradio demo for [MSALab/PerceptionDLM](https://huggingface.co/MSALab/PerceptionDLM), a 9.2B parameter multimodal diffusion language model for parallel region captioning.

## How it works

Upload an image, then choose regions to caption. By default you **generate masks with SAM 3** by clicking point(s) on the image; SAM 3 turns each click set into a binary mask. You can also switch to the secondary mode and **upload pre-existing binary mask files**. The model then generates descriptions for all provided regions **simultaneously** in a single denoising process — avoiding the linear latency growth of autoregressive region captioners.

The decoding animation replays each diffusion step so you can watch captions emerge token by token.

SAM 3 ([facebook/sam3](https://huggingface.co/facebook/sam3)) is used only as a mask-generation tool; the region captioning itself is always done by PerceptionDLM.

## Model details

- **Base:** LLaDA-8B (diffusion language model) + SigLIP2 vision encoder
- **Precision:** bfloat16
- **Region prompts:** up to 6 per image
- **Default inference:** 32 diffusion steps, generation length 32 per mask
- **Paper:** [arXiv:2606.19534](https://arxiv.org/abs/2606.19534)
- **Code:** [GitHub](https://github.com/MSALab-PKU/PerceptionDLM)