README.md

---
library_name: transformers
license: apache-2.0
pipeline_tag: image-segmentation
base_model:
- Qwen/Qwen3.5-0.8B
tags:
- vision-language-grounding
- concept-correspondence
- referring-expression-segmentation
- phrase-grounding
- open-vocabulary-segmentation
---

# ConCor-1

**Vision-Language Grounding as Bidirectional Concept Correspondence**
Jieyu Zhang\*, Ziqi Gao\*, Luke Zettlemoyer, Ranjay Krishna
(\*equal contribution) — University of Washington, Allen Institute for AI, Microsoft SuperIntelligence, FAIR at Meta

ConCor-1 recovers **all** correspondences between visually referential text spans and
instance-level image segments in an image–text pair. Unlike conventional grounding, the
text spans are **not** given as queries: the model decides which parts of the text are
visually referential, groups co-referring mentions, segments the corresponding objects,
and decides how many correspondences exist.

This single formulation covers phrase grounding, referring-expression grounding and
open-vocabulary detection/segmentation — a caption, a referring expression and a
category list are all just "the text" for ConCor-1.

## How it works

ConCor-1 uses the pretrained **Qwen3.5-0.8B** vision-language model as a *contextual
image–text encoder* rather than as a generator, and appends **385 learnable bridge
tokens** to the multimodal sequence:

```
[<vision_start>, <image_pad> × N_v, <vision_end>, text tokens × N_t, bridge tokens × 385]
```

Each bridge token represents one candidate image–text correspondence. Three lightweight
heads read its final hidden state:

| Head | Prediction | Implementation |
|---|---|---|
| `presence_head` | is this a valid correspondence? | SwiGLU MLP → scalar logit |
| `text_segmentation_head` | binary mask over the input text tokens | bridge/text features → 256-d correspondence space → bilinear scorer |
| `vision_segmentation_head` | binary mask over the image | patch expander (4 MLPs, undoing 2×2 patch merging) + pre-merger ViT feature fusion + 2-block convolutional decoder → 4-px cell grid → bilinear scorer |

Bridge tokens are organised into **multi-scale spatial groups** — grids from 1×1 to
10×10, so `Q = Σ s² = 385` — which gives them location and scale priors (coarse levels
for large instances, fine levels for small ones). The backbone's *full-attention* layers
run with a **bidirectional** mask so bridge tokens can access the complete multimodal
context, while the linear-attention layers keep Qwen3.5's pretrained behaviour. The
model is therefore **not autoregressive**: `use_cache`, `past_key_values` and
`generate()` are not supported.

Parameters (Table 4 of the paper):

| Component | Params |
|---|---|
| Qwen3.5-0.8B backbone | 852.99 M |
| Presence head | 0.52 M |
| Text segmentation head | 1.11 M |
| Vision segmentation head | 12.01 M |
| Bridge-token embeddings | 0.145 M |
| **ConCor-1 total** | **866.79 M** (+1.62 %) |

## Usage

The example below is a real validation sample, bundled as `example.png`: image
`000000000285` from COCO `val2017` with its COCONut-PanCap caption.

```python
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

model_id = "UWGZQ/ConCor-1"  # or a local path to this repository

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # "sdpa" also works
).to("cuda").eval()

image = Image.open("example.png").convert("RGB")
text = (
    "This image depicts a close-up of a brown bear in a natural outdoor setting. "
    "The background consists of lush green grass. In the foreground, a large brown bear "
    "is positioned centrally."
)

inputs = processor(images=image, text=text, return_tensors="pt").to("cuda")
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    outputs = model(**inputs)

correspondences = processor.post_process_correspondences(
    outputs, text=text, target_sizes=[(image.height, image.width)]
)[0]

for correspondence in correspondences:
    print(
        f"{correspondence['presence_score']:.3f}",
        correspondence["text_phrases"],   # phrases of this correspondence's text mask
        correspondence["text_spans"],     # character spans into `text`
        correspondence["mask"].shape,     # bool array, (height, width)
    )
```

Output — two correspondences, which is exactly what this sample is annotated with:

```
1.000 ['a brown bear', 'a large brown bear']   [[33, 45], [140, 158]]   (640, 586)
0.999 ['green grass']                          [[108, 119]]             (640, 586)
```

Note the first line: the caption mentions the bear twice, and both mentions land in a
*single* text mask — which is why a text mask is a set of character spans rather than one
span. The ground-truth annotation for this sample groups the same two mentions
(`[[33, 45], [140, 158]]`) and pairs them with one bear mask.

`example_inference.py` wraps this up as a script, including a mask-overlay renderer:

```bash
python example_inference.py \
    --image example.png \
    --text "This image depicts a close-up of a brown bear in a natural outdoor setting. The background consists of lush green grass. In the foreground, a large brown bear is positioned centrally." \
    --output overlay.png
```

### Inference thresholds

Post-processing follows the paper: keep bridge tokens above a presence threshold,
threshold the text and image mask probabilities, then remove duplicate correspondences
with NMS (a candidate is suppressed only when *both* its image-mask IoU and its
text-span IoU with a higher-scoring candidate exceed the threshold).

| Threshold | Default | Argument |
|---|---|---|
| presence | 0.10 | `presence_threshold` |
| text mask | 0.45 | `text_threshold` |
| image mask | 0.45 | `image_threshold` |
| NMS IoU | 0.50 | `nms_iou_threshold` |

Image masks are produced by reshaping the per-bridge logits onto their 4-px cell grid,
bilinearly upsampling **in logit space** to `target_sizes`, and then applying the
sigmoid and threshold.

### Other input modes

```python
# A category list instead of a caption (open-vocabulary detection/segmentation).
inputs = processor(images=image, text="bear . grass . tree . person", return_tensors="pt")
# -> 0.995 ['bear'] [[0, 4]] ; 0.980 ['grass'] [[7, 12]]
#    the absent categories ("tree", "person") produce no correspondence

# A batch of image-text pairs (right-padded internally).
inputs = processor(images=[image_a, image_b], text=[text_a, text_b], return_tensors="pt")
results = processor.post_process_correspondences(
    outputs, text=[text_a, text_b],
    target_sizes=[(image_a.height, image_a.width), (image_b.height, image_b.width)],
)

# Text only: which spans are visually referential, and how do they group?
inputs = processor(images=None, text=text, return_tensors="pt")   # image_mask_logits is None
```

### Raw model outputs

`model(**inputs)` returns a `ConCor1Output`:

| Field | Shape | Meaning |
|---|---|---|
| `presence_logits` | `(B, 385)` | correspondence presence |
| `text_mask_logits` | `(B, 385, N_text)` | per-bridge mask over the text tokens |
| `image_mask_logits` | `(B, 385, N_cells)` | per-bridge mask over image cells, row-major |
| `image_mask_grid_hw` | `(B, 2)` | `(height, width)` of each sample's cell grid |

So `image_mask_logits[b, q, : h * w].reshape(h, w)` is bridge `q`'s mask map, where one
cell covers 4 px of the processed image.

## Training and evaluation data

Training converts grounding, segmentation and referring-expression datasets into a
unified correspondence format (image masks + text masks + correspondence labels), using
an LLM rewriting pipeline that groups co-referring mentions and drops mentions without a
visual referent: GoldG (Flickr30k + GQA, pseudo-masks from SAM3), COCONut-PanCap,
GroundedRef-train (LLM-written referring/compositional captions over COCO panoptic),
plus COCO, COCONut, LVIS, ADE20K instance segmentation and the counting subset of
PixMo-Point — 678 k caption-style images and 709 k instance-segmentation images.

Evaluation uses Flickr30k (4 k pairs), COCONut-PanCap (3 k) and GroundedRef (2 k),
scored along three axes — text segmentation, image segmentation and joint correspondence
precision/recall/F1. See the paper for the numbers, ablations and baselines.

This checkpoint is the released main model: 100 k steps, 385 bridge tokens, a fixed
~1.0 M-pixel image budget (`min_pixels = max_pixels = 1003520`, i.e. 1024 visual tokens).

## Files

| File | Purpose |
|---|---|
| `configuration_concor1.py` | `ConCor1Config` |
| `modeling_concor1.py` | `ConCor1ForConceptCorrespondence` and its heads |
| `processing_concor1.py` | `ConCor1Processor`: sequence construction + correspondence post-processing |
| `example_inference.py` | image + text → correspondences, with mask overlay |
| `example.png` | the example image: COCO `val2017/000000000285`, from the COCONut-PanCap validation split |
| `convert_original_checkpoint.py` | converts a research training checkpoint into this repository |
| `model.safetensors` | weights: bf16 backbone, fp32 prediction heads |

The prediction heads are stored and loaded in **fp32** (via
`_keep_in_fp32_modules_strict`) exactly as the training checkpoint holds them, while the
backbone is bf16. Every head casts its inputs to its own parameter dtype, so the model
runs with or without `torch.autocast`. With `attn_implementation="flash_attention_2"`
under `torch.autocast(bfloat16)`, this repository reproduces the research code's
presence, text-mask and image-mask logits **bit-for-bit** (verified single-sample and
batched). `sdpa` and running without autocast stay within bf16 kernel noise
(binary-mask IoU ≈ 0.98–0.995 against the exact path).

## Requirements

```
torch >= 2.8
transformers >= 5.3        # needs the Qwen3.5 architecture
flash-attn >= 2.8          # optional; use attn_implementation="sdpa" instead
fla / causal-conv1d        # Qwen3.5's gated-delta-net kernels
```

## Limitations

Mask quality remains behind specialized detection/segmentation models — the image-mask
head is deliberately lightweight and optimised for joint correspondence prediction
rather than segmentation quality alone. The model depends on correspondence-style
supervision that had to be converted from datasets not natively annotated that way,
which introduces noise. Grounding errors can propagate to downstream systems, and
performance may be uneven across categories, visual domains and languages, so validate
carefully before using it in human-facing or safety-critical settings.

## Demo

An interactive ZeroGPU demo lives at
[`UWGZQ/ConCor-1-demo`](https://huggingface.co/spaces/UWGZQ/ConCor-1-demo).

## Citation

```bibtex
@article{zhang2026concor,
  title   = {Vision-Language Grounding as Bidirectional Concept Correspondence},
  author  = {Zhang, Jieyu and Gao, Ziqi and Zettlemoyer, Luke and Krishna, Ranjay},
  year    = {2026}
}
```

The backbone is [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) (Apache 2.0).