README.md

---
license: cc-by-nc-4.0
language:
- en
tags:
- image-segmentation
- mask-refinement
- ultra-high-resolution
- crm
- resnet
- transformers
library_name: transformers
pipeline_tag: image-segmentation
---

# CRM Transformers Model

Self-contained Hugging Face checkpoint converted from the official Continuous Refinement Model (CRM) weights (`model_45705`) in [High Quality Segmentation for Ultra High-resolution Images](https://arxiv.org/abs/2111.14482).

The folder `crm-resnet50/` ships remote code (`modeling_crm.py`, image processor, pipeline) plus `model.safetensors` / `config.json` and loads with `trust_remote_code=True`.

**Original paper:** [High Quality Segmentation for Ultra High-resolution Images](https://arxiv.org/abs/2111.14482)  
**Original code:** [Entity / High-Quality-Segmention](https://github.com/dvlab-research/Entity)  
**Converted for Hugging Face by:** BiliSakura  
**License (weights / original code):** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/)

## What CRM does

CRM is a **mask refinement** model, not a from-scratch segmenter. It takes:

1. an RGB image
2. a coarse binary mask (from PSPNet, DeepLab, SAM, etc.)

and continuously aligns encoder features with the refinement target via an implicit MLP (LIIF-style local ensemble). Inference can stay at native resolution; the paper's multi-scale schedule is `0.125 → 0.25 → 0.5 → 1.0`.

## Checkpoint

| Folder | Backbone | Stem | Decoder | Source |
|--------|----------|------|---------|--------|
| `crm-resnet50` | Dilated ResNet-50 + 3-level ASPP | 4-channel (RGB + mask) | MLP `[32, 32, 32, 32]` | `model_45705` |

Synchronized BatchNorm from the original training code is stored as ordinary `nn.BatchNorm2d` (equivalent at eval time). Layer-4 weights are kept for checkpoint compatibility but are unused by the ASPP decoder, matching the original forward.

## Usage

```python
from transformers import pipeline
from PIL import Image

MODEL = "/path/to/CRM-transformers/crm-resnet50"

pipe = pipeline(
    task="image-segmentation",
    model=MODEL,
    trust_remote_code=True,
)

image = Image.open("image.png").convert("RGB")
coarse_mask = Image.open("coarse_mask.png").convert("L")  # 0/255
segments = pipe(image, mask=coarse_mask)

# Native Hugging Face image-segmentation schema:
# [{"score": None, "label": "foreground", "mask": PIL.Image}, ...]
for item in segments:
    print(item["label"], item["mask"].size)
```

The task is also inferred if you omit it (this repo registers a single custom pipeline):

```python
pipe = pipeline(model=MODEL, trust_remote_code=True)
```

Disable the paper's multi-scale schedule for a single full-resolution pass:

```python
segments = pipe(image, mask=coarse_mask, do_multi_scale=False)
```

Load components directly:

```python
from transformers import AutoImageProcessor, AutoModelForSemanticSegmentation

processor = AutoImageProcessor.from_pretrained(MODEL, trust_remote_code=True)
model = AutoModelForSemanticSegmentation.from_pretrained(MODEL, trust_remote_code=True)

inputs = processor(images=image, segmentation_maps=coarse_mask, return_tensors="pt")
outputs = model(
    pixel_values=inputs["pixel_values"],
    coarse_masks=inputs["coarse_masks"],
)
# logits: (1, 2, H', W')  — argmax is equivalent to sigmoid(raw) > 0.5
maps = processor.post_process_semantic_segmentation(
    outputs, target_sizes=inputs["original_sizes"]
)
```

## Preprocessing

`preprocessor_config.json` matches the original evaluation loader:

| Input | Transform |
|-------|-----------|
| RGB | `/255`, ImageNet mean/std `[0.485, 0.456, 0.406]` / `[0.229, 0.224, 0.225]` |
| Coarse mask | `/255`, mean/std `0.5` → `[-1, 1]` |
| Both | reflect pad 32 px (cropped back in post-process) |
| Resize | off by default (`do_resize: false`) so native `(H, W)` is preserved |


## Citation

```bibtex
@inproceedings{shen2022high,
  title={High Quality Segmentation for Ultra High-resolution Images},
  author={Shen, Tiancheng and Zhang, Yuechen and Qi, Lu and Kuen, Jason and Xie, Xingyu and Wu, Jianlong and Lin, Zhe and Jia, Jiaya},
  booktitle={CVPR},
  year={2022}
}
```