BiliSakura/CRM-transformers

CRM Transformers Model

Self-contained Hugging Face checkpoint converted from the official Continuous Refinement Model (CRM) weights (model_45705) in High Quality Segmentation for Ultra High-resolution Images.

The folder crm-resnet50/ ships remote code (modeling_crm.py, image processor, pipeline) plus model.safetensors / config.json and loads with trust_remote_code=True.

Original paper: High Quality Segmentation for Ultra High-resolution Images
Original code: Entity / High-Quality-Segmention
Converted for Hugging Face by: BiliSakura
License (weights / original code): CC BY-NC 4.0

What CRM does

CRM is a mask refinement model, not a from-scratch segmenter. It takes:

1. an RGB image
2. a coarse binary mask (from PSPNet, DeepLab, SAM, etc.)

and continuously aligns encoder features with the refinement target via an implicit MLP (LIIF-style local ensemble). Inference can stay at native resolution; the paper's multi-scale schedule is 0.125 → 0.25 → 0.5 → 1.0.

Checkpoint

Folder	Backbone	Stem	Decoder	Source
crm-resnet50	Dilated ResNet-50 + 3-level ASPP	4-channel (RGB + mask)	MLP [32, 32, 32, 32]	model_45705

Synchronized BatchNorm from the original training code is stored as ordinary nn.BatchNorm2d (equivalent at eval time). Layer-4 weights are kept for checkpoint compatibility but are unused by the ASPP decoder, matching the original forward.

Usage

from transformers import pipeline
from PIL import Image

MODEL = "/path/to/CRM-transformers/crm-resnet50"

pipe = pipeline(
    task="image-segmentation",
    model=MODEL,
    trust_remote_code=True,
)

image = Image.open("image.png").convert("RGB")
coarse_mask = Image.open("coarse_mask.png").convert("L")  # 0/255
segments = pipe(image, mask=coarse_mask)

# Native Hugging Face image-segmentation schema:
# [{"score": None, "label": "foreground", "mask": PIL.Image}, ...]
for item in segments:
    print(item["label"], item["mask"].size)

The task is also inferred if you omit it (this repo registers a single custom pipeline):

pipe = pipeline(model=MODEL, trust_remote_code=True)

Disable the paper's multi-scale schedule for a single full-resolution pass:

segments = pipe(image, mask=coarse_mask, do_multi_scale=False)

Load components directly:

from transformers import AutoImageProcessor, AutoModelForSemanticSegmentation

processor = AutoImageProcessor.from_pretrained(MODEL, trust_remote_code=True)
model = AutoModelForSemanticSegmentation.from_pretrained(MODEL, trust_remote_code=True)

inputs = processor(images=image, segmentation_maps=coarse_mask, return_tensors="pt")
outputs = model(
    pixel_values=inputs["pixel_values"],
    coarse_masks=inputs["coarse_masks"],
)
# logits: (1, 2, H', W')  — argmax is equivalent to sigmoid(raw) > 0.5
maps = processor.post_process_semantic_segmentation(
    outputs, target_sizes=inputs["original_sizes"]
)

Preprocessing

preprocessor_config.json matches the original evaluation loader:

Input	Transform
RGB	/255, ImageNet mean/std [0.485, 0.456, 0.406] / [0.229, 0.224, 0.225]
Coarse mask	/255, mean/std 0.5 → [-1, 1]
Both	reflect pad 32 px (cropped back in post-process)
Resize	off by default (do_resize: false) so native (H, W) is preserved

Citation

@inproceedings{shen2022high,
  title={High Quality Segmentation for Ultra High-resolution Images},
  author={Shen, Tiancheng and Zhang, Yuechen and Qi, Lu and Kuen, Jason and Xie, Xingyu and Wu, Jianlong and Lin, Zhe and Jia, Jiaya},
  booktitle={CVPR},
  year={2022}
}