zeromodels/detr-resnet-101-panoptic

See our collection for all versions of DETR.

Run DETR with Keras 3: JAX, PyTorch, or TensorFlow

kerasformers/detr-resnet-101-panoptic

Paper: End-to-End Object Detection with Transformers (arXiv:2005.12872) · HF Papers

DETR (DEtection TRansformer) treats object detection as direct set prediction. A ResNet backbone produces a feature map, a transformer encoder-decoder attends over it with a fixed set of learned object queries, and each query emits one class and one box. Training uses a bipartite (Hungarian) matching loss, so every ground-truth object is assigned exactly one query. That framing removes anchors and NMS. Panoptic checkpoints add a mask head for things and stuff.

For more details on the model, please go to Facebook's original model card.

Pure-Keras 3 conversion of facebook/detr-resnet-101-panoptic for kerasformers. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is a panoptic segmentation checkpoint (DETRPanopticSegment): each query predicts a class, box, and mask (COCO things + stuff).

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from kerasformers.models.detr import DETRPanopticSegment, DETRImageProcessor

model = DETRPanopticSegment.from_weights("kerasformers/detr-resnet-101-panoptic")
processor = DETRImageProcessor.from_weights("kerasformers/detr-resnet-101-panoptic")

image = Image.open("your_image.jpg").convert("RGB")
output = model(processor(image)["pixel_values"], training=False)
# output['logits'], output['pred_boxes'], output['pred_masks']

Load any DETR variant the same way with from_weights("kerasformers/<variant>") (use DETRPanopticSegment for this repo):

Variant	Hub	Task
detr-resnet-50	kerasformers/detr-resnet-50	object detection
detr-resnet-101	kerasformers/detr-resnet-101	object detection
detr-resnet-50-panoptic	kerasformers/detr-resnet-50-panoptic	panoptic segmentation
detr-resnet-101-panoptic	kerasformers/detr-resnet-101-panoptic	panoptic segmentation

Tips

• Set KERAS_BACKEND before importing Keras / kerasformers.
• Detection: DETRDetect + post_process_object_detection (try threshold=0.9 on clean COCO scenes).
• Panoptic: DETRPanopticSegment returns pred_masks per query; threshold at zero and resize to the image size yourself.
• See DETR docs and Loading Weights.
• Community / upstream safetensors still work via the hf: prefix, e.g. DETRPanopticSegment.from_weights("hf:facebook/detr-resnet-101-panoptic").

Special Thanks

A huge thank you to the Facebook AI Research DETR authors for creating and releasing these models.

License: Apache 2.0.