README.md

---
pipeline_tag: zero-shot-object-detection
license: apache-2.0
base_model: IDEA-Research/grounding-dino-tiny
library_name: kerasformers
tags:
- keras
- kerasformers
- grounding-dino
- zero-shot-object-detection
- arxiv:2303.05499
- pytorch
- jax
- tf
---

# Run Grounding DINO with Keras 3: JAX, PyTorch, or TensorFlow

[![GitHub](https://img.shields.io/badge/GitHub-KerasFormers-black?logo=github)](https://github.com/IMvision12/KerasFormers) [![Docs](https://img.shields.io/badge/Docs-Grounding%20DINO-blue)](https://imvision12.github.io/KerasFormers/grounding_dino/)

# kerasformers/grounding_dino_tiny

Paper: [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (arXiv:2303.05499)](https://arxiv.org/abs/2303.05499) · [HF Papers](https://huggingface.co/papers/2303.05499)

Grounding DINO performs **open-set, text-grounded** object detection: it finds the objects a free-form text prompt names, not a fixed label set. A Swin image backbone and a BERT text encoder feed a deformable cross-modality encoder that fuses vision and language, a contrastive query-selection stage picks object proposals, and a decoder with iterative box refinement emits one box per query scored against the prompt tokens. No anchors, no NMS, and categories that were never in a detection training set (here "Swin-Tiny" backbone).

For more details on the model, please go to IDEA-Research's original [model card](https://huggingface.co/IDEA-Research/grounding-dino-tiny).

Pure-**Keras 3** conversion of [`IDEA-Research/grounding-dino-tiny`](https://huggingface.co/IDEA-Research/grounding-dino-tiny) for [kerasformers](https://github.com/IMvision12/KerasFormers). One implementation runs unmodified on **TensorFlow / Torch / JAX**.

This is an **open-set object detection** checkpoint (`GroundingDinoForObjectDetection`, Swin-Tiny backbone): each query predicts a box and a score over the prompt tokens.

## ✨ Quick start

```python
import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

import torch
from PIL import Image
from kerasformers.models.grounding_dino import (
    GroundingDinoForObjectDetection,
    GroundingDinoProcessor,
)

model = GroundingDinoForObjectDetection.from_weights("kerasformers/grounding_dino_tiny")
processor = GroundingDinoProcessor.from_weights("kerasformers/grounding_dino_tiny")

image = Image.open("your_image.jpg").convert("RGB")
# Prompts are free text; pass a list of candidates (or one "a. b. c." string). Skip
# articles: in "a paddle" the "a" can outscore the noun.
inputs = processor(images=image, text=["person", "paddle", "board"])

with torch.no_grad():  # torch backend: avoids a large autograd graph (can OOM otherwise)
    output = model(inputs)
# output["logits"]:     (1, 900, 256)
# output["pred_boxes"]: (1, 900, 4)

results = processor.post_process_object_detection(
    output,
    threshold=0.3,
    target_sizes=[(image.height, image.width)],
    input_ids=inputs["input_ids"],
)[0]
for score, name, box in sorted(
    zip(results["scores"], results["text_labels"], results["boxes"]),
    key=lambda d: -float(d[0]),
):
    print(f"{name}: {float(score):.3f} {[round(float(v)) for v in box]}")
```

Load either Grounding DINO variant the same way with `from_weights("kerasformers/<variant>")`:

| Variant | Hub | Backbone |
|---|---|---|
| `grounding_dino_tiny` | [`kerasformers/grounding_dino_tiny`](https://huggingface.co/kerasformers/grounding_dino_tiny) | Swin-Tiny |
| `grounding_dino_base` | [`kerasformers/grounding_dino_base`](https://huggingface.co/kerasformers/grounding_dino_base) | Swin-Base |

## Tips

- Set `KERAS_BACKEND` **before** importing Keras / kerasformers.
- On the **torch** backend, wrap inference in `with torch.no_grad():` — the forward keeps a large autograd graph otherwise and can OOM. The JAX / TensorFlow backends need no such wrap.
- Write prompts as lower-case phrases separated as a list or by `.`; **drop articles** ("a", "the") so the noun scores highest. `post_process_object_detection` needs `input_ids=` to map scores back to prompt words (`text_labels`).
- `threshold=0.3` is a reasonable start; raise it for cleaner scenes.
- See [Grounding DINO docs](https://imvision12.github.io/KerasFormers/grounding_dino/) and [Loading Weights](https://imvision12.github.io/KerasFormers/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `GroundingDinoForObjectDetection.from_weights("hf:IDEA-Research/grounding-dino-tiny")`.

## Special Thanks

A huge thank you to the IDEA-Research authors for creating and releasing Grounding DINO.

License: Apache 2.0.