File size: 4,592 Bytes
1b447be | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
pipeline_tag: zero-shot-object-detection
license: apache-2.0
base_model: IDEA-Research/grounding-dino-tiny
library_name: zeromodels
tags:
- keras
- zeromodels
- grounding-dino
- zero-shot-object-detection
- arxiv:2303.05499
- pytorch
- jax
- tf
---
# Run Grounding DINO with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/grounding_dino/)
# zeromodels/grounding_dino_tiny
Paper: [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (arXiv:2303.05499)](https://arxiv.org/abs/2303.05499) · [HF Papers](https://huggingface.co/papers/2303.05499)
Grounding DINO performs **open-set, text-grounded** object detection: it finds the objects a free-form text prompt names, not a fixed label set. A Swin image backbone and a BERT text encoder feed a deformable cross-modality encoder that fuses vision and language, a contrastive query-selection stage picks object proposals, and a decoder with iterative box refinement emits one box per query scored against the prompt tokens. No anchors, no NMS, and categories that were never in a detection training set (here "Swin-Tiny" backbone).
For more details on the model, please go to IDEA-Research's original [model card](https://huggingface.co/IDEA-Research/grounding-dino-tiny).
Pure-**Keras 3** conversion of [`IDEA-Research/grounding-dino-tiny`](https://huggingface.co/IDEA-Research/grounding-dino-tiny) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
This is an **open-set object detection** checkpoint (`GroundingDinoForObjectDetection`, Swin-Tiny backbone): each query predicts a box and a score over the prompt tokens.
## ✨ Quick start
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
import torch
from PIL import Image
from zeromodels.models.grounding_dino import (
GroundingDinoForObjectDetection,
GroundingDinoProcessor,
)
model = GroundingDinoForObjectDetection.from_weights("zeromodels/grounding_dino_tiny")
processor = GroundingDinoProcessor.from_weights("zeromodels/grounding_dino_tiny")
image = Image.open("your_image.jpg").convert("RGB")
# Prompts are free text; pass a list of candidates (or one "a. b. c." string). Skip
# articles: in "a paddle" the "a" can outscore the noun.
inputs = processor(images=image, text=["person", "paddle", "board"])
with torch.no_grad(): # torch backend: avoids a large autograd graph (can OOM otherwise)
output = model(inputs)
# output["logits"]: (1, 900, 256)
# output["pred_boxes"]: (1, 900, 4)
results = processor.post_process_object_detection(
output,
threshold=0.3,
target_sizes=[(image.height, image.width)],
input_ids=inputs["input_ids"],
)[0]
for score, name, box in sorted(
zip(results["scores"], results["text_labels"], results["boxes"]),
key=lambda d: -float(d[0]),
):
print(f"{name}: {float(score):.3f} {[round(float(v)) for v in box]}")
```
Load either Grounding DINO variant the same way with `from_weights("zeromodels/<variant>")`:
| Variant | Hub | Backbone |
|---|---|---|
| `grounding_dino_tiny` | [`zeromodels/grounding_dino_tiny`](https://huggingface.co/zeromodels/grounding_dino_tiny) | Swin-Tiny |
| `grounding_dino_base` | [`zeromodels/grounding_dino_base`](https://huggingface.co/zeromodels/grounding_dino_base) | Swin-Base |
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- On the **torch** backend, wrap inference in `with torch.no_grad():` — the forward keeps a large autograd graph otherwise and can OOM. The JAX / TensorFlow backends need no such wrap.
- Write prompts as lower-case phrases separated as a list or by `.`; **drop articles** ("a", "the") so the noun scores highest. `post_process_object_detection` needs `input_ids=` to map scores back to prompt words (`text_labels`).
- `threshold=0.3` is a reasonable start; raise it for cleaner scenes.
- See [Grounding DINO docs](https://imvision12.github.io/ZeroModels/grounding_dino/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `GroundingDinoForObjectDetection.from_weights("hf:IDEA-Research/grounding-dino-tiny")`.
## Special Thanks
A huge thank you to the IDEA-Research authors for creating and releasing Grounding DINO.
License: Apache 2.0.
|