---
language:
- en
license: apache-2.0
pipeline_tag: object-detection
tags:
- OpenRAL
- rskill
- detector
- object-detection
- any
- open-vocabulary
- zero-shot
- omdet-turbo
- on-demand
- locate-in-view
inference: false
base_model:
- omlab/omdet-turbo-swin-tiny-hf
---

# rskill-omdet-turbo-locator

> **OpenRAL rSkill** — OmDet-Turbo (Swin-tiny) packaged as an Apache-2.0,
> **on-demand** open-vocabulary locator (`mode: on_demand`, ADR-0051). The
> reasoner invokes it via the read-only `locate_in_view` tool — "is object X in
> view right now?" — when it needs a specific object the continuous detector
> bank does not cover. A lightweight, real-time, in-process alternative to the
> 3B NVIDIA LocateAnything VLM for simple "find X" queries. **No actuators.**

This package wraps `hf://omlab/omdet-turbo-swin-tiny-hf` with a `rskill.yaml`
manifest. It does **not** copy model weights — they are the same Apache-2.0
checkpoint as its continuous sibling [`omdet-turbo-indoor`](../omdet-turbo-indoor/).

## What this skill does

Answers on-demand open-vocabulary localization queries from the reasoner: given
a free-text object (e.g. `"the red stapler"`), it runs one detection pass on the
current frame and reports whether that object is visible and where. It is **not**
a continuous background producer — it does not stream into world state every
frame; it responds when prompted (the `locate_in_view` service / the
`detector_query` topic). It emits no action chunks and drives no actuators.

| Field | Value |
| --- | --- |
| Actions | `detect` |
| Objects | open-vocabulary queried object (any free-text class the reasoner asks for) |
| Scenes  | tabletop, kitchen, indoor, household, office |
| Embodiment | embodiment-agnostic (any RGB camera ≥ 640×480) |

## How it works

OmDet-Turbo is a real-time `transformers` open-vocabulary detector
(`AutoModelForZeroShotObjectDetection`), run **in-process** by the
[`OmDetTurboDetector`](../../python/runner/src/openral_runner/backends/gstreamer/omdet_turbo_detector.py)
backend (`DetectorTier.ZEROSHOT_HF`). The same backend serves both detector
modes; this rSkill selects `mode: on_demand`, so the detector node exposes the
`locate_in_view` service and the `detector_query` retarget topic:

- `detect_with_query(frame, …, query)` — one-shot detection for a reasoner query
  without disturbing any persistent vocabulary (the `locate_in_view` path).
- `set_query(text)` — persistently retarget the query (the `detector_query` topic).

The free-text query is parsed into OmDet's multi-label class list by
`query_to_classes` (comma / `</c>` separated; a single phrase is one class).
`labels` in the manifest is only the static default used when no query is
supplied.

### Observation → action contract

| Direction | Key | Shape | Notes |
| --- | --- | --- | --- |
| in | any RGB camera | `(H, W, 3)` BGR `uint8` | latest frame cached per camera for the service; min 640×480 |
| in | query | text | object/description from the reasoner's `locate_in_view` call |
| out | `ObjectsMetadata` | list of `ObjectDetection2D` | `(label, confidence, bbox_xyxy)`; no action chunk |

## Upstream model and training

A thin wrapper around the upstream Apache-2.0 OmDet-Turbo checkpoint; weights
live upstream and are not copied here.

| Field | Value |
| --- | --- |
| Source repo | [`omlab/omdet-turbo-swin-tiny-hf`](https://huggingface.co/omlab/omdet-turbo-swin-tiny-hf) |
| Base model  | OmDet-Turbo, Swin-tiny backbone |
| Paper       | [arxiv:2403.06892](https://arxiv.org/abs/2403.06892) — *Real-time Transformer-based Open-Vocabulary Detection with Efficient Fusion Head* |
| License     | apache-2.0 (commercial use permitted) |
| Parameters  | ~115 M |
| Training data | upstream: Objects365 / GoldG and grounding data per the OmDet-Turbo release |

## Supported robots

Embodiment-agnostic — the only requirement is an RGB camera stream. All in-tree
embodiment tags are declared in `rskill.yaml`.

| Robot | Embodiment tag | Status | Notes |
| --- | --- | --- | --- |
| any with an RGB camera | `franka_panda`, `so100_follower`, `aloha`, … | ⚡ experimental | camera-only |

## Sensors required

Mirrors `rskill.yaml::sensors_required`.

| Key | Modality | Min resolution | Format |
| --- | --- | --- | --- |
| any RGB camera | RGB | 640 × 480 | `uint8` BGR frame |

## Manifest summary

| Field | Value |
| --- | --- |
| `name` | `OpenRAL/rskill-omdet-turbo-locator` |
| `version` | `0.1.0` |
| `license` | `apache-2.0` |
| `role` / `kind` | `s1` / `detector` |
| `runtime` / `quantization.dtype` | `pytorch` / `fp16` |
| `detector.engine` / `detector.mode` | `zeroshot_hf` / `on_demand` |
| `weights_uri` | `hf://omlab/omdet-turbo-swin-tiny-hf` |
| `latency_budget.per_chunk_ms` | 200 ms |
| `commercial_use_allowed` | yes (Apache-2.0 weights) |

Full schema: [`openral_core.schemas.RSkillManifest`](../../python/core/src/openral_core/schemas.py).

## Quick start

```bash
uv sync --group omdet   # torch + transformers for the in-process backend
```

```python
from openral_core.schemas import RSkillManifest, DetectorMode

manifest = RSkillManifest.from_yaml("rskills/omdet-turbo-locator/rskill.yaml")
assert manifest.detector.mode is DetectorMode.ON_DEMAND
```

## Reproduction

Packaging-only wrapper — no trained numbers to reproduce. Validate the wiring
(manifest + backend query path) without a GPU:

```bash
just bootstrap && uv sync --all-packages
uv run pytest tests/unit/test_omdet_turbo_detector.py
```

## Evaluation

No benchmarks shipped — packaging-only wrapper; see CLAUDE.md §6.4.

## License

This rSkill package (`rskill.yaml`, `README.md`) is **apache-2.0**. The wrapped
weights at `hf://omlab/omdet-turbo-swin-tiny-hf` are also **apache-2.0**, so the
locator is fully commercial-safe (CLAUDE.md §1.9).

## See also

- [`rskills/omdet-turbo-indoor/`](../omdet-turbo-indoor/) — the continuous
  background sibling (same weights, `mode: continuous`, fixed 266-class vocab).
- [`rskills/locateanything-3b-nf4/`](../locateanything-3b-nf4/) — higher-quality
  3B open-vocab locator (NVIDIA non-commercial; `VLM_SIDECAR` tier).
- [`docs/adr/0051-detector-invocation-mode.md`](../../docs/adr/0051-detector-invocation-mode.md) — continuous vs on-demand detector mode.
- [CLAUDE.md §6.4](../../CLAUDE.md) — rSkill packaging contract.