README.md

---
license: mit
license_link: LICENSE
library_name: openvino
pipeline_tag: image-classification
tags:
  - openvino
  - intel
  - person-detection
  - person-attributes
  - appearance-search
  - attribute-recognition
  - edge-ai
  - metro
  - dlstreamer
language:
  - en
---

# Appearance-Based Search

| Property | Value |
|---|---|
| **Category** | Person Detection + Appearance Attribute Search |
| **Base Model** | [person-detection-retail-0013](https://docs.openvino.ai/2024/omz_models_model_person_detection_retail_0013.html) + [person-attributes-recognition-crossroad-0230](https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/intel/person-attributes-recognition-crossroad-0230/README.md) (Open Model Zoo) |
| **Source Framework** | Caffe / PyTorch (Open Model Zoo) |
| **Supported Precisions** | FP32, FP16 |
| **Inference Engine** | OpenVINO |
| **Hardware** | CPU, GPU, NPU |
| **Detected Class(es)** | Persons (detection) + 8 appearance attributes (gender, bag, backpack, hat, sleeve length, trouser length, hair length, coat/jacket) |

---

## Overview

Appearance-Based Search is a Metro Analytics use case that finds every person
in a video whose visible appearance matches a described set of attributes.
Instead of comparing against a reference photo, the operator specifies *what a
person looks like* -- for example "a person with long hair" -- and the pipeline
highlights only the people who match that description.

It uses a two-stage pipeline:

- **person-detection-retail-0013** -- detects every person in the scene.
- **person-attributes-recognition-crossroad-0230** -- classifies each detected
  person with eight binary appearance attributes:
  `is_male`, `has_bag`, `has_backpack`, `has_hat`, `has_longsleeves`,
  `has_longpants`, `has_longhair`, and `has_coat_jacket`.

Each detected person is scored on all eight attributes, thresholded, and
compared to the search query. Only people who satisfy every requested attribute
are boxed, so the annotated output shows the search result directly.

The search query is expressed as a small dictionary. Set an attribute to `True`
to require it or `False` to require its absence, and omit the attributes you do
not care about:

```python
QUERY = {"has_longhair": True}                    # anyone with long hair
QUERY = {"is_male": True, "has_hat": True}         # men wearing a hat
QUERY = {"has_backpack": True}                     # anyone with a backpack
```

Typical Metro deployments include:

- **Suspect / Person-of-Interest Search** -- scan recorded footage for people
  matching a witness description ("man with a backpack and a hat").
- **Lost Property** -- locate the person who was carrying a particular bag.
- **Retail and Transit Analytics** -- count shoppers or passengers with
  specific appearance traits over time.
- **Operational Triage** -- narrow a large camera archive to a short list of
  candidate clips before manual review.

> **Privacy Note:** Appearance attribute recognition processes
> biometric-adjacent data.
> Ensure your deployment complies with applicable privacy regulations
> (GDPR, BIPA, etc.) and has proper consent and retention policies in place.

---

## Prerequisites

- Python 3.11+
- [Install OpenVINO](https://docs.openvino.ai/2026/get-started/install-openvino.html) (latest version)
- [Install Intel DLStreamer](https://docs.openedgeplatform.intel.com/2026.0/edge-ai-libraries/dlstreamer/get_started/install/install_guide_ubuntu.html) (latest version)

Create and activate a Python virtual environment before running the scripts:

```bash
python3 -m venv .venv --system-site-packages
source .venv/bin/activate
```

> **Note:** The `--system-site-packages` flag is required so the virtual
> environment can access the system-installed OpenVINO and DLStreamer Python
> packages.

---

## Getting Started

### Download Models

Run the provided script to download the person detection and person attributes
models from the Open Model Zoo:

```bash
chmod +x export_and_quantize.sh
./export_and_quantize.sh
```

The script downloads `person-detection-retail-0013` and
`person-attributes-recognition-crossroad-0230` in FP16 and downloads the sample
retail-aisle video (`test_video.mp4`) that the samples search.

### OpenVINO Sample

The sample below searches the video for people whose appearance matches the
`QUERY`. In each sampled frame it detects every person, classifies the eight
appearance attributes for each one, and keeps only the people who satisfy the
query. It saves the frame containing the most matching people, drawing a green
box on every matched person.
Change the `device` string to run on CPU, GPU, or NPU.

```python
import cv2
import numpy as np
import openvino as ov

DETECTION_MODEL = "intel/person-detection-retail-0013/FP16/person-detection-retail-0013.xml"
ATTRIBUTES_MODEL = "intel/person-attributes-recognition-crossroad-0230/FP16/person-attributes-recognition-crossroad-0230.xml"
SCENE_VIDEO = "test_video.mp4"

# person-attributes-recognition-crossroad-0230 emits eight binary appearance
# attributes (output layer "453"), in this order:
ATTRIBUTE_NAMES = [
    "is_male", "has_bag", "has_backpack", "has_hat",
    "has_longsleeves", "has_longpants", "has_longhair", "has_coat_jacket",
]

# The appearance being searched for. Set an attribute to True to require it or
# False to require its absence; omit attributes you do not care about.
QUERY = {"has_longhair": True}

CONF_THRESHOLD = 0.6      # person-detection confidence
ATTR_THRESHOLD = 0.5      # attribute presence threshold

core = ov.Core()

# Change device to "GPU" or "NPU" to run on integrated GPU or NPU.
det_compiled = core.compile_model(core.read_model(DETECTION_MODEL), "CPU")
attr_compiled = core.compile_model(core.read_model(ATTRIBUTES_MODEL), "CPU")

det_input = det_compiled.input(0)
det_h, det_w = det_input.shape[2], det_input.shape[3]
attr_input = attr_compiled.input(0)
attr_h, attr_w = attr_input.shape[2], attr_input.shape[3]
attr_output = attr_compiled.output("453")


def detect_persons(img):
    h0, w0 = img.shape[:2]
    blob = cv2.resize(img, (det_w, det_h))
    blob = blob.transpose(2, 0, 1)[np.newaxis, ...].astype(np.float32)
    dets = det_compiled([blob])[det_compiled.output(0)][0][0]
    persons = []
    for d in dets:
        if float(d[2]) < CONF_THRESHOLD:
            continue
        x1 = max(0, int(d[3] * w0))
        y1 = max(0, int(d[4] * h0))
        x2 = min(w0, int(d[5] * w0))
        y2 = min(h0, int(d[6] * h0))
        if x2 > x1 and y2 > y1:
            persons.append((x1, y1, x2, y2))
    return persons


def get_attributes(img, bbox):
    x1, y1, x2, y2 = bbox
    crop = img[y1:y2, x1:x2]
    blob = cv2.resize(crop, (attr_w, attr_h))
    blob = blob.transpose(2, 0, 1)[np.newaxis, ...].astype(np.float32)
    values = attr_compiled([blob])[attr_output].flatten()
    return {name: float(values[i]) for i, name in enumerate(ATTRIBUTE_NAMES)}


def matches_query(attributes):
    return all((attributes[name] >= ATTR_THRESHOLD) == wanted
               for name, wanted in QUERY.items())


# Scan the scene and keep the frame containing the most people whose appearance
# matches the query, so the saved image best illustrates the search result.
cap = cv2.VideoCapture(SCENE_VIDEO)
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) or 900
best = {"count": 0, "frame": None, "boxes": []}
for frame_idx in range(0, total, 15):
    cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx)
    ok, frame = cap.read()
    if not ok:
        break
    boxes = [bbox for bbox in detect_persons(frame)
             if matches_query(get_attributes(frame, bbox))]
    if len(boxes) > best["count"]:
        best = {"count": len(boxes), "frame": frame.copy(), "boxes": boxes}
cap.release()

if best["frame"] is None or best["count"] == 0:
    raise SystemExit("No person matching the appearance query was found")

# Draw a green box on every person that matches the searched appearance.
query_text = ", ".join(k if v else f"no {k}" for k, v in QUERY.items())
frame = best["frame"]
for x1, y1, x2, y2 in best["boxes"]:
    cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
    cv2.putText(frame, f"MATCH: {query_text}", (x1, max(15, y1 - 8)),
                cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)
print(f"Found {best['count']} person(s) matching [{query_text}]")

cv2.imwrite("output_openvino.jpg", frame)
print("Saved: output_openvino.jpg")
```

**Device targets:**

- `"CPU"` -- default, works on all Intel platforms.
- `"GPU"` -- Intel integrated or discrete GPU.
- `"NPU"` -- Intel NPU; both FP16 models are NPU-compatible.

#### Expected Output

![OpenVINO expected output](expected_output_openvino.jpg)

### DLStreamer Sample

The pipeline below runs the person detector via `gvadetect` and the appearance
attribute classifier via `gvaclassify` on the video.
Frames are pulled through an `appsink`, where each detected person's eight
appearance attributes are read from the classification tensor, thresholded, and
compared to the `QUERY`.
Only people whose appearance matches the query are boxed, so the annotated
`output_dlstreamer.mp4` highlights exactly the people the search is looking for.

> **Notes on running this sample:**
>
> - Export `PYTHONPATH` so the DLStreamer Python modules (`gi`, `gstgva`) are
>   importable:
>
>   ```bash
>   source /opt/intel/openvino_2026/setupvars.sh
>   source /opt/intel/dlstreamer/scripts/setup_dls_env.sh
>   export PYTHONPATH=/opt/intel/dlstreamer/python:\
>   /opt/intel/dlstreamer/gstreamer/lib/python3/dist-packages:${PYTHONPATH:-}
>   ```
>
> - The attribute scores are attached as a tensor on each person's
>   region-of-interest metadata. Convert the stream to `BGR` **before**
>   `gvadetect`/`gvaclassify` so a downstream format conversion does not strip
>   those tensors before the `appsink` reads them.

```python
import gi

gi.require_version("Gst", "1.0")
from gi.repository import Gst

Gst.init([])

import numpy as np
import cv2
from gstgva import VideoFrame

INPUT_VIDEO = "test_video.mp4"
OUTPUT_VIDEO = "output_dlstreamer.mp4"
DETECTION_MODEL = "intel/person-detection-retail-0013/FP16/person-detection-retail-0013.xml"
ATTRIBUTES_MODEL = "intel/person-attributes-recognition-crossroad-0230/FP16/person-attributes-recognition-crossroad-0230.xml"
# For CPU: change "GPU" to "CPU". For NPU: change "GPU" to "NPU".
DEVICE = "GPU"
DET_THRESHOLD = 0.6
ATTR_THRESHOLD = 0.5

# person-attributes-recognition-crossroad-0230 emits eight binary appearance
# attributes (output layer "453"), in this order:
ATTRIBUTE_NAMES = [
    "is_male", "has_bag", "has_backpack", "has_hat",
    "has_longsleeves", "has_longpants", "has_longhair", "has_coat_jacket",
]

# The appearance being searched for. Set an attribute to True to require it or
# False to require its absence; omit attributes you do not care about.
QUERY = {"has_longhair": True}


def person_attributes(video_frame):
    """Yield ((x, y, w, h), {attribute: score}) for each classified person."""
    for region in video_frame.regions():
        rect = region.rect()
        scores = None
        for tensor in region.tensors():
            if tensor.is_detection():
                continue
            data = np.array(tensor.data(), dtype=np.float32)
            # The attributes vector is the length-8 output ("453"); the model
            # also emits two length-2 colour points that are ignored here.
            if data.size == len(ATTRIBUTE_NAMES):
                scores = {name: float(data[i])
                          for i, name in enumerate(ATTRIBUTE_NAMES)}
        if scores is None:
            continue
        yield (int(rect.x), int(rect.y), int(rect.w), int(rect.h)), scores


def matches_query(scores):
    return all((scores[name] >= ATTR_THRESHOLD) == wanted
               for name, wanted in QUERY.items())


query_text = ", ".join(k if v else f"no {k}" for k, v in QUERY.items())
writer = {"w": None}
match_frames = 0


def on_video(sink):
    global match_frames
    sample = sink.emit("pull-sample")
    if sample is None:
        return Gst.FlowReturn.OK
    vf = VideoFrame(sample.get_buffer(), caps=sample.get_caps())
    matches = [(x, y, w, h) for (x, y, w, h), scores in person_attributes(vf)
               if matches_query(scores)]

    with vf.data() as mat:
        frame = mat.copy()

    if writer["w"] is None:
        frame_h, frame_w = frame.shape[:2]
        structure = sample.get_caps().get_structure(0)
        ok_fr, fps_n, fps_d = structure.get_fraction("framerate")
        fps = fps_n / fps_d if ok_fr and fps_d else 12
        writer["w"] = cv2.VideoWriter(
            OUTPUT_VIDEO, cv2.VideoWriter_fourcc(*"mp4v"), fps, (frame_w, frame_h))

    for x, y, w, h in matches:
        cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)
        cv2.putText(frame, f"MATCH: {query_text}", (x, max(15, y - 8)),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)
    if matches:
        match_frames += 1
    writer["w"].write(frame)
    return Gst.FlowReturn.OK


# Convert to BGR before inference so gvaclassify's attribute tensors survive to
# the appsink (a later format-changing videoconvert would strip them).
pipeline = Gst.parse_launch(
    f"filesrc location={INPUT_VIDEO} ! decodebin3 ! videoconvert ! "
    f"video/x-raw,format=BGR ! "
    f"gvadetect model={DETECTION_MODEL} device={DEVICE} "
    f"threshold={DET_THRESHOLD} ! queue ! "
    f"gvaclassify model={ATTRIBUTES_MODEL} device={DEVICE} ! queue ! "
    "appsink name=sink emit-signals=true sync=false max-buffers=4 drop=false"
)
sink = pipeline.get_by_name("sink")
sink.connect("new-sample", on_video)
pipeline.set_state(Gst.State.PLAYING)
pipeline.get_bus().timed_pop_filtered(
    Gst.CLOCK_TIME_NONE, Gst.MessageType.EOS | Gst.MessageType.ERROR)
pipeline.set_state(Gst.State.NULL)
if writer["w"] is not None:
    writer["w"].release()
print(f"Frames with a person matching [{query_text}]: {match_frames}", flush=True)
print(f"Saved: {OUTPUT_VIDEO}", flush=True)
```

**Device targets:**

- `DEVICE = "GPU"` -- default in the sample code.
- `DEVICE = "CPU"` -- change `"GPU"` to `"CPU"`.
- `DEVICE = "NPU"` -- change `"GPU"` to `"NPU"`; use `batch-size=1` and `nireq=4` for best NPU utilization.

#### Expected Output

![DLStreamer expected output](expected_output_dlstreamer.gif)

---

## License

Licensed under the MIT License. See [LICENSE](LICENSE) for details.

## References

- [person-detection-retail-0013](https://docs.openvino.ai/2024/omz_models_model_person_detection_retail_0013.html)
- [person-attributes-recognition-crossroad-0230](https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/intel/person-attributes-recognition-crossroad-0230/README.md)
- [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo)
- [OpenVINO Documentation](https://docs.openvino.ai/)
- [Intel DLStreamer](https://docs.openedgeplatform.intel.com/2026.0/edge-ai-libraries/dlstreamer/index.html)