--- license: mit license_link: LICENSE library_name: openvino pipeline_tag: object-detection tags: - openvino - intel - yolo - yolo26 - detr - motion-tracking - multi-object-tracking - edge-ai - metro - dlstreamer language: - en --- # Motion Tracking | Property | Value | |---|---| | **Category** | Object Detection + Multi-Object Tracking | | **Base Model** | [YOLO26](https://docs.ultralytics.com/models/yolo26/) (Ultralytics, default) or [DETR](https://huggingface.co/facebook/detr-resnet-50) (Facebook, alternative) + DLStreamer `gvatrack` (Kalman filter tracker) | | **Source Framework** | PyTorch (Ultralytics, Hugging Face Transformers) | | **Supported Precisions** | FP32, FP16, INT8 (mixed-precision) | | **Inference Engine** | OpenVINO | | **Hardware** | CPU, GPU, NPU | | **Detected Class(es)** | Configurable (default: all 80 COCO classes) | --- ## Overview Motion Tracking is a Metro Analytics use case that detects objects and assigns persistent track IDs across frames, enabling trajectory analysis and temporal event detection. It is built on [YOLO26](https://docs.ultralytics.com/models/yolo26/), a state-of-the-art real-time object detector quantized to INT8, paired with a multi-object tracker: - **DLStreamer pipeline:** YOLO26 FP16 detection via `gvadetect` + `gvatrack` element with `tracking-type=short-term-imageless`. Each detected object receives a unique `track_id` that persists across frames as long as the object remains visible. Outputs include per-object trajectories suitable for path analysis, dwell-time computation, and zone-based event triggers. Typical Metro deployments include: - **Pedestrian Trajectory Analysis** -- map walking paths through stations for flow optimization. - **Dwell-Time Measurement** -- measure how long individuals stay in specific zones. - **Zone-Based Event Detection** -- trigger alerts when tracked objects enter or exit defined areas. - **Traffic Flow Analytics** -- track vehicles through intersections for signal timing optimization. - **Incident Replay** -- reconstruct object paths for post-event forensic review. Available YOLO26 variants: `yolo26n`, `yolo26s`, `yolo26m`, `yolo26l`. Smaller variants (`yolo26n`, `yolo26s`) are recommended for high-FPS edge deployment. The default tracker is `short-term-imageless` (Kalman filter-based, no image data required). DLStreamer also supports `tracking-type=deep-sort` for more robust re-identification using a feature extraction model (e.g., mars-small128). ### Model Selection [DETR](https://huggingface.co/facebook/detr-resnet-50) (DEtection TRansformer) is available as an alternative to YOLO26. DETR is a transformer-based, NMS-free detector trained on COCO and released under the Apache-2.0 license. The export script embeds DETR normalization, box decoding, and the class remap into the OpenVINO IR, so the DETR model produces the same output layout and class ids as YOLO26. The same DLStreamer sample run either model; select one with the `MODEL` constant at the top of the sample. | Model | Variants | Input Size | Output | License | |---|---|---|---|---| | YOLO26 (default) | `yolo26n`, `yolo26s`, `yolo26m`, `yolo26l` | 640 x 640 | `[1, 300, 6]` | AGPL-3.0 | | DETR | `detr-resnet-50` | 800 x 800 | `[1, 100, 6]` | Apache-2.0 | Choose YOLO26 for maximum throughput on edge devices. Choose DETR when a permissively licensed transformer detector is preferred; it is heavier and runs at a lower frame rate. --- ## Prerequisites See the [released runtime setup](../README.md#released-runtimes) and [device support matrix](../DEVICE_SUPPORT_MATRIX.md) before choosing a runtime. The previous validation used the RC1 development stack. - Python 3.11+ - `ffmpeg` (`sudo apt install ffmpeg`) -- used by both samples to encode output video Create and activate a Python virtual environment before running the scripts: ```bash python3 -m venv .venv --system-site-packages source .venv/bin/activate ``` > **Note:** The `--system-site-packages` flag is required so the virtual > environment can access the system-installed OpenVINO and DLStreamer Python > packages. --- ## Getting Started ### Download and Quantize Model Run the provided script to download, export to OpenVINO IR, and optionally quantize: ```bash wget https://huggingface.co/open-edge-platform/motion-tracking/resolve/main/export_and_quantize.sh chmod +x export_and_quantize.sh ./export_and_quantize.sh ``` This exports the default **yolo26n** model in **FP16** precision. #### Optional: Select a Different Model, Variant, or Precision ```bash ./export_and_quantize.sh yolo26n FP32 # full-precision ./export_and_quantize.sh yolo26n INT8 # quantized ./export_and_quantize.sh yolo26s # larger variant, default FP16 ./export_and_quantize.sh detr-resnet-50 # DETR alternative, default FP16 ./export_and_quantize.sh detr-resnet-50 INT8 # DETR quantized ``` The first argument selects the model: any YOLO26 variant (`yolo26n`, `yolo26s`, `yolo26m`, `yolo26l`) or DETR (`detr-resnet-50`). The second argument selects the precision (`FP32`, `FP16`, `INT8`); the default is **FP16**. The script performs the following steps: 1. Installs dependencies (`openvino`, `ultralytics` for YOLO26 or `transformers`, `torch` for DETR; adds `nncf` for INT8). 2. Downloads the sample test video (`test_video.mp4`) and a sample test image (`test.jpg`). 3. Downloads the PyTorch weights and exports to OpenVINO IR. 4. *(INT8 only)* Quantizes the model using NNCF post-training quantization. Output files (replace `yolo26n` with the selected model, for example `detr-resnet-50`): - `yolo26n_openvino_model/` -- FP32 or FP16 OpenVINO IR model directory. - `yolo26n_tracking_int8.xml` / `yolo26n_tracking_int8.bin` -- INT8 quantized model *(only when `INT8` is selected)*. #### Precision / Device Compatibility | Precision | CPU | GPU | NPU | |---|---|---|---| | FP32 | Yes | Yes | No | | FP16 | Yes | Yes | Yes | | INT8 | Yes | Yes | Yes | > **Note:** The INT8 calibration uses frames from the bundled sample video. > For production accuracy, replace it with a representative set of frames from > the target deployment site. ### DLStreamer Sample The pipeline below runs the YOLO26 FP16 detector via `gvadetect` on `test_video.mp4`, attaches persistent track IDs with `gvatrack` (`short-term-imageless` tracker), and overlays bounding boxes with `gvawatermark`. Frames are pulled from an `appsink`, per-track trajectory polylines are drawn with OpenCV, and the result is muxed to `output_dlstreamer.mp4` (H.264 via ffmpeg). > **Notes on running this sample:** > > - Use the FP16 IR (`yolo26n_openvino_model/yolo26n.xml` or `detr-resnet-50_openvino_model/detr-resnet-50.xml`). Class names are > read automatically from the model's embedded `metadata.yaml` by > DLStreamer 2026.0+ -- no external `labels-file` is required. > - Export `PYTHONPATH` so the DLStreamer Python module is importable: > > ```bash > source /opt/intel/openvino_2026/setupvars.sh > source /opt/intel/dlstreamer/scripts/setup_dls_env.sh > export PYTHONPATH=/opt/intel/dlstreamer/python:\ > /opt/intel/dlstreamer/gstreamer/lib/python3/dist-packages:${PYTHONPATH:-} > ``` The full sample is in [dlstreamer_sample.py](dlstreamer_sample.py); run it with `python3 dlstreamer_sample.py` or copy the code below. ```python # SPDX-License-Identifier: MIT # Copyright (C) Intel Corporation # # Motion Tracking: DLStreamer pipeline sample. # See README.md for prerequisites and inputs. Run: python3 dlstreamer_sample.py import subprocess from collections import defaultdict import numpy as np import gi gi.require_version("Gst", "1.0") gi.require_version("GstAnalytics", "1.0") gi.require_version("GstVideo", "1.0") from gi.repository import Gst, GLib, GstAnalytics, GstVideo Gst.init([]) # Import cv2 after Gst.init to avoid GStreamer re-initialization conflicts. import cv2 # Select the model: "yolo26n" (default) or "detr-resnet-50". MODEL = "yolo26n" # For CPU: change device=GPU to device=CPU. # For NPU: change device=GPU to device=NPU (batch-size=1, nireq=4 recommended). pipeline_str = ( "filesrc location=test_video.mp4 ! decodebin3 ! " "videoconvert ! " f"gvadetect model={MODEL}_openvino_model/{MODEL}.xml " "device=GPU " "threshold=0.4 ! queue ! " "gvatrack tracking-type=short-term-imageless ! queue ! " "gvawatermark ! videoconvert ! video/x-raw,format=BGR ! " "appsink name=sink emit-signals=false sync=false" ) pipeline = Gst.parse_launch(pipeline_str) appsink = pipeline.get_by_name("sink") # Distinct colors for trajectory lines (one per track ID). COLORS = [ (255, 0, 0), (0, 255, 0), (0, 0, 255), (255, 255, 0), (255, 0, 255), (0, 255, 255), (128, 0, 255), (255, 128, 0), ] track_history: dict[int, list[tuple[int, int]]] = defaultdict(list) pipeline.set_state(Gst.State.PLAYING) proc = None while True: sample = appsink.emit("pull-sample") if sample is None: break buf = sample.get_buffer() caps = sample.get_caps() struct = caps.get_structure(0) width = struct.get_value("width") height = struct.get_value("height") # Start ffmpeg encoder on the first frame. if proc is None: ok, fps_num, fps_den = struct.get_fraction("framerate") fps = fps_num / fps_den if ok and fps_den > 0 else 30.0 proc = subprocess.Popen( ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "bgr24", "-s", f"{width}x{height}", "-r", str(fps), "-i", "pipe:0", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-movflags", "+faststart", "output_dlstreamer.mp4"], stdin=subprocess.PIPE, stderr=subprocess.DEVNULL, ) # Read detection / tracking metadata via GstAnalytics. rmeta = GstAnalytics.buffer_get_analytics_relation_meta(buf) regions_data = [] if rmeta is not None: od_entries = [] trk_map = {} # metadata_id -> tracking_id idx = 1 while True: ok_od, od = rmeta.get_od_mtd(idx) ok_trk, trk = rmeta.get_tracking_mtd(idx) if not ok_od and not ok_trk: break if ok_od: label = GLib.quark_to_string(od.get_obj_type()) _, x, y, w, h, conf = od.get_location() od_entries.append((idx, label, int(x + w / 2), int(y + h / 2))) if ok_trk: ok2, tid, _, _, _ = trk.get_info() if ok2: trk_map[idx] = tid idx += 1 for od_id, label, cx, cy in od_entries: tid = 0 for trk_meta_id, tracking_id in trk_map.items(): if rmeta.get_relation(od_id, trk_meta_id) != GstAnalytics.RelTypes.NONE: tid = tracking_id break regions_data.append((tid, label, cx, cy)) # Map buffer read-only and copy pixels to a writable numpy array. success, map_info = buf.map(Gst.MapFlags.READ) if not success: continue video_info = GstVideo.VideoInfo.new_from_caps(caps) stride = video_info.stride[0] arr = np.ndarray((height, width, 3), dtype=np.uint8, buffer=map_info.data, strides=(stride, 3, 1)).copy() buf.unmap(map_info) # Draw per-track trajectory polylines on the frame copy. for tid, label, cx, cy in regions_data: track = track_history[tid] track.append((cx, cy)) if len(track) > 30: track.pop(0) color = COLORS[tid % len(COLORS)] pts = np.array(track, dtype=np.int32).reshape((-1, 1, 2)) cv2.polylines(arr, [pts], False, color, 2) print(f" Track {tid}: {label} center=({cx},{cy})", flush=True) proc.stdin.write(arr.tobytes()) pipeline.set_state(Gst.State.NULL) if proc: proc.stdin.close() proc.wait() print("Wrote output_dlstreamer.mp4", flush=True) ``` #### Expected Output ![DLStreamer expected output](expected_output_dlstreamer.gif) **Device targets:** - `device=GPU` -- default in the sample code. - `device=CPU` -- change `device=GPU` to `device=CPU`. - `device=NPU` -- change `device=GPU` to `device=NPU`; use `batch-size=1` and `nireq=4` for best NPU utilization. --- ## License Licensed under the MIT License. See [LICENSE](LICENSE) for details. ## References - [YOLO26 Documentation](https://docs.ultralytics.com/models/yolo26/) - [DETR Model Card (facebook/detr-resnet-50)](https://huggingface.co/facebook/detr-resnet-50) - [End-to-End Object Detection with Transformers (DETR paper)](https://arxiv.org/abs/2005.12872) - [Ultralytics Multi-Object Tracking](https://docs.ultralytics.com/modes/track/) - [Intel DLStreamer gvatrack](https://docs.openedgeplatform.intel.com/dev/edge-ai-libraries/dlstreamer/elements/gvatrack.html) - [DLStreamer Object Tracking Guide](https://docs.openedgeplatform.intel.com/dev/edge-ai-libraries/dlstreamer/dev_guide/object_tracking.html) - [OpenVINO Documentation](https://docs.openvino.ai/) - [NNCF Post-Training Quantization](https://docs.openvino.ai/latest/nncf_ptq_introduction.html) - [Intel DLStreamer](https://docs.openedgeplatform.intel.com/dev/edge-ai-libraries/dlstreamer/index.html)