vagheshpatel commited on
Commit
de2b807
·
verified ·
1 Parent(s): 86f6487

Sync motion-tracking from metro-analytics-catalog

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ expected_output_dlstreamer.gif filter=lfs diff=lfs merge=lfs -text
37
+ expected_output_openvino.gif filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) Intel Corporation.
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE
README.md ADDED
@@ -0,0 +1,337 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ license_link: LICENSE
4
+ library_name: openvino
5
+ pipeline_tag: object-detection
6
+ tags:
7
+ - openvino
8
+ - intel
9
+ - yolo
10
+ - yolo26
11
+ - detr
12
+ - motion-tracking
13
+ - multi-object-tracking
14
+ - edge-ai
15
+ - metro
16
+ - dlstreamer
17
+ language:
18
+ - en
19
+ ---
20
+
21
+ # Motion Tracking
22
+
23
+ | Property | Value |
24
+ |---|---|
25
+ | **Category** | Object Detection + Multi-Object Tracking |
26
+ | **Base Model** | [YOLO26](https://docs.ultralytics.com/models/yolo26/) (Ultralytics, default) or [DETR](https://huggingface.co/facebook/detr-resnet-50) (Facebook, alternative) + DLStreamer `gvatrack` (Kalman filter tracker) |
27
+ | **Source Framework** | PyTorch (Ultralytics, Hugging Face Transformers) |
28
+ | **Supported Precisions** | FP32, FP16, INT8 (mixed-precision) |
29
+ | **Inference Engine** | OpenVINO |
30
+ | **Hardware** | CPU, GPU, NPU |
31
+ | **Detected Class(es)** | Configurable (default: all 80 COCO classes) |
32
+
33
+ ---
34
+
35
+ ## Overview
36
+
37
+ Motion Tracking is a Metro Analytics use case that detects objects and assigns persistent track IDs across frames, enabling trajectory analysis and temporal event detection.
38
+ It is built on [YOLO26](https://docs.ultralytics.com/models/yolo26/), a state-of-the-art real-time object detector quantized to INT8, paired with a multi-object tracker:
39
+
40
+ - **DLStreamer pipeline:** YOLO26 FP16 detection via `gvadetect` + `gvatrack` element with `tracking-type=short-term-imageless`.
41
+
42
+ Each detected object receives a unique `track_id` that persists across frames as long as the object remains visible.
43
+ Outputs include per-object trajectories suitable for path analysis, dwell-time computation, and zone-based event triggers.
44
+
45
+ Typical Metro deployments include:
46
+
47
+ - **Pedestrian Trajectory Analysis** -- map walking paths through stations for flow optimization.
48
+ - **Dwell-Time Measurement** -- measure how long individuals stay in specific zones.
49
+ - **Zone-Based Event Detection** -- trigger alerts when tracked objects enter or exit defined areas.
50
+ - **Traffic Flow Analytics** -- track vehicles through intersections for signal timing optimization.
51
+ - **Incident Replay** -- reconstruct object paths for post-event forensic review.
52
+
53
+ Available YOLO26 variants: `yolo26n`, `yolo26s`, `yolo26m`, `yolo26l`.
54
+ Smaller variants (`yolo26n`, `yolo26s`) are recommended for high-FPS edge deployment.
55
+ The default tracker is `short-term-imageless` (Kalman filter-based, no image data required).
56
+ DLStreamer also supports `tracking-type=deep-sort` for more robust re-identification using a feature extraction model (e.g., mars-small128).
57
+
58
+ ### Model Selection
59
+
60
+ [DETR](https://huggingface.co/facebook/detr-resnet-50) (DEtection TRansformer) is available as an alternative to YOLO26.
61
+ DETR is a transformer-based, NMS-free detector trained on COCO and released under the Apache-2.0 license.
62
+ The export script embeds DETR normalization, box decoding, and the class remap into the OpenVINO IR, so the DETR model produces the same output layout and class ids as YOLO26.
63
+ The same DLStreamer sample run either model; select one with the `MODEL` constant at the top of the sample.
64
+
65
+ | Model | Variants | Input Size | Output | License |
66
+ |---|---|---|---|---|
67
+ | YOLO26 (default) | `yolo26n`, `yolo26s`, `yolo26m`, `yolo26l` | 640 x 640 | `[1, 300, 6]` | AGPL-3.0 |
68
+ | DETR | `detr-resnet-50` | 800 x 800 | `[1, 100, 6]` | Apache-2.0 |
69
+
70
+ Choose YOLO26 for maximum throughput on edge devices.
71
+ Choose DETR when a permissively licensed transformer detector is preferred; it is heavier and runs at a lower frame rate.
72
+
73
+ ---
74
+
75
+ ## Prerequisites
76
+
77
+ Target stack: Intel DLStreamer **2026.3.0 RC1** with the matching OpenVINO GenAI bundle **2026.5.0.0.dev20261030** (OpenVINO Runtime **2026.5.0.dev20261030**).
78
+ Use the [native RC1 setup](../README.md#native-rc1-runtime) before running these samples; this is a development stack, not a stable release.
79
+ See the [device validation matrix](../DEVICE_SUPPORT_MATRIX.md) for measured CPU, GPU, and NPU results.
80
+
81
+ - Python 3.11+
82
+ - `ffmpeg` (`sudo apt install ffmpeg`) -- used by both samples to encode output video
83
+ - [Install OpenVINO](https://docs.openvino.ai/2026/get-started/install-openvino.html) (latest version)
84
+ - [Install Intel DLStreamer](https://docs.openedgeplatform.intel.com/dev/edge-ai-libraries/dlstreamer/get_started/install/install_guide_ubuntu.html) (latest version)
85
+
86
+ Create and activate a Python virtual environment before running the scripts:
87
+
88
+ ```bash
89
+ python3 -m venv .venv --system-site-packages
90
+ source .venv/bin/activate
91
+ ```
92
+
93
+ > **Note:** The `--system-site-packages` flag is required so the virtual
94
+ > environment can access the system-installed OpenVINO and DLStreamer Python
95
+ > packages.
96
+
97
+ ---
98
+
99
+ ## Getting Started
100
+
101
+ ### Download and Quantize Model
102
+
103
+ Run the provided script to download, export to OpenVINO IR, and optionally quantize:
104
+
105
+ ```bash
106
+ wget https://huggingface.co/open-edge-platform/motion-tracking/resolve/main/export_and_quantize.sh
107
+ chmod +x export_and_quantize.sh
108
+ ./export_and_quantize.sh
109
+ ```
110
+
111
+ This exports the default **yolo26n** model in **FP16** precision.
112
+
113
+ #### Optional: Select a Different Model, Variant, or Precision
114
+
115
+ ```bash
116
+ ./export_and_quantize.sh yolo26n FP32 # full-precision
117
+ ./export_and_quantize.sh yolo26n INT8 # quantized
118
+ ./export_and_quantize.sh yolo26s # larger variant, default FP16
119
+ ./export_and_quantize.sh detr-resnet-50 # DETR alternative, default FP16
120
+ ./export_and_quantize.sh detr-resnet-50 INT8 # DETR quantized
121
+ ```
122
+
123
+ The first argument selects the model: any YOLO26 variant (`yolo26n`, `yolo26s`, `yolo26m`, `yolo26l`) or DETR (`detr-resnet-50`).
124
+ The second argument selects the precision (`FP32`, `FP16`, `INT8`); the default is **FP16**.
125
+
126
+ The script performs the following steps:
127
+
128
+ 1. Installs dependencies (`openvino`, `ultralytics` for YOLO26 or `transformers`, `torch` for DETR; adds `nncf` for INT8).
129
+ 2. Downloads the sample test video (`test_video.mp4`) and a sample test image (`test.jpg`).
130
+ 3. Downloads the PyTorch weights and exports to OpenVINO IR.
131
+ 4. *(INT8 only)* Quantizes the model using NNCF post-training quantization.
132
+
133
+ Output files (replace `yolo26n` with the selected model, for example `detr-resnet-50`):
134
+
135
+ - `yolo26n_openvino_model/` -- FP32 or FP16 OpenVINO IR model directory.
136
+ - `yolo26n_tracking_int8.xml` / `yolo26n_tracking_int8.bin` -- INT8 quantized model *(only when `INT8` is selected)*.
137
+
138
+ #### Precision / Device Compatibility
139
+
140
+ | Precision | CPU | GPU | NPU |
141
+ |---|---|---|---|
142
+ | FP32 | Yes | Yes | No |
143
+ | FP16 | Yes | Yes | Yes |
144
+ | INT8 | Yes | Yes | Yes |
145
+
146
+ > **Note:** The INT8 calibration uses frames from the bundled sample video.
147
+ > For production accuracy, replace it with a representative set of frames from
148
+ > the target deployment site.
149
+
150
+ ### DLStreamer Sample
151
+
152
+ The pipeline below runs the YOLO26 FP16 detector via `gvadetect` on
153
+ `test_video.mp4`, attaches persistent track IDs with `gvatrack`
154
+ (`short-term-imageless` tracker), and overlays bounding boxes with
155
+ `gvawatermark`. Frames are pulled from an `appsink`, per-track trajectory
156
+ polylines are drawn with OpenCV, and the result is muxed to `output_dlstreamer.mp4`
157
+ (H.264 via ffmpeg).
158
+
159
+ > **Notes on running this sample:**
160
+ >
161
+ > - Use the FP16 IR (`yolo26n_openvino_model/yolo26n.xml` or `detr-resnet-50_openvino_model/detr-resnet-50.xml`). Class names are
162
+ > read automatically from the model's embedded `metadata.yaml` by
163
+ > DLStreamer 2026.0+ -- no external `labels-file` is required.
164
+ > - Export `PYTHONPATH` so the DLStreamer Python module is importable:
165
+ >
166
+ > ```bash
167
+ > source /opt/intel/openvino_2026/setupvars.sh
168
+ > source /opt/intel/dlstreamer/scripts/setup_dls_env.sh
169
+ > export PYTHONPATH=/opt/intel/dlstreamer/python:\
170
+ > /opt/intel/dlstreamer/gstreamer/lib/python3/dist-packages:${PYTHONPATH:-}
171
+ > ```
172
+
173
+ The full sample is in [dlstreamer_sample.py](dlstreamer_sample.py); run it with `python3 dlstreamer_sample.py` or copy the code below.
174
+
175
+ <!-- sample: dlstreamer_sample.py -->
176
+ ```python
177
+ # SPDX-License-Identifier: MIT
178
+ # Copyright (C) Intel Corporation
179
+ #
180
+ # Motion Tracking: DLStreamer pipeline sample.
181
+ # See README.md for prerequisites and inputs. Run: python3 dlstreamer_sample.py
182
+
183
+ import subprocess
184
+ from collections import defaultdict
185
+
186
+ import numpy as np
187
+ import gi
188
+
189
+ gi.require_version("Gst", "1.0")
190
+ gi.require_version("GstAnalytics", "1.0")
191
+ gi.require_version("GstVideo", "1.0")
192
+ from gi.repository import Gst, GLib, GstAnalytics, GstVideo
193
+
194
+ Gst.init([])
195
+
196
+ # Import cv2 after Gst.init to avoid GStreamer re-initialization conflicts.
197
+ import cv2
198
+
199
+ # Select the model: "yolo26n" (default) or "detr-resnet-50".
200
+ MODEL = "yolo26n"
201
+
202
+ # For CPU: change device=GPU to device=CPU.
203
+ # For NPU: change device=GPU to device=NPU (batch-size=1, nireq=4 recommended).
204
+ pipeline_str = (
205
+ "filesrc location=test_video.mp4 ! decodebin3 ! "
206
+ "videoconvert ! "
207
+ f"gvadetect model={MODEL}_openvino_model/{MODEL}.xml "
208
+ "device=GPU "
209
+ "threshold=0.4 ! queue ! "
210
+ "gvatrack tracking-type=short-term-imageless ! queue ! "
211
+ "gvawatermark ! videoconvert ! video/x-raw,format=BGR ! "
212
+ "appsink name=sink emit-signals=false sync=false"
213
+ )
214
+ pipeline = Gst.parse_launch(pipeline_str)
215
+ appsink = pipeline.get_by_name("sink")
216
+
217
+ # Distinct colors for trajectory lines (one per track ID).
218
+ COLORS = [
219
+ (255, 0, 0), (0, 255, 0), (0, 0, 255), (255, 255, 0),
220
+ (255, 0, 255), (0, 255, 255), (128, 0, 255), (255, 128, 0),
221
+ ]
222
+ track_history: dict[int, list[tuple[int, int]]] = defaultdict(list)
223
+
224
+ pipeline.set_state(Gst.State.PLAYING)
225
+
226
+ proc = None
227
+
228
+ while True:
229
+ sample = appsink.emit("pull-sample")
230
+ if sample is None:
231
+ break
232
+
233
+ buf = sample.get_buffer()
234
+ caps = sample.get_caps()
235
+ struct = caps.get_structure(0)
236
+ width = struct.get_value("width")
237
+ height = struct.get_value("height")
238
+
239
+ # Start ffmpeg encoder on the first frame.
240
+ if proc is None:
241
+ ok, fps_num, fps_den = struct.get_fraction("framerate")
242
+ fps = fps_num / fps_den if ok and fps_den > 0 else 30.0
243
+ proc = subprocess.Popen(
244
+ ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "bgr24",
245
+ "-s", f"{width}x{height}", "-r", str(fps),
246
+ "-i", "pipe:0", "-c:v", "libx264", "-pix_fmt", "yuv420p",
247
+ "-movflags", "+faststart", "output_dlstreamer.mp4"],
248
+ stdin=subprocess.PIPE, stderr=subprocess.DEVNULL,
249
+ )
250
+
251
+ # Read detection / tracking metadata via GstAnalytics.
252
+ rmeta = GstAnalytics.buffer_get_analytics_relation_meta(buf)
253
+ regions_data = []
254
+ if rmeta is not None:
255
+ od_entries = []
256
+ trk_map = {} # metadata_id -> tracking_id
257
+ idx = 1
258
+ while True:
259
+ ok_od, od = rmeta.get_od_mtd(idx)
260
+ ok_trk, trk = rmeta.get_tracking_mtd(idx)
261
+ if not ok_od and not ok_trk:
262
+ break
263
+ if ok_od:
264
+ label = GLib.quark_to_string(od.get_obj_type())
265
+ _, x, y, w, h, conf = od.get_location()
266
+ od_entries.append((idx, label, int(x + w / 2), int(y + h / 2)))
267
+ if ok_trk:
268
+ ok2, tid, _, _, _ = trk.get_info()
269
+ if ok2:
270
+ trk_map[idx] = tid
271
+ idx += 1
272
+ for od_id, label, cx, cy in od_entries:
273
+ tid = 0
274
+ for trk_meta_id, tracking_id in trk_map.items():
275
+ if rmeta.get_relation(od_id, trk_meta_id) != GstAnalytics.RelTypes.NONE:
276
+ tid = tracking_id
277
+ break
278
+ regions_data.append((tid, label, cx, cy))
279
+
280
+ # Map buffer read-only and copy pixels to a writable numpy array.
281
+ success, map_info = buf.map(Gst.MapFlags.READ)
282
+ if not success:
283
+ continue
284
+ video_info = GstVideo.VideoInfo.new_from_caps(caps)
285
+ stride = video_info.stride[0]
286
+ arr = np.ndarray((height, width, 3), dtype=np.uint8,
287
+ buffer=map_info.data, strides=(stride, 3, 1)).copy()
288
+ buf.unmap(map_info)
289
+
290
+ # Draw per-track trajectory polylines on the frame copy.
291
+ for tid, label, cx, cy in regions_data:
292
+ track = track_history[tid]
293
+ track.append((cx, cy))
294
+ if len(track) > 30:
295
+ track.pop(0)
296
+ color = COLORS[tid % len(COLORS)]
297
+ pts = np.array(track, dtype=np.int32).reshape((-1, 1, 2))
298
+ cv2.polylines(arr, [pts], False, color, 2)
299
+ print(f" Track {tid}: {label} center=({cx},{cy})", flush=True)
300
+
301
+ proc.stdin.write(arr.tobytes())
302
+
303
+ pipeline.set_state(Gst.State.NULL)
304
+ if proc:
305
+ proc.stdin.close()
306
+ proc.wait()
307
+ print("Wrote output_dlstreamer.mp4", flush=True)
308
+ ```
309
+ <!-- /sample -->
310
+
311
+ #### Expected Output
312
+
313
+ ![DLStreamer expected output](expected_output_dlstreamer.gif)
314
+
315
+ **Device targets:**
316
+
317
+ - `device=GPU` -- default in the sample code.
318
+ - `device=CPU` -- change `device=GPU` to `device=CPU`.
319
+ - `device=NPU` -- change `device=GPU` to `device=NPU`; use `batch-size=1` and `nireq=4` for best NPU utilization.
320
+
321
+ ---
322
+
323
+ ## License
324
+
325
+ Licensed under the MIT License. See [LICENSE](LICENSE) for details.
326
+
327
+ ## References
328
+
329
+ - [YOLO26 Documentation](https://docs.ultralytics.com/models/yolo26/)
330
+ - [DETR Model Card (facebook/detr-resnet-50)](https://huggingface.co/facebook/detr-resnet-50)
331
+ - [End-to-End Object Detection with Transformers (DETR paper)](https://arxiv.org/abs/2005.12872)
332
+ - [Ultralytics Multi-Object Tracking](https://docs.ultralytics.com/modes/track/)
333
+ - [Intel DLStreamer gvatrack](https://docs.openedgeplatform.intel.com/dev/edge-ai-libraries/dlstreamer/elements/gvatrack.html)
334
+ - [DLStreamer Object Tracking Guide](https://docs.openedgeplatform.intel.com/dev/edge-ai-libraries/dlstreamer/dev_guide/object_tracking.html)
335
+ - [OpenVINO Documentation](https://docs.openvino.ai/)
336
+ - [NNCF Post-Training Quantization](https://docs.openvino.ai/latest/nncf_ptq_introduction.html)
337
+ - [Intel DLStreamer](https://docs.openedgeplatform.intel.com/dev/edge-ai-libraries/dlstreamer/index.html)
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "motion-tracking",
3
+ "library_name": "openvino",
4
+ "pipeline_tag": "object-detection",
5
+ "category": "Object Detection + Multi-Object Tracking",
6
+ "base_model": "YOLO26 (Ultralytics, default) or DETR facebook/detr-resnet-50 (alternative) + DLStreamer `gvatrack` (Kalman filter tracker)",
7
+ "source_framework": "PyTorch (Ultralytics, Hugging Face Transformers)",
8
+ "supported_precisions": [
9
+ "FP32",
10
+ "FP16",
11
+ "INT8"
12
+ ],
13
+ "inference_engine": "OpenVINO",
14
+ "hardware": [
15
+ "CPU",
16
+ "GPU",
17
+ "NPU"
18
+ ],
19
+ "detected_classes": "Configurable (default: all 80 COCO classes)",
20
+ "hf_repo_id": "open-edge-platform/motion-tracking",
21
+ "description": "Multi-object motion tracking using YOLO26 (default) or DETR with BoT-SORT tracker",
22
+ "selectable_models": [
23
+ "yolo26n",
24
+ "yolo26s",
25
+ "yolo26m",
26
+ "yolo26l",
27
+ "detr-resnet-50"
28
+ ],
29
+ "default_model": "yolo26n"
30
+ }
dlstreamer_sample.py ADDED
@@ -0,0 +1,131 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # SPDX-License-Identifier: MIT
2
+ # Copyright (C) Intel Corporation
3
+ #
4
+ # Motion Tracking: DLStreamer pipeline sample.
5
+ # See README.md for prerequisites and inputs. Run: python3 dlstreamer_sample.py
6
+
7
+ import subprocess
8
+ from collections import defaultdict
9
+
10
+ import numpy as np
11
+ import gi
12
+
13
+ gi.require_version("Gst", "1.0")
14
+ gi.require_version("GstAnalytics", "1.0")
15
+ gi.require_version("GstVideo", "1.0")
16
+ from gi.repository import Gst, GLib, GstAnalytics, GstVideo
17
+
18
+ Gst.init([])
19
+
20
+ # Import cv2 after Gst.init to avoid GStreamer re-initialization conflicts.
21
+ import cv2
22
+
23
+ # Select the model: "yolo26n" (default) or "detr-resnet-50".
24
+ MODEL = "yolo26n"
25
+
26
+ # For CPU: change device=GPU to device=CPU.
27
+ # For NPU: change device=GPU to device=NPU (batch-size=1, nireq=4 recommended).
28
+ pipeline_str = (
29
+ "filesrc location=test_video.mp4 ! decodebin3 ! "
30
+ "videoconvert ! "
31
+ f"gvadetect model={MODEL}_openvino_model/{MODEL}.xml "
32
+ "device=GPU "
33
+ "threshold=0.4 ! queue ! "
34
+ "gvatrack tracking-type=short-term-imageless ! queue ! "
35
+ "gvawatermark ! videoconvert ! video/x-raw,format=BGR ! "
36
+ "appsink name=sink emit-signals=false sync=false"
37
+ )
38
+ pipeline = Gst.parse_launch(pipeline_str)
39
+ appsink = pipeline.get_by_name("sink")
40
+
41
+ # Distinct colors for trajectory lines (one per track ID).
42
+ COLORS = [
43
+ (255, 0, 0), (0, 255, 0), (0, 0, 255), (255, 255, 0),
44
+ (255, 0, 255), (0, 255, 255), (128, 0, 255), (255, 128, 0),
45
+ ]
46
+ track_history: dict[int, list[tuple[int, int]]] = defaultdict(list)
47
+
48
+ pipeline.set_state(Gst.State.PLAYING)
49
+
50
+ proc = None
51
+
52
+ while True:
53
+ sample = appsink.emit("pull-sample")
54
+ if sample is None:
55
+ break
56
+
57
+ buf = sample.get_buffer()
58
+ caps = sample.get_caps()
59
+ struct = caps.get_structure(0)
60
+ width = struct.get_value("width")
61
+ height = struct.get_value("height")
62
+
63
+ # Start ffmpeg encoder on the first frame.
64
+ if proc is None:
65
+ ok, fps_num, fps_den = struct.get_fraction("framerate")
66
+ fps = fps_num / fps_den if ok and fps_den > 0 else 30.0
67
+ proc = subprocess.Popen(
68
+ ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "bgr24",
69
+ "-s", f"{width}x{height}", "-r", str(fps),
70
+ "-i", "pipe:0", "-c:v", "libx264", "-pix_fmt", "yuv420p",
71
+ "-movflags", "+faststart", "output_dlstreamer.mp4"],
72
+ stdin=subprocess.PIPE, stderr=subprocess.DEVNULL,
73
+ )
74
+
75
+ # Read detection / tracking metadata via GstAnalytics.
76
+ rmeta = GstAnalytics.buffer_get_analytics_relation_meta(buf)
77
+ regions_data = []
78
+ if rmeta is not None:
79
+ od_entries = []
80
+ trk_map = {} # metadata_id -> tracking_id
81
+ idx = 1
82
+ while True:
83
+ ok_od, od = rmeta.get_od_mtd(idx)
84
+ ok_trk, trk = rmeta.get_tracking_mtd(idx)
85
+ if not ok_od and not ok_trk:
86
+ break
87
+ if ok_od:
88
+ label = GLib.quark_to_string(od.get_obj_type())
89
+ _, x, y, w, h, conf = od.get_location()
90
+ od_entries.append((idx, label, int(x + w / 2), int(y + h / 2)))
91
+ if ok_trk:
92
+ ok2, tid, _, _, _ = trk.get_info()
93
+ if ok2:
94
+ trk_map[idx] = tid
95
+ idx += 1
96
+ for od_id, label, cx, cy in od_entries:
97
+ tid = 0
98
+ for trk_meta_id, tracking_id in trk_map.items():
99
+ if rmeta.get_relation(od_id, trk_meta_id) != GstAnalytics.RelTypes.NONE:
100
+ tid = tracking_id
101
+ break
102
+ regions_data.append((tid, label, cx, cy))
103
+
104
+ # Map buffer read-only and copy pixels to a writable numpy array.
105
+ success, map_info = buf.map(Gst.MapFlags.READ)
106
+ if not success:
107
+ continue
108
+ video_info = GstVideo.VideoInfo.new_from_caps(caps)
109
+ stride = video_info.stride[0]
110
+ arr = np.ndarray((height, width, 3), dtype=np.uint8,
111
+ buffer=map_info.data, strides=(stride, 3, 1)).copy()
112
+ buf.unmap(map_info)
113
+
114
+ # Draw per-track trajectory polylines on the frame copy.
115
+ for tid, label, cx, cy in regions_data:
116
+ track = track_history[tid]
117
+ track.append((cx, cy))
118
+ if len(track) > 30:
119
+ track.pop(0)
120
+ color = COLORS[tid % len(COLORS)]
121
+ pts = np.array(track, dtype=np.int32).reshape((-1, 1, 2))
122
+ cv2.polylines(arr, [pts], False, color, 2)
123
+ print(f" Track {tid}: {label} center=({cx},{cy})", flush=True)
124
+
125
+ proc.stdin.write(arr.tobytes())
126
+
127
+ pipeline.set_state(Gst.State.NULL)
128
+ if proc:
129
+ proc.stdin.close()
130
+ proc.wait()
131
+ print("Wrote output_dlstreamer.mp4", flush=True)
expected_output_dlstreamer.gif ADDED

Git LFS Details

  • SHA256: 76fdac6b6bbe38bf4b8f744aa0803a3705142f4af990404005406a1135a06c2d
  • Pointer size: 131 Bytes
  • Size of remote file: 438 kB
expected_output_openvino.gif ADDED

Git LFS Details

  • SHA256: 351f44e1ca8578b3048f68e1db881136832303db1d8ee8cc80054fe6c0b579ba
  • Pointer size: 132 Bytes
  • Size of remote file: 7.28 MB
export_and_quantize.sh ADDED
@@ -0,0 +1,252 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # SPDX-License-Identifier: MIT
3
+ # Copyright (C) Intel Corporation
4
+ #
5
+ # Export a YOLO26 or DETR detector for motion tracking to OpenVINO IR.
6
+ # The tracker itself (BoT-SORT / ByteTrack) runs at inference time via
7
+ # Ultralytics or DLStreamer gvatrack; no separate model export is needed.
8
+ # Usage: ./export_and_quantize.sh [MODEL_VARIANT] [PRECISION]
9
+ # Example: ./export_and_quantize.sh yolo26n FP16
10
+ # Example: ./export_and_quantize.sh detr-resnet-50 FP16
11
+ #
12
+ # Supported model variants:
13
+ # yolo26n, yolo26s, yolo26m, yolo26l -- Ultralytics YOLO26 (default)
14
+ # detr-resnet-50 -- Facebook DETR (Hugging Face)
15
+ #
16
+ # DETR is exported end-to-end: ImageNet normalization, softmax,
17
+ # box decoding, and the COCO-91 to COCO-80 class remap are embedded in the
18
+ # graph, so the IR produces the same [1, N, 6] output layout and class ids as
19
+ # YOLO26 and runs with the same OpenVINO and DLStreamer samples.
20
+ #
21
+ # Supported precisions:
22
+ # FP32 -- Full-precision floating-point weights
23
+ # FP16 -- Half-precision floating-point weights (default)
24
+ # INT8 -- Quantized 8-bit integer weights (requires NNCF)
25
+ #
26
+ # Precision / device compatibility:
27
+ # | Precision | CPU | GPU | NPU |
28
+ # |-----------|-----|-----|-----|
29
+ # | FP32 | Yes | Yes | No |
30
+ # | FP16 | Yes | Yes | Yes |
31
+ # | INT8 | Yes | Yes | Yes |
32
+
33
+ set -euo pipefail
34
+
35
+ MODEL_NAME="${1:-yolo26n}"
36
+ PRECISION="${2:-FP16}"
37
+ PRECISION="$(echo "${PRECISION}" | tr '[:lower:]' '[:upper:]')"
38
+
39
+ if [[ "${PRECISION}" != "FP32" && "${PRECISION}" != "FP16" && "${PRECISION}" != "INT8" ]]; then
40
+ echo "ERROR: unsupported precision '${PRECISION}'. Choose FP32, FP16, or INT8." >&2
41
+ exit 1
42
+ fi
43
+
44
+ case "${MODEL_NAME}" in
45
+ yolo26n|yolo26s|yolo26m|yolo26l)
46
+ MODEL_FAMILY="yolo"
47
+ INPUT_SIZE=640
48
+ ;;
49
+ detr-resnet-50)
50
+ MODEL_FAMILY="detr"
51
+ INPUT_SIZE=800
52
+ ;;
53
+ *)
54
+ echo "ERROR: unsupported model '${MODEL_NAME}'. Choose yolo26[n|s|m|l] or detr-resnet-50." >&2
55
+ exit 1
56
+ ;;
57
+ esac
58
+
59
+ # Ask for approval before downloading models and sample files
60
+ echo ""
61
+ echo "This script will download:"
62
+ echo " - Model weights and/or sample files"
63
+ echo ""
64
+ read -p "Continue with downloads? (yes/no): " APPROVAL
65
+ if [[ "${APPROVAL}" != "yes" ]]; then
66
+ echo "Download cancelled by user."
67
+ exit 0
68
+ fi
69
+
70
+ # Ping the HuggingFace repo to register a tracked download of config.json.
71
+ # This is best-effort: a failure (offline, or before the repo is published)
72
+ # must not stop the export.
73
+ echo "--- Registering HuggingFace download (tracking ping) ---"
74
+ HF_REPO_ID="open-edge-platform/motion-tracking"
75
+ HF_CONFIG_URL="https://huggingface.co/${HF_REPO_ID}/resolve/main/config.json"
76
+ if curl -fsSL -o /dev/null "${HF_CONFIG_URL}"; then
77
+ echo "Registered HuggingFace download for ${HF_REPO_ID}"
78
+ else
79
+ echo "WARNING: HuggingFace tracking ping failed (offline?); continuing." >&2
80
+ fi
81
+ echo ""
82
+ echo "--- Installing dependencies ---"
83
+ if [[ "${MODEL_FAMILY}" == "detr" ]]; then
84
+ DEPS=(openvino transformers torch pyyaml opencv-python)
85
+ else
86
+ DEPS=(openvino ultralytics)
87
+ fi
88
+ if [[ "${PRECISION}" == "INT8" ]]; then
89
+ DEPS+=(nncf)
90
+ fi
91
+ pip install -qU "${DEPS[@]}"
92
+
93
+ echo "--- Downloading sample test video ---"
94
+ if [[ ! -f test_video.mp4 ]]; then
95
+ wget -q -O test_video.mp4 \
96
+ https://github.com/intel-iot-devkit/sample-videos/raw/master/people-detection.mp4
97
+ echo "Downloaded: test_video.mp4"
98
+ else
99
+ echo "Already present: test_video.mp4"
100
+ fi
101
+
102
+ echo "--- Downloading sample test image ---"
103
+ if [[ ! -f test.jpg ]]; then
104
+ wget -q -O test.jpg https://ultralytics.com/images/bus.jpg
105
+ echo "Downloaded: test.jpg"
106
+ else
107
+ echo "Already present: test.jpg"
108
+ fi
109
+
110
+ if [[ "${PRECISION}" == "FP32" ]]; then
111
+ HALF_FLAG="False"
112
+ EXPORT_LABEL="FP32"
113
+ else
114
+ HALF_FLAG="True"
115
+ EXPORT_LABEL="FP16"
116
+ fi
117
+
118
+ if [[ "${MODEL_FAMILY}" == "yolo" ]]; then
119
+ echo "--- Exporting ${MODEL_NAME} to OpenVINO IR (${EXPORT_LABEL}) ---"
120
+ python3 -c "
121
+ from ultralytics import YOLO
122
+
123
+ model = YOLO('${MODEL_NAME}.pt')
124
+ model.export(format='openvino', half=${HALF_FLAG}, dynamic=False, imgsz=${INPUT_SIZE}, nms=False)
125
+ print('Export complete: ${MODEL_NAME}_openvino_model/')
126
+ "
127
+ else
128
+ echo "--- Exporting facebook/${MODEL_NAME} to OpenVINO IR (${EXPORT_LABEL}) ---"
129
+ python3 - "${MODEL_NAME}" "${HALF_FLAG}" "${INPUT_SIZE}" <<'PYEXPORT'
130
+ import sys
131
+ from pathlib import Path
132
+
133
+ import openvino as ov
134
+ import torch
135
+ import yaml
136
+ from transformers import DetrConfig, DetrForObjectDetection
137
+
138
+ model_name, half_flag, size = sys.argv[1], sys.argv[2] == "True", int(sys.argv[3])
139
+ model_id = f"facebook/{model_name}"
140
+
141
+ # The "no_timm" revision uses the transformers ResNet backbone (no timm dependency).
142
+ config_dict, _ = DetrConfig.get_config_dict(model_id, revision="no_timm")
143
+ # Older no_timm configs serialize the standard non-dilated backbone as null.
144
+ if config_dict.get("dilation") is None:
145
+ config_dict["dilation"] = False
146
+ config = DetrConfig.from_dict(config_dict)
147
+ detr = DetrForObjectDetection.from_pretrained(model_id, revision="no_timm", config=config).eval()
148
+ id2label = detr.config.id2label
149
+ # DETR predicts 91 COCO category ids (with "N/A" gaps) plus a no-object class.
150
+ # Keep the 80 real categories so class ids match YOLO26 (person = 0).
151
+ coco80 = [k for k in sorted(id2label) if id2label[k] != "N/A"]
152
+ assert len(coco80) == 80, f"unexpected label map with {len(coco80)} classes"
153
+
154
+
155
+ class DetrEndToEnd(torch.nn.Module):
156
+ """Input: RGB [0, 1] NCHW. Output: [1, 100, 6] = [x1, y1, x2, y2, conf, class_id] in input pixels."""
157
+
158
+ def __init__(self, model: torch.nn.Module) -> None:
159
+ super().__init__()
160
+ self.model = model
161
+ self.register_buffer("mean", torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
162
+ self.register_buffer("std", torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
163
+ self.register_buffer("coco80", torch.tensor(coco80))
164
+
165
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
166
+ out = self.model(pixel_values=(x - self.mean) / self.std, return_dict=True)
167
+ probs = out.logits.softmax(-1).index_select(-1, self.coco80)
168
+ conf, cls = probs.max(-1)
169
+ cx, cy, w, h = (out.pred_boxes * size).unbind(-1)
170
+ return torch.stack([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2, conf, cls.float()], -1)
171
+
172
+
173
+ ov_model = ov.convert_model(
174
+ DetrEndToEnd(detr), example_input=torch.zeros(1, 3, size, size), input=[1, 3, size, size]
175
+ )
176
+ ov_model.outputs[0].get_tensor().set_names({"output0"})
177
+
178
+ names = {i: id2label[k] for i, k in enumerate(coco80)}
179
+ # DLStreamer reads preprocessing and the post-processor type from model_info.
180
+ model_info = {
181
+ "model_type": "YOLO",
182
+ "labels": " ".join(n.replace(" ", "_") for n in names.values()),
183
+ "resize_type": "standard",
184
+ "reverse_input_channels": "YES",
185
+ "scale_values": "255",
186
+ "iou_threshold": "0.7",
187
+ }
188
+ for key, value in model_info.items():
189
+ ov_model.set_rt_info(value, ["model_info", key])
190
+
191
+ out_dir = Path(f"{model_name}_openvino_model")
192
+ out_dir.mkdir(exist_ok=True)
193
+ ov.save_model(ov_model, out_dir / f"{model_name}.xml", compress_to_fp16=half_flag)
194
+ metadata = {
195
+ "description": f"{model_id} exported end-to-end with YOLO26-compatible output",
196
+ "license": "Apache-2.0",
197
+ "task": "detect",
198
+ "batch": 1,
199
+ "imgsz": [size, size],
200
+ "names": names,
201
+ "channels": 3,
202
+ "end2end": True,
203
+ }
204
+ with open(out_dir / "metadata.yaml", "w", encoding="utf-8") as f:
205
+ yaml.safe_dump(metadata, f, sort_keys=False)
206
+ print(f"Export complete: {out_dir}/")
207
+ PYEXPORT
208
+ fi
209
+
210
+ if [[ "${PRECISION}" == "INT8" ]]; then
211
+ echo "--- Quantizing to INT8 with NNCF ---"
212
+ python3 -c "
213
+ import nncf
214
+ import openvino as ov
215
+ import numpy as np
216
+ import cv2
217
+
218
+ core = ov.Core()
219
+ model = core.read_model('${MODEL_NAME}_openvino_model/${MODEL_NAME}.xml')
220
+
221
+ # Extract frames from the sample video for calibration.
222
+ cap = cv2.VideoCapture('test_video.mp4')
223
+ frames = []
224
+ while len(frames) < 300:
225
+ ret, frame = cap.read()
226
+ if not ret:
227
+ cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
228
+ continue
229
+ img = cv2.resize(frame, (${INPUT_SIZE}, ${INPUT_SIZE}))
230
+ img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB).astype(np.float32) / 255.0
231
+ img = img.transpose(2, 0, 1)[np.newaxis, ...]
232
+ frames.append(img)
233
+ cap.release()
234
+
235
+ def transform_fn(data_item):
236
+ return frames[data_item % len(frames)]
237
+
238
+ calibration_dataset = nncf.Dataset(list(range(300)), transform_fn)
239
+
240
+ quantized = nncf.quantize(
241
+ model,
242
+ calibration_dataset,
243
+ preset=nncf.QuantizationPreset.MIXED,
244
+ subset_size=300,
245
+ model_type=nncf.ModelType.TRANSFORMER if '${MODEL_FAMILY}' == 'detr' else None,
246
+ )
247
+
248
+ ov.save_model(quantized, '${MODEL_NAME}_tracking_int8.xml')
249
+ print('Quantization complete: ${MODEL_NAME}_tracking_int8.xml')
250
+ "
251
+ fi
252
+ echo "--- Done ---"