sunyijie commited on
Commit
efaa3d7
·
1 Parent(s): 7f13ce4

first commit

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ *.axmodel filter=lfs diff=lfs merge=lfs -text
37
+ *.jpg filter=lfs diff=lfs merge=lfs -text
.gitignore ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ results/**
2
+ datasets/**
README.md CHANGED
@@ -1,3 +1,104 @@
1
  ---
2
  license: mit
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
  ---
4
+
5
+ # DepthART (AX650 Relative Depth Estimation)
6
+
7
+ DepthART is a family of relative depth estimation models based on selective state
8
+ spaces (Selective Scan). This repository provides the ONNX model of the `small / 448`
9
+ variant, an `axmodel` compiled for **AX650**, and tools for host ONNX inference, board
10
+ axmodel inference, and model rewriting.
11
+
12
+ The model outputs **affine-invariant relative depth**: larger values mean closer,
13
+ smaller values mean farther. It does not directly produce metric depth; align it with
14
+ ground truth depth using a global scale/offset before use.
15
+
16
+ ## Single-image result
17
+
18
+ Left: input image; right: relative depth output by the model (INFERNO colormap,
19
+ stretched to the 2%–98% percentile range).
20
+
21
+ ![DepthART single-image inference result](host_depth.jpg)
22
+
23
+ ## Repository contents
24
+
25
+ | Path | Description |
26
+ | --- | --- |
27
+ | `models/relative_s_448_default.onnx` | Original FP32 ONNX model, containing the custom op `com.depthart::SelectiveScan` |
28
+ | `models/relative_s_448_default_ax650.axmodel` | Compiled AX650 model |
29
+ | `src/infer/depthart_infer_onnx.py` | Host ONNX inference (rewrites `SelectiveScan` automatically) |
30
+ | `src/infer/depthart_infer_axmodel.py` | AX650 board axmodel inference (standalone script) |
31
+ | `src/tool/depthart_export_axonnx.py` | Rewrites `SelectiveScan` into a pure ONNX subgraph that pulsar2 can compile |
32
+ | `src/tool/compare_onnx_axmodel.py`, `compare_depth_outputs.py` | Numerical comparison of outputs from different backends |
33
+ | `src/tool/evaluate_nyu654_depth.py` | Accuracy evaluation on the NYU Depth V2 eigen test split |
34
+ | `host.jpg`, `host_depth.jpg` | Single-image example input and side-by-side visualization result |
35
+
36
+ ## Model interface
37
+
38
+ ### ONNX
39
+
40
+ | Item | Value |
41
+ | --- | --- |
42
+ | Input | `image` `[1, 3, 448, 448]` float32, NCHW, RGB |
43
+ | Output | `depth` `[1, 1, 448, 448]` float32, relative depth (larger = closer) |
44
+
45
+ ## AX650 board inference
46
+
47
+ The script `src/infer/depthart_infer_axmodel.py` does not depend on other scripts in
48
+ the repository. For deployment, just place the following files in the same directory
49
+ on the board:
50
+
51
+ ```text
52
+ relative_s_448_default_ax650.axmodel
53
+ depthart_infer_axmodel.py
54
+ images or image directory to test
55
+ ```
56
+
57
+ Board runtime requires `axengine`.
58
+
59
+ ### Single-image validation
60
+
61
+ ```bash
62
+ python3 depthart_infer_axmodel.py \
63
+ --axmodel relative_s_448_default_ax650.axmodel \
64
+ --image ./host.jpg \
65
+ --output ./results/ax_single
66
+ ```
67
+
68
+ Outputs `results/ax_single/host_depth.jpg` (side-by-side visualization) and
69
+ `host_depth.npy`; the console prints depth statistics, NPU inference time, CPU
70
+ processing time, and per-stage timings. A single image is the first inference and
71
+ includes runtime first-execution overhead; use dataset mode for performance evaluation.
72
+
73
+ ### Dataset inference
74
+
75
+ By default, fast mode only measures `read / preprocess / npu` and skips depth
76
+ restoration, visualization, and file writing, for performance testing:
77
+
78
+ ```bash
79
+ python3 depthart_infer_axmodel.py \
80
+ --axmodel relative_s_448_default_ax650.axmodel \
81
+ --dataset ./images \
82
+ --output ./results/board
83
+ ```
84
+
85
+ Full output (depth restoration, visualization, summary NPZ), for accuracy evaluation:
86
+
87
+ ```bash
88
+ python3 depthart_infer_axmodel.py \
89
+ --axmodel relative_s_448_default_ax650.axmodel \
90
+ --dataset ./images \
91
+ --output ./results/board_full \
92
+ --full-output \
93
+ --dump-npz ./results/board_full/raw_outputs.npz
94
+ ```
95
+
96
+ Image-by-image comparison against the host ONNX baseline summary NPZ:
97
+
98
+ ```bash
99
+ python3 depthart_infer_axmodel.py \
100
+ --axmodel relative_s_448_default_ax650.axmodel \
101
+ --dataset ./images \
102
+ --output ./results/board_check \
103
+ --ref-npz ./results/onnx_full/raw_outputs.npz
104
+ ```
config.json ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "ONNX",
3
+ "npu_mode": "NPU3",
4
+ "input": "onnx/relative_s_448_default_axonnx_t4.onnx",
5
+ "output_dir": "./output_axonnx_t4",
6
+ "output_name": "relative_s_448_default_ax650_t4.axmodel",
7
+ "quant": {
8
+ "input_configs": [
9
+ {
10
+ "tensor_name": "DEFAULT",
11
+ "calibration_dataset": "/data/datasets/coco2017val_500_calibration_dataset.tar",
12
+ "calibration_size": 8,
13
+ "calibration_mean": [
14
+ 123.675,
15
+ 116.28,
16
+ 103.53
17
+ ],
18
+ "calibration_std": [
19
+ 58.395,
20
+ 57.12,
21
+ 57.375
22
+ ]
23
+ }
24
+ ],
25
+ "calibration_method": "Percentile",
26
+ "precision_analysis": true,
27
+ "precision_analysis_method": "EndToEnd",
28
+ "conv_bias_data_type": "FP32",
29
+ "layer_configs": [
30
+ {
31
+ "start_tensor_names": [
32
+ "DEFAULT"
33
+ ],
34
+ "end_tensor_names": [
35
+ "DEFAULT"
36
+ ],
37
+ "data_type": "U16"
38
+ }
39
+ ]
40
+ },
41
+ "input_processors": [
42
+ {
43
+ "tensor_name": "DEFAULT",
44
+ "tensor_format": "RGB",
45
+ "src_format": "RGB",
46
+ "src_dtype": "U8",
47
+ "src_layout": "NCHW"
48
+ }
49
+ ],
50
+ "compiler": {
51
+ "check": 0
52
+ }
53
+ }
host.jpg ADDED

Git LFS Details

  • SHA256: d9fe245622dc52d9c977eb501aa70d23a9e0fb61e2873879db6c0a89b1bc6b88
  • Pointer size: 131 Bytes
  • Size of remote file: 158 kB
host_depth.jpg ADDED

Git LFS Details

  • SHA256: 4af7ef6f4e114398f2f871d3e259eb0acf8036bf268e8386e84cca4767a4aae4
  • Pointer size: 131 Bytes
  • Size of remote file: 158 kB
models/relative_s_448_default.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4773e2648803d207c470c86633c3059fd792bc87c5fdffce817005f6711abf06
3
+ size 24544478
models/relative_s_448_default_ax650.axmodel ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0e0082e21d0e5a42a37f615702bfe88c0718b0fa65bb29ef71b07086e7ec73e8
3
+ size 11560983
src/infer/__pycache__/depthart_infer_axmodel.cpython-310.pyc ADDED
Binary file (25.1 kB). View file
 
src/infer/__pycache__/depthart_infer_onnx.cpython-310.pyc ADDED
Binary file (29.2 kB). View file
 
src/infer/depthart_infer_axmodel.py ADDED
@@ -0,0 +1,639 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # -*- coding: utf-8 -*-
3
+ """
4
+ DepthART 相对深度 axmodel 板端推理(AX650 + axengine)。
5
+
6
+ 随板部署文件(建议放在同一目录):
7
+ relative_s_448_default_ax650.axmodel
8
+ depthart_infer_axmodel.py
9
+ 待测图片或图片目录
10
+
11
+ 模型接口(以运行时 get_inputs/get_outputs 实际返回为准):
12
+ input [1, 3, 448, 448] uint8 NCHW RGB
13
+ depth [1, 1, 448, 448] float32 仿射不变相对深度,数值越大表示越近
14
+
15
+ 预处理按编译配置 config_axonnx.json 的 input_processors 执行:tensor_format=RGB、
16
+ src_dtype=U8、mean/std 为 0-255 域的 ImageNet 值,归一化由编译前端完成。因此板端
17
+ 送 0-255 原始 RGB 像素即可,不要再做 /255 或减均值。若 axmodel 输入为 FP32(编译
18
+ 时未配置 input_processors),脚本默认改为 /255 + ImageNet mean/std,与 ONNX 基准
19
+ 一致;可用 --input-norm 显式覆盖。
20
+
21
+ 深度还原使用与 ONNX 脚本一致的 bilinear + align_corners=True(linspace + remap),
22
+ 保证板端与开发机基准可以逐像素对比。
23
+
24
+ 板端计时分为 NPU(session.run)和 CPU(读取、预处理、输出提取/校验、后处理、
25
+ 可视化、写盘),两者分别统计,避免弱 CPU 上的后处理掩盖真实 NPU 性能。
26
+
27
+ 单图模式:打印深度统计、NPU/CPU 各阶段耗时,保存并排可视化,不产出评测文件。
28
+ 数据集模式:tqdm 进度 + npu_timing.json;默认快速模式只统计读取/预处理/NPU,
29
+ --full-output 才做深度还原、可视化、原始 NPZ 和图片写盘。
30
+
31
+ 板端用法(所有路径均为必填):
32
+ # 单图验证
33
+ python3 depthart_infer_axmodel.py --axmodel relative_s_448_default_ax650.axmodel \
34
+ --image ./host.jpg --output ./results/ax_single
35
+
36
+ # 数据集快速性能模式
37
+ python3 depthart_infer_axmodel.py --axmodel relative_s_448_default_ax650.axmodel \
38
+ --dataset ./images --output ./results/board
39
+
40
+ # 数据集完整输出(原始深度 NPZ、可视化和大图深度)
41
+ python3 depthart_infer_axmodel.py --axmodel relative_s_448_default_ax650.axmodel \
42
+ --dataset ../../test_depth --output ./results/board_full --full-output
43
+
44
+ # 与开发机 ONNX 基准对比(--ref-npz 指向 ONNX 脚本的汇总 NPZ)
45
+ python3 depthart_infer_axmodel.py --axmodel relative_s_448_default_ax650.axmodel \
46
+ --dataset ../../test_depth --output ./results/board_check --ref-npz ./onnx_npz
47
+
48
+ 开发机用法(仅用于脚本语法/参数检查,实际推理需要板端 axengine):
49
+ ../../venv/bin/python src/infer/depthart_infer_axmodel.py \
50
+ --axmodel output_axonnx3/relative_s_448_default_ax650.axmodel \
51
+ --dataset /path/images --output results/axmodel_board --limit 50
52
+
53
+ 诊断:
54
+ --debug 打印输入/输出 tensor 的 shape、dtype、min、max、mean、nan 计数
55
+ --dump-npz 单图时为 .npz 文件路径;数据集时为汇总 .npz 文件路径,
56
+ 保存 names/depths/original_sizes,不再每张图一个小文件
57
+ --npz-chunk 数据集模式下每 N 张切一个文件(xxx_0000.npz 等),
58
+ 0 表示全部写入同一个文件
59
+ --ref-npz 单图时为参考 .npz,数据集时为参考汇总 .npz(按 names 匹配),
60
+ 报告 max/mean/p99 绝对误差、cosine 以及 scale-shift 对齐后误差
61
+ """
62
+
63
+ import argparse
64
+ import json
65
+ import logging
66
+ import time
67
+ from pathlib import Path
68
+
69
+ import cv2
70
+ import numpy as np
71
+ from tqdm import tqdm
72
+
73
+ MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
74
+ STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
75
+ EXTENSIONS = {'.jpg', '.jpeg', '.png', '.bmp', '.tif', '.tiff'}
76
+ STAGES = ('read', 'preprocess', 'npu', 'diagnostics',
77
+ 'postprocess', 'visualize', 'write', 'total')
78
+
79
+
80
+ def meta_type(meta):
81
+ value = getattr(meta, 'type', None)
82
+ if value is None:
83
+ value = getattr(meta, 'dtype', '')
84
+ return str(value).lower()
85
+
86
+
87
+ def input_spec(meta):
88
+ """从运行时输入元信息推导尺寸、layout 和 dtype,不依赖固定名称。"""
89
+ try:
90
+ shape = [int(v) for v in meta.shape]
91
+ except (TypeError, ValueError):
92
+ raise ValueError(f'输入必须是静态 4D NCHW/NHWC: {meta.name} {meta.shape}')
93
+ if len(shape) != 4 or any(v <= 0 for v in shape):
94
+ raise ValueError(f'输入必须是静态 4D NCHW/NHWC: {meta.name} {meta.shape}')
95
+ if shape[0] != 1:
96
+ raise ValueError(f'仅支持 batch=1: {meta.name} {meta.shape}')
97
+ if shape[1] == 3:
98
+ layout, size = 'NCHW', (shape[2], shape[3])
99
+ elif shape[3] == 3:
100
+ layout, size = 'NHWC', (shape[1], shape[2])
101
+ else:
102
+ raise ValueError(f'输入必须含 3 通道: {meta.name} {meta.shape}')
103
+ type_name = meta_type(meta)
104
+ if 'float16' in type_name or 'fp16' in type_name:
105
+ dtype = np.float16
106
+ elif 'float' in type_name or 'fp32' in type_name:
107
+ dtype = np.float32
108
+ elif 'uint8' in type_name or type_name.endswith('u8'):
109
+ dtype = np.uint8
110
+ elif 'int8' in type_name or type_name.endswith('s8'):
111
+ dtype = np.int8
112
+ else:
113
+ raise ValueError(f'不支持的输入类型: {meta.name} {type_name!r}')
114
+ return size, layout, dtype
115
+
116
+
117
+ def preprocess(image, size, color_order, layout, dtype, norm_mode):
118
+ """按编译契约生成输入 tensor。
119
+
120
+ uint8 直送路径沿用官方 axmodel 流程,在 uint8 域直接 resize;ImageNet
121
+ 归一化路径严格按 ONNX 基准的顺序(float 域 /255 -> resize -> mean/std)。
122
+ """
123
+ target_h, target_w = size
124
+ if color_order == 'rgb':
125
+ image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
126
+ if norm_mode == 'imagenet':
127
+ canvas = image.astype(np.float32) / 255.0
128
+ canvas = cv2.resize(canvas, (target_w, target_h),
129
+ interpolation=cv2.INTER_CUBIC)
130
+ canvas = (canvas - MEAN) / STD
131
+ elif dtype != np.uint8:
132
+ canvas = cv2.resize(image.astype(np.float32), (target_w, target_h),
133
+ interpolation=cv2.INTER_CUBIC)
134
+ else:
135
+ canvas = cv2.resize(image, (target_w, target_h),
136
+ interpolation=cv2.INTER_CUBIC).astype(np.float32)
137
+ if dtype == np.uint8:
138
+ canvas = np.clip(np.rint(canvas), 0, 255).astype(np.uint8)
139
+ elif dtype == np.int8:
140
+ raise ValueError('当前脚本不支持 INT8 输入:缺少反量化 scale/zero-point')
141
+ else:
142
+ canvas = canvas.astype(dtype)
143
+ tensor = canvas.transpose(2, 0, 1) if layout == 'NCHW' else canvas
144
+ return np.ascontiguousarray(tensor[None])
145
+
146
+
147
+ def select_output(outputs, metas, input_size):
148
+ """axmodel 输出名可能被编译器改写:先按名字,再按 shape 回退。"""
149
+ if not isinstance(outputs, (list, tuple)):
150
+ outputs = [outputs]
151
+ if len(outputs) != len(metas):
152
+ raise ValueError(f'输出数量不一致: run 返回 {len(outputs)},'
153
+ f'接口声明 {len(metas)}')
154
+ for index, meta in enumerate(metas):
155
+ if 'depth' in str(meta.name).lower():
156
+ return outputs[index], str(meta.name)
157
+ candidates = []
158
+ for index, array in enumerate(outputs):
159
+ shape = np.asarray(array).shape
160
+ if len(shape) == 4 and shape[1] == 1 and tuple(shape[2:]) == tuple(input_size):
161
+ candidates.append(index)
162
+ elif len(shape) == 3 and tuple(shape[1:]) == tuple(input_size):
163
+ candidates.append(index)
164
+ if len(candidates) != 1:
165
+ actual = [(m.name, np.asarray(o).shape) for m, o in zip(metas, outputs)]
166
+ raise ValueError(f'无法按 name 或 shape 唯一匹配 depth 输出;实际输出: {actual}')
167
+ return outputs[candidates[0]], str(metas[candidates[0]].name)
168
+
169
+
170
+ def read_depth(raw, input_size):
171
+ array = np.asarray(raw)
172
+ if array.ndim == 4 and array.shape[1] == 1:
173
+ array = array[0]
174
+ if array.ndim == 3 and array.shape[0] == 1:
175
+ array = array[0]
176
+ if array.shape != tuple(input_size):
177
+ raise ValueError(f'depth 输出无效: shape={array.shape},'
178
+ f'期望 {tuple(input_size)}')
179
+ if not np.issubdtype(array.dtype, np.floating):
180
+ raise ValueError(f'depth 输出应为浮点,实际 dtype={array.dtype},'
181
+ '整数输出需要已知反量化参数')
182
+ array = array.astype(np.float32, copy=False)
183
+ if not np.isfinite(array).all():
184
+ raise ValueError('depth 输出包含 nan/inf')
185
+ if float(array.std()) < 1e-6:
186
+ logging.warning('depth 输出近似常数(std=%.3g):请检查输入归一化/颜色顺序'
187
+ '是否与编译配置的 input_processors 一致。', float(array.std()))
188
+ return np.ascontiguousarray(array)
189
+
190
+
191
+ def restore_depth(depth, original_size):
192
+ """按官方推理的 bilinear + align_corners=True 还原到原图尺寸。"""
193
+ height, width = original_size
194
+ src_h, src_w = depth.shape
195
+ if (src_h, src_w) == (height, width):
196
+ return depth.copy()
197
+ xs = np.linspace(0.0, src_w - 1, width, dtype=np.float32)
198
+ ys = np.linspace(0.0, src_h - 1, height, dtype=np.float32)
199
+ map_x, map_y = np.meshgrid(xs, ys)
200
+ return cv2.remap(depth, map_x, map_y, interpolation=cv2.INTER_LINEAR,
201
+ borderMode=cv2.BORDER_REPLICATE)
202
+
203
+
204
+ def visualize(image, depth):
205
+ finite = depth[np.isfinite(depth)]
206
+ low, high = np.percentile(finite, (2, 98))
207
+ value = np.clip((depth - low) / max(float(high - low), 1e-6), 0, 1)
208
+ colored = cv2.applyColorMap((value * 255).astype(np.uint8), cv2.COLORMAP_INFERNO)
209
+ panel = np.concatenate((image, colored), axis=1)
210
+ legend = np.full((56, panel.shape[1], 3), 255, dtype=np.uint8)
211
+ ramp = np.tile(np.linspace(0, 255, max(panel.shape[1] - 20, 1),
212
+ dtype=np.uint8), (12, 1))
213
+ legend[5:17, 10:10 + ramp.shape[1]] = cv2.applyColorMap(ramp, cv2.COLORMAP_INFERNO)
214
+ cv2.putText(legend, f'Image | Relative depth (2-98%): {low:.3f} -> {high:.3f}',
215
+ (10, 42), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (30, 30, 30), 1, cv2.LINE_AA)
216
+ return np.concatenate((panel, legend), axis=0)
217
+
218
+
219
+ def log_tensor(name, value):
220
+ logging.info('%s: shape=%s dtype=%s min=%.6g max=%.6g mean=%.6g nan=%d inf=%d',
221
+ name, value.shape, value.dtype, float(np.min(value)),
222
+ float(np.max(value)), float(np.mean(value)),
223
+ int(np.isnan(value).sum()), int(np.isinf(value).sum()))
224
+
225
+
226
+ def compare_arrays(depth, expected, source='', preprocessing=''):
227
+ expected = np.asarray(expected)
228
+ if expected.ndim == 4 and expected.shape[1] == 1:
229
+ expected = expected[0]
230
+ if expected.shape != depth.shape:
231
+ raise ValueError(f'参考 depth shape 不符: {expected.shape} vs {depth.shape} ({source})')
232
+ if not np.isfinite(expected).all():
233
+ raise ValueError(f'参考 depth 含 nan/inf: {source}')
234
+ actual64 = depth.astype(np.float64).ravel()
235
+ expected64 = expected.astype(np.float64).ravel()
236
+ diff = np.abs(actual64 - expected64)
237
+ cosine = float(np.dot(actual64, expected64)
238
+ / (np.linalg.norm(actual64) * np.linalg.norm(expected64) + 1e-12))
239
+ # 相对深度存在全局 scale/offset 漂移,对齐后误差更能反映结构偏差。
240
+ slope, offset = np.polyfit(expected64, actual64, 1)
241
+ aligned = np.abs(actual64 - (slope * expected64 + offset))
242
+ metrics = {'cosine': cosine,
243
+ 'max_abs': float(diff.max()),
244
+ 'mean_abs': float(diff.mean()),
245
+ 'p99_abs': float(np.percentile(diff, 99)),
246
+ 'scale': float(slope), 'offset': float(offset),
247
+ 'aligned_mean_abs': float(aligned.mean()),
248
+ 'aligned_p99_abs': float(np.percentile(aligned, 99)),
249
+ 'reference_preprocessing': preprocessing}
250
+ logging.info('参考对比: %s', metrics)
251
+ return metrics
252
+
253
+
254
+ def compare_reference(depth, reference_path):
255
+ with np.load(reference_path, allow_pickle=False) as reference:
256
+ if 'depth' not in reference:
257
+ raise ValueError(f'参考 NPZ 缺少 depth: {reference_path}')
258
+ expected = np.asarray(reference['depth'])
259
+ preprocessing = (str(reference['preprocessing'])
260
+ if 'preprocessing' in reference else '')
261
+ return compare_arrays(depth, expected, str(reference_path), preprocessing)
262
+
263
+
264
+ def load_reference_dataset(path):
265
+ """读取汇总 NPZ,返回 {name: depth}。"""
266
+ path = Path(path)
267
+ if path.is_file():
268
+ with np.load(path, allow_pickle=False) as ref:
269
+ if 'depths' not in ref or 'names' not in ref:
270
+ raise ValueError(f'汇总 NPZ 需要 names/depths: {path}')
271
+ return {str(n): d for n, d in zip(ref['names'], ref['depths'])}
272
+ raise ValueError(f'数据集模式 --ref-npz 需要汇总 NPZ 文件: {path}')
273
+
274
+
275
+ def preprocessing_description(color_order, norm_mode):
276
+ norm = ('imagenet' if norm_mode == 'imagenet' else 'raw-0-255')
277
+ return f'color={color_order}; resize=stretch; norm={norm}'
278
+
279
+
280
+ def save_raw_npz(path, depth, original_size, input_size, color_order, norm_mode):
281
+ # 板端 CPU 弱,savez_compressed 的 DEFLATE 对 float32 收益极小却耗时明显,用未压缩保存。
282
+ np.savez(path, depth=depth,
283
+ original_size=np.array(original_size, dtype=np.int32),
284
+ input_size=np.array(input_size, dtype=np.int32),
285
+ preprocessing=preprocessing_description(color_order, norm_mode))
286
+
287
+
288
+ def flush_dataset_npz(path, records, chunk_index, input_size, color_order, norm_mode):
289
+ """把缓冲区写成汇总 NPZ:names / depths / original_sizes。
290
+
291
+ 板端弱 CPU 上 DEFLATE 对 float32 收益极小却耗时明显,这里保持未压缩。
292
+ chunk_index 为 None 时写入 path 本身,否则按 chunk 编号追加后缀。
293
+ """
294
+ if not records:
295
+ return
296
+ target = path if chunk_index is None else path.with_name(
297
+ f'{path.stem}_{chunk_index:04d}{path.suffix}')
298
+ target.parent.mkdir(parents=True, exist_ok=True)
299
+ np.savez(
300
+ target,
301
+ names=np.array([r[0] for r in records]),
302
+ depths=np.stack([r[1] for r in records]),
303
+ original_sizes=np.array([r[2] for r in records], dtype=np.int32),
304
+ input_size=np.array(input_size, dtype=np.int32),
305
+ preprocessing=preprocessing_description(color_order, norm_mode))
306
+
307
+
308
+ def collect_images(dataset):
309
+ paths = sorted(p for p in dataset.rglob('*')
310
+ if p.is_file() and p.suffix.lower() in EXTENSIONS)
311
+ if not paths:
312
+ raise ValueError(f'目录中没有图片: {dataset}')
313
+ return paths
314
+
315
+
316
+ def summarize(samples):
317
+ result = {}
318
+ for stage in STAGES:
319
+ values = np.array([sample[stage] for sample in samples], dtype=np.float64)
320
+ result[stage] = {'mean_ms': float(values.mean()),
321
+ 'p50_ms': float(np.percentile(values, 50)),
322
+ 'p90_ms': float(np.percentile(values, 90)),
323
+ 'p99_ms': float(np.percentile(values, 99))}
324
+ return result
325
+
326
+
327
+ def run_single(session, input_meta, output_metas, input_size, layout, dtype,
328
+ norm_mode, args):
329
+ stem = args.image.stem
330
+ timings = dict.fromkeys(STAGES, 0.0)
331
+ total_start = time.perf_counter()
332
+
333
+ start = time.perf_counter()
334
+ image = cv2.imread(str(args.image), cv2.IMREAD_COLOR)
335
+ if image is None:
336
+ raise ValueError(f'读取图片失败: {args.image}')
337
+ timings['read'] = (time.perf_counter() - start) * 1000
338
+
339
+ start = time.perf_counter()
340
+ tensor = preprocess(image, input_size, args.color_order, layout, dtype, norm_mode)
341
+ timings['preprocess'] = (time.perf_counter() - start) * 1000
342
+
343
+ start = time.perf_counter()
344
+ outputs = session.run(None, {input_meta.name: tensor})
345
+ timings['npu'] = (time.perf_counter() - start) * 1000
346
+
347
+ start = time.perf_counter()
348
+ raw, output_name = select_output(outputs, output_metas, input_size)
349
+ depth = read_depth(raw, input_size)
350
+ comparison = None
351
+ if args.debug:
352
+ log_tensor(input_meta.name, tensor)
353
+ log_tensor('depth', depth)
354
+ if args.ref_npz:
355
+ comparison = compare_reference(depth, args.ref_npz)
356
+ timings['diagnostics'] = (time.perf_counter() - start) * 1000
357
+
358
+ start = time.perf_counter()
359
+ restored = restore_depth(depth, image.shape[:2])
360
+ timings['postprocess'] = (time.perf_counter() - start) * 1000
361
+
362
+ start = time.perf_counter()
363
+ panel = visualize(image, restored)
364
+ timings['visualize'] = (time.perf_counter() - start) * 1000
365
+
366
+ start = time.perf_counter()
367
+ args.output.mkdir(parents=True, exist_ok=True)
368
+ image_path = args.output / f'{stem}_depth.jpg'
369
+ if not cv2.imwrite(str(image_path), panel, [cv2.IMWRITE_JPEG_QUALITY, 90]):
370
+ raise OSError(f'保存可视化失败: {image_path}')
371
+ np.save(str(args.output / f'{stem}_depth.npy'), restored)
372
+ npz_path = None
373
+ if args.dump_npz:
374
+ npz_path = Path(args.dump_npz)
375
+ npz_path.parent.mkdir(parents=True, exist_ok=True)
376
+ save_raw_npz(npz_path, depth, image.shape[:2], input_size,
377
+ args.color_order, norm_mode)
378
+ timings['write'] = (time.perf_counter() - start) * 1000
379
+ timings['total'] = (time.perf_counter() - total_start) * 1000
380
+
381
+ cpu_ms = (timings['read'] + timings['preprocess'] + timings['diagnostics']
382
+ + timings['postprocess'] + timings['visualize'] + timings['write'])
383
+ logging.info('深度(模型分辨率): min=%.4f max=%.4f mean=%.4f p50=%.4f',
384
+ float(depth.min()), float(depth.max()), float(depth.mean()),
385
+ float(np.percentile(depth, 50)))
386
+ logging.info('深度(原图分辨率): min=%.4f max=%.4f mean=%.4f',
387
+ float(restored.min()), float(restored.max()), float(restored.mean()))
388
+ logging.info('输出绑定: %s', output_name)
389
+ logging.info('NPU 推理: %.2f ms;CPU 处理: %.2f ms;端到端: %.2f ms'
390
+ '(单图为首次推理,含运行时首次执行开销;性能请用数据集模式)',
391
+ timings['npu'], cpu_ms, timings['total'])
392
+ logging.info('各阶段耗时: %s',
393
+ ' '.join(f'{key}={value:.2f} ms' for key, value in timings.items()))
394
+ if comparison is not None:
395
+ logging.info('参考对比结果: %s', comparison)
396
+ logging.info('结果目录: %s(可视化 %s%s)', args.output, image_path.name,
397
+ f',原始 NPZ {npz_path.name}' if npz_path else '')
398
+
399
+
400
+ def run_dataset(session, input_meta, output_metas, input_size, layout, dtype,
401
+ norm_mode, runtime_version, args):
402
+ images = collect_images(args.dataset)
403
+ if args.limit is not None:
404
+ images = images[:args.limit]
405
+ args.output.mkdir(parents=True, exist_ok=True)
406
+ full_output = args.full_output
407
+ if full_output:
408
+ logging.warning('--full-output: CPU 侧深度还原、可视化与写盘会拉高端到端耗时,'
409
+ '不能只看 NPU 时间。')
410
+ else:
411
+ logging.info('数据集快速模式:只统计 read/preprocess/npu,'
412
+ '跳过深度还原、可视化和结果写盘。')
413
+ ref_map = load_reference_dataset(args.ref_npz) if args.ref_npz else None
414
+ preprocessing = preprocessing_description(args.color_order, norm_mode)
415
+ dump_path = Path(args.dump_npz) if args.dump_npz else None
416
+ if dump_path is not None and dump_path.is_dir():
417
+ raise ValueError('数据集模式 --dump-npz 请传 .npz 文件路径,'
418
+ '不再按目录为每张图生成小文件')
419
+ dump_records, dump_chunk = [], 0
420
+
421
+ first = cv2.imread(str(images[0]), cv2.IMREAD_COLOR)
422
+ if first is None:
423
+ raise ValueError(f'读取图片失败: {images[0]}')
424
+ warm = preprocess(first, input_size, args.color_order, layout, dtype, norm_mode)
425
+ warmup_ms = 0.0
426
+ if args.warmup:
427
+ start = time.perf_counter()
428
+ for _ in range(args.warmup):
429
+ session.run(None, {input_meta.name: warm})
430
+ warmup_ms = (time.perf_counter() - start) * 1000
431
+
432
+ samples = []
433
+ sample_info = []
434
+ for path in tqdm(images, desc='AX 相对深度', unit='image'):
435
+ relative = path.relative_to(args.dataset)
436
+ timings = dict.fromkeys(STAGES, 0.0)
437
+ total_start = time.perf_counter()
438
+
439
+ start = time.perf_counter()
440
+ image = cv2.imread(str(path), cv2.IMREAD_COLOR)
441
+ if image is None:
442
+ raise ValueError(f'读取图片失败: {path}')
443
+ timings['read'] = (time.perf_counter() - start) * 1000
444
+
445
+ start = time.perf_counter()
446
+ tensor = preprocess(image, input_size, args.color_order, layout, dtype, norm_mode)
447
+ timings['preprocess'] = (time.perf_counter() - start) * 1000
448
+
449
+ start = time.perf_counter()
450
+ outputs = session.run(None, {input_meta.name: tensor})
451
+ timings['npu'] = (time.perf_counter() - start) * 1000
452
+
453
+ start = time.perf_counter()
454
+ raw, output_name = select_output(outputs, output_metas, input_size)
455
+ depth = read_depth(raw, input_size)
456
+ comparison = None
457
+ if args.debug and path == images[0]:
458
+ log_tensor(input_meta.name, tensor)
459
+ log_tensor('depth', depth)
460
+ if dump_path is not None:
461
+ dump_records.append((str(relative), depth,
462
+ np.array(image.shape[:2], dtype=np.int32)))
463
+ if args.npz_chunk and len(dump_records) >= args.npz_chunk:
464
+ flush_dataset_npz(dump_path, dump_records, dump_chunk,
465
+ input_size, args.color_order, norm_mode)
466
+ dump_records.clear()
467
+ dump_chunk += 1
468
+ if ref_map is not None:
469
+ key = str(relative)
470
+ if key not in ref_map:
471
+ raise ValueError(f'参考 NPZ 缺少样本: {key}')
472
+ comparison = compare_arrays(depth, ref_map[key],
473
+ f'{args.ref_npz}:{key}', preprocessing)
474
+ timings['diagnostics'] = (time.perf_counter() - start) * 1000
475
+
476
+ restored = None
477
+ if full_output:
478
+ start = time.perf_counter()
479
+ restored = restore_depth(depth, image.shape[:2])
480
+ timings['postprocess'] = (time.perf_counter() - start) * 1000
481
+ start = time.perf_counter()
482
+ panel = visualize(image, restored)
483
+ timings['visualize'] = (time.perf_counter() - start) * 1000
484
+ start = time.perf_counter()
485
+ prefix = args.output / relative
486
+ prefix.parent.mkdir(parents=True, exist_ok=True)
487
+ if not cv2.imwrite(str(prefix) + '.depth.jpg', panel,
488
+ [cv2.IMWRITE_JPEG_QUALITY, 90]):
489
+ raise OSError(f'保存可视化失败: {prefix}.depth.jpg')
490
+ timings['write'] = (time.perf_counter() - start) * 1000
491
+ timings['total'] = (time.perf_counter() - total_start) * 1000
492
+
493
+ samples.append(timings)
494
+ sample_info.append({'name': str(relative),
495
+ 'original_size': list(image.shape[:2]),
496
+ 'depth_min': float((restored if restored is not None
497
+ else depth).min()),
498
+ 'depth_max': float((restored if restored is not None
499
+ else depth).max()),
500
+ 'timing_ms': timings,
501
+ 'reference_comparison': comparison})
502
+
503
+ if dump_path is not None:
504
+ flush_dataset_npz(dump_path, dump_records,
505
+ None if not args.npz_chunk else dump_chunk,
506
+ input_size, args.color_order, norm_mode)
507
+
508
+ stats = summarize(samples)
509
+ total_ms = sum(sample['total'] for sample in samples)
510
+ cpu_stages = ('read', 'preprocess', 'diagnostics', 'postprocess', 'visualize', 'write')
511
+ cpu_total_ms = sum(stats[stage]['mean_ms'] for stage in cpu_stages)
512
+ report = {'model': str(Path(args.axmodel).resolve()),
513
+ 'runtime': runtime_version,
514
+ 'task': 'relative-depth',
515
+ 'input_name': input_meta.name, 'output_name': output_name,
516
+ 'input_size': list(input_size), 'input_layout': layout,
517
+ 'input_dtype': np.dtype(dtype).name,
518
+ 'color_order': args.color_order,
519
+ 'preprocessing': preprocessing_description(args.color_order, norm_mode),
520
+ 'batch_size': 1, 'sample_count': len(samples),
521
+ 'full_output': full_output, 'warmup_count': args.warmup,
522
+ 'warmup_ms': warmup_ms, 'total_ms': total_ms,
523
+ 'fps': len(samples) * 1000 / total_ms,
524
+ 'npu_mean_ms': stats['npu']['mean_ms'],
525
+ 'cpu_total_mean_ms': cpu_total_ms,
526
+ 'timing_scope': 'npu=仅 session.run;cpu=读取+预处理+输出提取/校验+'
527
+ '诊断+后处理+可视化+写盘;total=单样本端到端。排除'
528
+ '模型加载、session 初始化和 warmup。快速模式的 '
529
+ 'postprocess/visualize/write 记为 0。',
530
+ 'stages': stats, 'samples': sample_info}
531
+ with (args.output / 'npu_timing.json').open('w', encoding='utf-8') as file:
532
+ json.dump(report, file, ensure_ascii=False, indent=2, allow_nan=False)
533
+ for stage, values in stats.items():
534
+ logging.info('%s: mean=%.2f p50=%.2f p90=%.2f p99=%.2f ms',
535
+ stage, values['mean_ms'], values['p50_ms'],
536
+ values['p90_ms'], values['p99_ms'])
537
+ logging.info('%d 张图片,NPU mean=%.2f ms,CPU mean=%.2f ms,'
538
+ 'total=%.2f s,FPS=%.3f,结果目录: %s',
539
+ len(samples), stats['npu']['mean_ms'], cpu_total_ms,
540
+ total_ms / 1000, report['fps'], args.output)
541
+
542
+
543
+ def main():
544
+ parser = argparse.ArgumentParser(
545
+ description='DepthART 相对深度 axmodel 板端推理(AX650)',
546
+ formatter_class=argparse.RawDescriptionHelpFormatter, epilog=__doc__)
547
+ parser.add_argument('--axmodel', type=Path, required=True, help='编译后的 axmodel')
548
+ source = parser.add_mutually_exclusive_group(required=True)
549
+ source.add_argument('--image', type=Path, help='单张图片')
550
+ source.add_argument('--dataset', type=Path, help='图片目录(递归)')
551
+ parser.add_argument('--output', type=Path, required=True, help='结果目录')
552
+ parser.add_argument('--color-order', choices=('rgb', 'bgr'), default='rgb',
553
+ help='rgb=与官方预处理一致(BGR->RGB);bgr=按编译配置'
554
+ 'input_processors.src_format 直送')
555
+ parser.add_argument('--input-norm', choices=('auto', 'raw', 'imagenet'),
556
+ default='auto',
557
+ help='auto=U8 送 0-255、浮点用 ImageNet mean/std;'
558
+ 'raw=直送 0-255;imagenet=/255+mean/std(仅浮点输入)')
559
+ parser.add_argument('--warmup', type=int, default=3,
560
+ help='数据集模式 warmup 次数,默认 3')
561
+ parser.add_argument('--limit', type=int, help='数据集模式只处理前 N 张')
562
+ parser.add_argument('--full-output', action='store_true',
563
+ help='数据集模式执行完整后处理、可视化和结果写盘(含 NPZ)')
564
+ parser.add_argument('--debug', action='store_true', help='打印输入/输出统计')
565
+ parser.add_argument('--dump-npz',
566
+ help='单图:NPZ 文件;数据集:汇总 NPZ 文件(names/depths/original_sizes)')
567
+ parser.add_argument('--npz-chunk', type=int, default=0,
568
+ help='数据集模式每 N 张切一个 NPZ 文件,0 表示全部写入一个文件')
569
+ parser.add_argument('--ref-npz', type=Path, help='参考 NPZ 文件或目录')
570
+ args = parser.parse_args()
571
+ logging.basicConfig(level=logging.INFO, format='%(message)s')
572
+ if args.warmup < 0:
573
+ parser.error('--warmup 不能为负数')
574
+ if args.npz_chunk < 0:
575
+ parser.error('--npz-chunk 不能为负数')
576
+ if args.limit is not None and args.limit < 1:
577
+ parser.error('--limit 必须为正数')
578
+ if args.image:
579
+ if not args.image.is_file():
580
+ parser.error(f'--image 不是文件: {args.image}')
581
+ if args.ref_npz and not args.ref_npz.is_file():
582
+ parser.error('单图模式 --ref-npz 必须是文件')
583
+ if args.limit is not None or args.full_output:
584
+ logging.warning('单图模式忽略 --limit/--full-output')
585
+ else:
586
+ if not args.dataset.is_dir():
587
+ parser.error(f'--dataset 不是目录: {args.dataset}')
588
+
589
+ try:
590
+ import axengine
591
+ except ImportError as error:
592
+ raise SystemExit('未找到 axengine:该脚本需在板端运行(pip 安装 axengine/'
593
+ 'pyaxengine 后重试)') from error
594
+
595
+ session = axengine.InferenceSession(str(args.axmodel))
596
+ inputs = session.get_inputs()
597
+ output_metas = session.get_outputs()
598
+ logging.info('模型输入: %s', [(m.name, m.shape, meta_type(m)) for m in inputs])
599
+ logging.info('模型输出: %s', [(m.name, m.shape, meta_type(m)) for m in output_metas])
600
+ if len(inputs) != 1:
601
+ raise ValueError(f'模型应有 1 个输入,实际 '
602
+ f'{[(m.name, m.shape) for m in inputs]}')
603
+ input_meta = inputs[0]
604
+ input_size, layout, dtype = input_spec(input_meta)
605
+ if dtype == np.int8:
606
+ raise ValueError('当前脚本不支持 INT8 输入(缺少反量化参数);请把编译配置的 '
607
+ 'input_processors.src_dtype 设为 U8 后重编译')
608
+
609
+ if args.input_norm == 'auto':
610
+ norm_mode = 'raw' if dtype == np.uint8 else 'imagenet'
611
+ else:
612
+ norm_mode = args.input_norm
613
+ if dtype == np.uint8 and norm_mode == 'imagenet':
614
+ raise ValueError('输入类型为 uint8,不能在推理时做 ImageNet 归一化;'
615
+ '归一化必须由编译前端 input_processors 或校准口径保证')
616
+ if dtype != np.uint8 and norm_mode == 'raw':
617
+ logging.warning('输入为浮点但按 0-255 直送(--input-norm raw),'
618
+ '仅用于诊断,需与编译前端/校准口径一致。')
619
+
620
+ logging.info('batch=1,输入尺寸: %s,layout=%s,dtype=%s,颜色顺序=%s,'
621
+ '预处理=%s,resize=stretch', input_size, layout,
622
+ np.dtype(dtype).name, args.color_order,
623
+ 'ImageNet mean/std (/255)' if norm_mode == 'imagenet'
624
+ else '0-255 原始像素')
625
+ if dtype == np.uint8 and norm_mode == 'raw':
626
+ logging.info('提示:uint8 输入按编译契约送 0-255;ImageNet 归一化应由 '
627
+ 'config_axonnx.json 的 input_processors 在模型内完成。')
628
+
629
+ runtime_version = getattr(axengine, '__version__', 'unknown')
630
+ if args.image:
631
+ run_single(session, input_meta, output_metas, input_size, layout, dtype,
632
+ norm_mode, args)
633
+ else:
634
+ run_dataset(session, input_meta, output_metas, input_size, layout, dtype,
635
+ norm_mode, runtime_version, args)
636
+
637
+
638
+ if __name__ == '__main__':
639
+ main()
src/infer/depthart_infer_onnx.py ADDED
@@ -0,0 +1,721 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # -*- coding: utf-8 -*-
3
+ """
4
+ DepthART 相对深度 ONNX 推理(开发机 FP32 基准)。
5
+
6
+ 模型接口(脚本按实际 ONNX 读取并校验):
7
+ input [1, 3, 448, 448] float32 NCHW RGB
8
+ depth [1, 1, 448, 448] float32 仿射不变相对深度,数值越大表示越近
9
+
10
+ 预处理按官方相对深度推理流程:BGR->RGB,直接拉伸到 448x448(INTER_CUBIC),
11
+ /255 后按 ImageNet mean/std 归一化。归一化必须保留,否则输出会退化。
12
+
13
+ 自定义算子:
14
+ relative_s_448_default.onnx 含 com.depthart::SelectiveScan(官方只提供
15
+ TensorRT 插件),onnxruntime 无法直接加载。本脚本在内存中把每个
16
+ SelectiveScan 改写为等价的纯 ONNX 子图(Loop + 逐 t 递归),语义与官方
17
+ reference 实现一致(含 delta_bias/softplus、按组 repeat_interleave)。
18
+ 改写后整模型输出与 PyTorch 参考的最大绝对误差约 5e-5。
19
+ --save-rewritten 可导出改写后的标准 ONNX 便于排查。
20
+ 注意:pulsar2 不支持 Loop,要编译 axmodel 请用
21
+ src/tool/depthart_export_axonnx.py 生成分块并行扫描版本。
22
+
23
+ 单图模式:打印深度统计和各阶段耗时,保存并排可视化,不产出评测文件。
24
+ 数据集模式:tqdm 进度 + onnx_timing.json;默认快速模式只统计读取/预处理/推理,
25
+ --full-output 才做深度还原、可视化、原始 NPZ 和图片写盘。
26
+
27
+ 用法(在 hs/DepthART 目录下执行):
28
+ # 单图验证,输出 results/onnx_demo/xxx_depth.jpg
29
+ ../../venv/bin/python src/infer/depthart_infer_onnx.py --image /path/a.jpg --output results/onnx_demo
30
+
31
+ # 指定其他尺寸/变的模型
32
+ ../../venv/bin/python src/infer/depthart_infer_onnx.py --model onnx/relative_b_448_default.onnx --image /path/a.jpg
33
+
34
+ # 数据集快速性能模式
35
+ ../../venv/bin/python src/infer/depthart_infer_onnx.py --dataset /path/images --output results/onnx_dataset --limit 50
36
+
37
+ # 数据集完整输出(可视化 + 汇总 NPZ)
38
+ ../../venv/bin/python src/infer/depthart_infer_onnx.py --dataset ../../datasets/hs_ai_tv/test_depth \
39
+ --output results/onnx_full --full-output --dump-npz results/onnx_full/raw_outputs.npz
40
+
41
+ # 与参考原始输出对比(单图,--ref-npz 指向单图 NPZ)
42
+ ../../venv/bin/python src/infer/depthart_infer_onnx.py --image /path/a.jpg --output results/onnx_check --ref-npz results/onnx_demo/a.jpg.npz
43
+
44
+ # 导出改写后的标准 ONNX(保留原始文件不变)
45
+ ../../venv/bin/python src/infer/depthart_infer_onnx.py --image /path/a.jpg --save-rewritten results/relative_s_448_default_std.onnx
46
+
47
+ 诊断:
48
+ --debug 打印输入/输出 tensor 的 shape、dtype、min、max、mean、nan 计数,
49
+ 以及自定义算子的改写数量
50
+ --dump-npz 单图时为 .npz 文件路径;数据集时为汇总 .npz 文件路径,
51
+ 保存 names/depths/original_sizes,不再每张图一个小文件
52
+ --npz-chunk 数据集模式下每 N 张切一个文件(xxx_0000.npz 等),
53
+ 0 表示全部写入同一个文件
54
+ --ref-npz 单图时为参考 .npz,数据集时为参考汇总 .npz(按 names 匹配),
55
+ 报告 max/mean/p99 绝对误差、cosine 以及 scale-shift 对齐后误差
56
+ """
57
+
58
+ import argparse
59
+ import json
60
+ import logging
61
+ import time
62
+ from pathlib import Path
63
+
64
+ import cv2
65
+ import numpy as np
66
+ import onnx
67
+ import onnxruntime as ort
68
+ from onnx import helper, numpy_helper, TensorProto
69
+ from tqdm import tqdm
70
+
71
+ PROJECT_ROOT = Path(__file__).resolve().parents[2]
72
+ DEFAULT_MODEL = PROJECT_ROOT / 'onnx' / 'relative_s_448_default.onnx'
73
+ MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
74
+ STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
75
+ EXTENSIONS = {'.jpg', '.jpeg', '.png', '.bmp', '.tif', '.tiff'}
76
+ STAGES = ('read', 'preprocess', 'inference', 'diagnostics',
77
+ 'postprocess', 'visualize', 'write', 'total')
78
+
79
+ SCAN_DOMAIN = 'com.depthart'
80
+ SCAN_OP = 'SelectiveScan'
81
+
82
+
83
+ # ---------------------------------------------------------------------------
84
+ # 自定义算子改写:com.depthart::SelectiveScan -> 纯 ONNX
85
+ # ---------------------------------------------------------------------------
86
+
87
+ class _ScanGraph:
88
+ """为一个 SelectiveScan 节点生成改写子图。
89
+
90
+ 语义(与官方 reference 一致,u 为 [B, D, L]):
91
+ delta = softplus(delta + delta_bias) # 可选前两项
92
+ transition = exp(delta_t * A) # [B, D, N]
93
+ state_t = transition * state_{t-1}
94
+ + delta_t * B_t * u_t # B/C 按组 repeat_interleave
95
+ y_t = sum_n(state_t * C_t) + D * u_t
96
+ 组映射:group = channel // (D / G),即把 [B, G, N, L] 的每个组连续复制
97
+ D/G 次到 [B, D, N, L]。
98
+ """
99
+
100
+ def __init__(self, prefix):
101
+ self.prefix = prefix
102
+ self.nodes = []
103
+ self.inits = []
104
+ self._cache = {}
105
+
106
+ def const(self, name, value):
107
+ tensor_name = f'{self.prefix}_{name}'
108
+ if tensor_name not in self._cache:
109
+ self.inits.append(numpy_helper.from_array(np.asarray(value), tensor_name))
110
+ self._cache[tensor_name] = True
111
+ return tensor_name
112
+
113
+ def node(self, op, inputs, outputs, **kwargs):
114
+ self.nodes.append(helper.make_node(op, inputs, outputs, **kwargs))
115
+ return outputs[0]
116
+
117
+ def build(self, u, delta, a, b, c, d, bias, attrs, out_name):
118
+ p = self.prefix
119
+ one = self.const('one', np.array([1], dtype=np.int64))
120
+
121
+ # delta (+ bias) -> softplus
122
+ dt = delta
123
+ if bias is not None:
124
+ bias3 = self.node('Unsqueeze', [bias, self.const('uax02', [0, 2])], [f'{p}_bias3'])
125
+ dt = self.node('Add', [delta, bias3], [f'{p}_dt_bias'])
126
+ if int(attrs.get('delta_softplus', 0)):
127
+ dt = self.node('Softplus', [dt], [f'{p}_dt_softplus'])
128
+
129
+ # 形状:D = u.shape[1],G = B.shape[1],gs = D / G,L = u.shape[2]
130
+ u_shape = self.node('Shape', [u], [f'{p}_u_shape'])
131
+ b_shape = self.node('Shape', [b], [f'{p}_b_shape'])
132
+ d_vec = self.node('Gather', [u_shape, self.const('i1', [1])], [f'{p}_d'], axis=0)
133
+ l_vec = self.node('Gather', [u_shape, self.const('i2', [2])], [f'{p}_l'], axis=0)
134
+ g_vec = self.node('Gather', [b_shape, self.const('i1b', [1])], [f'{p}_g'], axis=0)
135
+ n_vec = self.node('Gather', [b_shape, self.const('i2b', [2])], [f'{p}_n'], axis=0)
136
+ l_vec_b = self.node('Gather', [b_shape, self.const('i3b', [3])], [f'{p}_lb'], axis=0)
137
+ gs_vec = self.node('Div', [d_vec, g_vec], [f'{p}_gs'])
138
+ l_scalar = self.node('Gather', [b_shape, self.const('i3s', 3)], [f'{p}_l_scalar'], axis=0)
139
+
140
+ def expand_groups(x, tag):
141
+ # [B,G,N,L] -> [B,G,1,N,L] -> [B,G,gs,N,L] -> [B,D,N,L]
142
+ x5 = self.node('Unsqueeze', [x, self.const(f'{tag}_uax', [2])], [f'{p}_{tag}_x5'])
143
+ target5 = self.node('Concat', [one, g_vec, gs_vec, n_vec, l_vec_b],
144
+ [f'{p}_{tag}_target5'], axis=0)
145
+ expanded = self.node('Expand', [x5, target5], [f'{p}_{tag}_expanded'])
146
+ target4 = self.node('Concat', [one, d_vec, n_vec, l_vec],
147
+ [f'{p}_{tag}_target4'], axis=0)
148
+ return self.node('Reshape', [expanded, target4], [f'{p}_{tag}_4d'])
149
+
150
+ b_full = expand_groups(b, 'b')
151
+ c_full = expand_groups(c, 'c')
152
+
153
+ # 初始状态 [1, D, N]
154
+ state_shape = self.node('Concat', [one, d_vec, n_vec], [f'{p}_state_shape'], axis=0)
155
+ state0 = self.node('ConstantOfShape', [state_shape], [f'{p}_state0'],
156
+ value=helper.make_tensor('value', TensorProto.FLOAT, [1], [0.0]))
157
+
158
+ # Loop body:按时间步推进状态,扫描输出自动在轴 0 堆叠
159
+ bp = f'{p}_body'
160
+ body_inputs = [helper.make_tensor_value_info(bp + '_iter', TensorProto.INT64, []),
161
+ helper.make_tensor_value_info(bp + '_cond', TensorProto.BOOL, []),
162
+ helper.make_tensor_value_info(bp + '_state', TensorProto.FLOAT, None)]
163
+ body_outputs = [helper.make_tensor_value_info(bp + '_cond_out', TensorProto.BOOL, []),
164
+ helper.make_tensor_value_info(bp + '_state_out', TensorProto.FLOAT, None),
165
+ helper.make_tensor_value_info(bp + '_y', TensorProto.FLOAT, None)]
166
+ body = _ScanGraph.body_graph(p, bp, u, dt, b_full, c_full, a, d, body_inputs, body_outputs)
167
+
168
+ final_state = f'{p}_final_state'
169
+ y_seq = f'{p}_y_seq'
170
+ self.node('Loop', [l_scalar, self.const('cond', np.array(True)), state0],
171
+ [final_state, y_seq], body=body, name=f'{p}_loop')
172
+ # y_seq: [L,1,D,1] -> [1,D,L]
173
+ y_sq = self.node('Squeeze', [y_seq, self.const('sq_axes', [1, 3])], [f'{p}_y_sq'])
174
+ y_tr = self.node('Transpose', [y_sq], [f'{p}_y_tr'], perm=[1, 0])
175
+ self.node('Unsqueeze', [y_tr, self.const('uax_out', [0])], [out_name])
176
+
177
+ @staticmethod
178
+ def body_graph(p, bp, u, dt, b_full, c_full, a, d, body_inputs, body_outputs):
179
+ """构造 Loop body;外层张量通过闭包名字被 body 引用。"""
180
+ g = _ScanGraph(p)
181
+ iter_name, cond_name, state_name = (x.name for x in body_inputs)
182
+ cond_out, state_out, y_out = (x.name for x in body_outputs)
183
+
184
+ u_t = g.node('Gather', [u, iter_name], [bp + '_u_t'], axis=2) # [1,D]
185
+ dt_t = g.node('Gather', [dt, iter_name], [bp + '_dt_t'], axis=2) # [1,D]
186
+ b_t = g.node('Gather', [b_full, iter_name], [bp + '_b_t'], axis=3) # [1,D,N]
187
+ c_t = g.node('Gather', [c_full, iter_name], [bp + '_c_t'], axis=3) # [1,D,N]
188
+ dt3 = g.node('Unsqueeze', [dt_t, g.const('uax2', [2])], [bp + '_dt3'])
189
+ u3 = g.node('Unsqueeze', [u_t, g.const('uax2b', [2])], [bp + '_u3'])
190
+ a3 = g.node('Unsqueeze', [a, g.const('uax0', [0])], [bp + '_a3'])
191
+ trans = g.node('Exp', [g.node('Mul', [dt3, a3], [bp + '_dt_a'])], [bp + '_trans'])
192
+ state_keep = g.node('Mul', [trans, state_name], [bp + '_state_keep'])
193
+ bu = g.node('Mul', [g.node('Mul', [dt3, u3], [bp + '_dt_u']), b_t], [bp + '_dt_u_b'])
194
+ state_new = g.node('Add', [state_keep, bu], [bp + '_state_new'])
195
+ y = g.node('ReduceSum', [g.node('Mul', [state_new, c_t], [bp + '_state_c']),
196
+ g.const('rs_axes', [2])], [bp + '_y_val'], keepdims=0)
197
+ if d is not None:
198
+ y = g.node('Add', [y, g.node('Mul', [d, u_t], [bp + '_d_u'])], [bp + '_y_d'])
199
+ y3 = g.node('Unsqueeze', [y, g.const('uax2c', [2])], [bp + '_y3'])
200
+ g.node('Identity', [cond_name], [cond_out])
201
+ g.node('Identity', [state_new], [state_out])
202
+ g.node('Identity', [y3], [y_out])
203
+ return helper.make_graph(g.nodes, bp + '_graph', body_inputs, body_outputs,
204
+ initializer=g.inits)
205
+
206
+
207
+ def _static_rank(value_info_map, name):
208
+ info = value_info_map.get(name)
209
+ if info is None or not info.type.HasField('tensor_type'):
210
+ return None
211
+ return len(info.type.tensor_type.shape.dim)
212
+
213
+
214
+ def rewrite_selective_scan(model):
215
+ """把 com.depthart::SelectiveScan 替换成纯 ONNX 子图,返回替换数量。
216
+
217
+ 仅处理 domain 为 com.depthart、op_type 为 SelectiveScan 的节点;
218
+ 其余自定义 domain 节点保持原样,由调用方检查后报错。
219
+ """
220
+ try:
221
+ inferred = onnx.shape_inference.infer_shapes(model)
222
+ shape_map = {v.name: v for v in list(inferred.graph.value_info)
223
+ + list(inferred.graph.input) + list(inferred.graph.output)}
224
+ except Exception as error: # 自定义算子会阻断部分形状推导,不影响 4D B/C 的常规情形
225
+ logging.warning('形状推导失败,按 4D B/C 处理: %s', error)
226
+ shape_map = {}
227
+ new_nodes, extra_inits, count = [], [], 0
228
+ for node in model.graph.node:
229
+ if node.domain != SCAN_DOMAIN or node.op_type != SCAN_OP:
230
+ new_nodes.append(node)
231
+ continue
232
+ count += 1
233
+ prefix = f'depthart_scan_{count - 1}'
234
+ if not 5 <= len(node.input) <= 7:
235
+ raise ValueError(f'{node.name}: 期望 5~7 个输入,实际 {len(node.input)}')
236
+ u, delta, a, b, c = node.input[:5]
237
+ d = node.input[5] if len(node.input) > 5 and node.input[5] else None
238
+ bias = node.input[6] if len(node.input) > 6 and node.input[6] else None
239
+ # 3D 的 B/C 按 [B,N,L] 处理,补一个 group 轴
240
+ if _static_rank(shape_map, b) == 3:
241
+ b4 = prefix + '_b4'
242
+ new_nodes.append(helper.make_node('Unsqueeze', [b, prefix + '_uax_g'],
243
+ [b4], name=prefix + '_b_unsqueeze'))
244
+ extra_inits.append(numpy_helper.from_array(np.array([1], dtype=np.int64),
245
+ prefix + '_uax_g'))
246
+ b = b4
247
+ if _static_rank(shape_map, c) == 3:
248
+ c4 = prefix + '_c4'
249
+ new_nodes.append(helper.make_node('Unsqueeze', [c, prefix + '_uax_g2'],
250
+ [c4], name=prefix + '_c_unsqueeze'))
251
+ extra_inits.append(numpy_helper.from_array(np.array([1], dtype=np.int64),
252
+ prefix + '_uax_g2'))
253
+ c = c4
254
+ attrs = {item.name: item.i for item in node.attribute}
255
+ graph = _ScanGraph(prefix)
256
+ graph.build(u, delta, a, b, c, d, bias, attrs, node.output[0])
257
+ new_nodes.extend(graph.nodes)
258
+ extra_inits.extend(graph.inits)
259
+
260
+ if count:
261
+ del model.graph.node[:]
262
+ model.graph.node.extend(new_nodes)
263
+ model.graph.initializer.extend(extra_inits)
264
+ return count
265
+
266
+
267
+ def unsupported_custom_ops(model):
268
+ return [(n.domain, n.op_type, n.name) for n in model.graph.node
269
+ if n.domain not in ('', 'ai.onnx') and n.domain != SCAN_DOMAIN]
270
+
271
+
272
+ def create_session(model_path, save_rewritten=None):
273
+ """加载 ONNX;存在 SelectiveScan 时先在内存中改写成标准 ONNX。"""
274
+ model = onnx.load(str(model_path))
275
+ scan_count = rewrite_selective_scan(model)
276
+ if scan_count:
277
+ logging.info('检测到 %d 个 %s::%s,已改写为等价纯 ONNX 子图(Loop)',
278
+ scan_count, SCAN_DOMAIN, SCAN_OP)
279
+ remaining = unsupported_custom_ops(model)
280
+ if remaining:
281
+ raise ValueError(f'模型仍含未支持的自定义算子: {remaining[:5]}')
282
+ if save_rewritten is not None:
283
+ save_rewritten.parent.mkdir(parents=True, exist_ok=True)
284
+ onnx.save(model, str(save_rewritten))
285
+ logging.info('已保存改写后的标准 ONNX: %s', save_rewritten)
286
+ if scan_count:
287
+ return ort.InferenceSession(model.SerializeToString()), scan_count
288
+ return ort.InferenceSession(str(model_path)), 0
289
+
290
+
291
+ # ---------------------------------------------------------------------------
292
+ # 预处理 / 后处理
293
+ # ---------------------------------------------------------------------------
294
+
295
+ def check_interface(session):
296
+ inputs, outputs = session.get_inputs(), session.get_outputs()
297
+ logging.info('模型输入: %s', [(m.name, m.shape, m.type) for m in inputs])
298
+ logging.info('模型输出: %s', [(m.name, m.shape, m.type) for m in outputs])
299
+ if len(inputs) != 1:
300
+ raise ValueError(f'模型应有 1 个输入,实际 {[(m.name, m.shape) for m in inputs]}')
301
+ meta = inputs[0]
302
+ shape = list(meta.shape)
303
+ if len(shape) != 4 or not all(isinstance(v, int) and v > 0 for v in shape):
304
+ raise ValueError(f'输入必须是静态 4D NCHW: {meta.name} {meta.shape}')
305
+ if shape[0] != 1 or shape[1] != 3:
306
+ raise ValueError(f'仅支持 [1,3,H,W] 输入: {meta.name} {meta.shape}')
307
+ if 'float' not in meta.type:
308
+ raise ValueError(f'当前实现要求 float32 输入: {meta.name} {meta.type}')
309
+ depth_metas = [m for m in outputs if 'depth' in m.name.lower()] or outputs
310
+ if len(depth_metas) != 1:
311
+ raise ValueError(f'无法确定 depth 输出: {[(m.name, m.shape) for m in outputs]}')
312
+ return meta.name, (shape[2], shape[3]), depth_metas[0].name
313
+
314
+
315
+ def preprocess(image, size):
316
+ target_h, target_w = size
317
+ rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32) / 255.0
318
+ resized = cv2.resize(rgb, (target_w, target_h), interpolation=cv2.INTER_CUBIC)
319
+ normalized = (resized - MEAN) / STD
320
+ tensor = normalized.transpose(2, 0, 1)
321
+ return np.ascontiguousarray(tensor[None])
322
+
323
+
324
+ def read_depth(raw, input_size):
325
+ array = np.asarray(raw)
326
+ if array.ndim == 4 and array.shape[1] == 1:
327
+ array = array[0]
328
+ if array.ndim == 3 and array.shape[0] == 1:
329
+ array = array[0]
330
+ if array.ndim != 2 or array.shape != tuple(input_size):
331
+ raise ValueError(f'depth 输出无效: shape={array.shape},期望 {tuple(input_size)}')
332
+ if array.dtype != np.float32:
333
+ raise ValueError(f'depth 输出必须是 float32: {array.dtype}')
334
+ if not np.isfinite(array).all():
335
+ raise ValueError('depth 输出包含 nan/inf')
336
+ return array
337
+
338
+
339
+ def restore_depth(depth, original_size):
340
+ """按官方推理的 bilinear + align_corners=True 还原到原图尺寸。"""
341
+ height, width = original_size
342
+ src_h, src_w = depth.shape
343
+ if (src_h, src_w) == (height, width):
344
+ return depth.copy()
345
+ xs = np.linspace(0.0, src_w - 1, width, dtype=np.float32)
346
+ ys = np.linspace(0.0, src_h - 1, height, dtype=np.float32)
347
+ map_x, map_y = np.meshgrid(xs, ys)
348
+ return cv2.remap(depth, map_x, map_y, interpolation=cv2.INTER_LINEAR,
349
+ borderMode=cv2.BORDER_REPLICATE)
350
+
351
+
352
+ def visualize(image, depth):
353
+ finite = depth[np.isfinite(depth)]
354
+ low, high = np.percentile(finite, (2, 98))
355
+ value = np.clip((depth - low) / max(float(high - low), 1e-6), 0, 1)
356
+ colored = cv2.applyColorMap((value * 255).astype(np.uint8), cv2.COLORMAP_INFERNO)
357
+ panel = np.concatenate((image, colored), axis=1)
358
+ legend = np.full((56, panel.shape[1], 3), 255, dtype=np.uint8)
359
+ ramp = np.tile(np.linspace(0, 255, max(panel.shape[1] - 20, 1),
360
+ dtype=np.uint8), (12, 1))
361
+ legend[5:17, 10:10 + ramp.shape[1]] = cv2.applyColorMap(ramp, cv2.COLORMAP_INFERNO)
362
+ cv2.putText(legend, f'Image | Relative depth (2-98%): {low:.3f} -> {high:.3f}',
363
+ (10, 42), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (30, 30, 30), 1, cv2.LINE_AA)
364
+ return np.concatenate((panel, legend), axis=0)
365
+
366
+
367
+ def log_tensor(name, value):
368
+ logging.info('%s: shape=%s dtype=%s min=%.6g max=%.6g mean=%.6g nan=%d inf=%d',
369
+ name, value.shape, value.dtype, float(np.min(value)),
370
+ float(np.max(value)), float(np.mean(value)),
371
+ int(np.isnan(value).sum()), int(np.isinf(value).sum()))
372
+
373
+
374
+ def compare_arrays(depth, expected, source=''):
375
+ expected = np.asarray(expected)
376
+ if expected.ndim == 4 and expected.shape[1] == 1:
377
+ expected = expected[0]
378
+ if expected.shape != depth.shape:
379
+ raise ValueError(f'参考 depth shape 不符: {expected.shape} vs {depth.shape} ({source})')
380
+ if not np.isfinite(expected).all():
381
+ raise ValueError(f'参考 depth 含 nan/inf: {source}')
382
+ actual64 = depth.astype(np.float64).ravel()
383
+ expected64 = expected.astype(np.float64).ravel()
384
+ diff = np.abs(actual64 - expected64)
385
+ cosine = float(np.dot(actual64, expected64)
386
+ / (np.linalg.norm(actual64) * np.linalg.norm(expected64) + 1e-12))
387
+ # 相对深度存在全局 scale/offset 漂移,对齐后误差更能反映结构偏差。
388
+ slope, offset = np.polyfit(expected64, actual64, 1)
389
+ aligned = np.abs(actual64 - (slope * expected64 + offset))
390
+ metrics = {'cosine': cosine,
391
+ 'max_abs': float(diff.max()),
392
+ 'mean_abs': float(diff.mean()),
393
+ 'p99_abs': float(np.percentile(diff, 99)),
394
+ 'scale': float(slope), 'offset': float(offset),
395
+ 'aligned_mean_abs': float(aligned.mean()),
396
+ 'aligned_p99_abs': float(np.percentile(aligned, 99))}
397
+ logging.info('参考对比: %s', metrics)
398
+ return metrics
399
+
400
+
401
+ def compare_reference(depth, reference_path):
402
+ with np.load(reference_path, allow_pickle=False) as reference:
403
+ if 'depth' not in reference:
404
+ raise ValueError(f'参考 NPZ 缺少 depth: {reference_path}')
405
+ expected = np.asarray(reference['depth'])
406
+ return compare_arrays(depth, expected, str(reference_path))
407
+
408
+
409
+ def load_reference_dataset(path):
410
+ """读取汇总 NPZ,返回 {name: depth}。"""
411
+ path = Path(path)
412
+ if path.is_file():
413
+ with np.load(path, allow_pickle=False) as ref:
414
+ if 'depths' not in ref or 'names' not in ref:
415
+ raise ValueError(f'汇总 NPZ 需要 names/depths: {path}')
416
+ return {str(n): d for n, d in zip(ref['names'], ref['depths'])}
417
+ raise ValueError(f'数据集模式 --ref-npz 需要汇总 NPZ 文件: {path}')
418
+
419
+
420
+ def save_raw_npz(path, depth, original_size, input_size):
421
+ np.savez_compressed(path, depth=depth,
422
+ original_size=np.array(original_size, dtype=np.int32),
423
+ input_size=np.array(input_size, dtype=np.int32),
424
+ preprocessing='RGB /255 + ImageNet mean/std; resize=stretch')
425
+
426
+
427
+ def flush_dataset_npz(path, records, chunk_index, input_size, preprocessing):
428
+ """把缓冲区写成汇总 NPZ:names / depths / original_sizes。
429
+
430
+ chunk_index 为 None 时写入 path 本身,否则按 chunk 编号追加后缀。
431
+ """
432
+ if not records:
433
+ return
434
+ target = path if chunk_index is None else path.with_name(
435
+ f'{path.stem}_{chunk_index:04d}{path.suffix}')
436
+ target.parent.mkdir(parents=True, exist_ok=True)
437
+ np.savez_compressed(
438
+ target,
439
+ names=np.array([r[0] for r in records]),
440
+ depths=np.stack([r[1] for r in records]),
441
+ original_sizes=np.array([r[2] for r in records], dtype=np.int32),
442
+ input_size=np.array(input_size, dtype=np.int32),
443
+ preprocessing=preprocessing)
444
+
445
+
446
+ def collect_images(dataset):
447
+ paths = sorted(p for p in dataset.rglob('*')
448
+ if p.is_file() and p.suffix.lower() in EXTENSIONS)
449
+ if not paths:
450
+ raise ValueError(f'目录中没有图片: {dataset}')
451
+ return paths
452
+
453
+
454
+ def summarize(samples):
455
+ result = {}
456
+ for stage in STAGES:
457
+ values = np.array([sample[stage] for sample in samples], dtype=np.float64)
458
+ result[stage] = {'mean_ms': float(values.mean()),
459
+ 'p50_ms': float(np.percentile(values, 50)),
460
+ 'p90_ms': float(np.percentile(values, 90)),
461
+ 'p99_ms': float(np.percentile(values, 99))}
462
+ return result
463
+
464
+
465
+ # ---------------------------------------------------------------------------
466
+ # 推理流程
467
+ # ---------------------------------------------------------------------------
468
+
469
+ def run_single(session, input_name, output_name, input_size, args):
470
+ stem = args.image.stem
471
+ timings = dict.fromkeys(STAGES, 0.0)
472
+ total_start = time.perf_counter()
473
+
474
+ start = time.perf_counter()
475
+ image = cv2.imread(str(args.image), cv2.IMREAD_COLOR)
476
+ if image is None:
477
+ raise ValueError(f'读取图片失败: {args.image}')
478
+ timings['read'] = (time.perf_counter() - start) * 1000
479
+
480
+ start = time.perf_counter()
481
+ tensor = preprocess(image, input_size)
482
+ timings['preprocess'] = (time.perf_counter() - start) * 1000
483
+
484
+ start = time.perf_counter()
485
+ raw = session.run([output_name], {input_name: tensor})[0]
486
+ timings['inference'] = (time.perf_counter() - start) * 1000
487
+ depth = read_depth(raw, input_size)
488
+
489
+ start = time.perf_counter()
490
+ comparison = None
491
+ if args.debug:
492
+ log_tensor(input_name, tensor)
493
+ log_tensor('depth', depth)
494
+ if args.ref_npz:
495
+ comparison = compare_reference(depth, args.ref_npz)
496
+ timings['diagnostics'] = (time.perf_counter() - start) * 1000
497
+
498
+ start = time.perf_counter()
499
+ restored = restore_depth(depth, image.shape[:2])
500
+ timings['postprocess'] = (time.perf_counter() - start) * 1000
501
+
502
+ start = time.perf_counter()
503
+ panel = visualize(image, restored)
504
+ timings['visualize'] = (time.perf_counter() - start) * 1000
505
+
506
+ start = time.perf_counter()
507
+ args.output.mkdir(parents=True, exist_ok=True)
508
+ image_path = args.output / f'{stem}_depth.jpg'
509
+ if not cv2.imwrite(str(image_path), panel):
510
+ raise OSError(f'保存可视化失败: {image_path}')
511
+ np.save(str(args.output / f'{stem}_depth.npy'), restored)
512
+ npz_path = None
513
+ if args.dump_npz:
514
+ npz_path = Path(args.dump_npz)
515
+ npz_path.parent.mkdir(parents=True, exist_ok=True)
516
+ save_raw_npz(npz_path, depth, image.shape[:2], input_size)
517
+ timings['write'] = (time.perf_counter() - start) * 1000
518
+ timings['total'] = (time.perf_counter() - total_start) * 1000
519
+
520
+ logging.info('深度(模型分辨率): min=%.4f max=%.4f mean=%.4f p50=%.4f',
521
+ float(depth.min()), float(depth.max()), float(depth.mean()),
522
+ float(np.percentile(depth, 50)))
523
+ logging.info('深度(原图分辨率): min=%.4f max=%.4f mean=%.4f',
524
+ float(restored.min()), float(restored.max()), float(restored.mean()))
525
+ logging.info('各阶段耗时: %s',
526
+ ' '.join(f'{key}={value:.2f} ms' for key, value in timings.items()))
527
+ logging.info('结果目录: %s(可视化 %s%s)', args.output, image_path.name,
528
+ f',原始 NPZ {npz_path.name}' if npz_path else '')
529
+
530
+
531
+ def run_dataset(session, input_name, output_name, input_size, args):
532
+ images = collect_images(args.dataset)
533
+ if args.limit is not None:
534
+ images = images[:args.limit]
535
+ args.output.mkdir(parents=True, exist_ok=True)
536
+ full_output = args.full_output
537
+ if full_output:
538
+ logging.warning('--full-output: CPU 侧深度还原、可视化与写盘会拉高端到端耗时,'
539
+ '不能只看 inference 时间。')
540
+ else:
541
+ logging.info('数据集快速模式:只统计 read/preprocess/inference,'
542
+ '跳过深度还原、可视化和结果写盘。')
543
+ ref_map = load_reference_dataset(args.ref_npz) if args.ref_npz else None
544
+
545
+ preprocessing = 'RGB /255 + ImageNet mean/std; resize=stretch'
546
+ dump_path = Path(args.dump_npz) if args.dump_npz else None
547
+ if dump_path is not None and dump_path.is_dir():
548
+ raise ValueError('数据集模式 --dump-npz 请传 .npz 文件路径,'
549
+ '不再按目录为每张图生成小文件')
550
+ dump_records, dump_chunk = [], 0
551
+
552
+ warmup_ms = 0.0
553
+ if args.warmup:
554
+ first = cv2.imread(str(images[0]), cv2.IMREAD_COLOR)
555
+ if first is None:
556
+ raise ValueError(f'读取图片失败: {images[0]}')
557
+ warm = preprocess(first, input_size)
558
+ start = time.perf_counter()
559
+ for _ in range(args.warmup):
560
+ session.run([output_name], {input_name: warm})
561
+ warmup_ms = (time.perf_counter() - start) * 1000
562
+
563
+ samples, sample_info = [], []
564
+ for index, path in enumerate(tqdm(images, desc='ONNX 相对深度', unit='image')):
565
+ relative = path.relative_to(args.dataset)
566
+ timings = dict.fromkeys(STAGES, 0.0)
567
+ total_start = time.perf_counter()
568
+
569
+ start = time.perf_counter()
570
+ image = cv2.imread(str(path), cv2.IMREAD_COLOR)
571
+ if image is None:
572
+ raise ValueError(f'读取图片失败: {path}')
573
+ timings['read'] = (time.perf_counter() - start) * 1000
574
+
575
+ start = time.perf_counter()
576
+ tensor = preprocess(image, input_size)
577
+ timings['preprocess'] = (time.perf_counter() - start) * 1000
578
+
579
+ start = time.perf_counter()
580
+ raw = session.run([output_name], {input_name: tensor})[0]
581
+ timings['inference'] = (time.perf_counter() - start) * 1000
582
+ depth = read_depth(raw, input_size)
583
+
584
+ start = time.perf_counter()
585
+ comparison = None
586
+ if args.debug and index == 0:
587
+ log_tensor(input_name, tensor)
588
+ log_tensor('depth', depth)
589
+ if dump_path is not None:
590
+ dump_records.append((str(relative), depth,
591
+ np.array(image.shape[:2], dtype=np.int32)))
592
+ if args.npz_chunk and len(dump_records) >= args.npz_chunk:
593
+ flush_dataset_npz(dump_path, dump_records, dump_chunk,
594
+ input_size, preprocessing)
595
+ dump_records.clear()
596
+ dump_chunk += 1
597
+ if ref_map is not None:
598
+ key = str(relative)
599
+ if key not in ref_map:
600
+ raise ValueError(f'参考 NPZ 缺少样本: {key}')
601
+ comparison = compare_arrays(depth, ref_map[key], f'{args.ref_npz}:{key}')
602
+ timings['diagnostics'] = (time.perf_counter() - start) * 1000
603
+
604
+ restored = None
605
+ if full_output:
606
+ start = time.perf_counter()
607
+ restored = restore_depth(depth, image.shape[:2])
608
+ timings['postprocess'] = (time.perf_counter() - start) * 1000
609
+ start = time.perf_counter()
610
+ panel = visualize(image, restored)
611
+ timings['visualize'] = (time.perf_counter() - start) * 1000
612
+ start = time.perf_counter()
613
+ prefix = args.output / relative
614
+ prefix.parent.mkdir(parents=True, exist_ok=True)
615
+ np.save(str(prefix) + '.depth.npy', restored)
616
+ if not cv2.imwrite(str(prefix) + '.depth.jpg', panel):
617
+ raise OSError(f'保存可视化失败: {prefix}.depth.jpg')
618
+ timings['write'] = (time.perf_counter() - start) * 1000
619
+ timings['total'] = (time.perf_counter() - total_start) * 1000
620
+
621
+ samples.append(timings)
622
+ sample_info.append({'name': str(relative),
623
+ 'original_size': list(image.shape[:2]),
624
+ 'depth_min': float((restored if restored is not None else depth).min()),
625
+ 'depth_max': float((restored if restored is not None else depth).max()),
626
+ 'timing_ms': timings,
627
+ 'reference_comparison': comparison})
628
+
629
+ if dump_path is not None:
630
+ flush_dataset_npz(dump_path, dump_records,
631
+ None if not args.npz_chunk else dump_chunk,
632
+ input_size, preprocessing)
633
+
634
+ stats = summarize(samples)
635
+ total_ms = sum(sample['total'] for sample in samples)
636
+ report = {'model': str(args.model.resolve()),
637
+ 'runtime': 'onnxruntime ' + ort.__version__,
638
+ 'providers': session.get_providers(),
639
+ 'task': 'relative-depth',
640
+ 'custom_ops_rewritten': args.scan_count,
641
+ 'input_name': input_name, 'output_name': output_name,
642
+ 'input_size': list(input_size), 'input_layout': 'NCHW',
643
+ 'preprocessing': 'RGB /255 + ImageNet mean/std; resize=stretch',
644
+ 'batch_size': 1, 'sample_count': len(samples),
645
+ 'full_output': full_output, 'warmup_count': args.warmup,
646
+ 'warmup_ms': warmup_ms, 'total_ms': total_ms,
647
+ 'fps': len(samples) * 1000 / total_ms,
648
+ 'timing_scope': '样本 read 至 write,包含 diagnostics;排除模型加载、'
649
+ 'session 初始化、自定义算子改写、warmup、进度条和最终性能 JSON 写盘。'
650
+ '快速模式的 postprocess/visualize/write 记为 0。',
651
+ 'stages': stats, 'samples': sample_info}
652
+ with (args.output / 'onnx_timing.json').open('w', encoding='utf-8') as file:
653
+ json.dump(report, file, ensure_ascii=False, indent=2, allow_nan=False)
654
+ for stage, values in stats.items():
655
+ logging.info('%s: mean=%.2f p50=%.2f p90=%.2f p99=%.2f ms',
656
+ stage, values['mean_ms'], values['p50_ms'],
657
+ values['p90_ms'], values['p99_ms'])
658
+ logging.info('%d 张图片,total=%.2f s,FPS=%.3f,结果目录: %s',
659
+ len(samples), total_ms / 1000, report['fps'], args.output)
660
+
661
+
662
+ def main():
663
+ parser = argparse.ArgumentParser(
664
+ description='DepthART 相对深度 ONNX 推理',
665
+ formatter_class=argparse.RawDescriptionHelpFormatter, epilog=__doc__)
666
+ parser.add_argument('--model', type=Path, default=DEFAULT_MODEL,
667
+ help=f'FP32 ONNX 模型,默认 {DEFAULT_MODEL.relative_to(PROJECT_ROOT)}')
668
+ source = parser.add_mutually_exclusive_group(required=True)
669
+ source.add_argument('--image', type=Path, help='单张图片')
670
+ source.add_argument('--dataset', type=Path, help='图片目录(递归)')
671
+ parser.add_argument('--output', type=Path,
672
+ default=PROJECT_ROOT / 'results' / 'onnx',
673
+ help='结果目录,默认 results/onnx')
674
+ parser.add_argument('--warmup', type=int, default=3,
675
+ help='数据集模式首次输入 warmup 次数,默认 3')
676
+ parser.add_argument('--limit', type=int, help='数据集模式只处理前 N 张')
677
+ parser.add_argument('--full-output', action='store_true',
678
+ help='数据集模式执行完整后处理、可视化和结果写盘')
679
+ parser.add_argument('--debug', action='store_true', help='打印输入/输出统计')
680
+ parser.add_argument('--dump-npz',
681
+ help='单图:NPZ 文件;数据集:汇总 NPZ 文件(names/depths/original_sizes)')
682
+ parser.add_argument('--npz-chunk', type=int, default=0,
683
+ help='数据集模式每 N 张切一个 NPZ 文件,0 表示全部写入一个文件')
684
+ parser.add_argument('--ref-npz', type=Path,
685
+ help='单图:参考 NPZ 文件;数据集:参考汇总 NPZ 文件')
686
+ parser.add_argument('--save-rewritten', type=Path,
687
+ help='保存去除自定义算子后的标准 ONNX(不影响原模型)')
688
+ args = parser.parse_args()
689
+ logging.basicConfig(level=logging.INFO, format='%(message)s')
690
+ if args.warmup < 0:
691
+ parser.error('--warmup 不能为负数')
692
+ if args.limit is not None and args.limit < 1:
693
+ parser.error('--limit 必须为正数')
694
+ if args.npz_chunk < 0:
695
+ parser.error('--npz-chunk 不能为负数')
696
+ if not args.model.is_file():
697
+ parser.error(f'--model 不存在: {args.model}')
698
+ if args.image:
699
+ if not args.image.is_file():
700
+ parser.error(f'--image 不是文件: {args.image}')
701
+ if args.ref_npz and not args.ref_npz.is_file():
702
+ parser.error('单图模式 --ref-npz 必须是文件')
703
+ if args.limit is not None or args.full_output:
704
+ logging.warning('单图模式忽略 --limit/--full-output')
705
+ else:
706
+ if not args.dataset.is_dir():
707
+ parser.error(f'--dataset 不是目录: {args.dataset}')
708
+
709
+ session, args.scan_count = create_session(args.model, args.save_rewritten)
710
+ input_name, input_size, output_name = check_interface(session)
711
+ logging.info('batch=1���输入尺寸: %s,预处理: RGB /255 + ImageNet mean/std(stretch)',
712
+ input_size)
713
+
714
+ if args.image:
715
+ run_single(session, input_name, output_name, input_size, args)
716
+ else:
717
+ run_dataset(session, input_name, output_name, input_size, args)
718
+
719
+
720
+ if __name__ == '__main__':
721
+ main()
src/tool/__pycache__/evaluate_nyu654_depth.cpython-310.pyc ADDED
Binary file (8.21 kB). View file
 
src/tool/compare_depth_outputs.py ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # -*- coding: utf-8 -*-
3
+ """
4
+ DepthART 两路深度输出的相关性/一致性对比。
5
+
6
+ 支持三种输入格式,自动识别:
7
+ - 汇总 NPZ(names/depths),如 ONNX 脚本 --dump-npz 生成的文件
8
+ - 分块汇总目录(*.npz 中含 names/depths)
9
+ - 旧的逐图 NPZ 目录(<图片名>.npz 中含 depth)
10
+
11
+ 指标(逐图计算后汇总 mean/p50/p90/min):
12
+ pearson 皮尔逊相关系数
13
+ spearman 斯皮尔曼秩相关
14
+ cosine 余弦相似度
15
+ mae / rmse 原始量纲误差
16
+ aligned_* 最小二乘 scale-shift 对齐后的误差
17
+ rel_error mae / mean(|ref|)
18
+
19
+ 用法:
20
+ ../../venv/bin/python src/tool/compare_depth_outputs.py \
21
+ --ref results/nyu654/fp32/npz --pred results/nyu654/axmodel \
22
+ --out results/nyu654/eval/corr_fp32_axmodel.json
23
+ """
24
+
25
+ import argparse
26
+ import json
27
+ from pathlib import Path
28
+
29
+ import numpy as np
30
+ from scipy.stats import spearmanr
31
+ from tqdm import tqdm
32
+
33
+
34
+ def load_predictions(path):
35
+ path = Path(path)
36
+ if path.is_file():
37
+ files = [path]
38
+ elif path.is_dir():
39
+ files = sorted(path.glob('*.npz'))
40
+ if not files:
41
+ raise ValueError(f'目录中没有 NPZ: {path}')
42
+ else:
43
+ raise FileNotFoundError(f'路径不存在: {path}')
44
+
45
+ preds = {}
46
+ for fp in files:
47
+ with np.load(fp, allow_pickle=False) as data:
48
+ if 'depths' in data and 'names' in data:
49
+ for name, depth in zip(data['names'], data['depths']):
50
+ preds[str(name)] = np.asarray(depth)
51
+ elif 'depth' in data:
52
+ preds[fp.name[:-len('.npz')]] = np.asarray(data['depth'])
53
+ else:
54
+ raise ValueError(f'无法识别的 NPZ: {fp}')
55
+ return preds
56
+
57
+
58
+ def compare(ref, pred):
59
+ a = np.asarray(ref, dtype=np.float64).ravel()
60
+ b = np.asarray(pred, dtype=np.float64).ravel()
61
+ if a.shape != b.shape:
62
+ raise ValueError(f'shape 不一致: {a.shape} vs {b.shape}')
63
+ diff = b - a
64
+ mae = float(np.mean(np.abs(diff)))
65
+ rmse = float(np.sqrt(np.mean(diff ** 2)))
66
+ cosine = float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-12))
67
+ pearson = float(np.corrcoef(a, b)[0, 1])
68
+ spearman = float(spearmanr(a, b).statistic)
69
+ slope, offset = np.polyfit(a, b, 1)
70
+ aligned = b - (slope * a + offset)
71
+ return {
72
+ 'pearson': pearson,
73
+ 'spearman': spearman,
74
+ 'cosine': cosine,
75
+ 'mae': mae,
76
+ 'rmse': rmse,
77
+ 'rel_error': mae / (np.mean(np.abs(a)) + 1e-12),
78
+ 'scale': float(slope),
79
+ 'offset': float(offset),
80
+ 'aligned_mae': float(np.mean(np.abs(aligned))),
81
+ 'aligned_rmse': float(np.sqrt(np.mean(aligned ** 2))),
82
+ }
83
+
84
+
85
+ def summarize(rows):
86
+ keys = ('pearson', 'spearman', 'cosine', 'rel_error', 'aligned_rmse')
87
+ out = {}
88
+ for k in keys:
89
+ values = np.array([r[k] for r in rows])
90
+ out[k] = {'mean': float(values.mean()), 'p50': float(np.percentile(values, 50)),
91
+ 'p90': float(np.percentile(values, 90)), 'min': float(values.min())}
92
+ return out
93
+
94
+
95
+ def main():
96
+ parser = argparse.ArgumentParser(description='两路深度输出相关性对比')
97
+ parser.add_argument('--ref', required=True, help='参考输出(基准后端)')
98
+ parser.add_argument('--pred', required=True, help='待比较输出')
99
+ parser.add_argument('--out', help='结果 JSON 路径')
100
+ args = parser.parse_args()
101
+
102
+ ref = load_predictions(args.ref)
103
+ pred = load_predictions(args.pred)
104
+ names = sorted(set(ref) & set(pred))
105
+ if not names:
106
+ raise ValueError('两路输出没有同名样本')
107
+ missing = (len(ref) - len(names)) + (len(pred) - len(names))
108
+ if missing:
109
+ print(f'警告: {missing} 个样本名称不匹配,已忽略')
110
+
111
+ rows = []
112
+ for name in tqdm(names, desc='compare', ncols=80):
113
+ rows.append({'name': name, **compare(ref[name], pred[name])})
114
+
115
+ summary = summarize(rows)
116
+ low = [r for r in rows if r['pearson'] < 0.9]
117
+ worst = sorted(rows, key=lambda r: r['pearson'])[:5]
118
+ report = {
119
+ 'ref': str(args.ref), 'pred': str(args.pred),
120
+ 'num_samples': len(rows),
121
+ 'summary': summary,
122
+ 'pearson_below_0.9': len(low),
123
+ 'worst': [{'name': r['name'], 'pearson': r['pearson'],
124
+ 'mae': r['mae'], 'aligned_rmse': r['aligned_rmse']} for r in worst],
125
+ 'per_image': rows,
126
+ }
127
+ print(f'\n样本数 {len(rows)},pearson<0.9 的 {len(low)} 张')
128
+ print(f'{"指标":<14}{"mean":>12}{"p50":>12}{"p90":>12}{"min":>12}')
129
+ for k, v in summary.items():
130
+ print(f'{k:<14}{v["mean"]:>12.4f}{v["p50"]:>12.4f}{v["p90"]:>12.4f}{v["min"]:>12.4f}')
131
+ print('\n最差 5 张:')
132
+ for r in worst:
133
+ print(f' {r["name"]:<32} pearson={r["pearson"]:.4f} mae={r["mae"]:.4f}')
134
+
135
+ if args.out:
136
+ Path(args.out).parent.mkdir(parents=True, exist_ok=True)
137
+ with open(args.out, 'w') as fp:
138
+ json.dump(report, fp, indent=2)
139
+ print(f'\n结果已保存: {args.out}')
140
+
141
+
142
+ if __name__ == '__main__':
143
+ main()
src/tool/compare_onnx_axmodel.py ADDED
@@ -0,0 +1,546 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # -*- coding: utf-8 -*-
3
+ """
4
+ Depth Anything V2:ONNX 与 AXMODEL 原始输出一致性对比。
5
+
6
+ 对同一批图片分别运行 ONNX 和 axmodel,在模型分辨率下比较两路深度输出
7
+ (相对逆深度,数值越大越近),逐样本计算相似度指标,并汇总为 Markdown 文档。
8
+
9
+ 两个后端的输入预处理可以不同,但 resize 与颜色顺序必须一致:
10
+ 默认 ONNX 做 /255 + ImageNet mean/std,axmodel 送 0-255 原始像素
11
+ (对应编译时把归一化写进 input_processors / calibration_mean/std 的方案)。
12
+ 如果 axmodel 重编译后仍需脚本侧归一化,使用 --axmodel-norm imagenet。
13
+
14
+ 指标:
15
+ cosine 两路输出的余弦相似度
16
+ pearson 皮尔逊相关系数
17
+ mae / rmse 绝对误差均值 / 均方根
18
+ max_abs/p99_abs 最大误差 / 99 分位误差
19
+ aligned_mae 最小二乘 scale-shift 对齐后的 MAE(区分全局尺度漂移与结构误差)
20
+ aligned_rmse 对齐后的 RMSE
21
+ rel_error mae / mean(|onnx|)
22
+
23
+ 依赖:numpy、opencv-python、onnxruntime、axengine(板端 pyaxengine)。
24
+
25
+ 用法(在 hs/Depth-Anything-V2 目录下执行):
26
+ # 离线比较已有结果(开发机,无需 axengine 或模型文件)
27
+ python3 src/tool/compare_onnx_axmodel.py \
28
+ --onnx-results results/onnx_full --axmodel-results results/board_full \
29
+ --output results/compare_full/onnx_vs_axmodel.md
30
+
31
+ # 单图
32
+ python3 src/tool/compare_onnx_axmodel.py \
33
+ --onnx depth_anything_v2_vits.onnx \
34
+ --axmodel depth_anything_v2_vits_ax650.axmodel \
35
+ --image /path/a.jpg --output results/compare/single.md
36
+
37
+ # 数据集(递归收集图片),默认 axmodel 送 0-255
38
+ python3 src/tool/compare_onnx_axmodel.py \
39
+ --onnx depth_anything_v2_vits.onnx \
40
+ --axmodel depth_anything_v2_vits_ax650.axmodel \
41
+ --dataset /path/images --limit 50 \
42
+ --output results/compare/onnx_vs_axmodel.md
43
+
44
+ # axmodel 仍需要脚本侧做 ImageNet 归一化时
45
+ python3 src/tool/compare_onnx_axmodel.py \
46
+ --onnx depth_anything_v2_vits.onnx --axmodel other.axmodel \
47
+ --dataset /path/images --axmodel-norm imagenet \
48
+ --output results/compare/onnx_vs_axmodel.md
49
+
50
+ 注意:
51
+ - axmodel 的输出名可能被编译器改写,脚本先按名字、再按 shape 回退绑定。
52
+ - 两个后端的输出 shape 必须一致;不一致时直接报错并打印实际接口。
53
+ - 结果默认输出到 results/compare/,不写预测 NPZ。
54
+ """
55
+
56
+ import argparse
57
+ import json
58
+ import logging
59
+ import time
60
+ from datetime import datetime
61
+ from pathlib import Path
62
+
63
+ import cv2
64
+ import numpy as np
65
+
66
+ MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
67
+ STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
68
+ EXTENSIONS = {'.jpg', '.jpeg', '.png', '.bmp', '.tif', '.tiff'}
69
+ METRICS = ('cosine', 'pearson', 'mae', 'rmse', 'max_abs', 'p99_abs',
70
+ 'aligned_mae', 'aligned_rmse', 'rel_error')
71
+
72
+
73
+ def meta_type(meta):
74
+ return str(getattr(meta, 'type', getattr(meta, 'dtype', ''))).lower()
75
+
76
+
77
+ def input_spec(shape, type_name, model_name):
78
+ shape = list(shape)
79
+ if len(shape) != 4 or not all(isinstance(v, int) and v > 0 for v in shape):
80
+ raise ValueError(f'{model_name} 输入必须是静态 4D NHWC/NCHW: {shape}')
81
+ if shape[0] != 1:
82
+ raise ValueError(f'{model_name} 仅支持 batch=1: {shape}')
83
+ if shape[1] == 3:
84
+ layout, size = 'NCHW', (shape[2], shape[3])
85
+ elif shape[3] == 3:
86
+ layout, size = 'NHWC', (shape[1], shape[2])
87
+ else:
88
+ raise ValueError(f'{model_name} 输入必须含 3 通道: {shape}')
89
+ if 'float' in type_name:
90
+ dtype = np.float32
91
+ elif 'uint8' in type_name or type_name.endswith('u8'):
92
+ dtype = np.uint8
93
+ elif 'int8' in type_name:
94
+ dtype = np.int8
95
+ else:
96
+ raise ValueError(f'{model_name} 不支持的输入类型: {type_name!r}')
97
+ return size, layout, dtype
98
+
99
+
100
+ def onnx_interface(session):
101
+ inputs, outputs = session.get_inputs(), session.get_outputs()
102
+ logging.info('ONNX 输入: %s', [(m.name, m.shape, m.type) for m in inputs])
103
+ logging.info('ONNX 输出: %s', [(m.name, m.shape, m.type) for m in outputs])
104
+ if len(inputs) != 1:
105
+ raise ValueError(f'ONNX 应有 1 个输入: {[(m.name, m.shape) for m in inputs]}')
106
+ meta = inputs[0]
107
+ if 'float' not in str(meta.type):
108
+ raise ValueError(f'ONNX 输入要求 float32: {meta.name} {meta.type}')
109
+ size, layout, dtype = input_spec(meta.shape, str(meta.type), 'ONNX')
110
+ return meta.name, size, layout, dtype, outputs[0].name
111
+
112
+
113
+ def axmodel_interface(session):
114
+ inputs, outputs = session.get_inputs(), session.get_outputs()
115
+ logging.info('AXMODEL 输入: %s', [(m.name, m.shape, meta_type(m)) for m in inputs])
116
+ logging.info('AXMODEL 输出: %s', [(m.name, m.shape, meta_type(m)) for m in outputs])
117
+ if len(inputs) != 1:
118
+ raise ValueError(f'axmodel 应有 1 个输入: {[(m.name, m.shape) for m in inputs]}')
119
+ meta = inputs[0]
120
+ size, layout, dtype = input_spec(meta.shape, meta_type(meta), 'AXMODEL')
121
+ return meta, size, layout, dtype, outputs
122
+
123
+
124
+ def preprocess(image, size, layout, dtype, resize_mode, color_order, norm):
125
+ target_h, target_w = size
126
+ original_h, original_w = image.shape[:2]
127
+ if color_order == 'rgb':
128
+ image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
129
+ if resize_mode == 'pad':
130
+ scale = min(target_h / original_h, target_w / original_w)
131
+ new_h = max(1, min(target_h, int(round(original_h * scale))))
132
+ new_w = max(1, min(target_w, int(round(original_w * scale))))
133
+ resized = cv2.resize(image, (new_w, new_h), interpolation=cv2.INTER_LINEAR)
134
+ canvas = np.zeros((target_h, target_w, 3), dtype=np.float32)
135
+ canvas[:new_h, :new_w] = resized
136
+ image_size = (new_h, new_w)
137
+ else:
138
+ canvas = cv2.resize(image, (target_w, target_h),
139
+ interpolation=cv2.INTER_LINEAR).astype(np.float32)
140
+ image_size = (target_h, target_w)
141
+ canvas = canvas.astype(np.float32, copy=False)
142
+ if norm == 'imagenet':
143
+ if dtype != np.float32:
144
+ raise ValueError('整数输入不能使用 ImageNet 归一化')
145
+ canvas = (canvas / 255.0 - MEAN) / STD
146
+ if dtype != np.float32:
147
+ canvas = np.rint(np.clip(canvas, 0, 255)).astype(dtype)
148
+ tensor = canvas.transpose(2, 0, 1) if layout == 'NCHW' else canvas
149
+ return np.ascontiguousarray(tensor[None]), image_size
150
+
151
+
152
+ def to_2d(array, model_name):
153
+ array = np.asarray(array)
154
+ if array.ndim == 4:
155
+ if array.shape[0] != 1:
156
+ raise ValueError(f'{model_name} 输出 batch 必须为 1: {array.shape}')
157
+ if array.shape[1] == 1:
158
+ array = array[0, 0]
159
+ elif array.shape[3] == 1:
160
+ array = array[0, :, :, 0]
161
+ else:
162
+ raise ValueError(f'{model_name} 输出无法压缩为 2D: {array.shape}')
163
+ elif array.ndim == 3:
164
+ if array.shape[0] == 1:
165
+ array = array[0]
166
+ elif array.shape[2] == 1:
167
+ array = array[:, :, 0]
168
+ else:
169
+ raise ValueError(f'{model_name} 输出无法压缩为 2D: {array.shape}')
170
+ if array.ndim != 2:
171
+ raise ValueError(f'{model_name} 输出必须是 2D 深度图: {array.shape}')
172
+ array = array.astype(np.float32, copy=False)
173
+ if not np.isfinite(array).all():
174
+ raise ValueError(f'{model_name} 输出包含 nan/inf')
175
+ return array
176
+
177
+
178
+ def select_ax_output(outputs, metas, expected_shape):
179
+ if not isinstance(outputs, (list, tuple)):
180
+ outputs = [outputs]
181
+ if len(outputs) != len(metas):
182
+ raise ValueError(f'axmodel 输出数量不一致: {len(outputs)} vs {len(metas)}')
183
+ for index, meta in enumerate(metas):
184
+ if any(token in str(meta.name).lower() for token in ('depth', 'output')):
185
+ array = to_2d(outputs[index], 'AXMODEL')
186
+ if array.shape == expected_shape:
187
+ return array, str(meta.name)
188
+ candidates = []
189
+ for index, array in enumerate(outputs):
190
+ candidate = to_2d(array, 'AXMODEL')
191
+ if candidate.shape == expected_shape:
192
+ candidates.append((index, candidate))
193
+ if len(candidates) != 1:
194
+ raise ValueError('axmodel 输出无法按 name 或 shape 唯一匹配'
195
+ f' {expected_shape}: {[(m.name, np.asarray(o).shape) for m, o in zip(metas, outputs)]}')
196
+ index, candidate = candidates[0]
197
+ return candidate, str(metas[index].name)
198
+
199
+
200
+ def compare_arrays(reference, actual):
201
+ ref = reference.astype(np.float64).ravel()
202
+ act = actual.astype(np.float64).ravel()
203
+ diff = act - ref
204
+ ref_norm = np.linalg.norm(ref)
205
+ act_norm = np.linalg.norm(act)
206
+ cosine = float(np.dot(ref, act) / (ref_norm * act_norm + 1e-12))
207
+ if ref.std() > 1e-12 and act.std() > 1e-12:
208
+ pearson = float(np.corrcoef(ref, act)[0, 1])
209
+ else:
210
+ pearson = float('nan')
211
+ slope, offset = np.polyfit(ref, act, 1)
212
+ aligned = act - (slope * ref + offset)
213
+ abs_diff = np.abs(diff)
214
+ return {
215
+ 'cosine': cosine,
216
+ 'pearson': pearson,
217
+ 'mae': float(abs_diff.mean()),
218
+ 'rmse': float(np.sqrt(np.mean(diff ** 2))),
219
+ 'max_abs': float(abs_diff.max()),
220
+ 'p99_abs': float(np.percentile(abs_diff, 99)),
221
+ 'aligned_mae': float(np.abs(aligned).mean()),
222
+ 'aligned_rmse': float(np.sqrt(np.mean(aligned ** 2))),
223
+ 'rel_error': float(abs_diff.mean() / (np.abs(ref).mean() + 1e-12)),
224
+ }
225
+
226
+
227
+ def summarize(records):
228
+ summary = {}
229
+ for key in METRICS:
230
+ values = np.array([record[key] for record in records], dtype=np.float64)
231
+ values = values[np.isfinite(values)]
232
+ if values.size == 0:
233
+ summary[key] = None
234
+ continue
235
+ summary[key] = {'mean': float(values.mean()),
236
+ 'median': float(np.percentile(values, 50)),
237
+ 'p90': float(np.percentile(values, 90)),
238
+ 'p99': float(np.percentile(values, 99)),
239
+ 'min': float(values.min()),
240
+ 'max': float(values.max())}
241
+ return summary
242
+
243
+
244
+ def format_value(value):
245
+ if value is None:
246
+ return '-'
247
+ if isinstance(value, float) and not np.isfinite(value):
248
+ return '-'
249
+ return f'{value:.6g}'
250
+
251
+
252
+ def write_markdown(path, report):
253
+ lines = ['# Depth Anything V2:ONNX 与 AXMODEL 输出一致性', '']
254
+ lines.append(f'- 生成时间:{report["created"]}')
255
+ lines.append(f'- 样本数:{report["sample_count"]}')
256
+ lines.append(f'- 结果目录:`{report["output_dir"]}`')
257
+ if report['limit'] is not None:
258
+ lines.append(f'- 样本上限:{report["limit"]}')
259
+ lines.append('')
260
+ if report.get('mode') == 'results':
261
+ lines.extend([
262
+ '## 结果来源', '',
263
+ f'- ONNX:`{report["onnx_results"]}`',
264
+ f'- AXMODEL:`{report["axmodel_results"]}`',
265
+ '- 按相对路径匹配同名 `.depth.npy`,比较保存的深度图,不重新推理或 resize。',
266
+ '- 这些文件由推理脚本还原至原图尺寸,指标对应保存结果的像素空间。',
267
+ '- 离线比较不测量推理耗时;预处理配置以原推理记录为准。', '',
268
+ ])
269
+ else:
270
+ lines.append('## 模型与预处理')
271
+ lines.append('')
272
+ lines.append('| 项 | ONNX | AXMODEL |')
273
+ lines.append('| --- | --- | --- |')
274
+ lines.append(f'| 模型 | `{report["onnx"]["path"]}` | `{report["axmodel"]["path"]}` |')
275
+ lines.append(f'| 运行时 | {report["onnx"]["runtime"]} | {report["axmodel"]["runtime"]} |')
276
+ lines.append(f'| 输入 | {report["onnx"]["input"]} | {report["axmodel"]["input"]} |')
277
+ lines.append(f'| 输出 | {report["onnx"]["output"]} | {report["axmodel"]["output"]} |')
278
+ lines.append(f'| 归一化 | {report["onnx"]["norm"]} | {report["axmodel"]["norm"]} |')
279
+ lines.append(f'| resize | {report["resize_mode"]} | {report["resize_mode"]} |')
280
+ lines.append(f'| 颜色顺序 | {report["color_order"]} | {report["color_order"]} |')
281
+ lines.append(f'| warmup | {report["warmup"]} | {report["warmup"]} |')
282
+ lines.append('')
283
+ lines.append('## 汇总指标')
284
+ lines.append('')
285
+ lines.append('| 指标 | mean | median | p90 | p99 | min | max |')
286
+ lines.append('| --- | --- | --- | --- | --- | --- | --- |')
287
+ for key in METRICS:
288
+ stats = report['summary'][key]
289
+ if stats is None:
290
+ lines.append(f'| {key} | - | - | - | - | - | - |')
291
+ else:
292
+ lines.append(f'| {key} | {format_value(stats["mean"])} | '
293
+ f'{format_value(stats["median"])} | {format_value(stats["p90"])} | '
294
+ f'{format_value(stats["p99"])} | {format_value(stats["min"])} | '
295
+ f'{format_value(stats["max"])} |')
296
+ lines.append('')
297
+ lines.append('## 逐样本结果')
298
+ lines.append('')
299
+ lines.append('| # | 图片 | cosine | pearson | mae | rmse | max_abs | p99_abs | '
300
+ 'aligned_mae | aligned_rmse | rel_error | ONNX ms | AXMODEL ms |')
301
+ lines.append('| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |')
302
+ for index, sample in enumerate(report['samples'], 1):
303
+ metrics = sample['metrics']
304
+ lines.append(
305
+ f'| {index} | `{sample["name"]}` | {format_value(metrics["cosine"])} | '
306
+ f'{format_value(metrics["pearson"])} | {format_value(metrics["mae"])} | '
307
+ f'{format_value(metrics["rmse"])} | {format_value(metrics["max_abs"])} | '
308
+ f'{format_value(metrics["p99_abs"])} | {format_value(metrics["aligned_mae"])} | '
309
+ f'{format_value(metrics["aligned_rmse"])} | {format_value(metrics["rel_error"])} | '
310
+ f'{format_value(sample["onnx_ms"])} | {format_value(sample["axmodel_ms"])} |')
311
+ lines.append('')
312
+ lines.append('## 指标说明')
313
+ lines.append('')
314
+ lines.append('- `cosine`:两路输出的余弦相似度,1 表示方向完全一致。')
315
+ lines.append('- `pearson`:线性相关系数,衡量结构一致性,对全局尺度不敏感。')
316
+ lines.append('- `mae` / `rmse`:未对齐的绝对误差,包含量化误差与整体尺度漂移。')
317
+ lines.append('- `max_abs` / `p99_abs`:最大误差和 99 分位误差,反映局部离群点。')
318
+ lines.append('- `aligned_mae` / `aligned_rmse`:先做最小二乘 scale-shift 对齐再计算,'
319
+ '用于区分全局尺度漂移和真正的结构偏差。')
320
+ lines.append('- `rel_error`:`mae / mean(|ONNX|)`,便于跨图片比较。')
321
+ lines.append('')
322
+ lines.append('## 判定参考')
323
+ lines.append('')
324
+ lines.append(f'- 余弦相似度均值:{format_value(report["summary"]["cosine"]["mean"])}')
325
+ lines.append(f'- 对齐后 MAE 均值:{format_value(report["summary"]["aligned_mae"]["mean"])}')
326
+ lines.append(f'- 未���齐相对误差均值:'
327
+ f'{format_value(report["summary"]["rel_error"]["mean"])}')
328
+ lines.append('')
329
+ lines.append('若 `cosine` 接近 1 但 `mae` 明显偏大、`aligned_mae` 很小,说明两路输出只存在'
330
+ '整体尺度/偏移差异;若 `aligned_mae` 也明显偏大,则更可能是量化本身造成的'
331
+ '结构失真,需要结合逐层精度分析继续定位。')
332
+ path.parent.mkdir(parents=True, exist_ok=True)
333
+ path.write_text('\n'.join(lines) + '\n', encoding='utf-8')
334
+
335
+
336
+ def collect_images(dataset):
337
+ paths = sorted(p for p in dataset.rglob('*')
338
+ if p.is_file() and p.suffix.lower() in EXTENSIONS)
339
+ if not paths:
340
+ raise ValueError(f'目录中没有图片: {dataset}')
341
+ return paths
342
+
343
+
344
+ def compare_result_dirs(args):
345
+ from tqdm import tqdm
346
+
347
+ files = []
348
+ for directory in (args.onnx_results, args.axmodel_results):
349
+ if not directory.is_dir():
350
+ raise ValueError(f'结果目录不存在: {directory}')
351
+ mapping = {p.relative_to(directory).as_posix(): p
352
+ for p in directory.rglob('*.depth.npy') if p.is_file()}
353
+ if not mapping:
354
+ raise ValueError(f'结果目录中没有 *.depth.npy: {directory}')
355
+ files.append(mapping)
356
+ onnx_files, axmodel_files = files
357
+ if onnx_files.keys() != axmodel_files.keys():
358
+ onnx_only = sorted(onnx_files.keys() - axmodel_files.keys())
359
+ axmodel_only = sorted(axmodel_files.keys() - onnx_files.keys())
360
+ raise ValueError(f'结果文件未完全配对: ONNX 独有 {len(onnx_only)} 个 '
361
+ f'{onnx_only[:5]}, AXMODEL 独有 {len(axmodel_only)} 个 '
362
+ f'{axmodel_only[:5]}')
363
+ names = sorted(onnx_files)
364
+ if args.limit is not None:
365
+ names = names[:args.limit]
366
+ records = []
367
+ for name in tqdm(names, desc='比较深度输出', disable=len(names) == 1):
368
+ reference = to_2d(np.load(onnx_files[name], allow_pickle=False), f'ONNX {name}')
369
+ actual = to_2d(np.load(axmodel_files[name], allow_pickle=False), f'AXMODEL {name}')
370
+ if reference.size == 0 or reference.shape != actual.shape:
371
+ raise ValueError(f'{name} 深度图为空或 shape 不一致: '
372
+ f'ONNX {reference.shape} vs AXMODEL {actual.shape}')
373
+ metrics = compare_arrays(reference, actual)
374
+ if args.debug:
375
+ logging.info('%s: %s', name, metrics)
376
+ records.append({'name': name[:-len('.depth.npy')], 'metrics': metrics,
377
+ 'onnx_ms': None, 'axmodel_ms': None})
378
+ report = {
379
+ 'mode': 'results',
380
+ 'created': datetime.now().astimezone().isoformat(timespec='seconds'),
381
+ 'output_dir': str(args.output.resolve().parent),
382
+ 'onnx_results': str(args.onnx_results.resolve()),
383
+ 'axmodel_results': str(args.axmodel_results.resolve()),
384
+ 'sample_count': len(records), 'limit': args.limit,
385
+ 'summary': summarize([record['metrics'] for record in records]),
386
+ 'samples': records,
387
+ }
388
+ write_markdown(args.output, report)
389
+ logging.info('样本数 %d,cosine mean=%.6f,aligned_mae mean=%.6g,报告: %s',
390
+ len(records), report['summary']['cosine']['mean'],
391
+ report['summary']['aligned_mae']['mean'], args.output)
392
+
393
+
394
+ def main():
395
+ parser = argparse.ArgumentParser(
396
+ description='计算 ONNX 与 AXMODEL 深度输出的相似度并生成 Markdown 报告',
397
+ formatter_class=argparse.RawDescriptionHelpFormatter, epilog=__doc__)
398
+ parser.add_argument('--onnx', type=Path, help='FP32 ONNX 模型')
399
+ parser.add_argument('--axmodel', type=Path, help='编译后的 axmodel')
400
+ parser.add_argument('--onnx-results', type=Path, help='ONNX 结果目录(递归读取 *.depth.npy)')
401
+ parser.add_argument('--axmodel-results', type=Path, help='AXMODEL 结果目录(相对路径必须完全配对)')
402
+ source = parser.add_mutually_exclusive_group()
403
+ source.add_argument('--image', type=Path, help='单张图片')
404
+ source.add_argument('--dataset', type=Path, help='图片目录(递归)')
405
+ parser.add_argument('--output', type=Path,
406
+ default=Path(__file__).resolve().parents[2] / 'results' /
407
+ 'compare' / 'onnx_vs_axmodel.md',
408
+ help='Markdown 输出路径,默认 results/compare/onnx_vs_axmodel.md')
409
+ parser.add_argument('--resize-mode', choices=('stretch', 'pad'), default='stretch',
410
+ help='两个后端共用的 resize 方式,默认 stretch')
411
+ parser.add_argument('--color-order', choices=('rgb', 'bgr'), default='rgb',
412
+ help='两个后端共用的颜色顺序,默认 rgb')
413
+ parser.add_argument('--onnx-norm', choices=('imagenet', 'raw'), default='imagenet',
414
+ help='ONNX 输入归一化,默认 imagenet')
415
+ parser.add_argument('--axmodel-norm', choices=('imagenet', 'raw'), default='raw',
416
+ help='axmodel 输入归一化,默认 raw(编译时已吸收归一化)')
417
+ parser.add_argument('--warmup', type=int, default=3,
418
+ help='数据集模式每个后端的 warmup 次数,默认 3')
419
+ parser.add_argument('--limit', type=int, help='数据集模式只处理前 N 张')
420
+ parser.add_argument('--debug', action='store_true', help='打印逐样本输入/输出统计')
421
+ args = parser.parse_args()
422
+ logging.basicConfig(level=logging.INFO, format='%(message)s')
423
+ if args.warmup < 0:
424
+ parser.error('--warmup 不能为负数')
425
+ if args.limit is not None and args.limit < 1:
426
+ parser.error('--limit 必须为正数')
427
+ if args.onnx_results is not None or args.axmodel_results is not None:
428
+ if args.onnx_results is None or args.axmodel_results is None:
429
+ parser.error('--onnx-results 和 --axmodel-results 必须同时提供')
430
+ if any(value is not None for value in (args.onnx, args.axmodel, args.image, args.dataset)):
431
+ parser.error('结果目录模式不能同时指定模型或图片输入')
432
+ compare_result_dirs(args)
433
+ return
434
+ if args.onnx is None or args.axmodel is None or (args.image is None and args.dataset is None):
435
+ parser.error('推理模式必须提供 --onnx、--axmodel 和 --image/--dataset;'
436
+ '或使用 --onnx-results 和 --axmodel-results 进行离线比较')
437
+ if not args.onnx.is_file():
438
+ parser.error(f'--onnx 不存在: {args.onnx}')
439
+ if not args.axmodel.is_file():
440
+ parser.error(f'--axmodel 不存在: {args.axmodel}')
441
+
442
+ import onnxruntime as ort
443
+ import axengine
444
+ onnx_session = ort.InferenceSession(str(args.onnx))
445
+ axmodel_session = axengine.InferenceSession(str(args.axmodel))
446
+ onnx_input, onnx_size, onnx_layout, onnx_dtype, onnx_output = onnx_interface(onnx_session)
447
+ ax_meta, ax_size, ax_layout, ax_dtype, ax_outputs = axmodel_interface(axmodel_session)
448
+ if onnx_size != ax_size:
449
+ raise ValueError(f'两个后端输入尺寸不一致: ONNX {onnx_size} vs AXMODEL {ax_size}')
450
+
451
+ if args.image:
452
+ if not args.image.is_file():
453
+ parser.error(f'--image 不是文件: {args.image}')
454
+ images = [args.image]
455
+ root = args.image.parent
456
+ else:
457
+ if not args.dataset.is_dir():
458
+ parser.error(f'--dataset 不是目录: {args.dataset}')
459
+ images = collect_images(args.dataset)
460
+ if args.limit is not None:
461
+ images = images[:args.limit]
462
+ root = args.dataset
463
+
464
+ if args.axmodel_norm == 'imagenet' and ax_dtype != np.float32:
465
+ parser.error('axmodel 输入为整数类型,不能使用 --axmodel-norm imagenet')
466
+
467
+ # 数据集模式先 warmup,使用第一张图。
468
+ if len(images) > 1 and args.warmup:
469
+ first = cv2.imread(str(images[0]), cv2.IMREAD_COLOR)
470
+ if first is None:
471
+ raise ValueError(f'读取图片失败: {images[0]}')
472
+ onnx_tensor, _ = preprocess(first, onnx_size, onnx_layout, onnx_dtype,
473
+ args.resize_mode, args.color_order, args.onnx_norm)
474
+ ax_tensor, _ = preprocess(first, ax_size, ax_layout, ax_dtype,
475
+ args.resize_mode, args.color_order, args.axmodel_norm)
476
+ for _ in range(args.warmup):
477
+ onnx_session.run([onnx_output], {onnx_input: onnx_tensor})
478
+ axmodel_session.run(None, {ax_meta.name: ax_tensor})
479
+ logging.info('两个后端各 warmup %d 次', args.warmup)
480
+
481
+ records = []
482
+ for path in images:
483
+ name = str(path.relative_to(root)) if path != root else path.name
484
+ image = cv2.imread(str(path), cv2.IMREAD_COLOR)
485
+ if image is None:
486
+ raise ValueError(f'读取图片失败: {path}')
487
+ onnx_tensor, _ = preprocess(image, onnx_size, onnx_layout, onnx_dtype,
488
+ args.resize_mode, args.color_order, args.onnx_norm)
489
+ ax_tensor, _ = preprocess(image, ax_size, ax_layout, ax_dtype,
490
+ args.resize_mode, args.color_order, args.axmodel_norm)
491
+
492
+ start = time.perf_counter()
493
+ onnx_raw = onnx_session.run([onnx_output], {onnx_input: onnx_tensor})[0]
494
+ onnx_ms = (time.perf_counter() - start) * 1000
495
+ start = time.perf_counter()
496
+ ax_outputs_data = axmodel_session.run(None, {ax_meta.name: ax_tensor})
497
+ axmodel_ms = (time.perf_counter() - start) * 1000
498
+
499
+ reference = to_2d(onnx_raw, 'ONNX')
500
+ actual, ax_output_name = select_ax_output(ax_outputs_data, ax_outputs,
501
+ reference.shape)
502
+ metrics = compare_arrays(reference, actual)
503
+ if not np.isfinite(metrics['cosine']):
504
+ raise ValueError(f'{name} 无法计算相似度,请检查输出是否常数')
505
+ if args.debug:
506
+ logging.info('%s: ONNX min=%.4f max=%.4f mean=%.4f | AXMODEL min=%.4f '
507
+ 'max=%.4f mean=%.4f | cosine=%.6f mae=%.6g',
508
+ name, float(reference.min()), float(reference.max()),
509
+ float(reference.mean()), float(actual.min()),
510
+ float(actual.max()), float(actual.mean()),
511
+ metrics['cosine'], metrics['mae'])
512
+ records.append({'name': name, 'metrics': metrics,
513
+ 'onnx_ms': onnx_ms, 'axmodel_ms': axmodel_ms})
514
+ if len(images) == 1:
515
+ logging.info('输出绑定: ONNX=%s AXMODEL=%s', onnx_output, ax_output_name)
516
+ logging.info('指标: %s', json.dumps(metrics, ensure_ascii=False))
517
+
518
+ report = {
519
+ 'created': datetime.now().astimezone().isoformat(timespec='seconds'),
520
+ 'output_dir': str(args.output.resolve().parent),
521
+ 'sample_count': len(records),
522
+ 'limit': args.limit if not args.image else None,
523
+ 'warmup': args.warmup if len(images) > 1 else 0,
524
+ 'resize_mode': args.resize_mode,
525
+ 'color_order': args.color_order,
526
+ 'onnx': {'path': str(args.onnx.resolve()),
527
+ 'runtime': 'onnxruntime ' + ort.__version__,
528
+ 'input': f'{onnx_input} {onnx_tensor.shape}',
529
+ 'output': onnx_output,
530
+ 'norm': args.onnx_norm},
531
+ 'axmodel': {'path': str(args.axmodel.resolve()),
532
+ 'runtime': getattr(axengine, '__version__', 'unknown'),
533
+ 'input': f'{ax_meta.name} {tuple(ax_meta.shape)} {np.dtype(ax_dtype).name}',
534
+ 'output': ax_output_name,
535
+ 'norm': args.axmodel_norm},
536
+ 'summary': summarize([record['metrics'] for record in records]),
537
+ 'samples': records,
538
+ }
539
+ write_markdown(args.output, report)
540
+ logging.info('样本数 %d,cosine mean=%.6f,aligned_mae mean=%.6g,报告: %s',
541
+ len(records), report['summary']['cosine']['mean'],
542
+ report['summary']['aligned_mae']['mean'], args.output)
543
+
544
+
545
+ if __name__ == '__main__':
546
+ main()
src/tool/depthart_export_axonnx.py ADDED
@@ -0,0 +1,555 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # -*- coding: utf-8 -*-
3
+ """
4
+ 把 DepthART ONNX 中的 com.depthart::SelectiveScan 改写为 pulsar2 可编译的纯 ONNX 子图。
5
+
6
+ 背景:
7
+ 官方 relative_* ONNX 含自定义算子 SelectiveScan,pulsar2 前端直接报
8
+ "dont support SelectiveScan opr"。官方只提供 TensorRT 插件,没有 NPU 实现。
9
+ 推理脚本里的 Loop 改写可以跑通 onnxruntime,但 AX 工具链不支持 Loop 控制流。
10
+
11
+ 做法:
12
+ 选择性扫描是线性递推 h_t = exp(dt_t*A) * h_{t-1} + dt_t*B_t*u_t,可写成
13
+ h_t = exp(L_t) * h_{-1} + sum_{s<=t} exp(L_t - L_s) * dt_s*B_s*u_s
14
+ 其中 L_t = cumsum_{r<=t}(dt_r*A)。把时间轴切成固定长度 T(默认 14),每个
15
+ 分块内用 CumSum + 下三角 mask + MatMul 一次算出,块间只传一个状态。所有
16
+ 指数项的指数都 <= 0,不会溢出。所有算子(CumSum/Exp/Where/MatMul 等)都在
17
+ AX 前端支持列表内,不含 Loop/Scan/自定义算子。
18
+
19
+ 校验:
20
+ - 本工具用 Identity 替换 scan 跑一遍,拿到每个 scan 的 D/G/N/L,再按实际
21
+ 尺寸生成静态图;时间轴不是 T 整数倍时自动补零并在输出处裁掉。
22
+ - 改写后的整模型输出与已验证的 Loop 版本最大绝对误差约 6e-6(FP32)。
23
+ - 同时把 4D `Einsum` 换成按 k 的批 MatMul(`b k d l, k c d -> b k c l`
24
+ 和 `b k r l, k d r -> b k d l`):pulsar2 的量化执行器不支持 4D Einsum,
25
+ 会报 Op Execution Error。
26
+
27
+ 用法(在 hs/DepthART 目录下执行):
28
+ # 默认:448 模型,块大小 14,输出 onnx/relative_s_448_default_axonnx.onnx
29
+ ../../venv/bin/python src/tool/depthart_export_axonnx.py
30
+
31
+ # 指定模型/输出/块大小
32
+ ../../venv/bin/python src/tool/depthart_export_axonnx.py \\
33
+ --model onnx/relative_b_448_default.onnx \\
34
+ --output onnx/relative_b_448_default_axonnx.onnx --chunk-size 14
35
+
36
+ # 改写后用推理脚本验证数值
37
+ ../../venv/bin/python src/infer/depthart_infer_onnx.py \\
38
+ --model onnx/relative_s_448_default_axonnx.onnx --image /path/a.jpg
39
+
40
+ 注意:
41
+ - 块内两两指数矩阵形状为 [D*N, T, T],T 越大显存/内存占用越大;
42
+ T=14 时 672 通道层约 1M 元素(4 MB),推荐保持默认。
43
+ - 输出是标准 ONNX,不再含自定义算子;原模型文件不会被修改。
44
+ """
45
+
46
+ import argparse
47
+ import logging
48
+ from pathlib import Path
49
+
50
+ import numpy as np
51
+ import onnx
52
+ import onnxruntime as ort
53
+ from onnx import helper, numpy_helper, TensorProto
54
+
55
+ PROJECT_ROOT = Path(__file__).resolve().parents[2]
56
+ DEFAULT_MODEL = PROJECT_ROOT / 'onnx' / 'relative_s_448_default.onnx'
57
+ DOMAIN = 'com.depthart'
58
+ OP = 'SelectiveScan'
59
+
60
+
61
+ EINSUM_EQUATIONS = {
62
+ 'b k d l, k c d -> b k c l',
63
+ 'b k r l, k d r -> b k d l',
64
+ }
65
+
66
+
67
+ def collect_runtime_shapes(model):
68
+ """用 ORT 实际跑一遍,读取 scan 与 Einsum 的静态形状。
69
+
70
+ scan 会被替换成 Identity(输出形状等于 u,不影响下游形状);Einsum 正常执行。
71
+ 返回 (scan_shapes, einsum_shapes):
72
+ scan_shapes : 按图顺序的 [(D, G, N, L), ...]
73
+ einsum_shapes: {节点名: (x_shape, y_shape)}
74
+ """
75
+ nodes = []
76
+ scan_outputs, b_aliases = [], {}
77
+ einsum_x, einsum_y = {}, {}
78
+ for index, node in enumerate(model.graph.node):
79
+ if node.domain == DOMAIN and node.op_type == OP:
80
+ nodes.append(helper.make_node('Identity', [node.input[0]], [node.output[0]],
81
+ name=f'depthart_stub_scan_{index}'))
82
+ alias = f'depthart_stub_b_{index}'
83
+ nodes.append(helper.make_node('Identity', [node.input[3]], [alias],
84
+ name=f'depthart_stub_bnode_{index}'))
85
+ scan_outputs.append(node.output[0])
86
+ b_aliases[node.output[0]] = alias
87
+ else:
88
+ if node.op_type == 'Einsum':
89
+ x_alias = f'depthart_stub_einsum_x_{index}'
90
+ y_alias = f'depthart_stub_einsum_y_{index}'
91
+ nodes.append(helper.make_node('Identity', [node.input[0]], [x_alias],
92
+ name=f'{x_alias}_node'))
93
+ nodes.append(helper.make_node('Identity', [node.output[0]], [y_alias],
94
+ name=f'{y_alias}_node'))
95
+ einsum_x[node.name] = x_alias
96
+ einsum_y[node.name] = y_alias
97
+ nodes.append(node)
98
+
99
+ stub = onnx.ModelProto()
100
+ stub.CopyFrom(model)
101
+ del stub.graph.node[:]
102
+ stub.graph.node.extend(nodes)
103
+ for name in scan_outputs:
104
+ stub.graph.output.append(helper.make_tensor_value_info(name, TensorProto.FLOAT, None))
105
+ stub.graph.output.append(helper.make_tensor_value_info(b_aliases[name],
106
+ TensorProto.FLOAT, None))
107
+ for x_alias, y_alias in zip(einsum_x.values(), einsum_y.values()):
108
+ stub.graph.output.append(helper.make_tensor_value_info(x_alias, TensorProto.FLOAT, None))
109
+ stub.graph.output.append(helper.make_tensor_value_info(y_alias, TensorProto.FLOAT, None))
110
+
111
+ meta = model.graph.input[0]
112
+ shape = [dim.dim_value for dim in meta.type.tensor_type.shape.dim]
113
+ if not all(shape):
114
+ raise ValueError(f'输入必须是静态形状: {meta.name} {shape}')
115
+ session = ort.InferenceSession(stub.SerializeToString())
116
+ values = dict(zip([out.name for out in session.get_outputs()],
117
+ session.run(None, {meta.name: np.zeros(shape, dtype=np.float32)})))
118
+
119
+ scan_map = {}
120
+ for name in scan_outputs:
121
+ u_shape = values[name].shape
122
+ b_shape = values[b_aliases[name]].shape
123
+ if len(u_shape) != 3 or len(b_shape) != 4:
124
+ raise ValueError(f'scan 形状不符合预期: u={u_shape} B={b_shape}')
125
+ scan_map[name] = (int(u_shape[1]), int(b_shape[1]), int(b_shape[2]), int(u_shape[2]))
126
+ scan_shapes = [scan_map[node.output[0]] for node in model.graph.node
127
+ if node.domain == DOMAIN and node.op_type == OP]
128
+ einsum_shapes = {name: (values[einsum_x[name]].shape, values[einsum_y[name]].shape)
129
+ for name in einsum_x}
130
+ return scan_shapes, einsum_shapes
131
+
132
+
133
+ class Rewriter:
134
+ def __init__(self, model, chunk_size, stub_scan=False, scan_method='pairwise'):
135
+ self.model = model
136
+ self.chunk_size = chunk_size
137
+ self.stub_scan = stub_scan
138
+ self.scan_method = scan_method
139
+ self.initializers = list(model.graph.initializer)
140
+ self.names = {tensor.name for tensor in self.initializers}
141
+ self.initializer_shapes = {tensor.name: tuple(tensor.dims)
142
+ for tensor in model.graph.initializer}
143
+ self.shared = {}
144
+
145
+ def build_einsum(self, prefix, node, x_shape, y_shape):
146
+ """b k d l, k c d -> b k c l (以及 b k r l, k d r -> b k d l)。
147
+
148
+ 两条式子都是按 k 的批 MatMul:W[k,out,in] @ x[b=1,k,in,l]。
149
+ 替换掉 4D Einsum,避免量化执行器不支持。
150
+ """
151
+ out_name = node.output[0]
152
+ w_shape = self.initializer_shapes.get(node.input[1])
153
+ if w_shape is None:
154
+ raise ValueError(f'{node.name}: 第二个输入不是常量权重,暂不支持')
155
+ if len(w_shape) != 3 or len(x_shape) != 4:
156
+ raise ValueError(f'{node.name}: Einsum 形状不符合预期: {w_shape} {x_shape}')
157
+ if int(x_shape[0]) != 1:
158
+ raise ValueError(f'{node.name}: 仅支持 batch=1,实际 {x_shape}')
159
+ K, out_dim, in_dim = (int(v) for v in w_shape)
160
+ length = int(x_shape[3])
161
+ if int(x_shape[1]) != K:
162
+ raise ValueError(f'{node.name}: K 不一致: {x_shape} vs {w_shape}')
163
+ expected = (1, K, out_dim, length)
164
+ if tuple(int(v) for v in y_shape) != expected:
165
+ raise ValueError(f'{node.name}: 输出形状 {y_shape} != {expected}')
166
+ nodes = []
167
+ # 用 Squeeze/Unsqueeze 而不是 Reshape 还原 batch 维:onnxslim 会把
168
+ # MatMul 前后的 Reshape 当成冗余节点合并/改名,导致下游张量断链。
169
+ nodes.append(helper.make_node(
170
+ 'Squeeze', [node.input[0], self.shared_const('axes0', np.array([0], np.int64))],
171
+ [f'{prefix}_x2']))
172
+ nodes.append(helper.make_node('MatMul', [node.input[1], f'{prefix}_x2'],
173
+ [f'{prefix}_y3']))
174
+ nodes.append(helper.make_node(
175
+ 'Unsqueeze', [f'{prefix}_y3', self.shared_const('axes0', np.array([0], np.int64))],
176
+ [out_name]))
177
+ return nodes
178
+
179
+ def const(self, name, value):
180
+ if name not in self.names:
181
+ self.initializers.append(numpy_helper.from_array(np.asarray(value), name))
182
+ self.names.add(name)
183
+ return name
184
+
185
+ def shared_const(self, key, value):
186
+ if key not in self.shared:
187
+ self.shared[key] = self.const(f'depthart_{key}', value)
188
+ return self.shared[key]
189
+
190
+ def build_logspace(self, prefix, shapes, inputs):
191
+ """对数空间分块递推:每块 O(T) 而非 O(T^2),不构造 [D*N,T,T] 两两矩阵。
192
+
193
+ h_t = exp(L_t) * ( h_prev + cumsum_s( exp(-L_s) * dt_s * B_s * u_s ) )
194
+ 其中 L 为块内相对累计衰减(L_0=0,L_t<=0)。要求块内 exp(-L) 不超 U16 量程,
195
+ 因此 T 要小(T=4 时最大约 e^9≈8e3)。
196
+ """
197
+ D, G, N, L = shapes
198
+ u, delta, a, b, c = inputs[:5]
199
+ d = inputs[5] if len(inputs) > 5 and inputs[5] else None
200
+ bias = inputs[6] if len(inputs) > 6 and inputs[6] else None
201
+ attrs = {item.name: item.i for item in self.node_attrs}
202
+ T = self.chunk_size
203
+ nodes = []
204
+
205
+ dt = delta
206
+ if bias is not None:
207
+ bias3 = self.const(f'{prefix}_bias3_shape', np.array([1, D, 1], np.int64))
208
+ nodes.append(helper.make_node('Reshape', [bias, bias3], [f'{prefix}_bias3']))
209
+ nodes.append(helper.make_node('Add', [delta, f'{prefix}_bias3'], [f'{prefix}_dtb']))
210
+ dt = f'{prefix}_dtb'
211
+ if int(attrs.get('delta_softplus', 0)):
212
+ nodes.append(helper.make_node('Softplus', [dt], [f'{prefix}_dt']))
213
+ dt = f'{prefix}_dt'
214
+
215
+ # 补零到 T 的整数倍,避免最后一块长度不足
216
+ pad = (-L) % T
217
+ Lp = L + pad
218
+ if pad:
219
+ nodes.append(helper.make_node(
220
+ 'Concat', [dt, self.const(f'{prefix}_pad_dt', np.zeros((1, D, pad), np.float32))],
221
+ [f'{prefix}_dt_pad'], axis=2))
222
+ nodes.append(helper.make_node(
223
+ 'Concat', [u, self.const(f'{prefix}_pad_u', np.zeros((1, D, pad), np.float32))],
224
+ [f'{prefix}_u_pad'], axis=2))
225
+ pad_bc = self.const(f'{prefix}_pad_bc', np.zeros((1, G, N, pad), np.float32))
226
+ nodes.append(helper.make_node('Concat', [b, pad_bc], [f'{prefix}_b_pad'], axis=3))
227
+ nodes.append(helper.make_node('Concat', [c, pad_bc], [f'{prefix}_c_pad'], axis=3))
228
+ dt, u, b, c = (f'{prefix}_dt_pad', f'{prefix}_u_pad',
229
+ f'{prefix}_b_pad', f'{prefix}_c_pad')
230
+
231
+ # dtA = dt * A -> [1,D,N,Lp]
232
+ a4 = self.const(f'{prefix}_a4_shape', np.array([1, D, N, 1], np.int64))
233
+ nodes.append(helper.make_node('Reshape', [a, a4], [f'{prefix}_a4']))
234
+ dt3 = self.const(f'{prefix}_dt3_shape', np.array([1, D, 1, Lp], np.int64))
235
+ nodes.append(helper.make_node('Reshape', [dt, dt3], [f'{prefix}_dt4']))
236
+ nodes.append(helper.make_node('Mul', [f'{prefix}_dt4', f'{prefix}_a4'],
237
+ [f'{prefix}_dtA']))
238
+
239
+ b_full = self.expand_groups(b, D, G, N, Lp, f'{prefix}_b', nodes)
240
+ c_full = self.expand_groups(c, D, G, N, Lp, f'{prefix}_c', nodes)
241
+
242
+ if d is not None:
243
+ nodes.append(helper.make_node('Reshape', [d,
244
+ self.const(f'{prefix}_d_shape',
245
+ np.array([1, D, 1], np.int64))],
246
+ [f'{prefix}_d3']))
247
+ d = f'{prefix}_d3'
248
+
249
+ axes_u2 = self.shared_const('axes_u2', np.array([2], np.int64))
250
+ axes_rs = self.shared_const('axes_rs2', np.array([2], np.int64))
251
+ axis_t = self.shared_const('axis2', np.array([2], np.int64))
252
+ axis_bc = self.shared_const('axis3', np.array([3], np.int64))
253
+ h_prev = self.const(f'{prefix}_h0', np.zeros((1, D, N, 1), np.float32))
254
+ outputs = []
255
+ for k in range(Lp // T):
256
+ start, end = k * T, (k + 1) * T
257
+ tag = f'{prefix}_c{k}'
258
+ starts = self.const(f'{tag}_starts', np.array([start], np.int64))
259
+ ends = self.const(f'{tag}_ends', np.array([end], np.int64))
260
+ for source, suffix, axis in ((f'{prefix}_dtA', 'dta', axis_bc),
261
+ (dt, 'dt', axis_t), (u, 'u', axis_t),
262
+ (b_full, 'b', axis_bc), (c_full, 'c', axis_bc)):
263
+ nodes.append(helper.make_node('Slice', [source, starts, ends, axis],
264
+ [f'{tag}_{suffix}']))
265
+
266
+ # L = cumsum(dtA)(块内,含首步);h = exp(L) * (h_prev + cumsum(exp(-L)*b))
267
+ nodes.append(helper.make_node('CumSum', [f'{tag}_dta',
268
+ self.shared_const('axis3s', np.array(3, np.int64))],
269
+ [f'{tag}_L']))
270
+ nodes.append(helper.make_node('Exp', [f'{tag}_L'], [f'{tag}_eL']))
271
+ nodes.append(helper.make_node('Neg', [f'{tag}_L'], [f'{tag}_negL']))
272
+ nodes.append(helper.make_node('Exp', [f'{tag}_negL'], [f'{tag}_eneg']))
273
+
274
+ # b = dt * u * B;G = cumsum(exp(-Lp) * b)
275
+ nodes.append(helper.make_node('Unsqueeze', [f'{tag}_dt', axes_u2], [f'{tag}_dt3']))
276
+ nodes.append(helper.make_node('Unsqueeze', [f'{tag}_u', axes_u2], [f'{tag}_u3']))
277
+ nodes.append(helper.make_node('Mul', [f'{tag}_dt3', f'{tag}_u3'], [f'{tag}_dtu']))
278
+ nodes.append(helper.make_node('Mul', [f'{tag}_dtu', f'{tag}_b'], [f'{tag}_bt']))
279
+ nodes.append(helper.make_node('Mul', [f'{tag}_eneg', f'{tag}_bt'], [f'{tag}_g']))
280
+ nodes.append(helper.make_node('CumSum', [f'{tag}_g',
281
+ self.shared_const('axis3s', np.array(3, np.int64))],
282
+ [f'{tag}_G']))
283
+ nodes.append(helper.make_node('Add', [f'{tag}_G', h_prev], [f'{tag}_hp']))
284
+ nodes.append(helper.make_node('Mul', [f'{tag}_eL', f'{tag}_hp'], [f'{tag}_h']))
285
+
286
+ nodes.append(helper.make_node('Mul', [f'{tag}_h', f'{tag}_c'], [f'{tag}_hc']))
287
+ nodes.append(helper.make_node('ReduceSum', [f'{tag}_hc', axes_rs],
288
+ [f'{tag}_ysum'], keepdims=0))
289
+ if d is not None:
290
+ nodes.append(helper.make_node('Mul', [d, f'{tag}_u'], [f'{tag}_du']))
291
+ nodes.append(helper.make_node('Add', [f'{tag}_ysum', f'{tag}_du'],
292
+ [f'{tag}_y']))
293
+ else:
294
+ nodes.append(helper.make_node('Identity', [f'{tag}_ysum'], [f'{tag}_y']))
295
+ outputs.append(f'{tag}_y')
296
+
297
+ nodes.append(helper.make_node('Slice', [f'{tag}_h',
298
+ self.const(f'{tag}_hl', np.array([T - 1], np.int64)),
299
+ self.const(f'{tag}_he', np.array([T], np.int64)),
300
+ axis_bc], [f'{tag}_hnext']))
301
+ h_prev = f'{tag}_hnext'
302
+
303
+ nodes.append(helper.make_node('Concat', outputs, [f'{prefix}_ycat'], axis=2))
304
+ if pad:
305
+ nodes.append(helper.make_node('Slice', [
306
+ f'{prefix}_ycat',
307
+ self.const(f'{prefix}_out_start', np.array([0], np.int64)),
308
+ self.const(f'{prefix}_out_end', np.array([L], np.int64)),
309
+ self.shared_const('axis2', np.array([2], np.int64))], [self.out_name]))
310
+ else:
311
+ nodes.append(helper.make_node('Identity', [f'{prefix}_ycat'], [self.out_name]))
312
+ return nodes
313
+
314
+ def build(self, prefix, shapes, inputs):
315
+ D, G, N, L = shapes
316
+ u, delta, a, b, c = inputs[:5]
317
+ d = inputs[5] if len(inputs) > 5 and inputs[5] else None
318
+ bias = inputs[6] if len(inputs) > 6 and inputs[6] else None
319
+ attrs = {item.name: item.i for item in self.node_attrs}
320
+ T = self.chunk_size
321
+ nodes = []
322
+
323
+ # dt = softplus(delta + bias)
324
+ dt = delta
325
+ if bias is not None:
326
+ bias3 = self.const(f'{prefix}_bias3_shape', np.array([1, D, 1], np.int64))
327
+ nodes.append(helper.make_node('Reshape', [bias, bias3], [f'{prefix}_bias3']))
328
+ nodes.append(helper.make_node('Add', [delta, f'{prefix}_bias3'], [f'{prefix}_dtb']))
329
+ dt = f'{prefix}_dtb'
330
+ if int(attrs.get('delta_softplus', 0)):
331
+ nodes.append(helper.make_node('Softplus', [dt], [f'{prefix}_dt']))
332
+ dt = f'{prefix}_dt'
333
+
334
+ # 时间轴补零到 T 的整数倍
335
+ pad = (-L) % T
336
+ Lp = L + pad
337
+ if pad:
338
+ nodes.append(helper.make_node(
339
+ 'Concat', [dt, self.const(f'{prefix}_pad_dt', np.zeros((1, D, pad), np.float32))],
340
+ [f'{prefix}_dt_pad'], axis=2))
341
+ nodes.append(helper.make_node(
342
+ 'Concat', [u, self.const(f'{prefix}_pad_u', np.zeros((1, D, pad), np.float32))],
343
+ [f'{prefix}_u_pad'], axis=2))
344
+ pad_bc = self.const(f'{prefix}_pad_bc', np.zeros((1, G, N, pad), np.float32))
345
+ nodes.append(helper.make_node('Concat', [b, pad_bc], [f'{prefix}_b_pad'], axis=3))
346
+ nodes.append(helper.make_node('Concat', [c, pad_bc], [f'{prefix}_c_pad'], axis=3))
347
+ dt, u, b, c = (f'{prefix}_dt_pad', f'{prefix}_u_pad',
348
+ f'{prefix}_b_pad', f'{prefix}_c_pad')
349
+
350
+ # B/C 按组连续复制:[1,G,N,Lp] -> [1,D,N,Lp]
351
+ b_full = self.expand_groups(b, D, G, N, Lp, f'{prefix}_b', nodes)
352
+ c_full = self.expand_groups(c, D, G, N, Lp, f'{prefix}_c', nodes)
353
+
354
+ # A -> [1,D,N,1],dt -> [1,D,N,T],再摊平成 [D*N,T] 做 CumSum
355
+ a4 = self.const(f'{prefix}_a_shape', np.array([1, D, N, 1], np.int64))
356
+ nodes.append(helper.make_node('Reshape', [a, a4], [f'{prefix}_a4']))
357
+ if d is not None:
358
+ d3 = self.const(f'{prefix}_d_shape', np.array([1, D, 1], np.int64))
359
+ nodes.append(helper.make_node('Reshape', [d, d3], [f'{prefix}_d3']))
360
+ d = f'{prefix}_d3'
361
+
362
+ mask = np.tril(np.ones((T, T), dtype=bool)).reshape(1, T, T)
363
+ mask_name = self.shared_const(f'mask_{T}', mask)
364
+ neg_name = self.shared_const('mask_fill', np.array(-1e4, np.float32))
365
+ flat_axis = self.shared_const('axis1', np.array(1, np.int64))
366
+ axes_u2 = self.shared_const('axes_u2', np.array([2], np.int64))
367
+ axes_u3 = self.shared_const('axes_u3', np.array([3], np.int64))
368
+ axes_rs = self.shared_const('axes_rs2', np.array([2], np.int64))
369
+
370
+ h_prev = self.const(f'{prefix}_h0', np.zeros((1, D, N, 1), np.float32))
371
+ outputs = []
372
+ for k in range(Lp // T):
373
+ start, end = k * T, (k + 1) * T
374
+ tag = f'{prefix}_c{k}'
375
+ starts = self.const(f'{tag}_starts', np.array([start], np.int64))
376
+ ends = self.const(f'{tag}_ends', np.array([end], np.int64))
377
+ axis_t = self.shared_const('axis2', np.array([2], np.int64))
378
+ axis_bc = self.shared_const('axis3', np.array([3], np.int64))
379
+ for source, suffix, axis in ((dt, 'dt', axis_t), (u, 'u', axis_t),
380
+ (b_full, 'b', axis_bc), (c_full, 'c', axis_bc)):
381
+ nodes.append(helper.make_node('Slice', [source, starts, ends, axis],
382
+ [f'{tag}_{suffix}']))
383
+
384
+ # dtA = dt * A -> [1,D,N,T] -> [D*N,T],L = cumsum_t(dtA)
385
+ nodes.append(helper.make_node('Unsqueeze', [f'{tag}_dt', axes_u2], [f'{tag}_dt3']))
386
+ nodes.append(helper.make_node('Mul', [f'{tag}_dt3', f'{prefix}_a4'], [f'{tag}_dtA']))
387
+ nodes.append(helper.make_node('Reshape', [f'{tag}_dtA',
388
+ self.const(f'{tag}_dtA2_shape',
389
+ np.array([D * N, T], np.int64))],
390
+ [f'{tag}_dtA2']))
391
+ nodes.append(helper.make_node('CumSum', [f'{tag}_dtA2', flat_axis], [f'{tag}_L2']))
392
+ nodes.append(helper.make_node('Reshape', [f'{tag}_L2',
393
+ self.const(f'{tag}_L_shape',
394
+ np.array([1, D, N, T], np.int64))],
395
+ [f'{tag}_L']))
396
+
397
+ # 两两指数矩阵 E[t,s] = exp(L_t - L_s)(s>t 置为 -1e4 后 exp 为 0)
398
+ nodes.append(helper.make_node('Reshape', [f'{tag}_L2',
399
+ self.const(f'{tag}_Lt_shape',
400
+ np.array([D * N, T, 1], np.int64))],
401
+ [f'{tag}_Lt']))
402
+ nodes.append(helper.make_node('Reshape', [f'{tag}_L2',
403
+ self.const(f'{tag}_Ls_shape',
404
+ np.array([D * N, 1, T], np.int64))],
405
+ [f'{tag}_Ls']))
406
+ nodes.append(helper.make_node('Sub', [f'{tag}_Lt', f'{tag}_Ls'], [f'{tag}_diff']))
407
+ nodes.append(helper.make_node('Where', [mask_name, f'{tag}_diff', neg_name],
408
+ [f'{tag}_diffm']))
409
+ nodes.append(helper.make_node('Exp', [f'{tag}_diffm'], [f'{tag}_E']))
410
+
411
+ # b = dt * u * B,形状 [D*N,1,T];contrib[t] = sum_{s<=t} E[t,s]*b[s]
412
+ nodes.append(helper.make_node('Unsqueeze', [f'{tag}_u', axes_u2], [f'{tag}_u3']))
413
+ nodes.append(helper.make_node('Mul', [f'{tag}_dt3', f'{tag}_u3'], [f'{tag}_dtu']))
414
+ nodes.append(helper.make_node('Mul', [f'{tag}_dtu', f'{tag}_b'], [f'{tag}_bt']))
415
+ nodes.append(helper.make_node('Reshape', [f'{tag}_bt',
416
+ self.const(f'{tag}_bt2_shape',
417
+ np.array([D * N, 1, T], np.int64))],
418
+ [f'{tag}_bt2']))
419
+ nodes.append(helper.make_node('Mul', [f'{tag}_E', f'{tag}_bt2'], [f'{tag}_eb']))
420
+ nodes.append(helper.make_node('ReduceSum', [f'{tag}_eb',
421
+ self.shared_const('axis2s', np.array([2], np.int64))],
422
+ [f'{tag}_contrib2'], keepdims=0))
423
+ nodes.append(helper.make_node('Reshape', [f'{tag}_contrib2',
424
+ self.const(f'{tag}_contrib_shape',
425
+ np.array([1, D, N, T], np.int64))],
426
+ [f'{tag}_contrib']))
427
+
428
+ # 块间状态:h_t = exp(L_t) * h_prev + contrib
429
+ nodes.append(helper.make_node('Exp', [f'{tag}_L'], [f'{tag}_expL']))
430
+ nodes.append(helper.make_node('Mul', [f'{tag}_expL', h_prev], [f'{tag}_carry']))
431
+ nodes.append(helper.make_node('Add', [f'{tag}_carry', f'{tag}_contrib'],
432
+ [f'{tag}_h']))
433
+
434
+ # y = sum_n(h*C) + D*u
435
+ nodes.append(helper.make_node('Mul', [f'{tag}_h', f'{tag}_c'], [f'{tag}_hc']))
436
+ nodes.append(helper.make_node('ReduceSum', [f'{tag}_hc', axes_rs],
437
+ [f'{tag}_ysum'], keepdims=0))
438
+ if d is not None:
439
+ nodes.append(helper.make_node('Mul', [d, f'{tag}_u'], [f'{tag}_du']))
440
+ nodes.append(helper.make_node('Add', [f'{tag}_ysum', f'{tag}_du'],
441
+ [f'{tag}_y']))
442
+ else:
443
+ nodes.append(helper.make_node('Identity', [f'{tag}_ysum'], [f'{tag}_y']))
444
+ outputs.append(f'{tag}_y')
445
+
446
+ nodes.append(helper.make_node('Gather', [f'{tag}_h',
447
+ self.const(f'{tag}_last', np.array(T - 1, np.int64))],
448
+ [f'{tag}_hlast'], axis=3))
449
+ nodes.append(helper.make_node('Unsqueeze', [f'{tag}_hlast', axes_u3],
450
+ [f'{tag}_hnext']))
451
+ h_prev = f'{tag}_hnext'
452
+
453
+ nodes.append(helper.make_node('Concat', outputs, [f'{prefix}_ycat'], axis=2))
454
+ if pad:
455
+ nodes.append(helper.make_node('Slice', [
456
+ f'{prefix}_ycat',
457
+ self.const(f'{prefix}_out_start', np.array([0], np.int64)),
458
+ self.const(f'{prefix}_out_end', np.array([L], np.int64)),
459
+ self.shared_const('axis2', np.array([2], np.int64))], [self.out_name]))
460
+ else:
461
+ nodes.append(helper.make_node('Identity', [f'{prefix}_ycat'], [self.out_name]))
462
+ return nodes
463
+
464
+ def expand_groups(self, x, D, G, N, L, prefix, nodes):
465
+ gs = D // G
466
+ r1_shape = self.const(f'{prefix}_r1_shape', np.array([1, G, 1, N, L], np.int64))
467
+ nodes.append(helper.make_node('Reshape', [x, r1_shape], [f'{prefix}_r1']))
468
+ target = self.const(f'{prefix}_tgt', np.array([1, G, gs, N, L], np.int64))
469
+ nodes.append(helper.make_node('Expand', [f'{prefix}_r1', target], [f'{prefix}_e']))
470
+ r2_shape = self.const(f'{prefix}_r2_shape', np.array([1, D, N, L], np.int64))
471
+ nodes.append(helper.make_node('Reshape', [f'{prefix}_e', r2_shape], [f'{prefix}_bc4']))
472
+ return f'{prefix}_bc4'
473
+
474
+ def rewrite(self):
475
+ shapes, einsum_shapes = collect_runtime_shapes(self.model)
476
+ if not shapes:
477
+ raise ValueError('模型中没有 SelectiveScan 节点,无需改写')
478
+ new_nodes, index, einsum_index = [], 0, 0
479
+ for node in self.model.graph.node:
480
+ if node.domain == DOMAIN and node.op_type == OP:
481
+ if self.stub_scan:
482
+ # 仅用于测 NPU 耗时的对照模型:扫描直接透传 u,精度无意义。
483
+ stub_node = helper.make_node('Identity', [node.input[0]], [node.output[0]],
484
+ name=f'depthart_stub_scan_{index}')
485
+ new_nodes.append(stub_node)
486
+ index += 1
487
+ continue
488
+ self.node_attrs = node.attribute
489
+ self.out_name = node.output[0]
490
+ builder = self.build_logspace if self.scan_method == 'log' else self.build
491
+ nodes = builder(f'depthart_pscan_{index}', shapes[index], node.input)
492
+ new_nodes.extend(nodes)
493
+ index += 1
494
+ elif node.op_type == 'Einsum':
495
+ equation = None
496
+ for attr in node.attribute:
497
+ if attr.name == 'equation':
498
+ equation = attr.s.decode()
499
+ if equation not in EINSUM_EQUATIONS:
500
+ raise ValueError(f'{node.name}: 不支持的 Einsum 方程 {equation!r}')
501
+ x_shape, y_shape = einsum_shapes[node.name]
502
+ nodes = self.build_einsum(f'depthart_einsum_{einsum_index}',
503
+ node, x_shape, y_shape)
504
+ new_nodes.extend(nodes)
505
+ einsum_index += 1
506
+ else:
507
+ new_nodes.append(node)
508
+ out = onnx.ModelProto()
509
+ out.CopyFrom(self.model)
510
+ del out.graph.node[:]
511
+ for position, node in enumerate(new_nodes):
512
+ if not node.name:
513
+ node.name = f'depthart_gen_{position}'
514
+ out.graph.node.extend(new_nodes)
515
+ out.graph.initializer.extend(self.initializers[len(self.model.graph.initializer):])
516
+ return out
517
+
518
+
519
+ def main():
520
+ parser = argparse.ArgumentParser(
521
+ description='把 DepthART SelectiveScan 改写为 pulsar2 可编译的纯 ONNX 子图')
522
+ parser.add_argument('--model', type=Path, default=DEFAULT_MODEL,
523
+ help=f'输入 ONNX,默认 {DEFAULT_MODEL.relative_to(PROJECT_ROOT)}')
524
+ parser.add_argument('--output', type=Path,
525
+ default=PROJECT_ROOT / 'onnx' / 'relative_s_448_default_axonnx.onnx',
526
+ help='输出 ONNX(不会覆盖输入模型)')
527
+ parser.add_argument('--chunk-size', type=int, default=14,
528
+ help='分块长度 T,默认 14;越大显存占用越高')
529
+ parser.add_argument('--stub-scan', action='store_true',
530
+ help='把 SelectiveScan 替换为 Identity,生成仅用于测 NPU 耗时的对照模型(精度无意义)')
531
+ parser.add_argument('--scan-method', choices=('pairwise', 'log'), default='pairwise',
532
+ help='pairwise=两两指数矩阵(默认);log=对数空间递推,无 T^2 矩阵,T 需小')
533
+ args = parser.parse_args()
534
+ logging.basicConfig(level=logging.INFO, format='%(message)s')
535
+ if args.chunk_size < 1:
536
+ parser.error('--chunk-size 必须为正数')
537
+ if not args.model.is_file():
538
+ parser.error(f'--model 不存在: {args.model}')
539
+ if args.output.resolve() == args.model.resolve():
540
+ parser.error('--output 不能与 --model 相同')
541
+
542
+ model = onnx.load(str(args.model))
543
+ if not any(node.domain == DOMAIN and node.op_type == OP for node in model.graph.node):
544
+ raise ValueError(f'{args.model} 中没有 {DOMAIN}::{OP} 节点')
545
+ rewriter = Rewriter(model, args.chunk_size, args.stub_scan, args.scan_method)
546
+ rewritten = rewriter.rewrite()
547
+ args.output.parent.mkdir(parents=True, exist_ok=True)
548
+ onnx.save(rewritten, str(args.output))
549
+ logging.info('已写出标准 ONNX: %s(节点数 %d)', args.output, len(rewritten.graph.node))
550
+ logging.info('下一步:把 pulsar2 配置的 input 指向该文件;'
551
+ '可先用 src/infer/depthart_infer_onnx.py 验证数值。')
552
+
553
+
554
+ if __name__ == '__main__':
555
+ main()
src/tool/evaluate_nyu654_depth.py ADDED
@@ -0,0 +1,210 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # -*- coding: utf-8 -*-
3
+ """
4
+ NYU Depth V2 Eigen test split (654 张) 相对深度精度评估。
5
+
6
+ 输入:
7
+ --mat nyu_depth_v2_labeled.mat(GT 直接从这里读,不落盘)
8
+ --split export_nyu654_test.py 生成的 split.json
9
+ --pred 一个或多个 <名称>=<NPZ 目录>,NPZ 由 infer 脚本 --dump-npz 生成
10
+
11
+ 评估协议:
12
+ 1. 有效像素:rawDepths > 0 且 depths 在 (0, max_depth];
13
+ 2. 预测上采样到 GT 分辨率(bilinear);
14
+ 3. 相对深度对齐:
15
+ - disparity(默认,MiDaS 协议):拟合 a*pred + b ≈ 1/gt,再取倒数得米制深度;
16
+ - depth:拟合 a*pred + b ≈ gt;
17
+ 最小二乘在有效像素上求解;
18
+ 4. 逐图计算 AbsRel / SqRel / RMSE / RMSE_log / δ<1.25 / 1.25² / 1.25³,
19
+ 对 654 张取平均;
20
+ 5. 第一个 --pred 作为基准,其余输出相对掉点 (%)。
21
+
22
+ 注意:未做 Eigen crop,绝对数值不能直接与论文表格对比;同一批图上不同后端的
23
+ 相对掉点不受影响。
24
+
25
+ 用法:
26
+ ../../venv/bin/python src/tool/evaluate_nyu654_depth.py \
27
+ --mat ../../datasets/nyu_depth_v2_labeled.mat \
28
+ --split datasets/nyu654_test/split.json \
29
+ --pred fp32=results/nyu654_fp32/npz axonnx=results/nyu654_axonnx/npz \
30
+ --out results/nyu654_eval
31
+ """
32
+
33
+ import argparse
34
+ import json
35
+ from pathlib import Path
36
+
37
+ import cv2
38
+ import h5py
39
+ import numpy as np
40
+ from tqdm import tqdm
41
+
42
+ METRICS = ('AbsRel', 'SqRel', 'RMSE', 'RMSE_log', 'd1', 'd2', 'd3')
43
+ SIGMA = (1.25, 1.25 ** 2, 1.25 ** 3)
44
+
45
+
46
+ def solve_scale_shift(pred, target, mask):
47
+ """在 mask 上最小二乘求解 a, b 使 a*pred + b ≈ target。"""
48
+ p = pred[mask]
49
+ t = target[mask]
50
+ A = np.stack([p, np.ones_like(p)], axis=1)
51
+ (a, b), *_ = np.linalg.lstsq(A, t, rcond=None)
52
+ return float(a), float(b)
53
+
54
+
55
+ def compute_metrics(pred_depth, gt_depth, mask, align, max_depth):
56
+ clipped = 0.0
57
+ if align == 'disparity':
58
+ gt_target = 1.0 / gt_depth
59
+ a, b = solve_scale_shift(pred_depth, gt_target, mask)
60
+ if a <= 0:
61
+ raise ValueError(f'disparity 对齐斜率为负 (a={a:.4f}),'
62
+ f'预测不是"越大越近"的视差量')
63
+ aligned = a * pred_depth + b
64
+ # 极少数图(低纹理/异常预测)会拟出负视差,裁剪到最大深度的下限
65
+ neg = aligned[mask] <= 0
66
+ clipped = float(neg.mean())
67
+ aligned = np.maximum(aligned, 1.0 / max_depth)
68
+ pred = 1.0 / aligned
69
+ else:
70
+ a, b = solve_scale_shift(pred_depth, gt_depth, mask)
71
+ pred = a * pred_depth + b
72
+
73
+ pred = pred[mask]
74
+ gt = gt_depth[mask]
75
+ if np.any(pred <= 0):
76
+ raise ValueError('对齐后预测深度出现非正值')
77
+
78
+ diff = pred - gt
79
+ out = {
80
+ 'AbsRel': float(np.mean(np.abs(diff) / gt)),
81
+ 'SqRel': float(np.mean(diff ** 2 / gt)),
82
+ 'RMSE': float(np.sqrt(np.mean(diff ** 2))),
83
+ 'RMSE_log': float(np.sqrt(np.mean((np.log(pred) - np.log(gt)) ** 2))),
84
+ }
85
+ ratio = np.maximum(pred / gt, gt / pred)
86
+ for name, s in zip(('d1', 'd2', 'd3'), SIGMA):
87
+ out[name] = float(np.mean(ratio < s))
88
+ return out, a, b, clipped
89
+
90
+
91
+ def load_predictions(path):
92
+ """读取预测:汇总 NPZ(names/depths)、分块汇总目录,或旧的逐图 NPZ 目录。"""
93
+ path = Path(path)
94
+ if path.is_file():
95
+ files = [path]
96
+ elif path.is_dir():
97
+ files = sorted(path.glob('*.npz'))
98
+ if not files:
99
+ raise ValueError(f'目录中没有 NPZ: {path}')
100
+ else:
101
+ raise FileNotFoundError(f'预测路径不存在: {path}')
102
+
103
+ preds = {}
104
+ for fp in files:
105
+ with np.load(fp, allow_pickle=False) as data:
106
+ if 'depths' in data and 'names' in data:
107
+ for name, depth in zip(data['names'], data['depths']):
108
+ preds[str(name)] = depth
109
+ elif 'depth' in data:
110
+ preds[fp.name[:-len('.npz')]] = data['depth']
111
+ else:
112
+ raise ValueError(f'无法识别的 NPZ 格式: {fp}')
113
+ return preds
114
+
115
+
116
+ def main():
117
+ parser = argparse.ArgumentParser(description='NYU 654 相对深度评测')
118
+ parser.add_argument('--mat', required=True)
119
+ parser.add_argument('--split', required=True, help='split.json')
120
+ parser.add_argument('--pred', nargs='+', required=True, help='名称=NPZ目录,第一个为基准')
121
+ parser.add_argument('--out', required=True, help='结果目录')
122
+ parser.add_argument('--align', choices=['disparity', 'depth'], default='disparity')
123
+ parser.add_argument('--max-depth', type=float, default=10.0)
124
+ args = parser.parse_args()
125
+
126
+ pred_sets = []
127
+ for item in args.pred:
128
+ name, _, path = item.partition('=')
129
+ if not path:
130
+ raise ValueError(f'--pred 需要 名称=路径 格式: {item}')
131
+ pred_sets.append((name, load_predictions(path)))
132
+
133
+ split = json.load(open(args.split))
134
+ out_dir = Path(args.out)
135
+ out_dir.mkdir(parents=True, exist_ok=True)
136
+
137
+ per_image = {name: [] for name, _ in pred_sets}
138
+ failed = {name: [] for name, _ in pred_sets}
139
+ with h5py.File(args.mat, 'r') as f:
140
+ for record in tqdm(split, desc=f'eval({args.align})', ncols=80):
141
+ i = record['mat_index']
142
+ gt = np.asarray(f['depths'][i], dtype=np.float32)
143
+ raw = np.asarray(f['rawDepths'][i], dtype=np.float32)
144
+ mask = (raw > 0) & (gt > 0) & (gt <= args.max_depth)
145
+
146
+ for name, preds in pred_sets:
147
+ if record['file'] not in preds:
148
+ raise KeyError(f'{name} 缺少预测: {record["file"]}')
149
+ pred = np.asarray(preds[record['file']], dtype=np.float32)
150
+ if pred.ndim == 3:
151
+ pred = pred[0]
152
+ pred = cv2.resize(pred, (gt.shape[1], gt.shape[0]), interpolation=cv2.INTER_LINEAR)
153
+ try:
154
+ m, a, b, clipped = compute_metrics(pred, gt, mask, args.align, args.max_depth)
155
+ except ValueError as error:
156
+ failed[name].append({'file': record['file'], 'reason': str(error)})
157
+ continue
158
+ m.update({'file': record['file'], 'scale': a, 'shift': b,
159
+ 'clipped_frac': clipped, 'valid_pixels': int(mask.sum())})
160
+ per_image[name].append(m)
161
+
162
+ results = {}
163
+ for name, _ in pred_sets:
164
+ rows = per_image[name]
165
+ results[name] = {k: float(np.mean([r[k] for r in rows])) for k in METRICS}
166
+ results[name]['p90_absrel'] = float(np.percentile([r['AbsRel'] for r in rows], 90))
167
+ results[name]['clipped_images'] = int(sum(r['clipped_frac'] > 0 for r in rows))
168
+ results[name]['clipped_pixel_frac'] = float(np.mean([r['clipped_frac'] for r in rows]))
169
+ results[name]['evaluated_images'] = len(rows)
170
+ results[name]['failed_images'] = len(failed[name])
171
+
172
+ base = pred_sets[0][0]
173
+ drops = {}
174
+ for name, _ in pred_sets[1:]:
175
+ drops[name] = {}
176
+ for k in METRICS:
177
+ b, v = results[base][k], results[name][k]
178
+ drops[name][k] = float((v - b) / b * 100)
179
+ b, v = results[base]['p90_absrel'], results[name]['p90_absrel']
180
+ drops[name]['p90_absrel'] = float((v - b) / b * 100)
181
+
182
+ summary = {
183
+ 'dataset': 'nyu_depth_v2_labeled / eigen test split',
184
+ 'num_images': len(split),
185
+ 'align': args.align,
186
+ 'max_depth': args.max_depth,
187
+ 'eigen_crop': False,
188
+ 'results': results,
189
+ 'baseline': base,
190
+ 'drop_percent': drops,
191
+ 'failed': failed,
192
+ 'per_image': {name: per_image[name] for name, _ in pred_sets},
193
+ }
194
+ with open(out_dir / 'metrics.json', 'w') as fp:
195
+ json.dump(summary, fp, indent=2)
196
+
197
+ print(f'\n{"指标":<10}' + ''.join(f'{n:>14}' for n, _ in pred_sets))
198
+ for k in METRICS + ('p90_absrel',):
199
+ print(f'{k:<10}' + ''.join(f'{results[n][k]:>14.4f}' for n, _ in pred_sets))
200
+ for name, d in drops.items():
201
+ print(f'\n{name} 相对 {base} 掉点 (%): ' +
202
+ ', '.join(f'{k}={v:+.2f}' for k, v in d.items()))
203
+ for name, _ in pred_sets:
204
+ if failed[name]:
205
+ print(f'\n{name}: {len(failed[name])} 张对齐失败(方向反了或输出异常),未计入均值')
206
+ print(f'\n结果已保存: {out_dir / "metrics.json"}')
207
+
208
+
209
+ if __name__ == '__main__':
210
+ main()
src/tool/export_nyu654_test.py ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # -*- coding: utf-8 -*-
3
+ """
4
+ 从 nyu_depth_v2_labeled.mat 导出官方 Eigen test split 的 654 张 RGB 图片。
5
+
6
+ 划分来源:BTS 仓库 utils/splits.mat 的 testNdxs(1-based,共 654 个),
7
+ 与 train_test_inputs/nyudepthv2_test_files_with_gt.txt 的 654 行逐场景帧数一致,
8
+ 已核对无重叠、并集覆盖 1..1449。
9
+
10
+ 只导出 RGB,GT 不落盘:评测脚本按 split.json 里的 mat_index 直接从 mat 读取,
11
+ 避免重复存储。
12
+
13
+ 用法:
14
+ # 导出 654 张到 datasets/nyu654_test/
15
+ ../../venv/bin/python src/tool/export_nyu654_test.py \
16
+ --mat ../../datasets/nyu_depth_v2_labeled.mat \
17
+ --splits datasets/nyu654_test/splits.mat \
18
+ --out-dir datasets/nyu654_test
19
+
20
+ 输出:
21
+ <out-dir>/images/{序号:04d}_{scene}.png
22
+ <out-dir>/split.json [{mat_index, scene, file}, ...]
23
+ """
24
+
25
+ import argparse
26
+ import json
27
+ from pathlib import Path
28
+
29
+ import cv2
30
+ import h5py
31
+ import numpy as np
32
+ import scipy.io as sio
33
+ from tqdm import tqdm
34
+
35
+
36
+ def decode_cell(f, ref):
37
+ return ''.join(chr(c) for c in np.asarray(f[ref]).ravel())
38
+
39
+
40
+ def main():
41
+ parser = argparse.ArgumentParser(description='导出 NYU Depth V2 Eigen test split 654 张')
42
+ parser.add_argument('--mat', required=True, help='nyu_depth_v2_labeled.mat')
43
+ parser.add_argument('--splits', required=True, help='含 testNdxs 的 splits.mat')
44
+ parser.add_argument('--out-dir', required=True, help='输出目录')
45
+ args = parser.parse_args()
46
+
47
+ out_dir = Path(args.out_dir)
48
+ image_dir = out_dir / 'images'
49
+ image_dir.mkdir(parents=True, exist_ok=True)
50
+
51
+ ndxs = sio.loadmat(args.splits)['testNdxs'].ravel().astype(int)
52
+ if len(ndxs) != 654:
53
+ raise ValueError(f'testNdxs 应为 654 个,实际 {len(ndxs)}')
54
+
55
+ records = []
56
+ with h5py.File(args.mat, 'r') as f:
57
+ for seq, mat_index in enumerate(tqdm(ndxs, desc='export', ncols=80)):
58
+ i = mat_index - 1 # matlab 1-based
59
+ scene = decode_cell(f, f['scenes'][0, i])
60
+ rgb = np.asarray(f['images'][i]).transpose(1, 2, 0) # CHW -> HWC
61
+ name = f'{seq:04d}_{scene}.png'
62
+ if not cv2.imwrite(str(image_dir / name), cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)):
63
+ raise RuntimeError(f'写入失败: {image_dir / name}')
64
+ records.append({'mat_index': int(i), 'scene': scene, 'file': name})
65
+
66
+ with open(out_dir / 'split.json', 'w') as fp:
67
+ json.dump(records, fp, indent=2)
68
+ print(f'已导出 {len(records)} 张到 {image_dir},划分文件 {out_dir / "split.json"}')
69
+
70
+
71
+ if __name__ == '__main__':
72
+ main()
src/tool/export_nyu_train795.py ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # -*- coding: utf-8 -*-
3
+ """
4
+ 从 nyu_depth_v2_labeled.mat 导出 795 张训练集(trainNdxs)图片。
5
+
6
+ 划分来源与 export_nyu654_test.py 相同:BTS utils/splits.mat。
7
+ trainNdxs 与 testNdxs(654)无重叠、并集覆盖 1..1449,可以与测试集严格隔离,
8
+ 用于量化校准或训练。
9
+
10
+ 只导出 RGB;GT 仍从 mat 按 split.json 的 mat_index 读取。
11
+
12
+ 用法:
13
+ ../../venv/bin/python src/tool/export_nyu_train795.py \
14
+ --mat ../../datasets/nyu_depth_v2_labeled.mat \
15
+ --splits datasets/nyu654_test/splits.mat \
16
+ --out-dir datasets/nyu_train795
17
+ """
18
+
19
+ import argparse
20
+ import json
21
+ from pathlib import Path
22
+
23
+ import cv2
24
+ import h5py
25
+ import numpy as np
26
+ import scipy.io as sio
27
+ from tqdm import tqdm
28
+
29
+
30
+ def decode_cell(f, ref):
31
+ return ''.join(chr(c) for c in np.asarray(f[ref]).ravel())
32
+
33
+
34
+ def main():
35
+ parser = argparse.ArgumentParser(description='导出 NYU Depth V2 训练集 795 张')
36
+ parser.add_argument('--mat', required=True, help='nyu_depth_v2_labeled.mat')
37
+ parser.add_argument('--splits', required=True, help='含 trainNdxs 的 splits.mat')
38
+ parser.add_argument('--out-dir', required=True, help='输出目录')
39
+ args = parser.parse_args()
40
+
41
+ image_dir = Path(args.out_dir) / 'images'
42
+ image_dir.mkdir(parents=True, exist_ok=True)
43
+
44
+ ndxs = sio.loadmat(args.splits)['trainNdxs'].ravel().astype(int)
45
+ if len(ndxs) != 795:
46
+ raise ValueError(f'trainNdxs 应为 795 个,实际 {len(ndxs)}')
47
+
48
+ records = []
49
+ with h5py.File(args.mat, 'r') as f:
50
+ for seq, mat_index in enumerate(tqdm(ndxs, desc='export', ncols=80)):
51
+ i = mat_index - 1
52
+ scene = decode_cell(f, f['scenes'][0, i])
53
+ rgb = np.asarray(f['images'][i]).transpose(1, 2, 0)
54
+ name = f'{seq:04d}_{scene}.png'
55
+ if not cv2.imwrite(str(image_dir / name), cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)):
56
+ raise RuntimeError(f'写入失败: {image_dir / name}')
57
+ records.append({'mat_index': int(i), 'scene': scene, 'file': name})
58
+
59
+ with open(Path(args.out_dir) / 'split.json', 'w') as fp:
60
+ json.dump(records, fp, indent=2)
61
+ print(f'已导出 {len(records)} 张到 {image_dir}')
62
+
63
+
64
+ if __name__ == '__main__':
65
+ main()
src/tool/inspect_nyu_depth_v2.py ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # -*- coding: utf-8 -*-
3
+ """
4
+ NYU Depth V2 (labeled) 数据集查看与导出工具。
5
+
6
+ nyu_depth_v2_labeled.mat 是 MATLAB v7.3 (HDF5) 格式,scipy.io.loadmat 无法读取,
7
+ 这里统一用 h5py 按需读取,避免一次性载入 2.8GB。
8
+
9
+ 数据内容(N = 1449,图像为竖版 640x480,H=640, W=480,RGB 与深度已对齐):
10
+ images [N, 3, 640, 480] uint8 RGB
11
+ depths [N, 640, 480] float32 已填补深度,单位米
12
+ rawDepths [N, 640, 480] float32 原始深度,0 表示缺失
13
+ labels [N, 640, 480] uint16 语义标签,0=未标注,1..894 对应 names
14
+ instances [N, 640, 480] uint8 实例标签,0=无实例
15
+ names [1, 894] cell 类别名(1-based 下标即 label id)
16
+ namesToIds containers.Map MATLAB Map,h5py 无法直接解析
17
+ scenes [1, N] cell 场景名,如 kitchen_0004
18
+ sceneTypes [1, N] cell 场景类型,如 kitchen
19
+ rawRgbFilenames [1, N] cell 原始 RGB 文件名
20
+ rawDepthFilenames [1, N] cell 原始深度文件名
21
+ accelData [4, N] float32 加速度计数据
22
+
23
+ 用法(在 hs/DepthART 目录下执行):
24
+ # 打印结构、场景/类别统计和深度数值范围(默认抽样 step=50)
25
+ ../../venv/bin/python src/tool/inspect_nyu_depth_v2.py --mat ../../datasets/nyu_depth_v2_labeled.mat
26
+
27
+ # 全量扫描统计(较慢)
28
+ ... --mat ../../datasets/nyu_depth_v2_labeled.mat --full-stats
29
+
30
+ # 列出全部 894 个类别名
31
+ ... --mat ... --list-classes
32
+
33
+ # 单样本导出:rgb.png / depth.npy / depth_vis.png / raw_depth.npy
34
+ ... --mat ... --index 0 --out-dir results/nyu_demo --with-labels
35
+
36
+ # 批量导出,例如某场景全部帧
37
+ ... --mat ... --export-dir results/nyu_export --scene kitchen_0004 --with-labels
38
+
39
+ # 批量导出前 50 张
40
+ ... --mat ... --export-dir results/nyu_export --count 50
41
+ """
42
+
43
+ import argparse
44
+ import collections
45
+ from pathlib import Path
46
+
47
+ import cv2
48
+ import h5py
49
+ import numpy as np
50
+ from tqdm import tqdm
51
+
52
+ # 深度可视化统一范围(米),超出部分截断
53
+ DEPTH_VIS_RANGE = (0.5, 10.0)
54
+
55
+
56
+ def decode_cell(f, ref):
57
+ """把 MATLAB cell 中的 char 数组解码为 str。"""
58
+ return ''.join(chr(c) for c in np.asarray(f[ref]).ravel())
59
+
60
+
61
+ def read_cell_list(dset):
62
+ return [decode_cell(dset.file, dset[0, i]) for i in range(dset.shape[1])]
63
+
64
+
65
+ def print_info(f, stride, full_stats):
66
+ print('== 变量 ==')
67
+ for k in f.keys():
68
+ if k.startswith('#'):
69
+ continue
70
+ o = f[k]
71
+ line = f'{k:20s} shape={str(o.shape):24s} dtype={o.dtype}'
72
+ if o.compression:
73
+ line += f' compression={o.compression}'
74
+ print(line)
75
+
76
+ n = f['depths'].shape[0]
77
+ h, w = f['depths'].shape[1:]
78
+ names = read_cell_list(f['names'])
79
+ scenes = read_cell_list(f['scenes'])
80
+ scene_types = read_cell_list(f['sceneTypes'])
81
+
82
+ print('\n== 规模 ==')
83
+ print(f'样本数 {n}')
84
+ print(f'分辨率 W={w}, H={h}(data layout 为 [N, C, H, W])')
85
+ print(f'场景数 {len(set(scenes))}')
86
+ print(f'场景类型数 {len(set(scene_types))}')
87
+ print(f'语义类别数 {len(names)}(label id 1..{len(names)},0 为未标注)')
88
+ print('场景类型分布 ' + ', '.join(f'{k}:{v}' for k, v in
89
+ collections.Counter(scene_types).most_common()))
90
+
91
+ idx = range(0, n) if full_stats else range(0, n, stride)
92
+ idx = list(idx)
93
+ print(f'\n== 数值统计({"全量" if full_stats else f"抽样 {len(idx)} 张, step={stride}"}) ==')
94
+ d_min, d_max = np.inf, -np.inf
95
+ raw_missing = 0
96
+ raw_total = 0
97
+ cls_count = collections.Counter()
98
+ inst_max = 0
99
+ for i in tqdm(idx, desc='scan', ncols=80):
100
+ d = f['depths'][i]
101
+ r = f['rawDepths'][i]
102
+ d_min = min(d_min, float(d.min()))
103
+ d_max = max(d_max, float(d.max()))
104
+ raw_missing += int((r == 0).sum())
105
+ raw_total += r.size
106
+ for c in np.unique(f['labels'][i]):
107
+ if c > 0:
108
+ cls_count[int(c)] += 1
109
+ inst_max = max(inst_max, int(f['instances'][i].max()))
110
+
111
+ print(f'depths 范围 [{d_min:.4f}, {d_max:.4f}] m')
112
+ print(f'rawDepths 缺失 {raw_missing / raw_total * 100:.2f}% (值为 0)')
113
+ print(f'出现类别数 {len(cls_count)} / {len(names)}')
114
+ print(f'instances 最大 {inst_max}')
115
+ print('\n最常见类别 ' + ', '.join(
116
+ f'{names[c - 1]}({cnt})' for c, cnt in cls_count.most_common(15)))
117
+
118
+
119
+ def depth_to_vis(depth, raw=False):
120
+ lo, hi = DEPTH_VIS_RANGE
121
+ norm = np.clip((depth - lo) / (hi - lo), 0.0, 1.0)
122
+ vis = cv2.applyColorMap((norm * 255).astype(np.uint8), cv2.COLORMAP_TURBO)
123
+ if raw:
124
+ vis[depth <= 0] = 0
125
+ return vis
126
+
127
+
128
+ def export_sample(f, index, out_dir, with_labels):
129
+ entry = out_dir / f'{index:05d}_{decode_cell(f, f["scenes"][0, index])}'
130
+ entry.mkdir(parents=True, exist_ok=True)
131
+
132
+ rgb = np.asarray(f['images'][index]).transpose(1, 2, 0) # CHW -> HWC, RGB
133
+ depth = np.asarray(f['depths'][index], dtype=np.float32)
134
+ raw = np.asarray(f['rawDepths'][index], dtype=np.float32)
135
+
136
+ cv2.imwrite(str(entry / 'rgb.png'), cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR))
137
+ cv2.imwrite(str(entry / 'depth_vis.png'), depth_to_vis(depth))
138
+ cv2.imwrite(str(entry / 'raw_depth_vis.png'), depth_to_vis(raw, raw=True))
139
+ np.save(entry / 'depth.npy', depth)
140
+ np.save(entry / 'raw_depth.npy', raw)
141
+
142
+ if with_labels:
143
+ labels = np.asarray(f['labels'][index])
144
+ np.save(entry / 'labels.npy', labels)
145
+ # 语义标签着色:按 label id 的哈希色,0 保持黑色
146
+ rng = np.random.default_rng(0)
147
+ lut = rng.integers(40, 255, size=(65536, 3), dtype=np.uint8)
148
+ lut[0] = 0
149
+ cv2.imwrite(str(entry / 'labels_vis.png'), lut[labels])
150
+
151
+ return entry
152
+
153
+
154
+ def main():
155
+ parser = argparse.ArgumentParser(description='查看/导出 NYU Depth V2 labeled 数据集')
156
+ parser.add_argument('--mat', required=True, help='nyu_depth_v2_labeled.mat 路径')
157
+ parser.add_argument('--list-classes', action='store_true', help='打印全部类别名及 id')
158
+ parser.add_argument('--index', type=int, help='导出单个样本的下标')
159
+ parser.add_argument('--out-dir', help='--index 模式的输出目录')
160
+ parser.add_argument('--export-dir', help='批量导出目录')
161
+ parser.add_argument('--count', type=int, help='批量导出数量,默认全部')
162
+ parser.add_argument('--step', type=int, default=1, help='批量导出间隔,默认 1')
163
+ parser.add_argument('--scene', help='只导出指定场景,如 kitchen_0004')
164
+ parser.add_argument('--with-labels', action='store_true', help='同时导出语义标签')
165
+ parser.add_argument('--full-stats', action='store_true', help='统计时扫描全部样本')
166
+ parser.add_argument('--stats-stride', type=int, default=50, help='抽样统计间隔,默认 50')
167
+ args = parser.parse_args()
168
+
169
+ with h5py.File(args.mat, 'r') as f:
170
+ if args.list_classes:
171
+ names = read_cell_list(f['names'])
172
+ for i, name in enumerate(names, 1):
173
+ print(f'{i:4d} {name}')
174
+ return
175
+
176
+ if args.index is not None:
177
+ out_dir = Path(args.out_dir or '.')
178
+ entry = export_sample(f, args.index, out_dir, args.with_labels)
179
+ print(f'已导出 {entry}')
180
+ return
181
+
182
+ if args.export_dir:
183
+ n = f['depths'].shape[0]
184
+ indices = list(range(0, n, args.step))
185
+ if args.scene:
186
+ scenes = read_cell_list(f['scenes'])
187
+ indices = [i for i in indices if scenes[i] == args.scene]
188
+ print(f'场景 {args.scene} 命中 {len(indices)} 张')
189
+ if args.count:
190
+ indices = indices[:args.count]
191
+ out_dir = Path(args.export_dir)
192
+ for i in tqdm(indices, desc='export', ncols=80):
193
+ export_sample(f, i, out_dir, args.with_labels)
194
+ print(f'已导出 {len(indices)} 张到 {out_dir}')
195
+ return
196
+
197
+ print_info(f, args.stats_stride, args.full_stats)
198
+
199
+
200
+ if __name__ == '__main__':
201
+ main()
src/tool/make_nyu_calibration.py ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ # -*- coding: utf-8 -*-
3
+ """
4
+ 从 NYU 训练集 795 张中按场景均匀抽取量化校准图片。
5
+
6
+ 抽样规则:
7
+ - 按场景类型(scene 去掉 _0001 后缀,如 kitchen、bedroom)分组;
8
+ - 每个场景至少 1 张,其余名额按场景图片数比例分配(最大余数法);
9
+ - 场景内按等距索引选取,结果确定可复现。
10
+
11
+ 输出:
12
+ <out-dir>/images/*.png 校准图片
13
+ <out-dir>/selection.json 每张图的来源 mat_index / scene / 源文件
14
+ <out-dir>/<name>.tar 给 pulsar2 的校准包,内部为 <name>/xxx.png
15
+
16
+ 用法:
17
+ ../../venv/bin/python src/tool/make_nyu_calibration.py \
18
+ --source-dir datasets/nyu_train795 --count 48 \
19
+ --out-dir datasets/nyu_calibration_48
20
+ """
21
+
22
+ import argparse
23
+ import json
24
+ import shutil
25
+ import subprocess
26
+ from collections import defaultdict
27
+ from pathlib import Path
28
+
29
+ import cv2
30
+ import numpy as np
31
+
32
+
33
+ def scene_type(scene):
34
+ return scene.rsplit('_', 1)[0]
35
+
36
+
37
+ def allocate(groups, count):
38
+ """最大余数法分配名额,每个场景至少 1 张。"""
39
+ total = sum(len(v) for v in groups.values())
40
+ if count < len(groups):
41
+ raise ValueError(f'抽取数量 {count} 少于场景数 {len(groups)},无法保证每场景 1 张')
42
+ quota = {k: 1 for k in groups}
43
+ left = count - len(groups)
44
+ exact = {k: left * len(v) / total for k, v in groups.items()}
45
+ for k, v in exact.items():
46
+ quota[k] += int(np.floor(v))
47
+ remain = count - sum(quota.values())
48
+ order = sorted(exact, key=lambda k: exact[k] - np.floor(exact[k]), reverse=True)
49
+ for k in order[:remain]:
50
+ quota[k] += 1
51
+ return quota
52
+
53
+
54
+ def main():
55
+ parser = argparse.ArgumentParser(description='按场景均匀抽取 NYU 校准集')
56
+ parser.add_argument('--source-dir', required=True, help='export_nyu_train795.py 的输出目录')
57
+ parser.add_argument('--count', type=int, default=48, help='抽取张数,默认 48')
58
+ parser.add_argument('--out-dir', required=True, help='校准集输出目录')
59
+ parser.add_argument('--name', help='tar 包内目录名,默认取 out-dir 名')
60
+ parser.add_argument('--ext', choices=('png', 'jpg'), default='png',
61
+ help='校准图片格式,默认 png;pulsar2 读取异常时可改 jpg')
62
+ args = parser.parse_args()
63
+
64
+ source = Path(args.source_dir)
65
+ split = json.load(open(source / 'split.json'))
66
+ groups = defaultdict(list)
67
+ for record in split:
68
+ groups[scene_type(record['scene'])].append(record)
69
+
70
+ quota = allocate(groups, args.count)
71
+ selected = []
72
+ for scene, records in sorted(groups.items()):
73
+ records = sorted(records, key=lambda r: r['file'])
74
+ idx = np.linspace(0, len(records) - 1, quota[scene]).round().astype(int)
75
+ selected.extend(records[i] for i in sorted(set(idx.tolist())))
76
+
77
+ if len(selected) != args.count:
78
+ raise RuntimeError(f'实际选出 {len(selected)} 张,与目标 {args.count} 不符')
79
+
80
+ out_dir = Path(args.out_dir)
81
+ image_dir = out_dir / 'images'
82
+ image_dir.mkdir(parents=True, exist_ok=True)
83
+ for record in selected:
84
+ src = source / 'images' / record['file']
85
+ if args.ext == 'png':
86
+ dst = image_dir / record['file']
87
+ shutil.copy2(src, dst)
88
+ else:
89
+ dst = image_dir / (Path(record['file']).stem + '.jpg')
90
+ image = cv2.imread(str(src), cv2.IMREAD_COLOR)
91
+ if image is None:
92
+ raise RuntimeError(f'读取失败: {src}')
93
+ cv2.imwrite(str(dst), image, [cv2.IMWRITE_JPEG_QUALITY, 95])
94
+
95
+ with open(out_dir / 'selection.json', 'w') as fp:
96
+ json.dump(selected, fp, indent=2)
97
+
98
+ name = args.name or out_dir.name
99
+ tar_path = out_dir / f'{name}.tar'
100
+ subprocess.run(
101
+ ['tar', '-cf', str(tar_path), '--transform', f's,^,{name}/,', '-C', str(image_dir)]
102
+ + sorted(p.name for p in image_dir.iterdir()),
103
+ check=True)
104
+
105
+ print(f'已抽取 {len(selected)} 张,覆盖 {len(groups)} 个场景类型')
106
+ for scene in sorted(groups):
107
+ print(f' {scene:<16} {quota[scene]:>3} 张 / 共 {len(groups[scene])}')
108
+ print(f'校准包: {tar_path}')
109
+
110
+
111
+ if __name__ == '__main__':
112
+ main()