sunyijie commited on
Commit ·
efaa3d7
1
Parent(s): 7f13ce4
first commit
Browse files- .gitattributes +2 -0
- .gitignore +2 -0
- README.md +101 -0
- config.json +53 -0
- host.jpg +3 -0
- host_depth.jpg +3 -0
- models/relative_s_448_default.onnx +3 -0
- models/relative_s_448_default_ax650.axmodel +3 -0
- src/infer/__pycache__/depthart_infer_axmodel.cpython-310.pyc +0 -0
- src/infer/__pycache__/depthart_infer_onnx.cpython-310.pyc +0 -0
- src/infer/depthart_infer_axmodel.py +639 -0
- src/infer/depthart_infer_onnx.py +721 -0
- src/tool/__pycache__/evaluate_nyu654_depth.cpython-310.pyc +0 -0
- src/tool/compare_depth_outputs.py +143 -0
- src/tool/compare_onnx_axmodel.py +546 -0
- src/tool/depthart_export_axonnx.py +555 -0
- src/tool/evaluate_nyu654_depth.py +210 -0
- src/tool/export_nyu654_test.py +72 -0
- src/tool/export_nyu_train795.py +65 -0
- src/tool/inspect_nyu_depth_v2.py +201 -0
- src/tool/make_nyu_calibration.py +112 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
*.axmodel filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
*.jpg filter=lfs diff=lfs merge=lfs -text
|
.gitignore
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
results/**
|
| 2 |
+
datasets/**
|
README.md
CHANGED
|
@@ -1,3 +1,104 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
---
|
| 4 |
+
|
| 5 |
+
# DepthART (AX650 Relative Depth Estimation)
|
| 6 |
+
|
| 7 |
+
DepthART is a family of relative depth estimation models based on selective state
|
| 8 |
+
spaces (Selective Scan). This repository provides the ONNX model of the `small / 448`
|
| 9 |
+
variant, an `axmodel` compiled for **AX650**, and tools for host ONNX inference, board
|
| 10 |
+
axmodel inference, and model rewriting.
|
| 11 |
+
|
| 12 |
+
The model outputs **affine-invariant relative depth**: larger values mean closer,
|
| 13 |
+
smaller values mean farther. It does not directly produce metric depth; align it with
|
| 14 |
+
ground truth depth using a global scale/offset before use.
|
| 15 |
+
|
| 16 |
+
## Single-image result
|
| 17 |
+
|
| 18 |
+
Left: input image; right: relative depth output by the model (INFERNO colormap,
|
| 19 |
+
stretched to the 2%–98% percentile range).
|
| 20 |
+
|
| 21 |
+

|
| 22 |
+
|
| 23 |
+
## Repository contents
|
| 24 |
+
|
| 25 |
+
| Path | Description |
|
| 26 |
+
| --- | --- |
|
| 27 |
+
| `models/relative_s_448_default.onnx` | Original FP32 ONNX model, containing the custom op `com.depthart::SelectiveScan` |
|
| 28 |
+
| `models/relative_s_448_default_ax650.axmodel` | Compiled AX650 model |
|
| 29 |
+
| `src/infer/depthart_infer_onnx.py` | Host ONNX inference (rewrites `SelectiveScan` automatically) |
|
| 30 |
+
| `src/infer/depthart_infer_axmodel.py` | AX650 board axmodel inference (standalone script) |
|
| 31 |
+
| `src/tool/depthart_export_axonnx.py` | Rewrites `SelectiveScan` into a pure ONNX subgraph that pulsar2 can compile |
|
| 32 |
+
| `src/tool/compare_onnx_axmodel.py`, `compare_depth_outputs.py` | Numerical comparison of outputs from different backends |
|
| 33 |
+
| `src/tool/evaluate_nyu654_depth.py` | Accuracy evaluation on the NYU Depth V2 eigen test split |
|
| 34 |
+
| `host.jpg`, `host_depth.jpg` | Single-image example input and side-by-side visualization result |
|
| 35 |
+
|
| 36 |
+
## Model interface
|
| 37 |
+
|
| 38 |
+
### ONNX
|
| 39 |
+
|
| 40 |
+
| Item | Value |
|
| 41 |
+
| --- | --- |
|
| 42 |
+
| Input | `image` `[1, 3, 448, 448]` float32, NCHW, RGB |
|
| 43 |
+
| Output | `depth` `[1, 1, 448, 448]` float32, relative depth (larger = closer) |
|
| 44 |
+
|
| 45 |
+
## AX650 board inference
|
| 46 |
+
|
| 47 |
+
The script `src/infer/depthart_infer_axmodel.py` does not depend on other scripts in
|
| 48 |
+
the repository. For deployment, just place the following files in the same directory
|
| 49 |
+
on the board:
|
| 50 |
+
|
| 51 |
+
```text
|
| 52 |
+
relative_s_448_default_ax650.axmodel
|
| 53 |
+
depthart_infer_axmodel.py
|
| 54 |
+
images or image directory to test
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
Board runtime requires `axengine`.
|
| 58 |
+
|
| 59 |
+
### Single-image validation
|
| 60 |
+
|
| 61 |
+
```bash
|
| 62 |
+
python3 depthart_infer_axmodel.py \
|
| 63 |
+
--axmodel relative_s_448_default_ax650.axmodel \
|
| 64 |
+
--image ./host.jpg \
|
| 65 |
+
--output ./results/ax_single
|
| 66 |
+
```
|
| 67 |
+
|
| 68 |
+
Outputs `results/ax_single/host_depth.jpg` (side-by-side visualization) and
|
| 69 |
+
`host_depth.npy`; the console prints depth statistics, NPU inference time, CPU
|
| 70 |
+
processing time, and per-stage timings. A single image is the first inference and
|
| 71 |
+
includes runtime first-execution overhead; use dataset mode for performance evaluation.
|
| 72 |
+
|
| 73 |
+
### Dataset inference
|
| 74 |
+
|
| 75 |
+
By default, fast mode only measures `read / preprocess / npu` and skips depth
|
| 76 |
+
restoration, visualization, and file writing, for performance testing:
|
| 77 |
+
|
| 78 |
+
```bash
|
| 79 |
+
python3 depthart_infer_axmodel.py \
|
| 80 |
+
--axmodel relative_s_448_default_ax650.axmodel \
|
| 81 |
+
--dataset ./images \
|
| 82 |
+
--output ./results/board
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
Full output (depth restoration, visualization, summary NPZ), for accuracy evaluation:
|
| 86 |
+
|
| 87 |
+
```bash
|
| 88 |
+
python3 depthart_infer_axmodel.py \
|
| 89 |
+
--axmodel relative_s_448_default_ax650.axmodel \
|
| 90 |
+
--dataset ./images \
|
| 91 |
+
--output ./results/board_full \
|
| 92 |
+
--full-output \
|
| 93 |
+
--dump-npz ./results/board_full/raw_outputs.npz
|
| 94 |
+
```
|
| 95 |
+
|
| 96 |
+
Image-by-image comparison against the host ONNX baseline summary NPZ:
|
| 97 |
+
|
| 98 |
+
```bash
|
| 99 |
+
python3 depthart_infer_axmodel.py \
|
| 100 |
+
--axmodel relative_s_448_default_ax650.axmodel \
|
| 101 |
+
--dataset ./images \
|
| 102 |
+
--output ./results/board_check \
|
| 103 |
+
--ref-npz ./results/onnx_full/raw_outputs.npz
|
| 104 |
+
```
|
config.json
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "ONNX",
|
| 3 |
+
"npu_mode": "NPU3",
|
| 4 |
+
"input": "onnx/relative_s_448_default_axonnx_t4.onnx",
|
| 5 |
+
"output_dir": "./output_axonnx_t4",
|
| 6 |
+
"output_name": "relative_s_448_default_ax650_t4.axmodel",
|
| 7 |
+
"quant": {
|
| 8 |
+
"input_configs": [
|
| 9 |
+
{
|
| 10 |
+
"tensor_name": "DEFAULT",
|
| 11 |
+
"calibration_dataset": "/data/datasets/coco2017val_500_calibration_dataset.tar",
|
| 12 |
+
"calibration_size": 8,
|
| 13 |
+
"calibration_mean": [
|
| 14 |
+
123.675,
|
| 15 |
+
116.28,
|
| 16 |
+
103.53
|
| 17 |
+
],
|
| 18 |
+
"calibration_std": [
|
| 19 |
+
58.395,
|
| 20 |
+
57.12,
|
| 21 |
+
57.375
|
| 22 |
+
]
|
| 23 |
+
}
|
| 24 |
+
],
|
| 25 |
+
"calibration_method": "Percentile",
|
| 26 |
+
"precision_analysis": true,
|
| 27 |
+
"precision_analysis_method": "EndToEnd",
|
| 28 |
+
"conv_bias_data_type": "FP32",
|
| 29 |
+
"layer_configs": [
|
| 30 |
+
{
|
| 31 |
+
"start_tensor_names": [
|
| 32 |
+
"DEFAULT"
|
| 33 |
+
],
|
| 34 |
+
"end_tensor_names": [
|
| 35 |
+
"DEFAULT"
|
| 36 |
+
],
|
| 37 |
+
"data_type": "U16"
|
| 38 |
+
}
|
| 39 |
+
]
|
| 40 |
+
},
|
| 41 |
+
"input_processors": [
|
| 42 |
+
{
|
| 43 |
+
"tensor_name": "DEFAULT",
|
| 44 |
+
"tensor_format": "RGB",
|
| 45 |
+
"src_format": "RGB",
|
| 46 |
+
"src_dtype": "U8",
|
| 47 |
+
"src_layout": "NCHW"
|
| 48 |
+
}
|
| 49 |
+
],
|
| 50 |
+
"compiler": {
|
| 51 |
+
"check": 0
|
| 52 |
+
}
|
| 53 |
+
}
|
host.jpg
ADDED
|
Git LFS Details
|
host_depth.jpg
ADDED
|
Git LFS Details
|
models/relative_s_448_default.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4773e2648803d207c470c86633c3059fd792bc87c5fdffce817005f6711abf06
|
| 3 |
+
size 24544478
|
models/relative_s_448_default_ax650.axmodel
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0e0082e21d0e5a42a37f615702bfe88c0718b0fa65bb29ef71b07086e7ec73e8
|
| 3 |
+
size 11560983
|
src/infer/__pycache__/depthart_infer_axmodel.cpython-310.pyc
ADDED
|
Binary file (25.1 kB). View file
|
|
|
src/infer/__pycache__/depthart_infer_onnx.cpython-310.pyc
ADDED
|
Binary file (29.2 kB). View file
|
|
|
src/infer/depthart_infer_axmodel.py
ADDED
|
@@ -0,0 +1,639 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# -*- coding: utf-8 -*-
|
| 3 |
+
"""
|
| 4 |
+
DepthART 相对深度 axmodel 板端推理(AX650 + axengine)。
|
| 5 |
+
|
| 6 |
+
随板部署文件(建议放在同一目录):
|
| 7 |
+
relative_s_448_default_ax650.axmodel
|
| 8 |
+
depthart_infer_axmodel.py
|
| 9 |
+
待测图片或图片目录
|
| 10 |
+
|
| 11 |
+
模型接口(以运行时 get_inputs/get_outputs 实际返回为准):
|
| 12 |
+
input [1, 3, 448, 448] uint8 NCHW RGB
|
| 13 |
+
depth [1, 1, 448, 448] float32 仿射不变相对深度,数值越大表示越近
|
| 14 |
+
|
| 15 |
+
预处理按编译配置 config_axonnx.json 的 input_processors 执行:tensor_format=RGB、
|
| 16 |
+
src_dtype=U8、mean/std 为 0-255 域的 ImageNet 值,归一化由编译前端完成。因此板端
|
| 17 |
+
送 0-255 原始 RGB 像素即可,不要再做 /255 或减均值。若 axmodel 输入为 FP32(编译
|
| 18 |
+
时未配置 input_processors),脚本默认改为 /255 + ImageNet mean/std,与 ONNX 基准
|
| 19 |
+
一致;可用 --input-norm 显式覆盖。
|
| 20 |
+
|
| 21 |
+
深度还原使用与 ONNX 脚本一致的 bilinear + align_corners=True(linspace + remap),
|
| 22 |
+
保证板端与开发机基准可以逐像素对比。
|
| 23 |
+
|
| 24 |
+
板端计时分为 NPU(session.run)和 CPU(读取、预处理、输出提取/校验、后处理、
|
| 25 |
+
可视化、写盘),两者分别统计,避免弱 CPU 上的后处理掩盖真实 NPU 性能。
|
| 26 |
+
|
| 27 |
+
单图模式:打印深度统计、NPU/CPU 各阶段耗时,保存并排可视化,不产出评测文件。
|
| 28 |
+
数据集模式:tqdm 进度 + npu_timing.json;默认快速模式只统计读取/预处理/NPU,
|
| 29 |
+
--full-output 才做深度还原、可视化、原始 NPZ 和图片写盘。
|
| 30 |
+
|
| 31 |
+
板端用法(所有路径均为必填):
|
| 32 |
+
# 单图验证
|
| 33 |
+
python3 depthart_infer_axmodel.py --axmodel relative_s_448_default_ax650.axmodel \
|
| 34 |
+
--image ./host.jpg --output ./results/ax_single
|
| 35 |
+
|
| 36 |
+
# 数据集快速性能模式
|
| 37 |
+
python3 depthart_infer_axmodel.py --axmodel relative_s_448_default_ax650.axmodel \
|
| 38 |
+
--dataset ./images --output ./results/board
|
| 39 |
+
|
| 40 |
+
# 数据集完整输出(原始深度 NPZ、可视化和大图深度)
|
| 41 |
+
python3 depthart_infer_axmodel.py --axmodel relative_s_448_default_ax650.axmodel \
|
| 42 |
+
--dataset ../../test_depth --output ./results/board_full --full-output
|
| 43 |
+
|
| 44 |
+
# 与开发机 ONNX 基准对比(--ref-npz 指向 ONNX 脚本的汇总 NPZ)
|
| 45 |
+
python3 depthart_infer_axmodel.py --axmodel relative_s_448_default_ax650.axmodel \
|
| 46 |
+
--dataset ../../test_depth --output ./results/board_check --ref-npz ./onnx_npz
|
| 47 |
+
|
| 48 |
+
开发机用法(仅用于脚本语法/参数检查,实际推理需要板端 axengine):
|
| 49 |
+
../../venv/bin/python src/infer/depthart_infer_axmodel.py \
|
| 50 |
+
--axmodel output_axonnx3/relative_s_448_default_ax650.axmodel \
|
| 51 |
+
--dataset /path/images --output results/axmodel_board --limit 50
|
| 52 |
+
|
| 53 |
+
诊断:
|
| 54 |
+
--debug 打印输入/输出 tensor 的 shape、dtype、min、max、mean、nan 计数
|
| 55 |
+
--dump-npz 单图时为 .npz 文件路径;数据集时为汇总 .npz 文件路径,
|
| 56 |
+
保存 names/depths/original_sizes,不再每张图一个小文件
|
| 57 |
+
--npz-chunk 数据集模式下每 N 张切一个文件(xxx_0000.npz 等),
|
| 58 |
+
0 表示全部写入同一个文件
|
| 59 |
+
--ref-npz 单图时为参考 .npz,数据集时为参考汇总 .npz(按 names 匹配),
|
| 60 |
+
报告 max/mean/p99 绝对误差、cosine 以及 scale-shift 对齐后误差
|
| 61 |
+
"""
|
| 62 |
+
|
| 63 |
+
import argparse
|
| 64 |
+
import json
|
| 65 |
+
import logging
|
| 66 |
+
import time
|
| 67 |
+
from pathlib import Path
|
| 68 |
+
|
| 69 |
+
import cv2
|
| 70 |
+
import numpy as np
|
| 71 |
+
from tqdm import tqdm
|
| 72 |
+
|
| 73 |
+
MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
|
| 74 |
+
STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
|
| 75 |
+
EXTENSIONS = {'.jpg', '.jpeg', '.png', '.bmp', '.tif', '.tiff'}
|
| 76 |
+
STAGES = ('read', 'preprocess', 'npu', 'diagnostics',
|
| 77 |
+
'postprocess', 'visualize', 'write', 'total')
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def meta_type(meta):
|
| 81 |
+
value = getattr(meta, 'type', None)
|
| 82 |
+
if value is None:
|
| 83 |
+
value = getattr(meta, 'dtype', '')
|
| 84 |
+
return str(value).lower()
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def input_spec(meta):
|
| 88 |
+
"""从运行时输入元信息推导尺寸、layout 和 dtype,不依赖固定名称。"""
|
| 89 |
+
try:
|
| 90 |
+
shape = [int(v) for v in meta.shape]
|
| 91 |
+
except (TypeError, ValueError):
|
| 92 |
+
raise ValueError(f'输入必须是静态 4D NCHW/NHWC: {meta.name} {meta.shape}')
|
| 93 |
+
if len(shape) != 4 or any(v <= 0 for v in shape):
|
| 94 |
+
raise ValueError(f'输入必须是静态 4D NCHW/NHWC: {meta.name} {meta.shape}')
|
| 95 |
+
if shape[0] != 1:
|
| 96 |
+
raise ValueError(f'仅支持 batch=1: {meta.name} {meta.shape}')
|
| 97 |
+
if shape[1] == 3:
|
| 98 |
+
layout, size = 'NCHW', (shape[2], shape[3])
|
| 99 |
+
elif shape[3] == 3:
|
| 100 |
+
layout, size = 'NHWC', (shape[1], shape[2])
|
| 101 |
+
else:
|
| 102 |
+
raise ValueError(f'输入必须含 3 通道: {meta.name} {meta.shape}')
|
| 103 |
+
type_name = meta_type(meta)
|
| 104 |
+
if 'float16' in type_name or 'fp16' in type_name:
|
| 105 |
+
dtype = np.float16
|
| 106 |
+
elif 'float' in type_name or 'fp32' in type_name:
|
| 107 |
+
dtype = np.float32
|
| 108 |
+
elif 'uint8' in type_name or type_name.endswith('u8'):
|
| 109 |
+
dtype = np.uint8
|
| 110 |
+
elif 'int8' in type_name or type_name.endswith('s8'):
|
| 111 |
+
dtype = np.int8
|
| 112 |
+
else:
|
| 113 |
+
raise ValueError(f'不支持的输入类型: {meta.name} {type_name!r}')
|
| 114 |
+
return size, layout, dtype
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
def preprocess(image, size, color_order, layout, dtype, norm_mode):
|
| 118 |
+
"""按编译契约生成输入 tensor。
|
| 119 |
+
|
| 120 |
+
uint8 直送路径沿用官方 axmodel 流程,在 uint8 域直接 resize;ImageNet
|
| 121 |
+
归一化路径严格按 ONNX 基准的顺序(float 域 /255 -> resize -> mean/std)。
|
| 122 |
+
"""
|
| 123 |
+
target_h, target_w = size
|
| 124 |
+
if color_order == 'rgb':
|
| 125 |
+
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
|
| 126 |
+
if norm_mode == 'imagenet':
|
| 127 |
+
canvas = image.astype(np.float32) / 255.0
|
| 128 |
+
canvas = cv2.resize(canvas, (target_w, target_h),
|
| 129 |
+
interpolation=cv2.INTER_CUBIC)
|
| 130 |
+
canvas = (canvas - MEAN) / STD
|
| 131 |
+
elif dtype != np.uint8:
|
| 132 |
+
canvas = cv2.resize(image.astype(np.float32), (target_w, target_h),
|
| 133 |
+
interpolation=cv2.INTER_CUBIC)
|
| 134 |
+
else:
|
| 135 |
+
canvas = cv2.resize(image, (target_w, target_h),
|
| 136 |
+
interpolation=cv2.INTER_CUBIC).astype(np.float32)
|
| 137 |
+
if dtype == np.uint8:
|
| 138 |
+
canvas = np.clip(np.rint(canvas), 0, 255).astype(np.uint8)
|
| 139 |
+
elif dtype == np.int8:
|
| 140 |
+
raise ValueError('当前脚本不支持 INT8 输入:缺少反量化 scale/zero-point')
|
| 141 |
+
else:
|
| 142 |
+
canvas = canvas.astype(dtype)
|
| 143 |
+
tensor = canvas.transpose(2, 0, 1) if layout == 'NCHW' else canvas
|
| 144 |
+
return np.ascontiguousarray(tensor[None])
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
def select_output(outputs, metas, input_size):
|
| 148 |
+
"""axmodel 输出名可能被编译器改写:先按名字,再按 shape 回退。"""
|
| 149 |
+
if not isinstance(outputs, (list, tuple)):
|
| 150 |
+
outputs = [outputs]
|
| 151 |
+
if len(outputs) != len(metas):
|
| 152 |
+
raise ValueError(f'输出数量不一致: run 返回 {len(outputs)},'
|
| 153 |
+
f'接口声明 {len(metas)}')
|
| 154 |
+
for index, meta in enumerate(metas):
|
| 155 |
+
if 'depth' in str(meta.name).lower():
|
| 156 |
+
return outputs[index], str(meta.name)
|
| 157 |
+
candidates = []
|
| 158 |
+
for index, array in enumerate(outputs):
|
| 159 |
+
shape = np.asarray(array).shape
|
| 160 |
+
if len(shape) == 4 and shape[1] == 1 and tuple(shape[2:]) == tuple(input_size):
|
| 161 |
+
candidates.append(index)
|
| 162 |
+
elif len(shape) == 3 and tuple(shape[1:]) == tuple(input_size):
|
| 163 |
+
candidates.append(index)
|
| 164 |
+
if len(candidates) != 1:
|
| 165 |
+
actual = [(m.name, np.asarray(o).shape) for m, o in zip(metas, outputs)]
|
| 166 |
+
raise ValueError(f'无法按 name 或 shape 唯一匹配 depth 输出;实际输出: {actual}')
|
| 167 |
+
return outputs[candidates[0]], str(metas[candidates[0]].name)
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
def read_depth(raw, input_size):
|
| 171 |
+
array = np.asarray(raw)
|
| 172 |
+
if array.ndim == 4 and array.shape[1] == 1:
|
| 173 |
+
array = array[0]
|
| 174 |
+
if array.ndim == 3 and array.shape[0] == 1:
|
| 175 |
+
array = array[0]
|
| 176 |
+
if array.shape != tuple(input_size):
|
| 177 |
+
raise ValueError(f'depth 输出无效: shape={array.shape},'
|
| 178 |
+
f'期望 {tuple(input_size)}')
|
| 179 |
+
if not np.issubdtype(array.dtype, np.floating):
|
| 180 |
+
raise ValueError(f'depth 输出应为浮点,实际 dtype={array.dtype},'
|
| 181 |
+
'整数输出需要已知反量化参数')
|
| 182 |
+
array = array.astype(np.float32, copy=False)
|
| 183 |
+
if not np.isfinite(array).all():
|
| 184 |
+
raise ValueError('depth 输出包含 nan/inf')
|
| 185 |
+
if float(array.std()) < 1e-6:
|
| 186 |
+
logging.warning('depth 输出近似常数(std=%.3g):请检查输入归一化/颜色顺序'
|
| 187 |
+
'是否与编译配置的 input_processors 一致。', float(array.std()))
|
| 188 |
+
return np.ascontiguousarray(array)
|
| 189 |
+
|
| 190 |
+
|
| 191 |
+
def restore_depth(depth, original_size):
|
| 192 |
+
"""按官方推理的 bilinear + align_corners=True 还原到原图尺寸。"""
|
| 193 |
+
height, width = original_size
|
| 194 |
+
src_h, src_w = depth.shape
|
| 195 |
+
if (src_h, src_w) == (height, width):
|
| 196 |
+
return depth.copy()
|
| 197 |
+
xs = np.linspace(0.0, src_w - 1, width, dtype=np.float32)
|
| 198 |
+
ys = np.linspace(0.0, src_h - 1, height, dtype=np.float32)
|
| 199 |
+
map_x, map_y = np.meshgrid(xs, ys)
|
| 200 |
+
return cv2.remap(depth, map_x, map_y, interpolation=cv2.INTER_LINEAR,
|
| 201 |
+
borderMode=cv2.BORDER_REPLICATE)
|
| 202 |
+
|
| 203 |
+
|
| 204 |
+
def visualize(image, depth):
|
| 205 |
+
finite = depth[np.isfinite(depth)]
|
| 206 |
+
low, high = np.percentile(finite, (2, 98))
|
| 207 |
+
value = np.clip((depth - low) / max(float(high - low), 1e-6), 0, 1)
|
| 208 |
+
colored = cv2.applyColorMap((value * 255).astype(np.uint8), cv2.COLORMAP_INFERNO)
|
| 209 |
+
panel = np.concatenate((image, colored), axis=1)
|
| 210 |
+
legend = np.full((56, panel.shape[1], 3), 255, dtype=np.uint8)
|
| 211 |
+
ramp = np.tile(np.linspace(0, 255, max(panel.shape[1] - 20, 1),
|
| 212 |
+
dtype=np.uint8), (12, 1))
|
| 213 |
+
legend[5:17, 10:10 + ramp.shape[1]] = cv2.applyColorMap(ramp, cv2.COLORMAP_INFERNO)
|
| 214 |
+
cv2.putText(legend, f'Image | Relative depth (2-98%): {low:.3f} -> {high:.3f}',
|
| 215 |
+
(10, 42), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (30, 30, 30), 1, cv2.LINE_AA)
|
| 216 |
+
return np.concatenate((panel, legend), axis=0)
|
| 217 |
+
|
| 218 |
+
|
| 219 |
+
def log_tensor(name, value):
|
| 220 |
+
logging.info('%s: shape=%s dtype=%s min=%.6g max=%.6g mean=%.6g nan=%d inf=%d',
|
| 221 |
+
name, value.shape, value.dtype, float(np.min(value)),
|
| 222 |
+
float(np.max(value)), float(np.mean(value)),
|
| 223 |
+
int(np.isnan(value).sum()), int(np.isinf(value).sum()))
|
| 224 |
+
|
| 225 |
+
|
| 226 |
+
def compare_arrays(depth, expected, source='', preprocessing=''):
|
| 227 |
+
expected = np.asarray(expected)
|
| 228 |
+
if expected.ndim == 4 and expected.shape[1] == 1:
|
| 229 |
+
expected = expected[0]
|
| 230 |
+
if expected.shape != depth.shape:
|
| 231 |
+
raise ValueError(f'参考 depth shape 不符: {expected.shape} vs {depth.shape} ({source})')
|
| 232 |
+
if not np.isfinite(expected).all():
|
| 233 |
+
raise ValueError(f'参考 depth 含 nan/inf: {source}')
|
| 234 |
+
actual64 = depth.astype(np.float64).ravel()
|
| 235 |
+
expected64 = expected.astype(np.float64).ravel()
|
| 236 |
+
diff = np.abs(actual64 - expected64)
|
| 237 |
+
cosine = float(np.dot(actual64, expected64)
|
| 238 |
+
/ (np.linalg.norm(actual64) * np.linalg.norm(expected64) + 1e-12))
|
| 239 |
+
# 相对深度存在全局 scale/offset 漂移,对齐后误差更能反映结构偏差。
|
| 240 |
+
slope, offset = np.polyfit(expected64, actual64, 1)
|
| 241 |
+
aligned = np.abs(actual64 - (slope * expected64 + offset))
|
| 242 |
+
metrics = {'cosine': cosine,
|
| 243 |
+
'max_abs': float(diff.max()),
|
| 244 |
+
'mean_abs': float(diff.mean()),
|
| 245 |
+
'p99_abs': float(np.percentile(diff, 99)),
|
| 246 |
+
'scale': float(slope), 'offset': float(offset),
|
| 247 |
+
'aligned_mean_abs': float(aligned.mean()),
|
| 248 |
+
'aligned_p99_abs': float(np.percentile(aligned, 99)),
|
| 249 |
+
'reference_preprocessing': preprocessing}
|
| 250 |
+
logging.info('参考对比: %s', metrics)
|
| 251 |
+
return metrics
|
| 252 |
+
|
| 253 |
+
|
| 254 |
+
def compare_reference(depth, reference_path):
|
| 255 |
+
with np.load(reference_path, allow_pickle=False) as reference:
|
| 256 |
+
if 'depth' not in reference:
|
| 257 |
+
raise ValueError(f'参考 NPZ 缺少 depth: {reference_path}')
|
| 258 |
+
expected = np.asarray(reference['depth'])
|
| 259 |
+
preprocessing = (str(reference['preprocessing'])
|
| 260 |
+
if 'preprocessing' in reference else '')
|
| 261 |
+
return compare_arrays(depth, expected, str(reference_path), preprocessing)
|
| 262 |
+
|
| 263 |
+
|
| 264 |
+
def load_reference_dataset(path):
|
| 265 |
+
"""读取汇总 NPZ,返回 {name: depth}。"""
|
| 266 |
+
path = Path(path)
|
| 267 |
+
if path.is_file():
|
| 268 |
+
with np.load(path, allow_pickle=False) as ref:
|
| 269 |
+
if 'depths' not in ref or 'names' not in ref:
|
| 270 |
+
raise ValueError(f'汇总 NPZ 需要 names/depths: {path}')
|
| 271 |
+
return {str(n): d for n, d in zip(ref['names'], ref['depths'])}
|
| 272 |
+
raise ValueError(f'数据集模式 --ref-npz 需要汇总 NPZ 文件: {path}')
|
| 273 |
+
|
| 274 |
+
|
| 275 |
+
def preprocessing_description(color_order, norm_mode):
|
| 276 |
+
norm = ('imagenet' if norm_mode == 'imagenet' else 'raw-0-255')
|
| 277 |
+
return f'color={color_order}; resize=stretch; norm={norm}'
|
| 278 |
+
|
| 279 |
+
|
| 280 |
+
def save_raw_npz(path, depth, original_size, input_size, color_order, norm_mode):
|
| 281 |
+
# 板端 CPU 弱,savez_compressed 的 DEFLATE 对 float32 收益极小却耗时明显,用未压缩保存。
|
| 282 |
+
np.savez(path, depth=depth,
|
| 283 |
+
original_size=np.array(original_size, dtype=np.int32),
|
| 284 |
+
input_size=np.array(input_size, dtype=np.int32),
|
| 285 |
+
preprocessing=preprocessing_description(color_order, norm_mode))
|
| 286 |
+
|
| 287 |
+
|
| 288 |
+
def flush_dataset_npz(path, records, chunk_index, input_size, color_order, norm_mode):
|
| 289 |
+
"""把缓冲区写成汇总 NPZ:names / depths / original_sizes。
|
| 290 |
+
|
| 291 |
+
板端弱 CPU 上 DEFLATE 对 float32 收益极小却耗时明显,这里保持未压缩。
|
| 292 |
+
chunk_index 为 None 时写入 path 本身,否则按 chunk 编号追加后缀。
|
| 293 |
+
"""
|
| 294 |
+
if not records:
|
| 295 |
+
return
|
| 296 |
+
target = path if chunk_index is None else path.with_name(
|
| 297 |
+
f'{path.stem}_{chunk_index:04d}{path.suffix}')
|
| 298 |
+
target.parent.mkdir(parents=True, exist_ok=True)
|
| 299 |
+
np.savez(
|
| 300 |
+
target,
|
| 301 |
+
names=np.array([r[0] for r in records]),
|
| 302 |
+
depths=np.stack([r[1] for r in records]),
|
| 303 |
+
original_sizes=np.array([r[2] for r in records], dtype=np.int32),
|
| 304 |
+
input_size=np.array(input_size, dtype=np.int32),
|
| 305 |
+
preprocessing=preprocessing_description(color_order, norm_mode))
|
| 306 |
+
|
| 307 |
+
|
| 308 |
+
def collect_images(dataset):
|
| 309 |
+
paths = sorted(p for p in dataset.rglob('*')
|
| 310 |
+
if p.is_file() and p.suffix.lower() in EXTENSIONS)
|
| 311 |
+
if not paths:
|
| 312 |
+
raise ValueError(f'目录中没有图片: {dataset}')
|
| 313 |
+
return paths
|
| 314 |
+
|
| 315 |
+
|
| 316 |
+
def summarize(samples):
|
| 317 |
+
result = {}
|
| 318 |
+
for stage in STAGES:
|
| 319 |
+
values = np.array([sample[stage] for sample in samples], dtype=np.float64)
|
| 320 |
+
result[stage] = {'mean_ms': float(values.mean()),
|
| 321 |
+
'p50_ms': float(np.percentile(values, 50)),
|
| 322 |
+
'p90_ms': float(np.percentile(values, 90)),
|
| 323 |
+
'p99_ms': float(np.percentile(values, 99))}
|
| 324 |
+
return result
|
| 325 |
+
|
| 326 |
+
|
| 327 |
+
def run_single(session, input_meta, output_metas, input_size, layout, dtype,
|
| 328 |
+
norm_mode, args):
|
| 329 |
+
stem = args.image.stem
|
| 330 |
+
timings = dict.fromkeys(STAGES, 0.0)
|
| 331 |
+
total_start = time.perf_counter()
|
| 332 |
+
|
| 333 |
+
start = time.perf_counter()
|
| 334 |
+
image = cv2.imread(str(args.image), cv2.IMREAD_COLOR)
|
| 335 |
+
if image is None:
|
| 336 |
+
raise ValueError(f'读取图片失败: {args.image}')
|
| 337 |
+
timings['read'] = (time.perf_counter() - start) * 1000
|
| 338 |
+
|
| 339 |
+
start = time.perf_counter()
|
| 340 |
+
tensor = preprocess(image, input_size, args.color_order, layout, dtype, norm_mode)
|
| 341 |
+
timings['preprocess'] = (time.perf_counter() - start) * 1000
|
| 342 |
+
|
| 343 |
+
start = time.perf_counter()
|
| 344 |
+
outputs = session.run(None, {input_meta.name: tensor})
|
| 345 |
+
timings['npu'] = (time.perf_counter() - start) * 1000
|
| 346 |
+
|
| 347 |
+
start = time.perf_counter()
|
| 348 |
+
raw, output_name = select_output(outputs, output_metas, input_size)
|
| 349 |
+
depth = read_depth(raw, input_size)
|
| 350 |
+
comparison = None
|
| 351 |
+
if args.debug:
|
| 352 |
+
log_tensor(input_meta.name, tensor)
|
| 353 |
+
log_tensor('depth', depth)
|
| 354 |
+
if args.ref_npz:
|
| 355 |
+
comparison = compare_reference(depth, args.ref_npz)
|
| 356 |
+
timings['diagnostics'] = (time.perf_counter() - start) * 1000
|
| 357 |
+
|
| 358 |
+
start = time.perf_counter()
|
| 359 |
+
restored = restore_depth(depth, image.shape[:2])
|
| 360 |
+
timings['postprocess'] = (time.perf_counter() - start) * 1000
|
| 361 |
+
|
| 362 |
+
start = time.perf_counter()
|
| 363 |
+
panel = visualize(image, restored)
|
| 364 |
+
timings['visualize'] = (time.perf_counter() - start) * 1000
|
| 365 |
+
|
| 366 |
+
start = time.perf_counter()
|
| 367 |
+
args.output.mkdir(parents=True, exist_ok=True)
|
| 368 |
+
image_path = args.output / f'{stem}_depth.jpg'
|
| 369 |
+
if not cv2.imwrite(str(image_path), panel, [cv2.IMWRITE_JPEG_QUALITY, 90]):
|
| 370 |
+
raise OSError(f'保存可视化失败: {image_path}')
|
| 371 |
+
np.save(str(args.output / f'{stem}_depth.npy'), restored)
|
| 372 |
+
npz_path = None
|
| 373 |
+
if args.dump_npz:
|
| 374 |
+
npz_path = Path(args.dump_npz)
|
| 375 |
+
npz_path.parent.mkdir(parents=True, exist_ok=True)
|
| 376 |
+
save_raw_npz(npz_path, depth, image.shape[:2], input_size,
|
| 377 |
+
args.color_order, norm_mode)
|
| 378 |
+
timings['write'] = (time.perf_counter() - start) * 1000
|
| 379 |
+
timings['total'] = (time.perf_counter() - total_start) * 1000
|
| 380 |
+
|
| 381 |
+
cpu_ms = (timings['read'] + timings['preprocess'] + timings['diagnostics']
|
| 382 |
+
+ timings['postprocess'] + timings['visualize'] + timings['write'])
|
| 383 |
+
logging.info('深度(模型分辨率): min=%.4f max=%.4f mean=%.4f p50=%.4f',
|
| 384 |
+
float(depth.min()), float(depth.max()), float(depth.mean()),
|
| 385 |
+
float(np.percentile(depth, 50)))
|
| 386 |
+
logging.info('深度(原图分辨率): min=%.4f max=%.4f mean=%.4f',
|
| 387 |
+
float(restored.min()), float(restored.max()), float(restored.mean()))
|
| 388 |
+
logging.info('输出绑定: %s', output_name)
|
| 389 |
+
logging.info('NPU 推理: %.2f ms;CPU 处理: %.2f ms;端到端: %.2f ms'
|
| 390 |
+
'(单图为首次推理,含运行时首次执行开销;性能请用数据集模式)',
|
| 391 |
+
timings['npu'], cpu_ms, timings['total'])
|
| 392 |
+
logging.info('各阶段耗时: %s',
|
| 393 |
+
' '.join(f'{key}={value:.2f} ms' for key, value in timings.items()))
|
| 394 |
+
if comparison is not None:
|
| 395 |
+
logging.info('参考对比结果: %s', comparison)
|
| 396 |
+
logging.info('结果目录: %s(可视化 %s%s)', args.output, image_path.name,
|
| 397 |
+
f',原始 NPZ {npz_path.name}' if npz_path else '')
|
| 398 |
+
|
| 399 |
+
|
| 400 |
+
def run_dataset(session, input_meta, output_metas, input_size, layout, dtype,
|
| 401 |
+
norm_mode, runtime_version, args):
|
| 402 |
+
images = collect_images(args.dataset)
|
| 403 |
+
if args.limit is not None:
|
| 404 |
+
images = images[:args.limit]
|
| 405 |
+
args.output.mkdir(parents=True, exist_ok=True)
|
| 406 |
+
full_output = args.full_output
|
| 407 |
+
if full_output:
|
| 408 |
+
logging.warning('--full-output: CPU 侧深度还原、可视化与写盘会拉高端到端耗时,'
|
| 409 |
+
'不能只看 NPU 时间。')
|
| 410 |
+
else:
|
| 411 |
+
logging.info('数据集快速模式:只统计 read/preprocess/npu,'
|
| 412 |
+
'跳过深度还原、可视化和结果写盘。')
|
| 413 |
+
ref_map = load_reference_dataset(args.ref_npz) if args.ref_npz else None
|
| 414 |
+
preprocessing = preprocessing_description(args.color_order, norm_mode)
|
| 415 |
+
dump_path = Path(args.dump_npz) if args.dump_npz else None
|
| 416 |
+
if dump_path is not None and dump_path.is_dir():
|
| 417 |
+
raise ValueError('数据集模式 --dump-npz 请传 .npz 文件路径,'
|
| 418 |
+
'不再按目录为每张图生成小文件')
|
| 419 |
+
dump_records, dump_chunk = [], 0
|
| 420 |
+
|
| 421 |
+
first = cv2.imread(str(images[0]), cv2.IMREAD_COLOR)
|
| 422 |
+
if first is None:
|
| 423 |
+
raise ValueError(f'读取图片失败: {images[0]}')
|
| 424 |
+
warm = preprocess(first, input_size, args.color_order, layout, dtype, norm_mode)
|
| 425 |
+
warmup_ms = 0.0
|
| 426 |
+
if args.warmup:
|
| 427 |
+
start = time.perf_counter()
|
| 428 |
+
for _ in range(args.warmup):
|
| 429 |
+
session.run(None, {input_meta.name: warm})
|
| 430 |
+
warmup_ms = (time.perf_counter() - start) * 1000
|
| 431 |
+
|
| 432 |
+
samples = []
|
| 433 |
+
sample_info = []
|
| 434 |
+
for path in tqdm(images, desc='AX 相对深度', unit='image'):
|
| 435 |
+
relative = path.relative_to(args.dataset)
|
| 436 |
+
timings = dict.fromkeys(STAGES, 0.0)
|
| 437 |
+
total_start = time.perf_counter()
|
| 438 |
+
|
| 439 |
+
start = time.perf_counter()
|
| 440 |
+
image = cv2.imread(str(path), cv2.IMREAD_COLOR)
|
| 441 |
+
if image is None:
|
| 442 |
+
raise ValueError(f'读取图片失败: {path}')
|
| 443 |
+
timings['read'] = (time.perf_counter() - start) * 1000
|
| 444 |
+
|
| 445 |
+
start = time.perf_counter()
|
| 446 |
+
tensor = preprocess(image, input_size, args.color_order, layout, dtype, norm_mode)
|
| 447 |
+
timings['preprocess'] = (time.perf_counter() - start) * 1000
|
| 448 |
+
|
| 449 |
+
start = time.perf_counter()
|
| 450 |
+
outputs = session.run(None, {input_meta.name: tensor})
|
| 451 |
+
timings['npu'] = (time.perf_counter() - start) * 1000
|
| 452 |
+
|
| 453 |
+
start = time.perf_counter()
|
| 454 |
+
raw, output_name = select_output(outputs, output_metas, input_size)
|
| 455 |
+
depth = read_depth(raw, input_size)
|
| 456 |
+
comparison = None
|
| 457 |
+
if args.debug and path == images[0]:
|
| 458 |
+
log_tensor(input_meta.name, tensor)
|
| 459 |
+
log_tensor('depth', depth)
|
| 460 |
+
if dump_path is not None:
|
| 461 |
+
dump_records.append((str(relative), depth,
|
| 462 |
+
np.array(image.shape[:2], dtype=np.int32)))
|
| 463 |
+
if args.npz_chunk and len(dump_records) >= args.npz_chunk:
|
| 464 |
+
flush_dataset_npz(dump_path, dump_records, dump_chunk,
|
| 465 |
+
input_size, args.color_order, norm_mode)
|
| 466 |
+
dump_records.clear()
|
| 467 |
+
dump_chunk += 1
|
| 468 |
+
if ref_map is not None:
|
| 469 |
+
key = str(relative)
|
| 470 |
+
if key not in ref_map:
|
| 471 |
+
raise ValueError(f'参考 NPZ 缺少样本: {key}')
|
| 472 |
+
comparison = compare_arrays(depth, ref_map[key],
|
| 473 |
+
f'{args.ref_npz}:{key}', preprocessing)
|
| 474 |
+
timings['diagnostics'] = (time.perf_counter() - start) * 1000
|
| 475 |
+
|
| 476 |
+
restored = None
|
| 477 |
+
if full_output:
|
| 478 |
+
start = time.perf_counter()
|
| 479 |
+
restored = restore_depth(depth, image.shape[:2])
|
| 480 |
+
timings['postprocess'] = (time.perf_counter() - start) * 1000
|
| 481 |
+
start = time.perf_counter()
|
| 482 |
+
panel = visualize(image, restored)
|
| 483 |
+
timings['visualize'] = (time.perf_counter() - start) * 1000
|
| 484 |
+
start = time.perf_counter()
|
| 485 |
+
prefix = args.output / relative
|
| 486 |
+
prefix.parent.mkdir(parents=True, exist_ok=True)
|
| 487 |
+
if not cv2.imwrite(str(prefix) + '.depth.jpg', panel,
|
| 488 |
+
[cv2.IMWRITE_JPEG_QUALITY, 90]):
|
| 489 |
+
raise OSError(f'保存可视化失败: {prefix}.depth.jpg')
|
| 490 |
+
timings['write'] = (time.perf_counter() - start) * 1000
|
| 491 |
+
timings['total'] = (time.perf_counter() - total_start) * 1000
|
| 492 |
+
|
| 493 |
+
samples.append(timings)
|
| 494 |
+
sample_info.append({'name': str(relative),
|
| 495 |
+
'original_size': list(image.shape[:2]),
|
| 496 |
+
'depth_min': float((restored if restored is not None
|
| 497 |
+
else depth).min()),
|
| 498 |
+
'depth_max': float((restored if restored is not None
|
| 499 |
+
else depth).max()),
|
| 500 |
+
'timing_ms': timings,
|
| 501 |
+
'reference_comparison': comparison})
|
| 502 |
+
|
| 503 |
+
if dump_path is not None:
|
| 504 |
+
flush_dataset_npz(dump_path, dump_records,
|
| 505 |
+
None if not args.npz_chunk else dump_chunk,
|
| 506 |
+
input_size, args.color_order, norm_mode)
|
| 507 |
+
|
| 508 |
+
stats = summarize(samples)
|
| 509 |
+
total_ms = sum(sample['total'] for sample in samples)
|
| 510 |
+
cpu_stages = ('read', 'preprocess', 'diagnostics', 'postprocess', 'visualize', 'write')
|
| 511 |
+
cpu_total_ms = sum(stats[stage]['mean_ms'] for stage in cpu_stages)
|
| 512 |
+
report = {'model': str(Path(args.axmodel).resolve()),
|
| 513 |
+
'runtime': runtime_version,
|
| 514 |
+
'task': 'relative-depth',
|
| 515 |
+
'input_name': input_meta.name, 'output_name': output_name,
|
| 516 |
+
'input_size': list(input_size), 'input_layout': layout,
|
| 517 |
+
'input_dtype': np.dtype(dtype).name,
|
| 518 |
+
'color_order': args.color_order,
|
| 519 |
+
'preprocessing': preprocessing_description(args.color_order, norm_mode),
|
| 520 |
+
'batch_size': 1, 'sample_count': len(samples),
|
| 521 |
+
'full_output': full_output, 'warmup_count': args.warmup,
|
| 522 |
+
'warmup_ms': warmup_ms, 'total_ms': total_ms,
|
| 523 |
+
'fps': len(samples) * 1000 / total_ms,
|
| 524 |
+
'npu_mean_ms': stats['npu']['mean_ms'],
|
| 525 |
+
'cpu_total_mean_ms': cpu_total_ms,
|
| 526 |
+
'timing_scope': 'npu=仅 session.run;cpu=读取+预处理+输出提取/校验+'
|
| 527 |
+
'诊断+后处理+可视化+写盘;total=单样本端到端。排除'
|
| 528 |
+
'模型加载、session 初始化和 warmup。快速模式的 '
|
| 529 |
+
'postprocess/visualize/write 记为 0。',
|
| 530 |
+
'stages': stats, 'samples': sample_info}
|
| 531 |
+
with (args.output / 'npu_timing.json').open('w', encoding='utf-8') as file:
|
| 532 |
+
json.dump(report, file, ensure_ascii=False, indent=2, allow_nan=False)
|
| 533 |
+
for stage, values in stats.items():
|
| 534 |
+
logging.info('%s: mean=%.2f p50=%.2f p90=%.2f p99=%.2f ms',
|
| 535 |
+
stage, values['mean_ms'], values['p50_ms'],
|
| 536 |
+
values['p90_ms'], values['p99_ms'])
|
| 537 |
+
logging.info('%d 张图片,NPU mean=%.2f ms,CPU mean=%.2f ms,'
|
| 538 |
+
'total=%.2f s,FPS=%.3f,结果目录: %s',
|
| 539 |
+
len(samples), stats['npu']['mean_ms'], cpu_total_ms,
|
| 540 |
+
total_ms / 1000, report['fps'], args.output)
|
| 541 |
+
|
| 542 |
+
|
| 543 |
+
def main():
|
| 544 |
+
parser = argparse.ArgumentParser(
|
| 545 |
+
description='DepthART 相对深度 axmodel 板端推理(AX650)',
|
| 546 |
+
formatter_class=argparse.RawDescriptionHelpFormatter, epilog=__doc__)
|
| 547 |
+
parser.add_argument('--axmodel', type=Path, required=True, help='编译后的 axmodel')
|
| 548 |
+
source = parser.add_mutually_exclusive_group(required=True)
|
| 549 |
+
source.add_argument('--image', type=Path, help='单张图片')
|
| 550 |
+
source.add_argument('--dataset', type=Path, help='图片目录(递归)')
|
| 551 |
+
parser.add_argument('--output', type=Path, required=True, help='结果目录')
|
| 552 |
+
parser.add_argument('--color-order', choices=('rgb', 'bgr'), default='rgb',
|
| 553 |
+
help='rgb=与官方预处理一致(BGR->RGB);bgr=按编译配置'
|
| 554 |
+
'input_processors.src_format 直送')
|
| 555 |
+
parser.add_argument('--input-norm', choices=('auto', 'raw', 'imagenet'),
|
| 556 |
+
default='auto',
|
| 557 |
+
help='auto=U8 送 0-255、浮点用 ImageNet mean/std;'
|
| 558 |
+
'raw=直送 0-255;imagenet=/255+mean/std(仅浮点输入)')
|
| 559 |
+
parser.add_argument('--warmup', type=int, default=3,
|
| 560 |
+
help='数据集模式 warmup 次数,默认 3')
|
| 561 |
+
parser.add_argument('--limit', type=int, help='数据集模式只处理前 N 张')
|
| 562 |
+
parser.add_argument('--full-output', action='store_true',
|
| 563 |
+
help='数据集模式执行完整后处理、可视化和结果写盘(含 NPZ)')
|
| 564 |
+
parser.add_argument('--debug', action='store_true', help='打印输入/输出统计')
|
| 565 |
+
parser.add_argument('--dump-npz',
|
| 566 |
+
help='单图:NPZ 文件;数据集:汇总 NPZ 文件(names/depths/original_sizes)')
|
| 567 |
+
parser.add_argument('--npz-chunk', type=int, default=0,
|
| 568 |
+
help='数据集模式每 N 张切一个 NPZ 文件,0 表示全部写入一个文件')
|
| 569 |
+
parser.add_argument('--ref-npz', type=Path, help='参考 NPZ 文件或目录')
|
| 570 |
+
args = parser.parse_args()
|
| 571 |
+
logging.basicConfig(level=logging.INFO, format='%(message)s')
|
| 572 |
+
if args.warmup < 0:
|
| 573 |
+
parser.error('--warmup 不能为负数')
|
| 574 |
+
if args.npz_chunk < 0:
|
| 575 |
+
parser.error('--npz-chunk 不能为负数')
|
| 576 |
+
if args.limit is not None and args.limit < 1:
|
| 577 |
+
parser.error('--limit 必须为正数')
|
| 578 |
+
if args.image:
|
| 579 |
+
if not args.image.is_file():
|
| 580 |
+
parser.error(f'--image 不是文件: {args.image}')
|
| 581 |
+
if args.ref_npz and not args.ref_npz.is_file():
|
| 582 |
+
parser.error('单图模式 --ref-npz 必须是文件')
|
| 583 |
+
if args.limit is not None or args.full_output:
|
| 584 |
+
logging.warning('单图模式忽略 --limit/--full-output')
|
| 585 |
+
else:
|
| 586 |
+
if not args.dataset.is_dir():
|
| 587 |
+
parser.error(f'--dataset 不是目录: {args.dataset}')
|
| 588 |
+
|
| 589 |
+
try:
|
| 590 |
+
import axengine
|
| 591 |
+
except ImportError as error:
|
| 592 |
+
raise SystemExit('未找到 axengine:该脚本需在板端运行(pip 安装 axengine/'
|
| 593 |
+
'pyaxengine 后重试)') from error
|
| 594 |
+
|
| 595 |
+
session = axengine.InferenceSession(str(args.axmodel))
|
| 596 |
+
inputs = session.get_inputs()
|
| 597 |
+
output_metas = session.get_outputs()
|
| 598 |
+
logging.info('模型输入: %s', [(m.name, m.shape, meta_type(m)) for m in inputs])
|
| 599 |
+
logging.info('模型输出: %s', [(m.name, m.shape, meta_type(m)) for m in output_metas])
|
| 600 |
+
if len(inputs) != 1:
|
| 601 |
+
raise ValueError(f'模型应有 1 个输入,实际 '
|
| 602 |
+
f'{[(m.name, m.shape) for m in inputs]}')
|
| 603 |
+
input_meta = inputs[0]
|
| 604 |
+
input_size, layout, dtype = input_spec(input_meta)
|
| 605 |
+
if dtype == np.int8:
|
| 606 |
+
raise ValueError('当前脚本不支持 INT8 输入(缺少反量化参数);请把编译配置的 '
|
| 607 |
+
'input_processors.src_dtype 设为 U8 后重编译')
|
| 608 |
+
|
| 609 |
+
if args.input_norm == 'auto':
|
| 610 |
+
norm_mode = 'raw' if dtype == np.uint8 else 'imagenet'
|
| 611 |
+
else:
|
| 612 |
+
norm_mode = args.input_norm
|
| 613 |
+
if dtype == np.uint8 and norm_mode == 'imagenet':
|
| 614 |
+
raise ValueError('输入类型为 uint8,不能在推理时做 ImageNet 归一化;'
|
| 615 |
+
'归一化必须由编译前端 input_processors 或校准口径保证')
|
| 616 |
+
if dtype != np.uint8 and norm_mode == 'raw':
|
| 617 |
+
logging.warning('输入为浮点但按 0-255 直送(--input-norm raw),'
|
| 618 |
+
'仅用于诊断,需与编译前端/校准口径一致。')
|
| 619 |
+
|
| 620 |
+
logging.info('batch=1,输入尺寸: %s,layout=%s,dtype=%s,颜色顺序=%s,'
|
| 621 |
+
'预处理=%s,resize=stretch', input_size, layout,
|
| 622 |
+
np.dtype(dtype).name, args.color_order,
|
| 623 |
+
'ImageNet mean/std (/255)' if norm_mode == 'imagenet'
|
| 624 |
+
else '0-255 原始像素')
|
| 625 |
+
if dtype == np.uint8 and norm_mode == 'raw':
|
| 626 |
+
logging.info('提示:uint8 输入按编译契约送 0-255;ImageNet 归一化应由 '
|
| 627 |
+
'config_axonnx.json 的 input_processors 在模型内完成。')
|
| 628 |
+
|
| 629 |
+
runtime_version = getattr(axengine, '__version__', 'unknown')
|
| 630 |
+
if args.image:
|
| 631 |
+
run_single(session, input_meta, output_metas, input_size, layout, dtype,
|
| 632 |
+
norm_mode, args)
|
| 633 |
+
else:
|
| 634 |
+
run_dataset(session, input_meta, output_metas, input_size, layout, dtype,
|
| 635 |
+
norm_mode, runtime_version, args)
|
| 636 |
+
|
| 637 |
+
|
| 638 |
+
if __name__ == '__main__':
|
| 639 |
+
main()
|
src/infer/depthart_infer_onnx.py
ADDED
|
@@ -0,0 +1,721 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# -*- coding: utf-8 -*-
|
| 3 |
+
"""
|
| 4 |
+
DepthART 相对深度 ONNX 推理(开发机 FP32 基准)。
|
| 5 |
+
|
| 6 |
+
模型接口(脚本按实际 ONNX 读取并校验):
|
| 7 |
+
input [1, 3, 448, 448] float32 NCHW RGB
|
| 8 |
+
depth [1, 1, 448, 448] float32 仿射不变相对深度,数值越大表示越近
|
| 9 |
+
|
| 10 |
+
预处理按官方相对深度推理流程:BGR->RGB,直接拉伸到 448x448(INTER_CUBIC),
|
| 11 |
+
/255 后按 ImageNet mean/std 归一化。归一化必须保留,否则输出会退化。
|
| 12 |
+
|
| 13 |
+
自定义算子:
|
| 14 |
+
relative_s_448_default.onnx 含 com.depthart::SelectiveScan(官方只提供
|
| 15 |
+
TensorRT 插件),onnxruntime 无法直接加载。本脚本在内存中把每个
|
| 16 |
+
SelectiveScan 改写为等价的纯 ONNX 子图(Loop + 逐 t 递归),语义与官方
|
| 17 |
+
reference 实现一致(含 delta_bias/softplus、按组 repeat_interleave)。
|
| 18 |
+
改写后整模型输出与 PyTorch 参考的最大绝对误差约 5e-5。
|
| 19 |
+
--save-rewritten 可导出改写后的标准 ONNX 便于排查。
|
| 20 |
+
注意:pulsar2 不支持 Loop,要编译 axmodel 请用
|
| 21 |
+
src/tool/depthart_export_axonnx.py 生成分块并行扫描版本。
|
| 22 |
+
|
| 23 |
+
单图模式:打印深度统计和各阶段耗时,保存并排可视化,不产出评测文件。
|
| 24 |
+
数据集模式:tqdm 进度 + onnx_timing.json;默认快速模式只统计读取/预处理/推理,
|
| 25 |
+
--full-output 才做深度还原、可视化、原始 NPZ 和图片写盘。
|
| 26 |
+
|
| 27 |
+
用法(在 hs/DepthART 目录下执行):
|
| 28 |
+
# 单图验证,输出 results/onnx_demo/xxx_depth.jpg
|
| 29 |
+
../../venv/bin/python src/infer/depthart_infer_onnx.py --image /path/a.jpg --output results/onnx_demo
|
| 30 |
+
|
| 31 |
+
# 指定其他尺寸/变的模型
|
| 32 |
+
../../venv/bin/python src/infer/depthart_infer_onnx.py --model onnx/relative_b_448_default.onnx --image /path/a.jpg
|
| 33 |
+
|
| 34 |
+
# 数据集快速性能模式
|
| 35 |
+
../../venv/bin/python src/infer/depthart_infer_onnx.py --dataset /path/images --output results/onnx_dataset --limit 50
|
| 36 |
+
|
| 37 |
+
# 数据集完整输出(可视化 + 汇总 NPZ)
|
| 38 |
+
../../venv/bin/python src/infer/depthart_infer_onnx.py --dataset ../../datasets/hs_ai_tv/test_depth \
|
| 39 |
+
--output results/onnx_full --full-output --dump-npz results/onnx_full/raw_outputs.npz
|
| 40 |
+
|
| 41 |
+
# 与参考原始输出对比(单图,--ref-npz 指向单图 NPZ)
|
| 42 |
+
../../venv/bin/python src/infer/depthart_infer_onnx.py --image /path/a.jpg --output results/onnx_check --ref-npz results/onnx_demo/a.jpg.npz
|
| 43 |
+
|
| 44 |
+
# 导出改写后的标准 ONNX(保留原始文件不变)
|
| 45 |
+
../../venv/bin/python src/infer/depthart_infer_onnx.py --image /path/a.jpg --save-rewritten results/relative_s_448_default_std.onnx
|
| 46 |
+
|
| 47 |
+
诊断:
|
| 48 |
+
--debug 打印输入/输出 tensor 的 shape、dtype、min、max、mean、nan 计数,
|
| 49 |
+
以及自定义算子的改写数量
|
| 50 |
+
--dump-npz 单图时为 .npz 文件路径;数据集时为汇总 .npz 文件路径,
|
| 51 |
+
保存 names/depths/original_sizes,不再每张图一个小文件
|
| 52 |
+
--npz-chunk 数据集模式下每 N 张切一个文件(xxx_0000.npz 等),
|
| 53 |
+
0 表示全部写入同一个文件
|
| 54 |
+
--ref-npz 单图时为参考 .npz,数据集时为参考汇总 .npz(按 names 匹配),
|
| 55 |
+
报告 max/mean/p99 绝对误差、cosine 以及 scale-shift 对齐后误差
|
| 56 |
+
"""
|
| 57 |
+
|
| 58 |
+
import argparse
|
| 59 |
+
import json
|
| 60 |
+
import logging
|
| 61 |
+
import time
|
| 62 |
+
from pathlib import Path
|
| 63 |
+
|
| 64 |
+
import cv2
|
| 65 |
+
import numpy as np
|
| 66 |
+
import onnx
|
| 67 |
+
import onnxruntime as ort
|
| 68 |
+
from onnx import helper, numpy_helper, TensorProto
|
| 69 |
+
from tqdm import tqdm
|
| 70 |
+
|
| 71 |
+
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
| 72 |
+
DEFAULT_MODEL = PROJECT_ROOT / 'onnx' / 'relative_s_448_default.onnx'
|
| 73 |
+
MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
|
| 74 |
+
STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
|
| 75 |
+
EXTENSIONS = {'.jpg', '.jpeg', '.png', '.bmp', '.tif', '.tiff'}
|
| 76 |
+
STAGES = ('read', 'preprocess', 'inference', 'diagnostics',
|
| 77 |
+
'postprocess', 'visualize', 'write', 'total')
|
| 78 |
+
|
| 79 |
+
SCAN_DOMAIN = 'com.depthart'
|
| 80 |
+
SCAN_OP = 'SelectiveScan'
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
# ---------------------------------------------------------------------------
|
| 84 |
+
# 自定义算子改写:com.depthart::SelectiveScan -> 纯 ONNX
|
| 85 |
+
# ---------------------------------------------------------------------------
|
| 86 |
+
|
| 87 |
+
class _ScanGraph:
|
| 88 |
+
"""为一个 SelectiveScan 节点生成改写子图。
|
| 89 |
+
|
| 90 |
+
语义(与官方 reference 一致,u 为 [B, D, L]):
|
| 91 |
+
delta = softplus(delta + delta_bias) # 可选前两项
|
| 92 |
+
transition = exp(delta_t * A) # [B, D, N]
|
| 93 |
+
state_t = transition * state_{t-1}
|
| 94 |
+
+ delta_t * B_t * u_t # B/C 按组 repeat_interleave
|
| 95 |
+
y_t = sum_n(state_t * C_t) + D * u_t
|
| 96 |
+
组映射:group = channel // (D / G),即把 [B, G, N, L] 的每个组连续复制
|
| 97 |
+
D/G 次到 [B, D, N, L]。
|
| 98 |
+
"""
|
| 99 |
+
|
| 100 |
+
def __init__(self, prefix):
|
| 101 |
+
self.prefix = prefix
|
| 102 |
+
self.nodes = []
|
| 103 |
+
self.inits = []
|
| 104 |
+
self._cache = {}
|
| 105 |
+
|
| 106 |
+
def const(self, name, value):
|
| 107 |
+
tensor_name = f'{self.prefix}_{name}'
|
| 108 |
+
if tensor_name not in self._cache:
|
| 109 |
+
self.inits.append(numpy_helper.from_array(np.asarray(value), tensor_name))
|
| 110 |
+
self._cache[tensor_name] = True
|
| 111 |
+
return tensor_name
|
| 112 |
+
|
| 113 |
+
def node(self, op, inputs, outputs, **kwargs):
|
| 114 |
+
self.nodes.append(helper.make_node(op, inputs, outputs, **kwargs))
|
| 115 |
+
return outputs[0]
|
| 116 |
+
|
| 117 |
+
def build(self, u, delta, a, b, c, d, bias, attrs, out_name):
|
| 118 |
+
p = self.prefix
|
| 119 |
+
one = self.const('one', np.array([1], dtype=np.int64))
|
| 120 |
+
|
| 121 |
+
# delta (+ bias) -> softplus
|
| 122 |
+
dt = delta
|
| 123 |
+
if bias is not None:
|
| 124 |
+
bias3 = self.node('Unsqueeze', [bias, self.const('uax02', [0, 2])], [f'{p}_bias3'])
|
| 125 |
+
dt = self.node('Add', [delta, bias3], [f'{p}_dt_bias'])
|
| 126 |
+
if int(attrs.get('delta_softplus', 0)):
|
| 127 |
+
dt = self.node('Softplus', [dt], [f'{p}_dt_softplus'])
|
| 128 |
+
|
| 129 |
+
# 形状:D = u.shape[1],G = B.shape[1],gs = D / G,L = u.shape[2]
|
| 130 |
+
u_shape = self.node('Shape', [u], [f'{p}_u_shape'])
|
| 131 |
+
b_shape = self.node('Shape', [b], [f'{p}_b_shape'])
|
| 132 |
+
d_vec = self.node('Gather', [u_shape, self.const('i1', [1])], [f'{p}_d'], axis=0)
|
| 133 |
+
l_vec = self.node('Gather', [u_shape, self.const('i2', [2])], [f'{p}_l'], axis=0)
|
| 134 |
+
g_vec = self.node('Gather', [b_shape, self.const('i1b', [1])], [f'{p}_g'], axis=0)
|
| 135 |
+
n_vec = self.node('Gather', [b_shape, self.const('i2b', [2])], [f'{p}_n'], axis=0)
|
| 136 |
+
l_vec_b = self.node('Gather', [b_shape, self.const('i3b', [3])], [f'{p}_lb'], axis=0)
|
| 137 |
+
gs_vec = self.node('Div', [d_vec, g_vec], [f'{p}_gs'])
|
| 138 |
+
l_scalar = self.node('Gather', [b_shape, self.const('i3s', 3)], [f'{p}_l_scalar'], axis=0)
|
| 139 |
+
|
| 140 |
+
def expand_groups(x, tag):
|
| 141 |
+
# [B,G,N,L] -> [B,G,1,N,L] -> [B,G,gs,N,L] -> [B,D,N,L]
|
| 142 |
+
x5 = self.node('Unsqueeze', [x, self.const(f'{tag}_uax', [2])], [f'{p}_{tag}_x5'])
|
| 143 |
+
target5 = self.node('Concat', [one, g_vec, gs_vec, n_vec, l_vec_b],
|
| 144 |
+
[f'{p}_{tag}_target5'], axis=0)
|
| 145 |
+
expanded = self.node('Expand', [x5, target5], [f'{p}_{tag}_expanded'])
|
| 146 |
+
target4 = self.node('Concat', [one, d_vec, n_vec, l_vec],
|
| 147 |
+
[f'{p}_{tag}_target4'], axis=0)
|
| 148 |
+
return self.node('Reshape', [expanded, target4], [f'{p}_{tag}_4d'])
|
| 149 |
+
|
| 150 |
+
b_full = expand_groups(b, 'b')
|
| 151 |
+
c_full = expand_groups(c, 'c')
|
| 152 |
+
|
| 153 |
+
# 初始状态 [1, D, N]
|
| 154 |
+
state_shape = self.node('Concat', [one, d_vec, n_vec], [f'{p}_state_shape'], axis=0)
|
| 155 |
+
state0 = self.node('ConstantOfShape', [state_shape], [f'{p}_state0'],
|
| 156 |
+
value=helper.make_tensor('value', TensorProto.FLOAT, [1], [0.0]))
|
| 157 |
+
|
| 158 |
+
# Loop body:按时间步推进状态,扫描输出自动在轴 0 堆叠
|
| 159 |
+
bp = f'{p}_body'
|
| 160 |
+
body_inputs = [helper.make_tensor_value_info(bp + '_iter', TensorProto.INT64, []),
|
| 161 |
+
helper.make_tensor_value_info(bp + '_cond', TensorProto.BOOL, []),
|
| 162 |
+
helper.make_tensor_value_info(bp + '_state', TensorProto.FLOAT, None)]
|
| 163 |
+
body_outputs = [helper.make_tensor_value_info(bp + '_cond_out', TensorProto.BOOL, []),
|
| 164 |
+
helper.make_tensor_value_info(bp + '_state_out', TensorProto.FLOAT, None),
|
| 165 |
+
helper.make_tensor_value_info(bp + '_y', TensorProto.FLOAT, None)]
|
| 166 |
+
body = _ScanGraph.body_graph(p, bp, u, dt, b_full, c_full, a, d, body_inputs, body_outputs)
|
| 167 |
+
|
| 168 |
+
final_state = f'{p}_final_state'
|
| 169 |
+
y_seq = f'{p}_y_seq'
|
| 170 |
+
self.node('Loop', [l_scalar, self.const('cond', np.array(True)), state0],
|
| 171 |
+
[final_state, y_seq], body=body, name=f'{p}_loop')
|
| 172 |
+
# y_seq: [L,1,D,1] -> [1,D,L]
|
| 173 |
+
y_sq = self.node('Squeeze', [y_seq, self.const('sq_axes', [1, 3])], [f'{p}_y_sq'])
|
| 174 |
+
y_tr = self.node('Transpose', [y_sq], [f'{p}_y_tr'], perm=[1, 0])
|
| 175 |
+
self.node('Unsqueeze', [y_tr, self.const('uax_out', [0])], [out_name])
|
| 176 |
+
|
| 177 |
+
@staticmethod
|
| 178 |
+
def body_graph(p, bp, u, dt, b_full, c_full, a, d, body_inputs, body_outputs):
|
| 179 |
+
"""构造 Loop body;外层张量通过闭包名字被 body 引用。"""
|
| 180 |
+
g = _ScanGraph(p)
|
| 181 |
+
iter_name, cond_name, state_name = (x.name for x in body_inputs)
|
| 182 |
+
cond_out, state_out, y_out = (x.name for x in body_outputs)
|
| 183 |
+
|
| 184 |
+
u_t = g.node('Gather', [u, iter_name], [bp + '_u_t'], axis=2) # [1,D]
|
| 185 |
+
dt_t = g.node('Gather', [dt, iter_name], [bp + '_dt_t'], axis=2) # [1,D]
|
| 186 |
+
b_t = g.node('Gather', [b_full, iter_name], [bp + '_b_t'], axis=3) # [1,D,N]
|
| 187 |
+
c_t = g.node('Gather', [c_full, iter_name], [bp + '_c_t'], axis=3) # [1,D,N]
|
| 188 |
+
dt3 = g.node('Unsqueeze', [dt_t, g.const('uax2', [2])], [bp + '_dt3'])
|
| 189 |
+
u3 = g.node('Unsqueeze', [u_t, g.const('uax2b', [2])], [bp + '_u3'])
|
| 190 |
+
a3 = g.node('Unsqueeze', [a, g.const('uax0', [0])], [bp + '_a3'])
|
| 191 |
+
trans = g.node('Exp', [g.node('Mul', [dt3, a3], [bp + '_dt_a'])], [bp + '_trans'])
|
| 192 |
+
state_keep = g.node('Mul', [trans, state_name], [bp + '_state_keep'])
|
| 193 |
+
bu = g.node('Mul', [g.node('Mul', [dt3, u3], [bp + '_dt_u']), b_t], [bp + '_dt_u_b'])
|
| 194 |
+
state_new = g.node('Add', [state_keep, bu], [bp + '_state_new'])
|
| 195 |
+
y = g.node('ReduceSum', [g.node('Mul', [state_new, c_t], [bp + '_state_c']),
|
| 196 |
+
g.const('rs_axes', [2])], [bp + '_y_val'], keepdims=0)
|
| 197 |
+
if d is not None:
|
| 198 |
+
y = g.node('Add', [y, g.node('Mul', [d, u_t], [bp + '_d_u'])], [bp + '_y_d'])
|
| 199 |
+
y3 = g.node('Unsqueeze', [y, g.const('uax2c', [2])], [bp + '_y3'])
|
| 200 |
+
g.node('Identity', [cond_name], [cond_out])
|
| 201 |
+
g.node('Identity', [state_new], [state_out])
|
| 202 |
+
g.node('Identity', [y3], [y_out])
|
| 203 |
+
return helper.make_graph(g.nodes, bp + '_graph', body_inputs, body_outputs,
|
| 204 |
+
initializer=g.inits)
|
| 205 |
+
|
| 206 |
+
|
| 207 |
+
def _static_rank(value_info_map, name):
|
| 208 |
+
info = value_info_map.get(name)
|
| 209 |
+
if info is None or not info.type.HasField('tensor_type'):
|
| 210 |
+
return None
|
| 211 |
+
return len(info.type.tensor_type.shape.dim)
|
| 212 |
+
|
| 213 |
+
|
| 214 |
+
def rewrite_selective_scan(model):
|
| 215 |
+
"""把 com.depthart::SelectiveScan 替换成纯 ONNX 子图,返回替换数量。
|
| 216 |
+
|
| 217 |
+
仅处理 domain 为 com.depthart、op_type 为 SelectiveScan 的节点;
|
| 218 |
+
其余自定义 domain 节点保持原样,由调用方检查后报错。
|
| 219 |
+
"""
|
| 220 |
+
try:
|
| 221 |
+
inferred = onnx.shape_inference.infer_shapes(model)
|
| 222 |
+
shape_map = {v.name: v for v in list(inferred.graph.value_info)
|
| 223 |
+
+ list(inferred.graph.input) + list(inferred.graph.output)}
|
| 224 |
+
except Exception as error: # 自定义算子会阻断部分形状推导,不影响 4D B/C 的常规情形
|
| 225 |
+
logging.warning('形状推导失败,按 4D B/C 处理: %s', error)
|
| 226 |
+
shape_map = {}
|
| 227 |
+
new_nodes, extra_inits, count = [], [], 0
|
| 228 |
+
for node in model.graph.node:
|
| 229 |
+
if node.domain != SCAN_DOMAIN or node.op_type != SCAN_OP:
|
| 230 |
+
new_nodes.append(node)
|
| 231 |
+
continue
|
| 232 |
+
count += 1
|
| 233 |
+
prefix = f'depthart_scan_{count - 1}'
|
| 234 |
+
if not 5 <= len(node.input) <= 7:
|
| 235 |
+
raise ValueError(f'{node.name}: 期望 5~7 个输入,实际 {len(node.input)}')
|
| 236 |
+
u, delta, a, b, c = node.input[:5]
|
| 237 |
+
d = node.input[5] if len(node.input) > 5 and node.input[5] else None
|
| 238 |
+
bias = node.input[6] if len(node.input) > 6 and node.input[6] else None
|
| 239 |
+
# 3D 的 B/C 按 [B,N,L] 处理,补一个 group 轴
|
| 240 |
+
if _static_rank(shape_map, b) == 3:
|
| 241 |
+
b4 = prefix + '_b4'
|
| 242 |
+
new_nodes.append(helper.make_node('Unsqueeze', [b, prefix + '_uax_g'],
|
| 243 |
+
[b4], name=prefix + '_b_unsqueeze'))
|
| 244 |
+
extra_inits.append(numpy_helper.from_array(np.array([1], dtype=np.int64),
|
| 245 |
+
prefix + '_uax_g'))
|
| 246 |
+
b = b4
|
| 247 |
+
if _static_rank(shape_map, c) == 3:
|
| 248 |
+
c4 = prefix + '_c4'
|
| 249 |
+
new_nodes.append(helper.make_node('Unsqueeze', [c, prefix + '_uax_g2'],
|
| 250 |
+
[c4], name=prefix + '_c_unsqueeze'))
|
| 251 |
+
extra_inits.append(numpy_helper.from_array(np.array([1], dtype=np.int64),
|
| 252 |
+
prefix + '_uax_g2'))
|
| 253 |
+
c = c4
|
| 254 |
+
attrs = {item.name: item.i for item in node.attribute}
|
| 255 |
+
graph = _ScanGraph(prefix)
|
| 256 |
+
graph.build(u, delta, a, b, c, d, bias, attrs, node.output[0])
|
| 257 |
+
new_nodes.extend(graph.nodes)
|
| 258 |
+
extra_inits.extend(graph.inits)
|
| 259 |
+
|
| 260 |
+
if count:
|
| 261 |
+
del model.graph.node[:]
|
| 262 |
+
model.graph.node.extend(new_nodes)
|
| 263 |
+
model.graph.initializer.extend(extra_inits)
|
| 264 |
+
return count
|
| 265 |
+
|
| 266 |
+
|
| 267 |
+
def unsupported_custom_ops(model):
|
| 268 |
+
return [(n.domain, n.op_type, n.name) for n in model.graph.node
|
| 269 |
+
if n.domain not in ('', 'ai.onnx') and n.domain != SCAN_DOMAIN]
|
| 270 |
+
|
| 271 |
+
|
| 272 |
+
def create_session(model_path, save_rewritten=None):
|
| 273 |
+
"""加载 ONNX;存在 SelectiveScan 时先在内存中改写成标准 ONNX。"""
|
| 274 |
+
model = onnx.load(str(model_path))
|
| 275 |
+
scan_count = rewrite_selective_scan(model)
|
| 276 |
+
if scan_count:
|
| 277 |
+
logging.info('检测到 %d 个 %s::%s,已改写为等价纯 ONNX 子图(Loop)',
|
| 278 |
+
scan_count, SCAN_DOMAIN, SCAN_OP)
|
| 279 |
+
remaining = unsupported_custom_ops(model)
|
| 280 |
+
if remaining:
|
| 281 |
+
raise ValueError(f'模型仍含未支持的自定义算子: {remaining[:5]}')
|
| 282 |
+
if save_rewritten is not None:
|
| 283 |
+
save_rewritten.parent.mkdir(parents=True, exist_ok=True)
|
| 284 |
+
onnx.save(model, str(save_rewritten))
|
| 285 |
+
logging.info('已保存改写后的标准 ONNX: %s', save_rewritten)
|
| 286 |
+
if scan_count:
|
| 287 |
+
return ort.InferenceSession(model.SerializeToString()), scan_count
|
| 288 |
+
return ort.InferenceSession(str(model_path)), 0
|
| 289 |
+
|
| 290 |
+
|
| 291 |
+
# ---------------------------------------------------------------------------
|
| 292 |
+
# 预处理 / 后处理
|
| 293 |
+
# ---------------------------------------------------------------------------
|
| 294 |
+
|
| 295 |
+
def check_interface(session):
|
| 296 |
+
inputs, outputs = session.get_inputs(), session.get_outputs()
|
| 297 |
+
logging.info('模型输入: %s', [(m.name, m.shape, m.type) for m in inputs])
|
| 298 |
+
logging.info('模型输出: %s', [(m.name, m.shape, m.type) for m in outputs])
|
| 299 |
+
if len(inputs) != 1:
|
| 300 |
+
raise ValueError(f'模型应有 1 个输入,实际 {[(m.name, m.shape) for m in inputs]}')
|
| 301 |
+
meta = inputs[0]
|
| 302 |
+
shape = list(meta.shape)
|
| 303 |
+
if len(shape) != 4 or not all(isinstance(v, int) and v > 0 for v in shape):
|
| 304 |
+
raise ValueError(f'输入必须是静态 4D NCHW: {meta.name} {meta.shape}')
|
| 305 |
+
if shape[0] != 1 or shape[1] != 3:
|
| 306 |
+
raise ValueError(f'仅支持 [1,3,H,W] 输入: {meta.name} {meta.shape}')
|
| 307 |
+
if 'float' not in meta.type:
|
| 308 |
+
raise ValueError(f'当前实现要求 float32 输入: {meta.name} {meta.type}')
|
| 309 |
+
depth_metas = [m for m in outputs if 'depth' in m.name.lower()] or outputs
|
| 310 |
+
if len(depth_metas) != 1:
|
| 311 |
+
raise ValueError(f'无法确定 depth 输出: {[(m.name, m.shape) for m in outputs]}')
|
| 312 |
+
return meta.name, (shape[2], shape[3]), depth_metas[0].name
|
| 313 |
+
|
| 314 |
+
|
| 315 |
+
def preprocess(image, size):
|
| 316 |
+
target_h, target_w = size
|
| 317 |
+
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32) / 255.0
|
| 318 |
+
resized = cv2.resize(rgb, (target_w, target_h), interpolation=cv2.INTER_CUBIC)
|
| 319 |
+
normalized = (resized - MEAN) / STD
|
| 320 |
+
tensor = normalized.transpose(2, 0, 1)
|
| 321 |
+
return np.ascontiguousarray(tensor[None])
|
| 322 |
+
|
| 323 |
+
|
| 324 |
+
def read_depth(raw, input_size):
|
| 325 |
+
array = np.asarray(raw)
|
| 326 |
+
if array.ndim == 4 and array.shape[1] == 1:
|
| 327 |
+
array = array[0]
|
| 328 |
+
if array.ndim == 3 and array.shape[0] == 1:
|
| 329 |
+
array = array[0]
|
| 330 |
+
if array.ndim != 2 or array.shape != tuple(input_size):
|
| 331 |
+
raise ValueError(f'depth 输出无效: shape={array.shape},期望 {tuple(input_size)}')
|
| 332 |
+
if array.dtype != np.float32:
|
| 333 |
+
raise ValueError(f'depth 输出必须是 float32: {array.dtype}')
|
| 334 |
+
if not np.isfinite(array).all():
|
| 335 |
+
raise ValueError('depth 输出包含 nan/inf')
|
| 336 |
+
return array
|
| 337 |
+
|
| 338 |
+
|
| 339 |
+
def restore_depth(depth, original_size):
|
| 340 |
+
"""按官方推理的 bilinear + align_corners=True 还原到原图尺寸。"""
|
| 341 |
+
height, width = original_size
|
| 342 |
+
src_h, src_w = depth.shape
|
| 343 |
+
if (src_h, src_w) == (height, width):
|
| 344 |
+
return depth.copy()
|
| 345 |
+
xs = np.linspace(0.0, src_w - 1, width, dtype=np.float32)
|
| 346 |
+
ys = np.linspace(0.0, src_h - 1, height, dtype=np.float32)
|
| 347 |
+
map_x, map_y = np.meshgrid(xs, ys)
|
| 348 |
+
return cv2.remap(depth, map_x, map_y, interpolation=cv2.INTER_LINEAR,
|
| 349 |
+
borderMode=cv2.BORDER_REPLICATE)
|
| 350 |
+
|
| 351 |
+
|
| 352 |
+
def visualize(image, depth):
|
| 353 |
+
finite = depth[np.isfinite(depth)]
|
| 354 |
+
low, high = np.percentile(finite, (2, 98))
|
| 355 |
+
value = np.clip((depth - low) / max(float(high - low), 1e-6), 0, 1)
|
| 356 |
+
colored = cv2.applyColorMap((value * 255).astype(np.uint8), cv2.COLORMAP_INFERNO)
|
| 357 |
+
panel = np.concatenate((image, colored), axis=1)
|
| 358 |
+
legend = np.full((56, panel.shape[1], 3), 255, dtype=np.uint8)
|
| 359 |
+
ramp = np.tile(np.linspace(0, 255, max(panel.shape[1] - 20, 1),
|
| 360 |
+
dtype=np.uint8), (12, 1))
|
| 361 |
+
legend[5:17, 10:10 + ramp.shape[1]] = cv2.applyColorMap(ramp, cv2.COLORMAP_INFERNO)
|
| 362 |
+
cv2.putText(legend, f'Image | Relative depth (2-98%): {low:.3f} -> {high:.3f}',
|
| 363 |
+
(10, 42), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (30, 30, 30), 1, cv2.LINE_AA)
|
| 364 |
+
return np.concatenate((panel, legend), axis=0)
|
| 365 |
+
|
| 366 |
+
|
| 367 |
+
def log_tensor(name, value):
|
| 368 |
+
logging.info('%s: shape=%s dtype=%s min=%.6g max=%.6g mean=%.6g nan=%d inf=%d',
|
| 369 |
+
name, value.shape, value.dtype, float(np.min(value)),
|
| 370 |
+
float(np.max(value)), float(np.mean(value)),
|
| 371 |
+
int(np.isnan(value).sum()), int(np.isinf(value).sum()))
|
| 372 |
+
|
| 373 |
+
|
| 374 |
+
def compare_arrays(depth, expected, source=''):
|
| 375 |
+
expected = np.asarray(expected)
|
| 376 |
+
if expected.ndim == 4 and expected.shape[1] == 1:
|
| 377 |
+
expected = expected[0]
|
| 378 |
+
if expected.shape != depth.shape:
|
| 379 |
+
raise ValueError(f'参考 depth shape 不符: {expected.shape} vs {depth.shape} ({source})')
|
| 380 |
+
if not np.isfinite(expected).all():
|
| 381 |
+
raise ValueError(f'参考 depth 含 nan/inf: {source}')
|
| 382 |
+
actual64 = depth.astype(np.float64).ravel()
|
| 383 |
+
expected64 = expected.astype(np.float64).ravel()
|
| 384 |
+
diff = np.abs(actual64 - expected64)
|
| 385 |
+
cosine = float(np.dot(actual64, expected64)
|
| 386 |
+
/ (np.linalg.norm(actual64) * np.linalg.norm(expected64) + 1e-12))
|
| 387 |
+
# 相对深度存在全局 scale/offset 漂移,对齐后误差更能反映结构偏差。
|
| 388 |
+
slope, offset = np.polyfit(expected64, actual64, 1)
|
| 389 |
+
aligned = np.abs(actual64 - (slope * expected64 + offset))
|
| 390 |
+
metrics = {'cosine': cosine,
|
| 391 |
+
'max_abs': float(diff.max()),
|
| 392 |
+
'mean_abs': float(diff.mean()),
|
| 393 |
+
'p99_abs': float(np.percentile(diff, 99)),
|
| 394 |
+
'scale': float(slope), 'offset': float(offset),
|
| 395 |
+
'aligned_mean_abs': float(aligned.mean()),
|
| 396 |
+
'aligned_p99_abs': float(np.percentile(aligned, 99))}
|
| 397 |
+
logging.info('参考对比: %s', metrics)
|
| 398 |
+
return metrics
|
| 399 |
+
|
| 400 |
+
|
| 401 |
+
def compare_reference(depth, reference_path):
|
| 402 |
+
with np.load(reference_path, allow_pickle=False) as reference:
|
| 403 |
+
if 'depth' not in reference:
|
| 404 |
+
raise ValueError(f'参考 NPZ 缺少 depth: {reference_path}')
|
| 405 |
+
expected = np.asarray(reference['depth'])
|
| 406 |
+
return compare_arrays(depth, expected, str(reference_path))
|
| 407 |
+
|
| 408 |
+
|
| 409 |
+
def load_reference_dataset(path):
|
| 410 |
+
"""读取汇总 NPZ,返回 {name: depth}。"""
|
| 411 |
+
path = Path(path)
|
| 412 |
+
if path.is_file():
|
| 413 |
+
with np.load(path, allow_pickle=False) as ref:
|
| 414 |
+
if 'depths' not in ref or 'names' not in ref:
|
| 415 |
+
raise ValueError(f'汇总 NPZ 需要 names/depths: {path}')
|
| 416 |
+
return {str(n): d for n, d in zip(ref['names'], ref['depths'])}
|
| 417 |
+
raise ValueError(f'数据集模式 --ref-npz 需要汇总 NPZ 文件: {path}')
|
| 418 |
+
|
| 419 |
+
|
| 420 |
+
def save_raw_npz(path, depth, original_size, input_size):
|
| 421 |
+
np.savez_compressed(path, depth=depth,
|
| 422 |
+
original_size=np.array(original_size, dtype=np.int32),
|
| 423 |
+
input_size=np.array(input_size, dtype=np.int32),
|
| 424 |
+
preprocessing='RGB /255 + ImageNet mean/std; resize=stretch')
|
| 425 |
+
|
| 426 |
+
|
| 427 |
+
def flush_dataset_npz(path, records, chunk_index, input_size, preprocessing):
|
| 428 |
+
"""把缓冲区写成汇总 NPZ:names / depths / original_sizes。
|
| 429 |
+
|
| 430 |
+
chunk_index 为 None 时写入 path 本身,否则按 chunk 编号追加后缀。
|
| 431 |
+
"""
|
| 432 |
+
if not records:
|
| 433 |
+
return
|
| 434 |
+
target = path if chunk_index is None else path.with_name(
|
| 435 |
+
f'{path.stem}_{chunk_index:04d}{path.suffix}')
|
| 436 |
+
target.parent.mkdir(parents=True, exist_ok=True)
|
| 437 |
+
np.savez_compressed(
|
| 438 |
+
target,
|
| 439 |
+
names=np.array([r[0] for r in records]),
|
| 440 |
+
depths=np.stack([r[1] for r in records]),
|
| 441 |
+
original_sizes=np.array([r[2] for r in records], dtype=np.int32),
|
| 442 |
+
input_size=np.array(input_size, dtype=np.int32),
|
| 443 |
+
preprocessing=preprocessing)
|
| 444 |
+
|
| 445 |
+
|
| 446 |
+
def collect_images(dataset):
|
| 447 |
+
paths = sorted(p for p in dataset.rglob('*')
|
| 448 |
+
if p.is_file() and p.suffix.lower() in EXTENSIONS)
|
| 449 |
+
if not paths:
|
| 450 |
+
raise ValueError(f'目录中没有图片: {dataset}')
|
| 451 |
+
return paths
|
| 452 |
+
|
| 453 |
+
|
| 454 |
+
def summarize(samples):
|
| 455 |
+
result = {}
|
| 456 |
+
for stage in STAGES:
|
| 457 |
+
values = np.array([sample[stage] for sample in samples], dtype=np.float64)
|
| 458 |
+
result[stage] = {'mean_ms': float(values.mean()),
|
| 459 |
+
'p50_ms': float(np.percentile(values, 50)),
|
| 460 |
+
'p90_ms': float(np.percentile(values, 90)),
|
| 461 |
+
'p99_ms': float(np.percentile(values, 99))}
|
| 462 |
+
return result
|
| 463 |
+
|
| 464 |
+
|
| 465 |
+
# ---------------------------------------------------------------------------
|
| 466 |
+
# 推理流程
|
| 467 |
+
# ---------------------------------------------------------------------------
|
| 468 |
+
|
| 469 |
+
def run_single(session, input_name, output_name, input_size, args):
|
| 470 |
+
stem = args.image.stem
|
| 471 |
+
timings = dict.fromkeys(STAGES, 0.0)
|
| 472 |
+
total_start = time.perf_counter()
|
| 473 |
+
|
| 474 |
+
start = time.perf_counter()
|
| 475 |
+
image = cv2.imread(str(args.image), cv2.IMREAD_COLOR)
|
| 476 |
+
if image is None:
|
| 477 |
+
raise ValueError(f'读取图片失败: {args.image}')
|
| 478 |
+
timings['read'] = (time.perf_counter() - start) * 1000
|
| 479 |
+
|
| 480 |
+
start = time.perf_counter()
|
| 481 |
+
tensor = preprocess(image, input_size)
|
| 482 |
+
timings['preprocess'] = (time.perf_counter() - start) * 1000
|
| 483 |
+
|
| 484 |
+
start = time.perf_counter()
|
| 485 |
+
raw = session.run([output_name], {input_name: tensor})[0]
|
| 486 |
+
timings['inference'] = (time.perf_counter() - start) * 1000
|
| 487 |
+
depth = read_depth(raw, input_size)
|
| 488 |
+
|
| 489 |
+
start = time.perf_counter()
|
| 490 |
+
comparison = None
|
| 491 |
+
if args.debug:
|
| 492 |
+
log_tensor(input_name, tensor)
|
| 493 |
+
log_tensor('depth', depth)
|
| 494 |
+
if args.ref_npz:
|
| 495 |
+
comparison = compare_reference(depth, args.ref_npz)
|
| 496 |
+
timings['diagnostics'] = (time.perf_counter() - start) * 1000
|
| 497 |
+
|
| 498 |
+
start = time.perf_counter()
|
| 499 |
+
restored = restore_depth(depth, image.shape[:2])
|
| 500 |
+
timings['postprocess'] = (time.perf_counter() - start) * 1000
|
| 501 |
+
|
| 502 |
+
start = time.perf_counter()
|
| 503 |
+
panel = visualize(image, restored)
|
| 504 |
+
timings['visualize'] = (time.perf_counter() - start) * 1000
|
| 505 |
+
|
| 506 |
+
start = time.perf_counter()
|
| 507 |
+
args.output.mkdir(parents=True, exist_ok=True)
|
| 508 |
+
image_path = args.output / f'{stem}_depth.jpg'
|
| 509 |
+
if not cv2.imwrite(str(image_path), panel):
|
| 510 |
+
raise OSError(f'保存可视化失败: {image_path}')
|
| 511 |
+
np.save(str(args.output / f'{stem}_depth.npy'), restored)
|
| 512 |
+
npz_path = None
|
| 513 |
+
if args.dump_npz:
|
| 514 |
+
npz_path = Path(args.dump_npz)
|
| 515 |
+
npz_path.parent.mkdir(parents=True, exist_ok=True)
|
| 516 |
+
save_raw_npz(npz_path, depth, image.shape[:2], input_size)
|
| 517 |
+
timings['write'] = (time.perf_counter() - start) * 1000
|
| 518 |
+
timings['total'] = (time.perf_counter() - total_start) * 1000
|
| 519 |
+
|
| 520 |
+
logging.info('深度(模型分辨率): min=%.4f max=%.4f mean=%.4f p50=%.4f',
|
| 521 |
+
float(depth.min()), float(depth.max()), float(depth.mean()),
|
| 522 |
+
float(np.percentile(depth, 50)))
|
| 523 |
+
logging.info('深度(原图分辨率): min=%.4f max=%.4f mean=%.4f',
|
| 524 |
+
float(restored.min()), float(restored.max()), float(restored.mean()))
|
| 525 |
+
logging.info('各阶段耗时: %s',
|
| 526 |
+
' '.join(f'{key}={value:.2f} ms' for key, value in timings.items()))
|
| 527 |
+
logging.info('结果目录: %s(可视化 %s%s)', args.output, image_path.name,
|
| 528 |
+
f',原始 NPZ {npz_path.name}' if npz_path else '')
|
| 529 |
+
|
| 530 |
+
|
| 531 |
+
def run_dataset(session, input_name, output_name, input_size, args):
|
| 532 |
+
images = collect_images(args.dataset)
|
| 533 |
+
if args.limit is not None:
|
| 534 |
+
images = images[:args.limit]
|
| 535 |
+
args.output.mkdir(parents=True, exist_ok=True)
|
| 536 |
+
full_output = args.full_output
|
| 537 |
+
if full_output:
|
| 538 |
+
logging.warning('--full-output: CPU 侧深度还原、可视化与写盘会拉高端到端耗时,'
|
| 539 |
+
'不能只看 inference 时间。')
|
| 540 |
+
else:
|
| 541 |
+
logging.info('数据集快速模式:只统计 read/preprocess/inference,'
|
| 542 |
+
'跳过深度还原、可视化和结果写盘。')
|
| 543 |
+
ref_map = load_reference_dataset(args.ref_npz) if args.ref_npz else None
|
| 544 |
+
|
| 545 |
+
preprocessing = 'RGB /255 + ImageNet mean/std; resize=stretch'
|
| 546 |
+
dump_path = Path(args.dump_npz) if args.dump_npz else None
|
| 547 |
+
if dump_path is not None and dump_path.is_dir():
|
| 548 |
+
raise ValueError('数据集模式 --dump-npz 请传 .npz 文件路径,'
|
| 549 |
+
'不再按目录为每张图生成小文件')
|
| 550 |
+
dump_records, dump_chunk = [], 0
|
| 551 |
+
|
| 552 |
+
warmup_ms = 0.0
|
| 553 |
+
if args.warmup:
|
| 554 |
+
first = cv2.imread(str(images[0]), cv2.IMREAD_COLOR)
|
| 555 |
+
if first is None:
|
| 556 |
+
raise ValueError(f'读取图片失败: {images[0]}')
|
| 557 |
+
warm = preprocess(first, input_size)
|
| 558 |
+
start = time.perf_counter()
|
| 559 |
+
for _ in range(args.warmup):
|
| 560 |
+
session.run([output_name], {input_name: warm})
|
| 561 |
+
warmup_ms = (time.perf_counter() - start) * 1000
|
| 562 |
+
|
| 563 |
+
samples, sample_info = [], []
|
| 564 |
+
for index, path in enumerate(tqdm(images, desc='ONNX 相对深度', unit='image')):
|
| 565 |
+
relative = path.relative_to(args.dataset)
|
| 566 |
+
timings = dict.fromkeys(STAGES, 0.0)
|
| 567 |
+
total_start = time.perf_counter()
|
| 568 |
+
|
| 569 |
+
start = time.perf_counter()
|
| 570 |
+
image = cv2.imread(str(path), cv2.IMREAD_COLOR)
|
| 571 |
+
if image is None:
|
| 572 |
+
raise ValueError(f'读取图片失败: {path}')
|
| 573 |
+
timings['read'] = (time.perf_counter() - start) * 1000
|
| 574 |
+
|
| 575 |
+
start = time.perf_counter()
|
| 576 |
+
tensor = preprocess(image, input_size)
|
| 577 |
+
timings['preprocess'] = (time.perf_counter() - start) * 1000
|
| 578 |
+
|
| 579 |
+
start = time.perf_counter()
|
| 580 |
+
raw = session.run([output_name], {input_name: tensor})[0]
|
| 581 |
+
timings['inference'] = (time.perf_counter() - start) * 1000
|
| 582 |
+
depth = read_depth(raw, input_size)
|
| 583 |
+
|
| 584 |
+
start = time.perf_counter()
|
| 585 |
+
comparison = None
|
| 586 |
+
if args.debug and index == 0:
|
| 587 |
+
log_tensor(input_name, tensor)
|
| 588 |
+
log_tensor('depth', depth)
|
| 589 |
+
if dump_path is not None:
|
| 590 |
+
dump_records.append((str(relative), depth,
|
| 591 |
+
np.array(image.shape[:2], dtype=np.int32)))
|
| 592 |
+
if args.npz_chunk and len(dump_records) >= args.npz_chunk:
|
| 593 |
+
flush_dataset_npz(dump_path, dump_records, dump_chunk,
|
| 594 |
+
input_size, preprocessing)
|
| 595 |
+
dump_records.clear()
|
| 596 |
+
dump_chunk += 1
|
| 597 |
+
if ref_map is not None:
|
| 598 |
+
key = str(relative)
|
| 599 |
+
if key not in ref_map:
|
| 600 |
+
raise ValueError(f'参考 NPZ 缺少样本: {key}')
|
| 601 |
+
comparison = compare_arrays(depth, ref_map[key], f'{args.ref_npz}:{key}')
|
| 602 |
+
timings['diagnostics'] = (time.perf_counter() - start) * 1000
|
| 603 |
+
|
| 604 |
+
restored = None
|
| 605 |
+
if full_output:
|
| 606 |
+
start = time.perf_counter()
|
| 607 |
+
restored = restore_depth(depth, image.shape[:2])
|
| 608 |
+
timings['postprocess'] = (time.perf_counter() - start) * 1000
|
| 609 |
+
start = time.perf_counter()
|
| 610 |
+
panel = visualize(image, restored)
|
| 611 |
+
timings['visualize'] = (time.perf_counter() - start) * 1000
|
| 612 |
+
start = time.perf_counter()
|
| 613 |
+
prefix = args.output / relative
|
| 614 |
+
prefix.parent.mkdir(parents=True, exist_ok=True)
|
| 615 |
+
np.save(str(prefix) + '.depth.npy', restored)
|
| 616 |
+
if not cv2.imwrite(str(prefix) + '.depth.jpg', panel):
|
| 617 |
+
raise OSError(f'保存可视化失败: {prefix}.depth.jpg')
|
| 618 |
+
timings['write'] = (time.perf_counter() - start) * 1000
|
| 619 |
+
timings['total'] = (time.perf_counter() - total_start) * 1000
|
| 620 |
+
|
| 621 |
+
samples.append(timings)
|
| 622 |
+
sample_info.append({'name': str(relative),
|
| 623 |
+
'original_size': list(image.shape[:2]),
|
| 624 |
+
'depth_min': float((restored if restored is not None else depth).min()),
|
| 625 |
+
'depth_max': float((restored if restored is not None else depth).max()),
|
| 626 |
+
'timing_ms': timings,
|
| 627 |
+
'reference_comparison': comparison})
|
| 628 |
+
|
| 629 |
+
if dump_path is not None:
|
| 630 |
+
flush_dataset_npz(dump_path, dump_records,
|
| 631 |
+
None if not args.npz_chunk else dump_chunk,
|
| 632 |
+
input_size, preprocessing)
|
| 633 |
+
|
| 634 |
+
stats = summarize(samples)
|
| 635 |
+
total_ms = sum(sample['total'] for sample in samples)
|
| 636 |
+
report = {'model': str(args.model.resolve()),
|
| 637 |
+
'runtime': 'onnxruntime ' + ort.__version__,
|
| 638 |
+
'providers': session.get_providers(),
|
| 639 |
+
'task': 'relative-depth',
|
| 640 |
+
'custom_ops_rewritten': args.scan_count,
|
| 641 |
+
'input_name': input_name, 'output_name': output_name,
|
| 642 |
+
'input_size': list(input_size), 'input_layout': 'NCHW',
|
| 643 |
+
'preprocessing': 'RGB /255 + ImageNet mean/std; resize=stretch',
|
| 644 |
+
'batch_size': 1, 'sample_count': len(samples),
|
| 645 |
+
'full_output': full_output, 'warmup_count': args.warmup,
|
| 646 |
+
'warmup_ms': warmup_ms, 'total_ms': total_ms,
|
| 647 |
+
'fps': len(samples) * 1000 / total_ms,
|
| 648 |
+
'timing_scope': '样本 read 至 write,包含 diagnostics;排除模型加载、'
|
| 649 |
+
'session 初始化、自定义算子改写、warmup、进度条和最终性能 JSON 写盘。'
|
| 650 |
+
'快速模式的 postprocess/visualize/write 记为 0。',
|
| 651 |
+
'stages': stats, 'samples': sample_info}
|
| 652 |
+
with (args.output / 'onnx_timing.json').open('w', encoding='utf-8') as file:
|
| 653 |
+
json.dump(report, file, ensure_ascii=False, indent=2, allow_nan=False)
|
| 654 |
+
for stage, values in stats.items():
|
| 655 |
+
logging.info('%s: mean=%.2f p50=%.2f p90=%.2f p99=%.2f ms',
|
| 656 |
+
stage, values['mean_ms'], values['p50_ms'],
|
| 657 |
+
values['p90_ms'], values['p99_ms'])
|
| 658 |
+
logging.info('%d 张图片,total=%.2f s,FPS=%.3f,结果目录: %s',
|
| 659 |
+
len(samples), total_ms / 1000, report['fps'], args.output)
|
| 660 |
+
|
| 661 |
+
|
| 662 |
+
def main():
|
| 663 |
+
parser = argparse.ArgumentParser(
|
| 664 |
+
description='DepthART 相对深度 ONNX 推理',
|
| 665 |
+
formatter_class=argparse.RawDescriptionHelpFormatter, epilog=__doc__)
|
| 666 |
+
parser.add_argument('--model', type=Path, default=DEFAULT_MODEL,
|
| 667 |
+
help=f'FP32 ONNX 模型,默认 {DEFAULT_MODEL.relative_to(PROJECT_ROOT)}')
|
| 668 |
+
source = parser.add_mutually_exclusive_group(required=True)
|
| 669 |
+
source.add_argument('--image', type=Path, help='单张图片')
|
| 670 |
+
source.add_argument('--dataset', type=Path, help='图片目录(递归)')
|
| 671 |
+
parser.add_argument('--output', type=Path,
|
| 672 |
+
default=PROJECT_ROOT / 'results' / 'onnx',
|
| 673 |
+
help='结果目录,默认 results/onnx')
|
| 674 |
+
parser.add_argument('--warmup', type=int, default=3,
|
| 675 |
+
help='数据集模式首次输入 warmup 次数,默认 3')
|
| 676 |
+
parser.add_argument('--limit', type=int, help='数据集模式只处理前 N 张')
|
| 677 |
+
parser.add_argument('--full-output', action='store_true',
|
| 678 |
+
help='数据集模式执行完整后处理、可视化和结果写盘')
|
| 679 |
+
parser.add_argument('--debug', action='store_true', help='打印输入/输出统计')
|
| 680 |
+
parser.add_argument('--dump-npz',
|
| 681 |
+
help='单图:NPZ 文件;数据集:汇总 NPZ 文件(names/depths/original_sizes)')
|
| 682 |
+
parser.add_argument('--npz-chunk', type=int, default=0,
|
| 683 |
+
help='数据集模式每 N 张切一个 NPZ 文件,0 表示全部写入一个文件')
|
| 684 |
+
parser.add_argument('--ref-npz', type=Path,
|
| 685 |
+
help='单图:参考 NPZ 文件;数据集:参考汇总 NPZ 文件')
|
| 686 |
+
parser.add_argument('--save-rewritten', type=Path,
|
| 687 |
+
help='保存去除自定义算子后的标准 ONNX(不影响原模型)')
|
| 688 |
+
args = parser.parse_args()
|
| 689 |
+
logging.basicConfig(level=logging.INFO, format='%(message)s')
|
| 690 |
+
if args.warmup < 0:
|
| 691 |
+
parser.error('--warmup 不能为负数')
|
| 692 |
+
if args.limit is not None and args.limit < 1:
|
| 693 |
+
parser.error('--limit 必须为正数')
|
| 694 |
+
if args.npz_chunk < 0:
|
| 695 |
+
parser.error('--npz-chunk 不能为负数')
|
| 696 |
+
if not args.model.is_file():
|
| 697 |
+
parser.error(f'--model 不存在: {args.model}')
|
| 698 |
+
if args.image:
|
| 699 |
+
if not args.image.is_file():
|
| 700 |
+
parser.error(f'--image 不是文件: {args.image}')
|
| 701 |
+
if args.ref_npz and not args.ref_npz.is_file():
|
| 702 |
+
parser.error('单图模式 --ref-npz 必须是文件')
|
| 703 |
+
if args.limit is not None or args.full_output:
|
| 704 |
+
logging.warning('单图模式忽略 --limit/--full-output')
|
| 705 |
+
else:
|
| 706 |
+
if not args.dataset.is_dir():
|
| 707 |
+
parser.error(f'--dataset 不是目录: {args.dataset}')
|
| 708 |
+
|
| 709 |
+
session, args.scan_count = create_session(args.model, args.save_rewritten)
|
| 710 |
+
input_name, input_size, output_name = check_interface(session)
|
| 711 |
+
logging.info('batch=1���输入尺寸: %s,预处理: RGB /255 + ImageNet mean/std(stretch)',
|
| 712 |
+
input_size)
|
| 713 |
+
|
| 714 |
+
if args.image:
|
| 715 |
+
run_single(session, input_name, output_name, input_size, args)
|
| 716 |
+
else:
|
| 717 |
+
run_dataset(session, input_name, output_name, input_size, args)
|
| 718 |
+
|
| 719 |
+
|
| 720 |
+
if __name__ == '__main__':
|
| 721 |
+
main()
|
src/tool/__pycache__/evaluate_nyu654_depth.cpython-310.pyc
ADDED
|
Binary file (8.21 kB). View file
|
|
|
src/tool/compare_depth_outputs.py
ADDED
|
@@ -0,0 +1,143 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# -*- coding: utf-8 -*-
|
| 3 |
+
"""
|
| 4 |
+
DepthART 两路深度输出的相关性/一致性对比。
|
| 5 |
+
|
| 6 |
+
支持三种输入格式,自动识别:
|
| 7 |
+
- 汇总 NPZ(names/depths),如 ONNX 脚本 --dump-npz 生成的文件
|
| 8 |
+
- 分块汇总目录(*.npz 中含 names/depths)
|
| 9 |
+
- 旧的逐图 NPZ 目录(<图片名>.npz 中含 depth)
|
| 10 |
+
|
| 11 |
+
指标(逐图计算后汇总 mean/p50/p90/min):
|
| 12 |
+
pearson 皮尔逊相关系数
|
| 13 |
+
spearman 斯皮尔曼秩相关
|
| 14 |
+
cosine 余弦相似度
|
| 15 |
+
mae / rmse 原始量纲误差
|
| 16 |
+
aligned_* 最小二乘 scale-shift 对齐后的误差
|
| 17 |
+
rel_error mae / mean(|ref|)
|
| 18 |
+
|
| 19 |
+
用法:
|
| 20 |
+
../../venv/bin/python src/tool/compare_depth_outputs.py \
|
| 21 |
+
--ref results/nyu654/fp32/npz --pred results/nyu654/axmodel \
|
| 22 |
+
--out results/nyu654/eval/corr_fp32_axmodel.json
|
| 23 |
+
"""
|
| 24 |
+
|
| 25 |
+
import argparse
|
| 26 |
+
import json
|
| 27 |
+
from pathlib import Path
|
| 28 |
+
|
| 29 |
+
import numpy as np
|
| 30 |
+
from scipy.stats import spearmanr
|
| 31 |
+
from tqdm import tqdm
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def load_predictions(path):
|
| 35 |
+
path = Path(path)
|
| 36 |
+
if path.is_file():
|
| 37 |
+
files = [path]
|
| 38 |
+
elif path.is_dir():
|
| 39 |
+
files = sorted(path.glob('*.npz'))
|
| 40 |
+
if not files:
|
| 41 |
+
raise ValueError(f'目录中没有 NPZ: {path}')
|
| 42 |
+
else:
|
| 43 |
+
raise FileNotFoundError(f'路径不存在: {path}')
|
| 44 |
+
|
| 45 |
+
preds = {}
|
| 46 |
+
for fp in files:
|
| 47 |
+
with np.load(fp, allow_pickle=False) as data:
|
| 48 |
+
if 'depths' in data and 'names' in data:
|
| 49 |
+
for name, depth in zip(data['names'], data['depths']):
|
| 50 |
+
preds[str(name)] = np.asarray(depth)
|
| 51 |
+
elif 'depth' in data:
|
| 52 |
+
preds[fp.name[:-len('.npz')]] = np.asarray(data['depth'])
|
| 53 |
+
else:
|
| 54 |
+
raise ValueError(f'无法识别的 NPZ: {fp}')
|
| 55 |
+
return preds
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def compare(ref, pred):
|
| 59 |
+
a = np.asarray(ref, dtype=np.float64).ravel()
|
| 60 |
+
b = np.asarray(pred, dtype=np.float64).ravel()
|
| 61 |
+
if a.shape != b.shape:
|
| 62 |
+
raise ValueError(f'shape 不一致: {a.shape} vs {b.shape}')
|
| 63 |
+
diff = b - a
|
| 64 |
+
mae = float(np.mean(np.abs(diff)))
|
| 65 |
+
rmse = float(np.sqrt(np.mean(diff ** 2)))
|
| 66 |
+
cosine = float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-12))
|
| 67 |
+
pearson = float(np.corrcoef(a, b)[0, 1])
|
| 68 |
+
spearman = float(spearmanr(a, b).statistic)
|
| 69 |
+
slope, offset = np.polyfit(a, b, 1)
|
| 70 |
+
aligned = b - (slope * a + offset)
|
| 71 |
+
return {
|
| 72 |
+
'pearson': pearson,
|
| 73 |
+
'spearman': spearman,
|
| 74 |
+
'cosine': cosine,
|
| 75 |
+
'mae': mae,
|
| 76 |
+
'rmse': rmse,
|
| 77 |
+
'rel_error': mae / (np.mean(np.abs(a)) + 1e-12),
|
| 78 |
+
'scale': float(slope),
|
| 79 |
+
'offset': float(offset),
|
| 80 |
+
'aligned_mae': float(np.mean(np.abs(aligned))),
|
| 81 |
+
'aligned_rmse': float(np.sqrt(np.mean(aligned ** 2))),
|
| 82 |
+
}
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def summarize(rows):
|
| 86 |
+
keys = ('pearson', 'spearman', 'cosine', 'rel_error', 'aligned_rmse')
|
| 87 |
+
out = {}
|
| 88 |
+
for k in keys:
|
| 89 |
+
values = np.array([r[k] for r in rows])
|
| 90 |
+
out[k] = {'mean': float(values.mean()), 'p50': float(np.percentile(values, 50)),
|
| 91 |
+
'p90': float(np.percentile(values, 90)), 'min': float(values.min())}
|
| 92 |
+
return out
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def main():
|
| 96 |
+
parser = argparse.ArgumentParser(description='两路深度输出相关性对比')
|
| 97 |
+
parser.add_argument('--ref', required=True, help='参考输出(基准后端)')
|
| 98 |
+
parser.add_argument('--pred', required=True, help='待比较输出')
|
| 99 |
+
parser.add_argument('--out', help='结果 JSON 路径')
|
| 100 |
+
args = parser.parse_args()
|
| 101 |
+
|
| 102 |
+
ref = load_predictions(args.ref)
|
| 103 |
+
pred = load_predictions(args.pred)
|
| 104 |
+
names = sorted(set(ref) & set(pred))
|
| 105 |
+
if not names:
|
| 106 |
+
raise ValueError('两路输出没有同名样本')
|
| 107 |
+
missing = (len(ref) - len(names)) + (len(pred) - len(names))
|
| 108 |
+
if missing:
|
| 109 |
+
print(f'警告: {missing} 个样本名称不匹配,已忽略')
|
| 110 |
+
|
| 111 |
+
rows = []
|
| 112 |
+
for name in tqdm(names, desc='compare', ncols=80):
|
| 113 |
+
rows.append({'name': name, **compare(ref[name], pred[name])})
|
| 114 |
+
|
| 115 |
+
summary = summarize(rows)
|
| 116 |
+
low = [r for r in rows if r['pearson'] < 0.9]
|
| 117 |
+
worst = sorted(rows, key=lambda r: r['pearson'])[:5]
|
| 118 |
+
report = {
|
| 119 |
+
'ref': str(args.ref), 'pred': str(args.pred),
|
| 120 |
+
'num_samples': len(rows),
|
| 121 |
+
'summary': summary,
|
| 122 |
+
'pearson_below_0.9': len(low),
|
| 123 |
+
'worst': [{'name': r['name'], 'pearson': r['pearson'],
|
| 124 |
+
'mae': r['mae'], 'aligned_rmse': r['aligned_rmse']} for r in worst],
|
| 125 |
+
'per_image': rows,
|
| 126 |
+
}
|
| 127 |
+
print(f'\n样本数 {len(rows)},pearson<0.9 的 {len(low)} 张')
|
| 128 |
+
print(f'{"指标":<14}{"mean":>12}{"p50":>12}{"p90":>12}{"min":>12}')
|
| 129 |
+
for k, v in summary.items():
|
| 130 |
+
print(f'{k:<14}{v["mean"]:>12.4f}{v["p50"]:>12.4f}{v["p90"]:>12.4f}{v["min"]:>12.4f}')
|
| 131 |
+
print('\n最差 5 张:')
|
| 132 |
+
for r in worst:
|
| 133 |
+
print(f' {r["name"]:<32} pearson={r["pearson"]:.4f} mae={r["mae"]:.4f}')
|
| 134 |
+
|
| 135 |
+
if args.out:
|
| 136 |
+
Path(args.out).parent.mkdir(parents=True, exist_ok=True)
|
| 137 |
+
with open(args.out, 'w') as fp:
|
| 138 |
+
json.dump(report, fp, indent=2)
|
| 139 |
+
print(f'\n结果已保存: {args.out}')
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
if __name__ == '__main__':
|
| 143 |
+
main()
|
src/tool/compare_onnx_axmodel.py
ADDED
|
@@ -0,0 +1,546 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# -*- coding: utf-8 -*-
|
| 3 |
+
"""
|
| 4 |
+
Depth Anything V2:ONNX 与 AXMODEL 原始输出一致性对比。
|
| 5 |
+
|
| 6 |
+
对同一批图片分别运行 ONNX 和 axmodel,在模型分辨率下比较两路深度输出
|
| 7 |
+
(相对逆深度,数值越大越近),逐样本计算相似度指标,并汇总为 Markdown 文档。
|
| 8 |
+
|
| 9 |
+
两个后端的输入预处理可以不同,但 resize 与颜色顺序必须一致:
|
| 10 |
+
默认 ONNX 做 /255 + ImageNet mean/std,axmodel 送 0-255 原始像素
|
| 11 |
+
(对应编译时把归一化写进 input_processors / calibration_mean/std 的方案)。
|
| 12 |
+
如果 axmodel 重编译后仍需脚本侧归一化,使用 --axmodel-norm imagenet。
|
| 13 |
+
|
| 14 |
+
指标:
|
| 15 |
+
cosine 两路输出的余弦相似度
|
| 16 |
+
pearson 皮尔逊相关系数
|
| 17 |
+
mae / rmse 绝对误差均值 / 均方根
|
| 18 |
+
max_abs/p99_abs 最大误差 / 99 分位误差
|
| 19 |
+
aligned_mae 最小二乘 scale-shift 对齐后的 MAE(区分全局尺度漂移与结构误差)
|
| 20 |
+
aligned_rmse 对齐后的 RMSE
|
| 21 |
+
rel_error mae / mean(|onnx|)
|
| 22 |
+
|
| 23 |
+
依赖:numpy、opencv-python、onnxruntime、axengine(板端 pyaxengine)。
|
| 24 |
+
|
| 25 |
+
用法(在 hs/Depth-Anything-V2 目录下执行):
|
| 26 |
+
# 离线比较已有结果(开发机,无需 axengine 或模型文件)
|
| 27 |
+
python3 src/tool/compare_onnx_axmodel.py \
|
| 28 |
+
--onnx-results results/onnx_full --axmodel-results results/board_full \
|
| 29 |
+
--output results/compare_full/onnx_vs_axmodel.md
|
| 30 |
+
|
| 31 |
+
# 单图
|
| 32 |
+
python3 src/tool/compare_onnx_axmodel.py \
|
| 33 |
+
--onnx depth_anything_v2_vits.onnx \
|
| 34 |
+
--axmodel depth_anything_v2_vits_ax650.axmodel \
|
| 35 |
+
--image /path/a.jpg --output results/compare/single.md
|
| 36 |
+
|
| 37 |
+
# 数据集(递归收集图片),默认 axmodel 送 0-255
|
| 38 |
+
python3 src/tool/compare_onnx_axmodel.py \
|
| 39 |
+
--onnx depth_anything_v2_vits.onnx \
|
| 40 |
+
--axmodel depth_anything_v2_vits_ax650.axmodel \
|
| 41 |
+
--dataset /path/images --limit 50 \
|
| 42 |
+
--output results/compare/onnx_vs_axmodel.md
|
| 43 |
+
|
| 44 |
+
# axmodel 仍需要脚本侧做 ImageNet 归一化时
|
| 45 |
+
python3 src/tool/compare_onnx_axmodel.py \
|
| 46 |
+
--onnx depth_anything_v2_vits.onnx --axmodel other.axmodel \
|
| 47 |
+
--dataset /path/images --axmodel-norm imagenet \
|
| 48 |
+
--output results/compare/onnx_vs_axmodel.md
|
| 49 |
+
|
| 50 |
+
注意:
|
| 51 |
+
- axmodel 的输出名可能被编译器改写,脚本先按名字、再按 shape 回退绑定。
|
| 52 |
+
- 两个后端的输出 shape 必须一致;不一致时直接报错并打印实际接口。
|
| 53 |
+
- 结果默认输出到 results/compare/,不写预测 NPZ。
|
| 54 |
+
"""
|
| 55 |
+
|
| 56 |
+
import argparse
|
| 57 |
+
import json
|
| 58 |
+
import logging
|
| 59 |
+
import time
|
| 60 |
+
from datetime import datetime
|
| 61 |
+
from pathlib import Path
|
| 62 |
+
|
| 63 |
+
import cv2
|
| 64 |
+
import numpy as np
|
| 65 |
+
|
| 66 |
+
MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
|
| 67 |
+
STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
|
| 68 |
+
EXTENSIONS = {'.jpg', '.jpeg', '.png', '.bmp', '.tif', '.tiff'}
|
| 69 |
+
METRICS = ('cosine', 'pearson', 'mae', 'rmse', 'max_abs', 'p99_abs',
|
| 70 |
+
'aligned_mae', 'aligned_rmse', 'rel_error')
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def meta_type(meta):
|
| 74 |
+
return str(getattr(meta, 'type', getattr(meta, 'dtype', ''))).lower()
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def input_spec(shape, type_name, model_name):
|
| 78 |
+
shape = list(shape)
|
| 79 |
+
if len(shape) != 4 or not all(isinstance(v, int) and v > 0 for v in shape):
|
| 80 |
+
raise ValueError(f'{model_name} 输入必须是静态 4D NHWC/NCHW: {shape}')
|
| 81 |
+
if shape[0] != 1:
|
| 82 |
+
raise ValueError(f'{model_name} 仅支持 batch=1: {shape}')
|
| 83 |
+
if shape[1] == 3:
|
| 84 |
+
layout, size = 'NCHW', (shape[2], shape[3])
|
| 85 |
+
elif shape[3] == 3:
|
| 86 |
+
layout, size = 'NHWC', (shape[1], shape[2])
|
| 87 |
+
else:
|
| 88 |
+
raise ValueError(f'{model_name} 输入必须含 3 通道: {shape}')
|
| 89 |
+
if 'float' in type_name:
|
| 90 |
+
dtype = np.float32
|
| 91 |
+
elif 'uint8' in type_name or type_name.endswith('u8'):
|
| 92 |
+
dtype = np.uint8
|
| 93 |
+
elif 'int8' in type_name:
|
| 94 |
+
dtype = np.int8
|
| 95 |
+
else:
|
| 96 |
+
raise ValueError(f'{model_name} 不支持的输入类型: {type_name!r}')
|
| 97 |
+
return size, layout, dtype
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
def onnx_interface(session):
|
| 101 |
+
inputs, outputs = session.get_inputs(), session.get_outputs()
|
| 102 |
+
logging.info('ONNX 输入: %s', [(m.name, m.shape, m.type) for m in inputs])
|
| 103 |
+
logging.info('ONNX 输出: %s', [(m.name, m.shape, m.type) for m in outputs])
|
| 104 |
+
if len(inputs) != 1:
|
| 105 |
+
raise ValueError(f'ONNX 应有 1 个输入: {[(m.name, m.shape) for m in inputs]}')
|
| 106 |
+
meta = inputs[0]
|
| 107 |
+
if 'float' not in str(meta.type):
|
| 108 |
+
raise ValueError(f'ONNX 输入要求 float32: {meta.name} {meta.type}')
|
| 109 |
+
size, layout, dtype = input_spec(meta.shape, str(meta.type), 'ONNX')
|
| 110 |
+
return meta.name, size, layout, dtype, outputs[0].name
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def axmodel_interface(session):
|
| 114 |
+
inputs, outputs = session.get_inputs(), session.get_outputs()
|
| 115 |
+
logging.info('AXMODEL 输入: %s', [(m.name, m.shape, meta_type(m)) for m in inputs])
|
| 116 |
+
logging.info('AXMODEL 输出: %s', [(m.name, m.shape, meta_type(m)) for m in outputs])
|
| 117 |
+
if len(inputs) != 1:
|
| 118 |
+
raise ValueError(f'axmodel 应有 1 个输入: {[(m.name, m.shape) for m in inputs]}')
|
| 119 |
+
meta = inputs[0]
|
| 120 |
+
size, layout, dtype = input_spec(meta.shape, meta_type(meta), 'AXMODEL')
|
| 121 |
+
return meta, size, layout, dtype, outputs
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
def preprocess(image, size, layout, dtype, resize_mode, color_order, norm):
|
| 125 |
+
target_h, target_w = size
|
| 126 |
+
original_h, original_w = image.shape[:2]
|
| 127 |
+
if color_order == 'rgb':
|
| 128 |
+
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
|
| 129 |
+
if resize_mode == 'pad':
|
| 130 |
+
scale = min(target_h / original_h, target_w / original_w)
|
| 131 |
+
new_h = max(1, min(target_h, int(round(original_h * scale))))
|
| 132 |
+
new_w = max(1, min(target_w, int(round(original_w * scale))))
|
| 133 |
+
resized = cv2.resize(image, (new_w, new_h), interpolation=cv2.INTER_LINEAR)
|
| 134 |
+
canvas = np.zeros((target_h, target_w, 3), dtype=np.float32)
|
| 135 |
+
canvas[:new_h, :new_w] = resized
|
| 136 |
+
image_size = (new_h, new_w)
|
| 137 |
+
else:
|
| 138 |
+
canvas = cv2.resize(image, (target_w, target_h),
|
| 139 |
+
interpolation=cv2.INTER_LINEAR).astype(np.float32)
|
| 140 |
+
image_size = (target_h, target_w)
|
| 141 |
+
canvas = canvas.astype(np.float32, copy=False)
|
| 142 |
+
if norm == 'imagenet':
|
| 143 |
+
if dtype != np.float32:
|
| 144 |
+
raise ValueError('整数输入不能使用 ImageNet 归一化')
|
| 145 |
+
canvas = (canvas / 255.0 - MEAN) / STD
|
| 146 |
+
if dtype != np.float32:
|
| 147 |
+
canvas = np.rint(np.clip(canvas, 0, 255)).astype(dtype)
|
| 148 |
+
tensor = canvas.transpose(2, 0, 1) if layout == 'NCHW' else canvas
|
| 149 |
+
return np.ascontiguousarray(tensor[None]), image_size
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
def to_2d(array, model_name):
|
| 153 |
+
array = np.asarray(array)
|
| 154 |
+
if array.ndim == 4:
|
| 155 |
+
if array.shape[0] != 1:
|
| 156 |
+
raise ValueError(f'{model_name} 输出 batch 必须为 1: {array.shape}')
|
| 157 |
+
if array.shape[1] == 1:
|
| 158 |
+
array = array[0, 0]
|
| 159 |
+
elif array.shape[3] == 1:
|
| 160 |
+
array = array[0, :, :, 0]
|
| 161 |
+
else:
|
| 162 |
+
raise ValueError(f'{model_name} 输出无法压缩为 2D: {array.shape}')
|
| 163 |
+
elif array.ndim == 3:
|
| 164 |
+
if array.shape[0] == 1:
|
| 165 |
+
array = array[0]
|
| 166 |
+
elif array.shape[2] == 1:
|
| 167 |
+
array = array[:, :, 0]
|
| 168 |
+
else:
|
| 169 |
+
raise ValueError(f'{model_name} 输出无法压缩为 2D: {array.shape}')
|
| 170 |
+
if array.ndim != 2:
|
| 171 |
+
raise ValueError(f'{model_name} 输出必须是 2D 深度图: {array.shape}')
|
| 172 |
+
array = array.astype(np.float32, copy=False)
|
| 173 |
+
if not np.isfinite(array).all():
|
| 174 |
+
raise ValueError(f'{model_name} 输出包含 nan/inf')
|
| 175 |
+
return array
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
def select_ax_output(outputs, metas, expected_shape):
|
| 179 |
+
if not isinstance(outputs, (list, tuple)):
|
| 180 |
+
outputs = [outputs]
|
| 181 |
+
if len(outputs) != len(metas):
|
| 182 |
+
raise ValueError(f'axmodel 输出数量不一致: {len(outputs)} vs {len(metas)}')
|
| 183 |
+
for index, meta in enumerate(metas):
|
| 184 |
+
if any(token in str(meta.name).lower() for token in ('depth', 'output')):
|
| 185 |
+
array = to_2d(outputs[index], 'AXMODEL')
|
| 186 |
+
if array.shape == expected_shape:
|
| 187 |
+
return array, str(meta.name)
|
| 188 |
+
candidates = []
|
| 189 |
+
for index, array in enumerate(outputs):
|
| 190 |
+
candidate = to_2d(array, 'AXMODEL')
|
| 191 |
+
if candidate.shape == expected_shape:
|
| 192 |
+
candidates.append((index, candidate))
|
| 193 |
+
if len(candidates) != 1:
|
| 194 |
+
raise ValueError('axmodel 输出无法按 name 或 shape 唯一匹配'
|
| 195 |
+
f' {expected_shape}: {[(m.name, np.asarray(o).shape) for m, o in zip(metas, outputs)]}')
|
| 196 |
+
index, candidate = candidates[0]
|
| 197 |
+
return candidate, str(metas[index].name)
|
| 198 |
+
|
| 199 |
+
|
| 200 |
+
def compare_arrays(reference, actual):
|
| 201 |
+
ref = reference.astype(np.float64).ravel()
|
| 202 |
+
act = actual.astype(np.float64).ravel()
|
| 203 |
+
diff = act - ref
|
| 204 |
+
ref_norm = np.linalg.norm(ref)
|
| 205 |
+
act_norm = np.linalg.norm(act)
|
| 206 |
+
cosine = float(np.dot(ref, act) / (ref_norm * act_norm + 1e-12))
|
| 207 |
+
if ref.std() > 1e-12 and act.std() > 1e-12:
|
| 208 |
+
pearson = float(np.corrcoef(ref, act)[0, 1])
|
| 209 |
+
else:
|
| 210 |
+
pearson = float('nan')
|
| 211 |
+
slope, offset = np.polyfit(ref, act, 1)
|
| 212 |
+
aligned = act - (slope * ref + offset)
|
| 213 |
+
abs_diff = np.abs(diff)
|
| 214 |
+
return {
|
| 215 |
+
'cosine': cosine,
|
| 216 |
+
'pearson': pearson,
|
| 217 |
+
'mae': float(abs_diff.mean()),
|
| 218 |
+
'rmse': float(np.sqrt(np.mean(diff ** 2))),
|
| 219 |
+
'max_abs': float(abs_diff.max()),
|
| 220 |
+
'p99_abs': float(np.percentile(abs_diff, 99)),
|
| 221 |
+
'aligned_mae': float(np.abs(aligned).mean()),
|
| 222 |
+
'aligned_rmse': float(np.sqrt(np.mean(aligned ** 2))),
|
| 223 |
+
'rel_error': float(abs_diff.mean() / (np.abs(ref).mean() + 1e-12)),
|
| 224 |
+
}
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
def summarize(records):
|
| 228 |
+
summary = {}
|
| 229 |
+
for key in METRICS:
|
| 230 |
+
values = np.array([record[key] for record in records], dtype=np.float64)
|
| 231 |
+
values = values[np.isfinite(values)]
|
| 232 |
+
if values.size == 0:
|
| 233 |
+
summary[key] = None
|
| 234 |
+
continue
|
| 235 |
+
summary[key] = {'mean': float(values.mean()),
|
| 236 |
+
'median': float(np.percentile(values, 50)),
|
| 237 |
+
'p90': float(np.percentile(values, 90)),
|
| 238 |
+
'p99': float(np.percentile(values, 99)),
|
| 239 |
+
'min': float(values.min()),
|
| 240 |
+
'max': float(values.max())}
|
| 241 |
+
return summary
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
def format_value(value):
|
| 245 |
+
if value is None:
|
| 246 |
+
return '-'
|
| 247 |
+
if isinstance(value, float) and not np.isfinite(value):
|
| 248 |
+
return '-'
|
| 249 |
+
return f'{value:.6g}'
|
| 250 |
+
|
| 251 |
+
|
| 252 |
+
def write_markdown(path, report):
|
| 253 |
+
lines = ['# Depth Anything V2:ONNX 与 AXMODEL 输出一致性', '']
|
| 254 |
+
lines.append(f'- 生成时间:{report["created"]}')
|
| 255 |
+
lines.append(f'- 样本数:{report["sample_count"]}')
|
| 256 |
+
lines.append(f'- 结果目录:`{report["output_dir"]}`')
|
| 257 |
+
if report['limit'] is not None:
|
| 258 |
+
lines.append(f'- 样本上限:{report["limit"]}')
|
| 259 |
+
lines.append('')
|
| 260 |
+
if report.get('mode') == 'results':
|
| 261 |
+
lines.extend([
|
| 262 |
+
'## 结果来源', '',
|
| 263 |
+
f'- ONNX:`{report["onnx_results"]}`',
|
| 264 |
+
f'- AXMODEL:`{report["axmodel_results"]}`',
|
| 265 |
+
'- 按相对路径匹配同名 `.depth.npy`,比较保存的深度图,不重新推理或 resize。',
|
| 266 |
+
'- 这些文件由推理脚本还原至原图尺寸,指标对应保存结果的像素空间。',
|
| 267 |
+
'- 离线比较不测量推理耗时;预处理配置以原推理记录为准。', '',
|
| 268 |
+
])
|
| 269 |
+
else:
|
| 270 |
+
lines.append('## 模型与预处理')
|
| 271 |
+
lines.append('')
|
| 272 |
+
lines.append('| 项 | ONNX | AXMODEL |')
|
| 273 |
+
lines.append('| --- | --- | --- |')
|
| 274 |
+
lines.append(f'| 模型 | `{report["onnx"]["path"]}` | `{report["axmodel"]["path"]}` |')
|
| 275 |
+
lines.append(f'| 运行时 | {report["onnx"]["runtime"]} | {report["axmodel"]["runtime"]} |')
|
| 276 |
+
lines.append(f'| 输入 | {report["onnx"]["input"]} | {report["axmodel"]["input"]} |')
|
| 277 |
+
lines.append(f'| 输出 | {report["onnx"]["output"]} | {report["axmodel"]["output"]} |')
|
| 278 |
+
lines.append(f'| 归一化 | {report["onnx"]["norm"]} | {report["axmodel"]["norm"]} |')
|
| 279 |
+
lines.append(f'| resize | {report["resize_mode"]} | {report["resize_mode"]} |')
|
| 280 |
+
lines.append(f'| 颜色顺序 | {report["color_order"]} | {report["color_order"]} |')
|
| 281 |
+
lines.append(f'| warmup | {report["warmup"]} | {report["warmup"]} |')
|
| 282 |
+
lines.append('')
|
| 283 |
+
lines.append('## 汇总指标')
|
| 284 |
+
lines.append('')
|
| 285 |
+
lines.append('| 指标 | mean | median | p90 | p99 | min | max |')
|
| 286 |
+
lines.append('| --- | --- | --- | --- | --- | --- | --- |')
|
| 287 |
+
for key in METRICS:
|
| 288 |
+
stats = report['summary'][key]
|
| 289 |
+
if stats is None:
|
| 290 |
+
lines.append(f'| {key} | - | - | - | - | - | - |')
|
| 291 |
+
else:
|
| 292 |
+
lines.append(f'| {key} | {format_value(stats["mean"])} | '
|
| 293 |
+
f'{format_value(stats["median"])} | {format_value(stats["p90"])} | '
|
| 294 |
+
f'{format_value(stats["p99"])} | {format_value(stats["min"])} | '
|
| 295 |
+
f'{format_value(stats["max"])} |')
|
| 296 |
+
lines.append('')
|
| 297 |
+
lines.append('## 逐样本结果')
|
| 298 |
+
lines.append('')
|
| 299 |
+
lines.append('| # | 图片 | cosine | pearson | mae | rmse | max_abs | p99_abs | '
|
| 300 |
+
'aligned_mae | aligned_rmse | rel_error | ONNX ms | AXMODEL ms |')
|
| 301 |
+
lines.append('| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |')
|
| 302 |
+
for index, sample in enumerate(report['samples'], 1):
|
| 303 |
+
metrics = sample['metrics']
|
| 304 |
+
lines.append(
|
| 305 |
+
f'| {index} | `{sample["name"]}` | {format_value(metrics["cosine"])} | '
|
| 306 |
+
f'{format_value(metrics["pearson"])} | {format_value(metrics["mae"])} | '
|
| 307 |
+
f'{format_value(metrics["rmse"])} | {format_value(metrics["max_abs"])} | '
|
| 308 |
+
f'{format_value(metrics["p99_abs"])} | {format_value(metrics["aligned_mae"])} | '
|
| 309 |
+
f'{format_value(metrics["aligned_rmse"])} | {format_value(metrics["rel_error"])} | '
|
| 310 |
+
f'{format_value(sample["onnx_ms"])} | {format_value(sample["axmodel_ms"])} |')
|
| 311 |
+
lines.append('')
|
| 312 |
+
lines.append('## 指标说明')
|
| 313 |
+
lines.append('')
|
| 314 |
+
lines.append('- `cosine`:两路输出的余弦相似度,1 表示方向完全一致。')
|
| 315 |
+
lines.append('- `pearson`:线性相关系数,衡量结构一致性,对全局尺度不敏感。')
|
| 316 |
+
lines.append('- `mae` / `rmse`:未对齐的绝对误差,包含量化误差与整体尺度漂移。')
|
| 317 |
+
lines.append('- `max_abs` / `p99_abs`:最大误差和 99 分位误差,反映局部离群点。')
|
| 318 |
+
lines.append('- `aligned_mae` / `aligned_rmse`:先做最小二乘 scale-shift 对齐再计算,'
|
| 319 |
+
'用于区分全局尺度漂移和真正的结构偏差。')
|
| 320 |
+
lines.append('- `rel_error`:`mae / mean(|ONNX|)`,便于跨图片比较。')
|
| 321 |
+
lines.append('')
|
| 322 |
+
lines.append('## 判定参考')
|
| 323 |
+
lines.append('')
|
| 324 |
+
lines.append(f'- 余弦相似度均值:{format_value(report["summary"]["cosine"]["mean"])}')
|
| 325 |
+
lines.append(f'- 对齐后 MAE 均值:{format_value(report["summary"]["aligned_mae"]["mean"])}')
|
| 326 |
+
lines.append(f'- 未���齐相对误差均值:'
|
| 327 |
+
f'{format_value(report["summary"]["rel_error"]["mean"])}')
|
| 328 |
+
lines.append('')
|
| 329 |
+
lines.append('若 `cosine` 接近 1 但 `mae` 明显偏大、`aligned_mae` 很小,说明两路输出只存在'
|
| 330 |
+
'整体尺度/偏移差异;若 `aligned_mae` 也明显偏大,则更可能是量化本身造成的'
|
| 331 |
+
'结构失真,需要结合逐层精度分析继续定位。')
|
| 332 |
+
path.parent.mkdir(parents=True, exist_ok=True)
|
| 333 |
+
path.write_text('\n'.join(lines) + '\n', encoding='utf-8')
|
| 334 |
+
|
| 335 |
+
|
| 336 |
+
def collect_images(dataset):
|
| 337 |
+
paths = sorted(p for p in dataset.rglob('*')
|
| 338 |
+
if p.is_file() and p.suffix.lower() in EXTENSIONS)
|
| 339 |
+
if not paths:
|
| 340 |
+
raise ValueError(f'目录中没有图片: {dataset}')
|
| 341 |
+
return paths
|
| 342 |
+
|
| 343 |
+
|
| 344 |
+
def compare_result_dirs(args):
|
| 345 |
+
from tqdm import tqdm
|
| 346 |
+
|
| 347 |
+
files = []
|
| 348 |
+
for directory in (args.onnx_results, args.axmodel_results):
|
| 349 |
+
if not directory.is_dir():
|
| 350 |
+
raise ValueError(f'结果目录不存在: {directory}')
|
| 351 |
+
mapping = {p.relative_to(directory).as_posix(): p
|
| 352 |
+
for p in directory.rglob('*.depth.npy') if p.is_file()}
|
| 353 |
+
if not mapping:
|
| 354 |
+
raise ValueError(f'结果目录中没有 *.depth.npy: {directory}')
|
| 355 |
+
files.append(mapping)
|
| 356 |
+
onnx_files, axmodel_files = files
|
| 357 |
+
if onnx_files.keys() != axmodel_files.keys():
|
| 358 |
+
onnx_only = sorted(onnx_files.keys() - axmodel_files.keys())
|
| 359 |
+
axmodel_only = sorted(axmodel_files.keys() - onnx_files.keys())
|
| 360 |
+
raise ValueError(f'结果文件未完全配对: ONNX 独有 {len(onnx_only)} 个 '
|
| 361 |
+
f'{onnx_only[:5]}, AXMODEL 独有 {len(axmodel_only)} 个 '
|
| 362 |
+
f'{axmodel_only[:5]}')
|
| 363 |
+
names = sorted(onnx_files)
|
| 364 |
+
if args.limit is not None:
|
| 365 |
+
names = names[:args.limit]
|
| 366 |
+
records = []
|
| 367 |
+
for name in tqdm(names, desc='比较深度输出', disable=len(names) == 1):
|
| 368 |
+
reference = to_2d(np.load(onnx_files[name], allow_pickle=False), f'ONNX {name}')
|
| 369 |
+
actual = to_2d(np.load(axmodel_files[name], allow_pickle=False), f'AXMODEL {name}')
|
| 370 |
+
if reference.size == 0 or reference.shape != actual.shape:
|
| 371 |
+
raise ValueError(f'{name} 深度图为空或 shape 不一致: '
|
| 372 |
+
f'ONNX {reference.shape} vs AXMODEL {actual.shape}')
|
| 373 |
+
metrics = compare_arrays(reference, actual)
|
| 374 |
+
if args.debug:
|
| 375 |
+
logging.info('%s: %s', name, metrics)
|
| 376 |
+
records.append({'name': name[:-len('.depth.npy')], 'metrics': metrics,
|
| 377 |
+
'onnx_ms': None, 'axmodel_ms': None})
|
| 378 |
+
report = {
|
| 379 |
+
'mode': 'results',
|
| 380 |
+
'created': datetime.now().astimezone().isoformat(timespec='seconds'),
|
| 381 |
+
'output_dir': str(args.output.resolve().parent),
|
| 382 |
+
'onnx_results': str(args.onnx_results.resolve()),
|
| 383 |
+
'axmodel_results': str(args.axmodel_results.resolve()),
|
| 384 |
+
'sample_count': len(records), 'limit': args.limit,
|
| 385 |
+
'summary': summarize([record['metrics'] for record in records]),
|
| 386 |
+
'samples': records,
|
| 387 |
+
}
|
| 388 |
+
write_markdown(args.output, report)
|
| 389 |
+
logging.info('样本数 %d,cosine mean=%.6f,aligned_mae mean=%.6g,报告: %s',
|
| 390 |
+
len(records), report['summary']['cosine']['mean'],
|
| 391 |
+
report['summary']['aligned_mae']['mean'], args.output)
|
| 392 |
+
|
| 393 |
+
|
| 394 |
+
def main():
|
| 395 |
+
parser = argparse.ArgumentParser(
|
| 396 |
+
description='计算 ONNX 与 AXMODEL 深度输出的相似度并生成 Markdown 报告',
|
| 397 |
+
formatter_class=argparse.RawDescriptionHelpFormatter, epilog=__doc__)
|
| 398 |
+
parser.add_argument('--onnx', type=Path, help='FP32 ONNX 模型')
|
| 399 |
+
parser.add_argument('--axmodel', type=Path, help='编译后的 axmodel')
|
| 400 |
+
parser.add_argument('--onnx-results', type=Path, help='ONNX 结果目录(递归读取 *.depth.npy)')
|
| 401 |
+
parser.add_argument('--axmodel-results', type=Path, help='AXMODEL 结果目录(相对路径必须完全配对)')
|
| 402 |
+
source = parser.add_mutually_exclusive_group()
|
| 403 |
+
source.add_argument('--image', type=Path, help='单张图片')
|
| 404 |
+
source.add_argument('--dataset', type=Path, help='图片目录(递归)')
|
| 405 |
+
parser.add_argument('--output', type=Path,
|
| 406 |
+
default=Path(__file__).resolve().parents[2] / 'results' /
|
| 407 |
+
'compare' / 'onnx_vs_axmodel.md',
|
| 408 |
+
help='Markdown 输出路径,默认 results/compare/onnx_vs_axmodel.md')
|
| 409 |
+
parser.add_argument('--resize-mode', choices=('stretch', 'pad'), default='stretch',
|
| 410 |
+
help='两个后端共用的 resize 方式,默认 stretch')
|
| 411 |
+
parser.add_argument('--color-order', choices=('rgb', 'bgr'), default='rgb',
|
| 412 |
+
help='两个后端共用的颜色顺序,默认 rgb')
|
| 413 |
+
parser.add_argument('--onnx-norm', choices=('imagenet', 'raw'), default='imagenet',
|
| 414 |
+
help='ONNX 输入归一化,默认 imagenet')
|
| 415 |
+
parser.add_argument('--axmodel-norm', choices=('imagenet', 'raw'), default='raw',
|
| 416 |
+
help='axmodel 输入归一化,默认 raw(编译时已吸收归一化)')
|
| 417 |
+
parser.add_argument('--warmup', type=int, default=3,
|
| 418 |
+
help='数据集模式每个后端的 warmup 次数,默认 3')
|
| 419 |
+
parser.add_argument('--limit', type=int, help='数据集模式只处理前 N 张')
|
| 420 |
+
parser.add_argument('--debug', action='store_true', help='打印逐样本输入/输出统计')
|
| 421 |
+
args = parser.parse_args()
|
| 422 |
+
logging.basicConfig(level=logging.INFO, format='%(message)s')
|
| 423 |
+
if args.warmup < 0:
|
| 424 |
+
parser.error('--warmup 不能为负数')
|
| 425 |
+
if args.limit is not None and args.limit < 1:
|
| 426 |
+
parser.error('--limit 必须为正数')
|
| 427 |
+
if args.onnx_results is not None or args.axmodel_results is not None:
|
| 428 |
+
if args.onnx_results is None or args.axmodel_results is None:
|
| 429 |
+
parser.error('--onnx-results 和 --axmodel-results 必须同时提供')
|
| 430 |
+
if any(value is not None for value in (args.onnx, args.axmodel, args.image, args.dataset)):
|
| 431 |
+
parser.error('结果目录模式不能同时指定模型或图片输入')
|
| 432 |
+
compare_result_dirs(args)
|
| 433 |
+
return
|
| 434 |
+
if args.onnx is None or args.axmodel is None or (args.image is None and args.dataset is None):
|
| 435 |
+
parser.error('推理模式必须提供 --onnx、--axmodel 和 --image/--dataset;'
|
| 436 |
+
'或使用 --onnx-results 和 --axmodel-results 进行离线比较')
|
| 437 |
+
if not args.onnx.is_file():
|
| 438 |
+
parser.error(f'--onnx 不存在: {args.onnx}')
|
| 439 |
+
if not args.axmodel.is_file():
|
| 440 |
+
parser.error(f'--axmodel 不存在: {args.axmodel}')
|
| 441 |
+
|
| 442 |
+
import onnxruntime as ort
|
| 443 |
+
import axengine
|
| 444 |
+
onnx_session = ort.InferenceSession(str(args.onnx))
|
| 445 |
+
axmodel_session = axengine.InferenceSession(str(args.axmodel))
|
| 446 |
+
onnx_input, onnx_size, onnx_layout, onnx_dtype, onnx_output = onnx_interface(onnx_session)
|
| 447 |
+
ax_meta, ax_size, ax_layout, ax_dtype, ax_outputs = axmodel_interface(axmodel_session)
|
| 448 |
+
if onnx_size != ax_size:
|
| 449 |
+
raise ValueError(f'两个后端输入尺寸不一致: ONNX {onnx_size} vs AXMODEL {ax_size}')
|
| 450 |
+
|
| 451 |
+
if args.image:
|
| 452 |
+
if not args.image.is_file():
|
| 453 |
+
parser.error(f'--image 不是文件: {args.image}')
|
| 454 |
+
images = [args.image]
|
| 455 |
+
root = args.image.parent
|
| 456 |
+
else:
|
| 457 |
+
if not args.dataset.is_dir():
|
| 458 |
+
parser.error(f'--dataset 不是目录: {args.dataset}')
|
| 459 |
+
images = collect_images(args.dataset)
|
| 460 |
+
if args.limit is not None:
|
| 461 |
+
images = images[:args.limit]
|
| 462 |
+
root = args.dataset
|
| 463 |
+
|
| 464 |
+
if args.axmodel_norm == 'imagenet' and ax_dtype != np.float32:
|
| 465 |
+
parser.error('axmodel 输入为整数类型,不能使用 --axmodel-norm imagenet')
|
| 466 |
+
|
| 467 |
+
# 数据集模式先 warmup,使用第一张图。
|
| 468 |
+
if len(images) > 1 and args.warmup:
|
| 469 |
+
first = cv2.imread(str(images[0]), cv2.IMREAD_COLOR)
|
| 470 |
+
if first is None:
|
| 471 |
+
raise ValueError(f'读取图片失败: {images[0]}')
|
| 472 |
+
onnx_tensor, _ = preprocess(first, onnx_size, onnx_layout, onnx_dtype,
|
| 473 |
+
args.resize_mode, args.color_order, args.onnx_norm)
|
| 474 |
+
ax_tensor, _ = preprocess(first, ax_size, ax_layout, ax_dtype,
|
| 475 |
+
args.resize_mode, args.color_order, args.axmodel_norm)
|
| 476 |
+
for _ in range(args.warmup):
|
| 477 |
+
onnx_session.run([onnx_output], {onnx_input: onnx_tensor})
|
| 478 |
+
axmodel_session.run(None, {ax_meta.name: ax_tensor})
|
| 479 |
+
logging.info('两个后端各 warmup %d 次', args.warmup)
|
| 480 |
+
|
| 481 |
+
records = []
|
| 482 |
+
for path in images:
|
| 483 |
+
name = str(path.relative_to(root)) if path != root else path.name
|
| 484 |
+
image = cv2.imread(str(path), cv2.IMREAD_COLOR)
|
| 485 |
+
if image is None:
|
| 486 |
+
raise ValueError(f'读取图片失败: {path}')
|
| 487 |
+
onnx_tensor, _ = preprocess(image, onnx_size, onnx_layout, onnx_dtype,
|
| 488 |
+
args.resize_mode, args.color_order, args.onnx_norm)
|
| 489 |
+
ax_tensor, _ = preprocess(image, ax_size, ax_layout, ax_dtype,
|
| 490 |
+
args.resize_mode, args.color_order, args.axmodel_norm)
|
| 491 |
+
|
| 492 |
+
start = time.perf_counter()
|
| 493 |
+
onnx_raw = onnx_session.run([onnx_output], {onnx_input: onnx_tensor})[0]
|
| 494 |
+
onnx_ms = (time.perf_counter() - start) * 1000
|
| 495 |
+
start = time.perf_counter()
|
| 496 |
+
ax_outputs_data = axmodel_session.run(None, {ax_meta.name: ax_tensor})
|
| 497 |
+
axmodel_ms = (time.perf_counter() - start) * 1000
|
| 498 |
+
|
| 499 |
+
reference = to_2d(onnx_raw, 'ONNX')
|
| 500 |
+
actual, ax_output_name = select_ax_output(ax_outputs_data, ax_outputs,
|
| 501 |
+
reference.shape)
|
| 502 |
+
metrics = compare_arrays(reference, actual)
|
| 503 |
+
if not np.isfinite(metrics['cosine']):
|
| 504 |
+
raise ValueError(f'{name} 无法计算相似度,请检查输出是否常数')
|
| 505 |
+
if args.debug:
|
| 506 |
+
logging.info('%s: ONNX min=%.4f max=%.4f mean=%.4f | AXMODEL min=%.4f '
|
| 507 |
+
'max=%.4f mean=%.4f | cosine=%.6f mae=%.6g',
|
| 508 |
+
name, float(reference.min()), float(reference.max()),
|
| 509 |
+
float(reference.mean()), float(actual.min()),
|
| 510 |
+
float(actual.max()), float(actual.mean()),
|
| 511 |
+
metrics['cosine'], metrics['mae'])
|
| 512 |
+
records.append({'name': name, 'metrics': metrics,
|
| 513 |
+
'onnx_ms': onnx_ms, 'axmodel_ms': axmodel_ms})
|
| 514 |
+
if len(images) == 1:
|
| 515 |
+
logging.info('输出绑定: ONNX=%s AXMODEL=%s', onnx_output, ax_output_name)
|
| 516 |
+
logging.info('指标: %s', json.dumps(metrics, ensure_ascii=False))
|
| 517 |
+
|
| 518 |
+
report = {
|
| 519 |
+
'created': datetime.now().astimezone().isoformat(timespec='seconds'),
|
| 520 |
+
'output_dir': str(args.output.resolve().parent),
|
| 521 |
+
'sample_count': len(records),
|
| 522 |
+
'limit': args.limit if not args.image else None,
|
| 523 |
+
'warmup': args.warmup if len(images) > 1 else 0,
|
| 524 |
+
'resize_mode': args.resize_mode,
|
| 525 |
+
'color_order': args.color_order,
|
| 526 |
+
'onnx': {'path': str(args.onnx.resolve()),
|
| 527 |
+
'runtime': 'onnxruntime ' + ort.__version__,
|
| 528 |
+
'input': f'{onnx_input} {onnx_tensor.shape}',
|
| 529 |
+
'output': onnx_output,
|
| 530 |
+
'norm': args.onnx_norm},
|
| 531 |
+
'axmodel': {'path': str(args.axmodel.resolve()),
|
| 532 |
+
'runtime': getattr(axengine, '__version__', 'unknown'),
|
| 533 |
+
'input': f'{ax_meta.name} {tuple(ax_meta.shape)} {np.dtype(ax_dtype).name}',
|
| 534 |
+
'output': ax_output_name,
|
| 535 |
+
'norm': args.axmodel_norm},
|
| 536 |
+
'summary': summarize([record['metrics'] for record in records]),
|
| 537 |
+
'samples': records,
|
| 538 |
+
}
|
| 539 |
+
write_markdown(args.output, report)
|
| 540 |
+
logging.info('样本数 %d,cosine mean=%.6f,aligned_mae mean=%.6g,报告: %s',
|
| 541 |
+
len(records), report['summary']['cosine']['mean'],
|
| 542 |
+
report['summary']['aligned_mae']['mean'], args.output)
|
| 543 |
+
|
| 544 |
+
|
| 545 |
+
if __name__ == '__main__':
|
| 546 |
+
main()
|
src/tool/depthart_export_axonnx.py
ADDED
|
@@ -0,0 +1,555 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# -*- coding: utf-8 -*-
|
| 3 |
+
"""
|
| 4 |
+
把 DepthART ONNX 中的 com.depthart::SelectiveScan 改写为 pulsar2 可编译的纯 ONNX 子图。
|
| 5 |
+
|
| 6 |
+
背景:
|
| 7 |
+
官方 relative_* ONNX 含自定义算子 SelectiveScan,pulsar2 前端直接报
|
| 8 |
+
"dont support SelectiveScan opr"。官方只提供 TensorRT 插件,没有 NPU 实现。
|
| 9 |
+
推理脚本里的 Loop 改写可以跑通 onnxruntime,但 AX 工具链不支持 Loop 控制流。
|
| 10 |
+
|
| 11 |
+
做法:
|
| 12 |
+
选择性扫描是线性递推 h_t = exp(dt_t*A) * h_{t-1} + dt_t*B_t*u_t,可写成
|
| 13 |
+
h_t = exp(L_t) * h_{-1} + sum_{s<=t} exp(L_t - L_s) * dt_s*B_s*u_s
|
| 14 |
+
其中 L_t = cumsum_{r<=t}(dt_r*A)。把时间轴切成固定长度 T(默认 14),每个
|
| 15 |
+
分块内用 CumSum + 下三角 mask + MatMul 一次算出,块间只传一个状态。所有
|
| 16 |
+
指数项的指数都 <= 0,不会溢出。所有算子(CumSum/Exp/Where/MatMul 等)都在
|
| 17 |
+
AX 前端支持列表内,不含 Loop/Scan/自定义算子。
|
| 18 |
+
|
| 19 |
+
校验:
|
| 20 |
+
- 本工具用 Identity 替换 scan 跑一遍,拿到每个 scan 的 D/G/N/L,再按实际
|
| 21 |
+
尺寸生成静态图;时间轴不是 T 整数倍时自动补零并在输出处裁掉。
|
| 22 |
+
- 改写后的整模型输出与已验证的 Loop 版本最大绝对误差约 6e-6(FP32)。
|
| 23 |
+
- 同时把 4D `Einsum` 换成按 k 的批 MatMul(`b k d l, k c d -> b k c l`
|
| 24 |
+
和 `b k r l, k d r -> b k d l`):pulsar2 的量化执行器不支持 4D Einsum,
|
| 25 |
+
会报 Op Execution Error。
|
| 26 |
+
|
| 27 |
+
用法(在 hs/DepthART 目录下执行):
|
| 28 |
+
# 默认:448 模型,块大小 14,输出 onnx/relative_s_448_default_axonnx.onnx
|
| 29 |
+
../../venv/bin/python src/tool/depthart_export_axonnx.py
|
| 30 |
+
|
| 31 |
+
# 指定模型/输出/块大小
|
| 32 |
+
../../venv/bin/python src/tool/depthart_export_axonnx.py \\
|
| 33 |
+
--model onnx/relative_b_448_default.onnx \\
|
| 34 |
+
--output onnx/relative_b_448_default_axonnx.onnx --chunk-size 14
|
| 35 |
+
|
| 36 |
+
# 改写后用推理脚本验证数值
|
| 37 |
+
../../venv/bin/python src/infer/depthart_infer_onnx.py \\
|
| 38 |
+
--model onnx/relative_s_448_default_axonnx.onnx --image /path/a.jpg
|
| 39 |
+
|
| 40 |
+
注意:
|
| 41 |
+
- 块内两两指数矩阵形状为 [D*N, T, T],T 越大显存/内存占用越大;
|
| 42 |
+
T=14 时 672 通道层约 1M 元素(4 MB),推荐保持默认。
|
| 43 |
+
- 输出是标准 ONNX,不再含自定义算子;原模型文件不会被修改。
|
| 44 |
+
"""
|
| 45 |
+
|
| 46 |
+
import argparse
|
| 47 |
+
import logging
|
| 48 |
+
from pathlib import Path
|
| 49 |
+
|
| 50 |
+
import numpy as np
|
| 51 |
+
import onnx
|
| 52 |
+
import onnxruntime as ort
|
| 53 |
+
from onnx import helper, numpy_helper, TensorProto
|
| 54 |
+
|
| 55 |
+
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
| 56 |
+
DEFAULT_MODEL = PROJECT_ROOT / 'onnx' / 'relative_s_448_default.onnx'
|
| 57 |
+
DOMAIN = 'com.depthart'
|
| 58 |
+
OP = 'SelectiveScan'
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
EINSUM_EQUATIONS = {
|
| 62 |
+
'b k d l, k c d -> b k c l',
|
| 63 |
+
'b k r l, k d r -> b k d l',
|
| 64 |
+
}
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def collect_runtime_shapes(model):
|
| 68 |
+
"""用 ORT 实际跑一遍,读取 scan 与 Einsum 的静态形状。
|
| 69 |
+
|
| 70 |
+
scan 会被替换成 Identity(输出形状等于 u,不影响下游形状);Einsum 正常执行。
|
| 71 |
+
返回 (scan_shapes, einsum_shapes):
|
| 72 |
+
scan_shapes : 按图顺序的 [(D, G, N, L), ...]
|
| 73 |
+
einsum_shapes: {节点名: (x_shape, y_shape)}
|
| 74 |
+
"""
|
| 75 |
+
nodes = []
|
| 76 |
+
scan_outputs, b_aliases = [], {}
|
| 77 |
+
einsum_x, einsum_y = {}, {}
|
| 78 |
+
for index, node in enumerate(model.graph.node):
|
| 79 |
+
if node.domain == DOMAIN and node.op_type == OP:
|
| 80 |
+
nodes.append(helper.make_node('Identity', [node.input[0]], [node.output[0]],
|
| 81 |
+
name=f'depthart_stub_scan_{index}'))
|
| 82 |
+
alias = f'depthart_stub_b_{index}'
|
| 83 |
+
nodes.append(helper.make_node('Identity', [node.input[3]], [alias],
|
| 84 |
+
name=f'depthart_stub_bnode_{index}'))
|
| 85 |
+
scan_outputs.append(node.output[0])
|
| 86 |
+
b_aliases[node.output[0]] = alias
|
| 87 |
+
else:
|
| 88 |
+
if node.op_type == 'Einsum':
|
| 89 |
+
x_alias = f'depthart_stub_einsum_x_{index}'
|
| 90 |
+
y_alias = f'depthart_stub_einsum_y_{index}'
|
| 91 |
+
nodes.append(helper.make_node('Identity', [node.input[0]], [x_alias],
|
| 92 |
+
name=f'{x_alias}_node'))
|
| 93 |
+
nodes.append(helper.make_node('Identity', [node.output[0]], [y_alias],
|
| 94 |
+
name=f'{y_alias}_node'))
|
| 95 |
+
einsum_x[node.name] = x_alias
|
| 96 |
+
einsum_y[node.name] = y_alias
|
| 97 |
+
nodes.append(node)
|
| 98 |
+
|
| 99 |
+
stub = onnx.ModelProto()
|
| 100 |
+
stub.CopyFrom(model)
|
| 101 |
+
del stub.graph.node[:]
|
| 102 |
+
stub.graph.node.extend(nodes)
|
| 103 |
+
for name in scan_outputs:
|
| 104 |
+
stub.graph.output.append(helper.make_tensor_value_info(name, TensorProto.FLOAT, None))
|
| 105 |
+
stub.graph.output.append(helper.make_tensor_value_info(b_aliases[name],
|
| 106 |
+
TensorProto.FLOAT, None))
|
| 107 |
+
for x_alias, y_alias in zip(einsum_x.values(), einsum_y.values()):
|
| 108 |
+
stub.graph.output.append(helper.make_tensor_value_info(x_alias, TensorProto.FLOAT, None))
|
| 109 |
+
stub.graph.output.append(helper.make_tensor_value_info(y_alias, TensorProto.FLOAT, None))
|
| 110 |
+
|
| 111 |
+
meta = model.graph.input[0]
|
| 112 |
+
shape = [dim.dim_value for dim in meta.type.tensor_type.shape.dim]
|
| 113 |
+
if not all(shape):
|
| 114 |
+
raise ValueError(f'输入必须是静态形状: {meta.name} {shape}')
|
| 115 |
+
session = ort.InferenceSession(stub.SerializeToString())
|
| 116 |
+
values = dict(zip([out.name for out in session.get_outputs()],
|
| 117 |
+
session.run(None, {meta.name: np.zeros(shape, dtype=np.float32)})))
|
| 118 |
+
|
| 119 |
+
scan_map = {}
|
| 120 |
+
for name in scan_outputs:
|
| 121 |
+
u_shape = values[name].shape
|
| 122 |
+
b_shape = values[b_aliases[name]].shape
|
| 123 |
+
if len(u_shape) != 3 or len(b_shape) != 4:
|
| 124 |
+
raise ValueError(f'scan 形状不符合预期: u={u_shape} B={b_shape}')
|
| 125 |
+
scan_map[name] = (int(u_shape[1]), int(b_shape[1]), int(b_shape[2]), int(u_shape[2]))
|
| 126 |
+
scan_shapes = [scan_map[node.output[0]] for node in model.graph.node
|
| 127 |
+
if node.domain == DOMAIN and node.op_type == OP]
|
| 128 |
+
einsum_shapes = {name: (values[einsum_x[name]].shape, values[einsum_y[name]].shape)
|
| 129 |
+
for name in einsum_x}
|
| 130 |
+
return scan_shapes, einsum_shapes
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
class Rewriter:
|
| 134 |
+
def __init__(self, model, chunk_size, stub_scan=False, scan_method='pairwise'):
|
| 135 |
+
self.model = model
|
| 136 |
+
self.chunk_size = chunk_size
|
| 137 |
+
self.stub_scan = stub_scan
|
| 138 |
+
self.scan_method = scan_method
|
| 139 |
+
self.initializers = list(model.graph.initializer)
|
| 140 |
+
self.names = {tensor.name for tensor in self.initializers}
|
| 141 |
+
self.initializer_shapes = {tensor.name: tuple(tensor.dims)
|
| 142 |
+
for tensor in model.graph.initializer}
|
| 143 |
+
self.shared = {}
|
| 144 |
+
|
| 145 |
+
def build_einsum(self, prefix, node, x_shape, y_shape):
|
| 146 |
+
"""b k d l, k c d -> b k c l (以及 b k r l, k d r -> b k d l)。
|
| 147 |
+
|
| 148 |
+
两条式子都是按 k 的批 MatMul:W[k,out,in] @ x[b=1,k,in,l]。
|
| 149 |
+
替换掉 4D Einsum,避免量化执行器不支持。
|
| 150 |
+
"""
|
| 151 |
+
out_name = node.output[0]
|
| 152 |
+
w_shape = self.initializer_shapes.get(node.input[1])
|
| 153 |
+
if w_shape is None:
|
| 154 |
+
raise ValueError(f'{node.name}: 第二个输入不是常量权重,暂不支持')
|
| 155 |
+
if len(w_shape) != 3 or len(x_shape) != 4:
|
| 156 |
+
raise ValueError(f'{node.name}: Einsum 形状不符合预期: {w_shape} {x_shape}')
|
| 157 |
+
if int(x_shape[0]) != 1:
|
| 158 |
+
raise ValueError(f'{node.name}: 仅支持 batch=1,实际 {x_shape}')
|
| 159 |
+
K, out_dim, in_dim = (int(v) for v in w_shape)
|
| 160 |
+
length = int(x_shape[3])
|
| 161 |
+
if int(x_shape[1]) != K:
|
| 162 |
+
raise ValueError(f'{node.name}: K 不一致: {x_shape} vs {w_shape}')
|
| 163 |
+
expected = (1, K, out_dim, length)
|
| 164 |
+
if tuple(int(v) for v in y_shape) != expected:
|
| 165 |
+
raise ValueError(f'{node.name}: 输出形状 {y_shape} != {expected}')
|
| 166 |
+
nodes = []
|
| 167 |
+
# 用 Squeeze/Unsqueeze 而不是 Reshape 还原 batch 维:onnxslim 会把
|
| 168 |
+
# MatMul 前后的 Reshape 当成冗余节点合并/改名,导致下游张量断链。
|
| 169 |
+
nodes.append(helper.make_node(
|
| 170 |
+
'Squeeze', [node.input[0], self.shared_const('axes0', np.array([0], np.int64))],
|
| 171 |
+
[f'{prefix}_x2']))
|
| 172 |
+
nodes.append(helper.make_node('MatMul', [node.input[1], f'{prefix}_x2'],
|
| 173 |
+
[f'{prefix}_y3']))
|
| 174 |
+
nodes.append(helper.make_node(
|
| 175 |
+
'Unsqueeze', [f'{prefix}_y3', self.shared_const('axes0', np.array([0], np.int64))],
|
| 176 |
+
[out_name]))
|
| 177 |
+
return nodes
|
| 178 |
+
|
| 179 |
+
def const(self, name, value):
|
| 180 |
+
if name not in self.names:
|
| 181 |
+
self.initializers.append(numpy_helper.from_array(np.asarray(value), name))
|
| 182 |
+
self.names.add(name)
|
| 183 |
+
return name
|
| 184 |
+
|
| 185 |
+
def shared_const(self, key, value):
|
| 186 |
+
if key not in self.shared:
|
| 187 |
+
self.shared[key] = self.const(f'depthart_{key}', value)
|
| 188 |
+
return self.shared[key]
|
| 189 |
+
|
| 190 |
+
def build_logspace(self, prefix, shapes, inputs):
|
| 191 |
+
"""对数空间分块递推:每块 O(T) 而非 O(T^2),不构造 [D*N,T,T] 两两矩阵。
|
| 192 |
+
|
| 193 |
+
h_t = exp(L_t) * ( h_prev + cumsum_s( exp(-L_s) * dt_s * B_s * u_s ) )
|
| 194 |
+
其中 L 为块内相对累计衰减(L_0=0,L_t<=0)。要求块内 exp(-L) 不超 U16 量程,
|
| 195 |
+
因此 T 要小(T=4 时最大约 e^9≈8e3)。
|
| 196 |
+
"""
|
| 197 |
+
D, G, N, L = shapes
|
| 198 |
+
u, delta, a, b, c = inputs[:5]
|
| 199 |
+
d = inputs[5] if len(inputs) > 5 and inputs[5] else None
|
| 200 |
+
bias = inputs[6] if len(inputs) > 6 and inputs[6] else None
|
| 201 |
+
attrs = {item.name: item.i for item in self.node_attrs}
|
| 202 |
+
T = self.chunk_size
|
| 203 |
+
nodes = []
|
| 204 |
+
|
| 205 |
+
dt = delta
|
| 206 |
+
if bias is not None:
|
| 207 |
+
bias3 = self.const(f'{prefix}_bias3_shape', np.array([1, D, 1], np.int64))
|
| 208 |
+
nodes.append(helper.make_node('Reshape', [bias, bias3], [f'{prefix}_bias3']))
|
| 209 |
+
nodes.append(helper.make_node('Add', [delta, f'{prefix}_bias3'], [f'{prefix}_dtb']))
|
| 210 |
+
dt = f'{prefix}_dtb'
|
| 211 |
+
if int(attrs.get('delta_softplus', 0)):
|
| 212 |
+
nodes.append(helper.make_node('Softplus', [dt], [f'{prefix}_dt']))
|
| 213 |
+
dt = f'{prefix}_dt'
|
| 214 |
+
|
| 215 |
+
# 补零到 T 的整数倍,避免最后一块长度不足
|
| 216 |
+
pad = (-L) % T
|
| 217 |
+
Lp = L + pad
|
| 218 |
+
if pad:
|
| 219 |
+
nodes.append(helper.make_node(
|
| 220 |
+
'Concat', [dt, self.const(f'{prefix}_pad_dt', np.zeros((1, D, pad), np.float32))],
|
| 221 |
+
[f'{prefix}_dt_pad'], axis=2))
|
| 222 |
+
nodes.append(helper.make_node(
|
| 223 |
+
'Concat', [u, self.const(f'{prefix}_pad_u', np.zeros((1, D, pad), np.float32))],
|
| 224 |
+
[f'{prefix}_u_pad'], axis=2))
|
| 225 |
+
pad_bc = self.const(f'{prefix}_pad_bc', np.zeros((1, G, N, pad), np.float32))
|
| 226 |
+
nodes.append(helper.make_node('Concat', [b, pad_bc], [f'{prefix}_b_pad'], axis=3))
|
| 227 |
+
nodes.append(helper.make_node('Concat', [c, pad_bc], [f'{prefix}_c_pad'], axis=3))
|
| 228 |
+
dt, u, b, c = (f'{prefix}_dt_pad', f'{prefix}_u_pad',
|
| 229 |
+
f'{prefix}_b_pad', f'{prefix}_c_pad')
|
| 230 |
+
|
| 231 |
+
# dtA = dt * A -> [1,D,N,Lp]
|
| 232 |
+
a4 = self.const(f'{prefix}_a4_shape', np.array([1, D, N, 1], np.int64))
|
| 233 |
+
nodes.append(helper.make_node('Reshape', [a, a4], [f'{prefix}_a4']))
|
| 234 |
+
dt3 = self.const(f'{prefix}_dt3_shape', np.array([1, D, 1, Lp], np.int64))
|
| 235 |
+
nodes.append(helper.make_node('Reshape', [dt, dt3], [f'{prefix}_dt4']))
|
| 236 |
+
nodes.append(helper.make_node('Mul', [f'{prefix}_dt4', f'{prefix}_a4'],
|
| 237 |
+
[f'{prefix}_dtA']))
|
| 238 |
+
|
| 239 |
+
b_full = self.expand_groups(b, D, G, N, Lp, f'{prefix}_b', nodes)
|
| 240 |
+
c_full = self.expand_groups(c, D, G, N, Lp, f'{prefix}_c', nodes)
|
| 241 |
+
|
| 242 |
+
if d is not None:
|
| 243 |
+
nodes.append(helper.make_node('Reshape', [d,
|
| 244 |
+
self.const(f'{prefix}_d_shape',
|
| 245 |
+
np.array([1, D, 1], np.int64))],
|
| 246 |
+
[f'{prefix}_d3']))
|
| 247 |
+
d = f'{prefix}_d3'
|
| 248 |
+
|
| 249 |
+
axes_u2 = self.shared_const('axes_u2', np.array([2], np.int64))
|
| 250 |
+
axes_rs = self.shared_const('axes_rs2', np.array([2], np.int64))
|
| 251 |
+
axis_t = self.shared_const('axis2', np.array([2], np.int64))
|
| 252 |
+
axis_bc = self.shared_const('axis3', np.array([3], np.int64))
|
| 253 |
+
h_prev = self.const(f'{prefix}_h0', np.zeros((1, D, N, 1), np.float32))
|
| 254 |
+
outputs = []
|
| 255 |
+
for k in range(Lp // T):
|
| 256 |
+
start, end = k * T, (k + 1) * T
|
| 257 |
+
tag = f'{prefix}_c{k}'
|
| 258 |
+
starts = self.const(f'{tag}_starts', np.array([start], np.int64))
|
| 259 |
+
ends = self.const(f'{tag}_ends', np.array([end], np.int64))
|
| 260 |
+
for source, suffix, axis in ((f'{prefix}_dtA', 'dta', axis_bc),
|
| 261 |
+
(dt, 'dt', axis_t), (u, 'u', axis_t),
|
| 262 |
+
(b_full, 'b', axis_bc), (c_full, 'c', axis_bc)):
|
| 263 |
+
nodes.append(helper.make_node('Slice', [source, starts, ends, axis],
|
| 264 |
+
[f'{tag}_{suffix}']))
|
| 265 |
+
|
| 266 |
+
# L = cumsum(dtA)(块内,含首步);h = exp(L) * (h_prev + cumsum(exp(-L)*b))
|
| 267 |
+
nodes.append(helper.make_node('CumSum', [f'{tag}_dta',
|
| 268 |
+
self.shared_const('axis3s', np.array(3, np.int64))],
|
| 269 |
+
[f'{tag}_L']))
|
| 270 |
+
nodes.append(helper.make_node('Exp', [f'{tag}_L'], [f'{tag}_eL']))
|
| 271 |
+
nodes.append(helper.make_node('Neg', [f'{tag}_L'], [f'{tag}_negL']))
|
| 272 |
+
nodes.append(helper.make_node('Exp', [f'{tag}_negL'], [f'{tag}_eneg']))
|
| 273 |
+
|
| 274 |
+
# b = dt * u * B;G = cumsum(exp(-Lp) * b)
|
| 275 |
+
nodes.append(helper.make_node('Unsqueeze', [f'{tag}_dt', axes_u2], [f'{tag}_dt3']))
|
| 276 |
+
nodes.append(helper.make_node('Unsqueeze', [f'{tag}_u', axes_u2], [f'{tag}_u3']))
|
| 277 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_dt3', f'{tag}_u3'], [f'{tag}_dtu']))
|
| 278 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_dtu', f'{tag}_b'], [f'{tag}_bt']))
|
| 279 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_eneg', f'{tag}_bt'], [f'{tag}_g']))
|
| 280 |
+
nodes.append(helper.make_node('CumSum', [f'{tag}_g',
|
| 281 |
+
self.shared_const('axis3s', np.array(3, np.int64))],
|
| 282 |
+
[f'{tag}_G']))
|
| 283 |
+
nodes.append(helper.make_node('Add', [f'{tag}_G', h_prev], [f'{tag}_hp']))
|
| 284 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_eL', f'{tag}_hp'], [f'{tag}_h']))
|
| 285 |
+
|
| 286 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_h', f'{tag}_c'], [f'{tag}_hc']))
|
| 287 |
+
nodes.append(helper.make_node('ReduceSum', [f'{tag}_hc', axes_rs],
|
| 288 |
+
[f'{tag}_ysum'], keepdims=0))
|
| 289 |
+
if d is not None:
|
| 290 |
+
nodes.append(helper.make_node('Mul', [d, f'{tag}_u'], [f'{tag}_du']))
|
| 291 |
+
nodes.append(helper.make_node('Add', [f'{tag}_ysum', f'{tag}_du'],
|
| 292 |
+
[f'{tag}_y']))
|
| 293 |
+
else:
|
| 294 |
+
nodes.append(helper.make_node('Identity', [f'{tag}_ysum'], [f'{tag}_y']))
|
| 295 |
+
outputs.append(f'{tag}_y')
|
| 296 |
+
|
| 297 |
+
nodes.append(helper.make_node('Slice', [f'{tag}_h',
|
| 298 |
+
self.const(f'{tag}_hl', np.array([T - 1], np.int64)),
|
| 299 |
+
self.const(f'{tag}_he', np.array([T], np.int64)),
|
| 300 |
+
axis_bc], [f'{tag}_hnext']))
|
| 301 |
+
h_prev = f'{tag}_hnext'
|
| 302 |
+
|
| 303 |
+
nodes.append(helper.make_node('Concat', outputs, [f'{prefix}_ycat'], axis=2))
|
| 304 |
+
if pad:
|
| 305 |
+
nodes.append(helper.make_node('Slice', [
|
| 306 |
+
f'{prefix}_ycat',
|
| 307 |
+
self.const(f'{prefix}_out_start', np.array([0], np.int64)),
|
| 308 |
+
self.const(f'{prefix}_out_end', np.array([L], np.int64)),
|
| 309 |
+
self.shared_const('axis2', np.array([2], np.int64))], [self.out_name]))
|
| 310 |
+
else:
|
| 311 |
+
nodes.append(helper.make_node('Identity', [f'{prefix}_ycat'], [self.out_name]))
|
| 312 |
+
return nodes
|
| 313 |
+
|
| 314 |
+
def build(self, prefix, shapes, inputs):
|
| 315 |
+
D, G, N, L = shapes
|
| 316 |
+
u, delta, a, b, c = inputs[:5]
|
| 317 |
+
d = inputs[5] if len(inputs) > 5 and inputs[5] else None
|
| 318 |
+
bias = inputs[6] if len(inputs) > 6 and inputs[6] else None
|
| 319 |
+
attrs = {item.name: item.i for item in self.node_attrs}
|
| 320 |
+
T = self.chunk_size
|
| 321 |
+
nodes = []
|
| 322 |
+
|
| 323 |
+
# dt = softplus(delta + bias)
|
| 324 |
+
dt = delta
|
| 325 |
+
if bias is not None:
|
| 326 |
+
bias3 = self.const(f'{prefix}_bias3_shape', np.array([1, D, 1], np.int64))
|
| 327 |
+
nodes.append(helper.make_node('Reshape', [bias, bias3], [f'{prefix}_bias3']))
|
| 328 |
+
nodes.append(helper.make_node('Add', [delta, f'{prefix}_bias3'], [f'{prefix}_dtb']))
|
| 329 |
+
dt = f'{prefix}_dtb'
|
| 330 |
+
if int(attrs.get('delta_softplus', 0)):
|
| 331 |
+
nodes.append(helper.make_node('Softplus', [dt], [f'{prefix}_dt']))
|
| 332 |
+
dt = f'{prefix}_dt'
|
| 333 |
+
|
| 334 |
+
# 时间轴补零到 T 的整数倍
|
| 335 |
+
pad = (-L) % T
|
| 336 |
+
Lp = L + pad
|
| 337 |
+
if pad:
|
| 338 |
+
nodes.append(helper.make_node(
|
| 339 |
+
'Concat', [dt, self.const(f'{prefix}_pad_dt', np.zeros((1, D, pad), np.float32))],
|
| 340 |
+
[f'{prefix}_dt_pad'], axis=2))
|
| 341 |
+
nodes.append(helper.make_node(
|
| 342 |
+
'Concat', [u, self.const(f'{prefix}_pad_u', np.zeros((1, D, pad), np.float32))],
|
| 343 |
+
[f'{prefix}_u_pad'], axis=2))
|
| 344 |
+
pad_bc = self.const(f'{prefix}_pad_bc', np.zeros((1, G, N, pad), np.float32))
|
| 345 |
+
nodes.append(helper.make_node('Concat', [b, pad_bc], [f'{prefix}_b_pad'], axis=3))
|
| 346 |
+
nodes.append(helper.make_node('Concat', [c, pad_bc], [f'{prefix}_c_pad'], axis=3))
|
| 347 |
+
dt, u, b, c = (f'{prefix}_dt_pad', f'{prefix}_u_pad',
|
| 348 |
+
f'{prefix}_b_pad', f'{prefix}_c_pad')
|
| 349 |
+
|
| 350 |
+
# B/C 按组连续复制:[1,G,N,Lp] -> [1,D,N,Lp]
|
| 351 |
+
b_full = self.expand_groups(b, D, G, N, Lp, f'{prefix}_b', nodes)
|
| 352 |
+
c_full = self.expand_groups(c, D, G, N, Lp, f'{prefix}_c', nodes)
|
| 353 |
+
|
| 354 |
+
# A -> [1,D,N,1],dt -> [1,D,N,T],再摊平成 [D*N,T] 做 CumSum
|
| 355 |
+
a4 = self.const(f'{prefix}_a_shape', np.array([1, D, N, 1], np.int64))
|
| 356 |
+
nodes.append(helper.make_node('Reshape', [a, a4], [f'{prefix}_a4']))
|
| 357 |
+
if d is not None:
|
| 358 |
+
d3 = self.const(f'{prefix}_d_shape', np.array([1, D, 1], np.int64))
|
| 359 |
+
nodes.append(helper.make_node('Reshape', [d, d3], [f'{prefix}_d3']))
|
| 360 |
+
d = f'{prefix}_d3'
|
| 361 |
+
|
| 362 |
+
mask = np.tril(np.ones((T, T), dtype=bool)).reshape(1, T, T)
|
| 363 |
+
mask_name = self.shared_const(f'mask_{T}', mask)
|
| 364 |
+
neg_name = self.shared_const('mask_fill', np.array(-1e4, np.float32))
|
| 365 |
+
flat_axis = self.shared_const('axis1', np.array(1, np.int64))
|
| 366 |
+
axes_u2 = self.shared_const('axes_u2', np.array([2], np.int64))
|
| 367 |
+
axes_u3 = self.shared_const('axes_u3', np.array([3], np.int64))
|
| 368 |
+
axes_rs = self.shared_const('axes_rs2', np.array([2], np.int64))
|
| 369 |
+
|
| 370 |
+
h_prev = self.const(f'{prefix}_h0', np.zeros((1, D, N, 1), np.float32))
|
| 371 |
+
outputs = []
|
| 372 |
+
for k in range(Lp // T):
|
| 373 |
+
start, end = k * T, (k + 1) * T
|
| 374 |
+
tag = f'{prefix}_c{k}'
|
| 375 |
+
starts = self.const(f'{tag}_starts', np.array([start], np.int64))
|
| 376 |
+
ends = self.const(f'{tag}_ends', np.array([end], np.int64))
|
| 377 |
+
axis_t = self.shared_const('axis2', np.array([2], np.int64))
|
| 378 |
+
axis_bc = self.shared_const('axis3', np.array([3], np.int64))
|
| 379 |
+
for source, suffix, axis in ((dt, 'dt', axis_t), (u, 'u', axis_t),
|
| 380 |
+
(b_full, 'b', axis_bc), (c_full, 'c', axis_bc)):
|
| 381 |
+
nodes.append(helper.make_node('Slice', [source, starts, ends, axis],
|
| 382 |
+
[f'{tag}_{suffix}']))
|
| 383 |
+
|
| 384 |
+
# dtA = dt * A -> [1,D,N,T] -> [D*N,T],L = cumsum_t(dtA)
|
| 385 |
+
nodes.append(helper.make_node('Unsqueeze', [f'{tag}_dt', axes_u2], [f'{tag}_dt3']))
|
| 386 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_dt3', f'{prefix}_a4'], [f'{tag}_dtA']))
|
| 387 |
+
nodes.append(helper.make_node('Reshape', [f'{tag}_dtA',
|
| 388 |
+
self.const(f'{tag}_dtA2_shape',
|
| 389 |
+
np.array([D * N, T], np.int64))],
|
| 390 |
+
[f'{tag}_dtA2']))
|
| 391 |
+
nodes.append(helper.make_node('CumSum', [f'{tag}_dtA2', flat_axis], [f'{tag}_L2']))
|
| 392 |
+
nodes.append(helper.make_node('Reshape', [f'{tag}_L2',
|
| 393 |
+
self.const(f'{tag}_L_shape',
|
| 394 |
+
np.array([1, D, N, T], np.int64))],
|
| 395 |
+
[f'{tag}_L']))
|
| 396 |
+
|
| 397 |
+
# 两两指数矩阵 E[t,s] = exp(L_t - L_s)(s>t 置为 -1e4 后 exp 为 0)
|
| 398 |
+
nodes.append(helper.make_node('Reshape', [f'{tag}_L2',
|
| 399 |
+
self.const(f'{tag}_Lt_shape',
|
| 400 |
+
np.array([D * N, T, 1], np.int64))],
|
| 401 |
+
[f'{tag}_Lt']))
|
| 402 |
+
nodes.append(helper.make_node('Reshape', [f'{tag}_L2',
|
| 403 |
+
self.const(f'{tag}_Ls_shape',
|
| 404 |
+
np.array([D * N, 1, T], np.int64))],
|
| 405 |
+
[f'{tag}_Ls']))
|
| 406 |
+
nodes.append(helper.make_node('Sub', [f'{tag}_Lt', f'{tag}_Ls'], [f'{tag}_diff']))
|
| 407 |
+
nodes.append(helper.make_node('Where', [mask_name, f'{tag}_diff', neg_name],
|
| 408 |
+
[f'{tag}_diffm']))
|
| 409 |
+
nodes.append(helper.make_node('Exp', [f'{tag}_diffm'], [f'{tag}_E']))
|
| 410 |
+
|
| 411 |
+
# b = dt * u * B,形状 [D*N,1,T];contrib[t] = sum_{s<=t} E[t,s]*b[s]
|
| 412 |
+
nodes.append(helper.make_node('Unsqueeze', [f'{tag}_u', axes_u2], [f'{tag}_u3']))
|
| 413 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_dt3', f'{tag}_u3'], [f'{tag}_dtu']))
|
| 414 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_dtu', f'{tag}_b'], [f'{tag}_bt']))
|
| 415 |
+
nodes.append(helper.make_node('Reshape', [f'{tag}_bt',
|
| 416 |
+
self.const(f'{tag}_bt2_shape',
|
| 417 |
+
np.array([D * N, 1, T], np.int64))],
|
| 418 |
+
[f'{tag}_bt2']))
|
| 419 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_E', f'{tag}_bt2'], [f'{tag}_eb']))
|
| 420 |
+
nodes.append(helper.make_node('ReduceSum', [f'{tag}_eb',
|
| 421 |
+
self.shared_const('axis2s', np.array([2], np.int64))],
|
| 422 |
+
[f'{tag}_contrib2'], keepdims=0))
|
| 423 |
+
nodes.append(helper.make_node('Reshape', [f'{tag}_contrib2',
|
| 424 |
+
self.const(f'{tag}_contrib_shape',
|
| 425 |
+
np.array([1, D, N, T], np.int64))],
|
| 426 |
+
[f'{tag}_contrib']))
|
| 427 |
+
|
| 428 |
+
# 块间状态:h_t = exp(L_t) * h_prev + contrib
|
| 429 |
+
nodes.append(helper.make_node('Exp', [f'{tag}_L'], [f'{tag}_expL']))
|
| 430 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_expL', h_prev], [f'{tag}_carry']))
|
| 431 |
+
nodes.append(helper.make_node('Add', [f'{tag}_carry', f'{tag}_contrib'],
|
| 432 |
+
[f'{tag}_h']))
|
| 433 |
+
|
| 434 |
+
# y = sum_n(h*C) + D*u
|
| 435 |
+
nodes.append(helper.make_node('Mul', [f'{tag}_h', f'{tag}_c'], [f'{tag}_hc']))
|
| 436 |
+
nodes.append(helper.make_node('ReduceSum', [f'{tag}_hc', axes_rs],
|
| 437 |
+
[f'{tag}_ysum'], keepdims=0))
|
| 438 |
+
if d is not None:
|
| 439 |
+
nodes.append(helper.make_node('Mul', [d, f'{tag}_u'], [f'{tag}_du']))
|
| 440 |
+
nodes.append(helper.make_node('Add', [f'{tag}_ysum', f'{tag}_du'],
|
| 441 |
+
[f'{tag}_y']))
|
| 442 |
+
else:
|
| 443 |
+
nodes.append(helper.make_node('Identity', [f'{tag}_ysum'], [f'{tag}_y']))
|
| 444 |
+
outputs.append(f'{tag}_y')
|
| 445 |
+
|
| 446 |
+
nodes.append(helper.make_node('Gather', [f'{tag}_h',
|
| 447 |
+
self.const(f'{tag}_last', np.array(T - 1, np.int64))],
|
| 448 |
+
[f'{tag}_hlast'], axis=3))
|
| 449 |
+
nodes.append(helper.make_node('Unsqueeze', [f'{tag}_hlast', axes_u3],
|
| 450 |
+
[f'{tag}_hnext']))
|
| 451 |
+
h_prev = f'{tag}_hnext'
|
| 452 |
+
|
| 453 |
+
nodes.append(helper.make_node('Concat', outputs, [f'{prefix}_ycat'], axis=2))
|
| 454 |
+
if pad:
|
| 455 |
+
nodes.append(helper.make_node('Slice', [
|
| 456 |
+
f'{prefix}_ycat',
|
| 457 |
+
self.const(f'{prefix}_out_start', np.array([0], np.int64)),
|
| 458 |
+
self.const(f'{prefix}_out_end', np.array([L], np.int64)),
|
| 459 |
+
self.shared_const('axis2', np.array([2], np.int64))], [self.out_name]))
|
| 460 |
+
else:
|
| 461 |
+
nodes.append(helper.make_node('Identity', [f'{prefix}_ycat'], [self.out_name]))
|
| 462 |
+
return nodes
|
| 463 |
+
|
| 464 |
+
def expand_groups(self, x, D, G, N, L, prefix, nodes):
|
| 465 |
+
gs = D // G
|
| 466 |
+
r1_shape = self.const(f'{prefix}_r1_shape', np.array([1, G, 1, N, L], np.int64))
|
| 467 |
+
nodes.append(helper.make_node('Reshape', [x, r1_shape], [f'{prefix}_r1']))
|
| 468 |
+
target = self.const(f'{prefix}_tgt', np.array([1, G, gs, N, L], np.int64))
|
| 469 |
+
nodes.append(helper.make_node('Expand', [f'{prefix}_r1', target], [f'{prefix}_e']))
|
| 470 |
+
r2_shape = self.const(f'{prefix}_r2_shape', np.array([1, D, N, L], np.int64))
|
| 471 |
+
nodes.append(helper.make_node('Reshape', [f'{prefix}_e', r2_shape], [f'{prefix}_bc4']))
|
| 472 |
+
return f'{prefix}_bc4'
|
| 473 |
+
|
| 474 |
+
def rewrite(self):
|
| 475 |
+
shapes, einsum_shapes = collect_runtime_shapes(self.model)
|
| 476 |
+
if not shapes:
|
| 477 |
+
raise ValueError('模型中没有 SelectiveScan 节点,无需改写')
|
| 478 |
+
new_nodes, index, einsum_index = [], 0, 0
|
| 479 |
+
for node in self.model.graph.node:
|
| 480 |
+
if node.domain == DOMAIN and node.op_type == OP:
|
| 481 |
+
if self.stub_scan:
|
| 482 |
+
# 仅用于测 NPU 耗时的对照模型:扫描直接透传 u,精度无意义。
|
| 483 |
+
stub_node = helper.make_node('Identity', [node.input[0]], [node.output[0]],
|
| 484 |
+
name=f'depthart_stub_scan_{index}')
|
| 485 |
+
new_nodes.append(stub_node)
|
| 486 |
+
index += 1
|
| 487 |
+
continue
|
| 488 |
+
self.node_attrs = node.attribute
|
| 489 |
+
self.out_name = node.output[0]
|
| 490 |
+
builder = self.build_logspace if self.scan_method == 'log' else self.build
|
| 491 |
+
nodes = builder(f'depthart_pscan_{index}', shapes[index], node.input)
|
| 492 |
+
new_nodes.extend(nodes)
|
| 493 |
+
index += 1
|
| 494 |
+
elif node.op_type == 'Einsum':
|
| 495 |
+
equation = None
|
| 496 |
+
for attr in node.attribute:
|
| 497 |
+
if attr.name == 'equation':
|
| 498 |
+
equation = attr.s.decode()
|
| 499 |
+
if equation not in EINSUM_EQUATIONS:
|
| 500 |
+
raise ValueError(f'{node.name}: 不支持的 Einsum 方程 {equation!r}')
|
| 501 |
+
x_shape, y_shape = einsum_shapes[node.name]
|
| 502 |
+
nodes = self.build_einsum(f'depthart_einsum_{einsum_index}',
|
| 503 |
+
node, x_shape, y_shape)
|
| 504 |
+
new_nodes.extend(nodes)
|
| 505 |
+
einsum_index += 1
|
| 506 |
+
else:
|
| 507 |
+
new_nodes.append(node)
|
| 508 |
+
out = onnx.ModelProto()
|
| 509 |
+
out.CopyFrom(self.model)
|
| 510 |
+
del out.graph.node[:]
|
| 511 |
+
for position, node in enumerate(new_nodes):
|
| 512 |
+
if not node.name:
|
| 513 |
+
node.name = f'depthart_gen_{position}'
|
| 514 |
+
out.graph.node.extend(new_nodes)
|
| 515 |
+
out.graph.initializer.extend(self.initializers[len(self.model.graph.initializer):])
|
| 516 |
+
return out
|
| 517 |
+
|
| 518 |
+
|
| 519 |
+
def main():
|
| 520 |
+
parser = argparse.ArgumentParser(
|
| 521 |
+
description='把 DepthART SelectiveScan 改写为 pulsar2 可编译的纯 ONNX 子图')
|
| 522 |
+
parser.add_argument('--model', type=Path, default=DEFAULT_MODEL,
|
| 523 |
+
help=f'输入 ONNX,默认 {DEFAULT_MODEL.relative_to(PROJECT_ROOT)}')
|
| 524 |
+
parser.add_argument('--output', type=Path,
|
| 525 |
+
default=PROJECT_ROOT / 'onnx' / 'relative_s_448_default_axonnx.onnx',
|
| 526 |
+
help='输出 ONNX(不会覆盖输入模型)')
|
| 527 |
+
parser.add_argument('--chunk-size', type=int, default=14,
|
| 528 |
+
help='分块长度 T,默认 14;越大显存占用越高')
|
| 529 |
+
parser.add_argument('--stub-scan', action='store_true',
|
| 530 |
+
help='把 SelectiveScan 替换为 Identity,生成仅用于测 NPU 耗时的对照模型(精度无意义)')
|
| 531 |
+
parser.add_argument('--scan-method', choices=('pairwise', 'log'), default='pairwise',
|
| 532 |
+
help='pairwise=两两指数矩阵(默认);log=对数空间递推,无 T^2 矩阵,T 需小')
|
| 533 |
+
args = parser.parse_args()
|
| 534 |
+
logging.basicConfig(level=logging.INFO, format='%(message)s')
|
| 535 |
+
if args.chunk_size < 1:
|
| 536 |
+
parser.error('--chunk-size 必须为正数')
|
| 537 |
+
if not args.model.is_file():
|
| 538 |
+
parser.error(f'--model 不存在: {args.model}')
|
| 539 |
+
if args.output.resolve() == args.model.resolve():
|
| 540 |
+
parser.error('--output 不能与 --model 相同')
|
| 541 |
+
|
| 542 |
+
model = onnx.load(str(args.model))
|
| 543 |
+
if not any(node.domain == DOMAIN and node.op_type == OP for node in model.graph.node):
|
| 544 |
+
raise ValueError(f'{args.model} 中没有 {DOMAIN}::{OP} 节点')
|
| 545 |
+
rewriter = Rewriter(model, args.chunk_size, args.stub_scan, args.scan_method)
|
| 546 |
+
rewritten = rewriter.rewrite()
|
| 547 |
+
args.output.parent.mkdir(parents=True, exist_ok=True)
|
| 548 |
+
onnx.save(rewritten, str(args.output))
|
| 549 |
+
logging.info('已写出标准 ONNX: %s(节点数 %d)', args.output, len(rewritten.graph.node))
|
| 550 |
+
logging.info('下一步:把 pulsar2 配置的 input 指向该文件;'
|
| 551 |
+
'可先用 src/infer/depthart_infer_onnx.py 验证数值。')
|
| 552 |
+
|
| 553 |
+
|
| 554 |
+
if __name__ == '__main__':
|
| 555 |
+
main()
|
src/tool/evaluate_nyu654_depth.py
ADDED
|
@@ -0,0 +1,210 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# -*- coding: utf-8 -*-
|
| 3 |
+
"""
|
| 4 |
+
NYU Depth V2 Eigen test split (654 张) 相对深度精度评估。
|
| 5 |
+
|
| 6 |
+
输入:
|
| 7 |
+
--mat nyu_depth_v2_labeled.mat(GT 直接从这里读,不落盘)
|
| 8 |
+
--split export_nyu654_test.py 生成的 split.json
|
| 9 |
+
--pred 一个或多个 <名称>=<NPZ 目录>,NPZ 由 infer 脚本 --dump-npz 生成
|
| 10 |
+
|
| 11 |
+
评估协议:
|
| 12 |
+
1. 有效像素:rawDepths > 0 且 depths 在 (0, max_depth];
|
| 13 |
+
2. 预测上采样到 GT 分辨率(bilinear);
|
| 14 |
+
3. 相对深度对齐:
|
| 15 |
+
- disparity(默认,MiDaS 协议):拟合 a*pred + b ≈ 1/gt,再取倒数得米制深度;
|
| 16 |
+
- depth:拟合 a*pred + b ≈ gt;
|
| 17 |
+
最小二乘在有效像素上求解;
|
| 18 |
+
4. 逐图计算 AbsRel / SqRel / RMSE / RMSE_log / δ<1.25 / 1.25² / 1.25³,
|
| 19 |
+
对 654 张取平均;
|
| 20 |
+
5. 第一个 --pred 作为基准,其余输出相对掉点 (%)。
|
| 21 |
+
|
| 22 |
+
注意:未做 Eigen crop,绝对数值不能直接与论文表格对比;同一批图上不同后端的
|
| 23 |
+
相对掉点不受影响。
|
| 24 |
+
|
| 25 |
+
用法:
|
| 26 |
+
../../venv/bin/python src/tool/evaluate_nyu654_depth.py \
|
| 27 |
+
--mat ../../datasets/nyu_depth_v2_labeled.mat \
|
| 28 |
+
--split datasets/nyu654_test/split.json \
|
| 29 |
+
--pred fp32=results/nyu654_fp32/npz axonnx=results/nyu654_axonnx/npz \
|
| 30 |
+
--out results/nyu654_eval
|
| 31 |
+
"""
|
| 32 |
+
|
| 33 |
+
import argparse
|
| 34 |
+
import json
|
| 35 |
+
from pathlib import Path
|
| 36 |
+
|
| 37 |
+
import cv2
|
| 38 |
+
import h5py
|
| 39 |
+
import numpy as np
|
| 40 |
+
from tqdm import tqdm
|
| 41 |
+
|
| 42 |
+
METRICS = ('AbsRel', 'SqRel', 'RMSE', 'RMSE_log', 'd1', 'd2', 'd3')
|
| 43 |
+
SIGMA = (1.25, 1.25 ** 2, 1.25 ** 3)
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def solve_scale_shift(pred, target, mask):
|
| 47 |
+
"""在 mask 上最小二乘求解 a, b 使 a*pred + b ≈ target。"""
|
| 48 |
+
p = pred[mask]
|
| 49 |
+
t = target[mask]
|
| 50 |
+
A = np.stack([p, np.ones_like(p)], axis=1)
|
| 51 |
+
(a, b), *_ = np.linalg.lstsq(A, t, rcond=None)
|
| 52 |
+
return float(a), float(b)
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
def compute_metrics(pred_depth, gt_depth, mask, align, max_depth):
|
| 56 |
+
clipped = 0.0
|
| 57 |
+
if align == 'disparity':
|
| 58 |
+
gt_target = 1.0 / gt_depth
|
| 59 |
+
a, b = solve_scale_shift(pred_depth, gt_target, mask)
|
| 60 |
+
if a <= 0:
|
| 61 |
+
raise ValueError(f'disparity 对齐斜率为负 (a={a:.4f}),'
|
| 62 |
+
f'预测不是"越大越近"的视差量')
|
| 63 |
+
aligned = a * pred_depth + b
|
| 64 |
+
# 极少数图(低纹理/异常预测)会拟出负视差,裁剪到最大深度的下限
|
| 65 |
+
neg = aligned[mask] <= 0
|
| 66 |
+
clipped = float(neg.mean())
|
| 67 |
+
aligned = np.maximum(aligned, 1.0 / max_depth)
|
| 68 |
+
pred = 1.0 / aligned
|
| 69 |
+
else:
|
| 70 |
+
a, b = solve_scale_shift(pred_depth, gt_depth, mask)
|
| 71 |
+
pred = a * pred_depth + b
|
| 72 |
+
|
| 73 |
+
pred = pred[mask]
|
| 74 |
+
gt = gt_depth[mask]
|
| 75 |
+
if np.any(pred <= 0):
|
| 76 |
+
raise ValueError('对齐后预测深度出现非正值')
|
| 77 |
+
|
| 78 |
+
diff = pred - gt
|
| 79 |
+
out = {
|
| 80 |
+
'AbsRel': float(np.mean(np.abs(diff) / gt)),
|
| 81 |
+
'SqRel': float(np.mean(diff ** 2 / gt)),
|
| 82 |
+
'RMSE': float(np.sqrt(np.mean(diff ** 2))),
|
| 83 |
+
'RMSE_log': float(np.sqrt(np.mean((np.log(pred) - np.log(gt)) ** 2))),
|
| 84 |
+
}
|
| 85 |
+
ratio = np.maximum(pred / gt, gt / pred)
|
| 86 |
+
for name, s in zip(('d1', 'd2', 'd3'), SIGMA):
|
| 87 |
+
out[name] = float(np.mean(ratio < s))
|
| 88 |
+
return out, a, b, clipped
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def load_predictions(path):
|
| 92 |
+
"""读取预测:汇总 NPZ(names/depths)、分块汇总目录,或旧的逐图 NPZ 目录。"""
|
| 93 |
+
path = Path(path)
|
| 94 |
+
if path.is_file():
|
| 95 |
+
files = [path]
|
| 96 |
+
elif path.is_dir():
|
| 97 |
+
files = sorted(path.glob('*.npz'))
|
| 98 |
+
if not files:
|
| 99 |
+
raise ValueError(f'目录中没有 NPZ: {path}')
|
| 100 |
+
else:
|
| 101 |
+
raise FileNotFoundError(f'预测路径不存在: {path}')
|
| 102 |
+
|
| 103 |
+
preds = {}
|
| 104 |
+
for fp in files:
|
| 105 |
+
with np.load(fp, allow_pickle=False) as data:
|
| 106 |
+
if 'depths' in data and 'names' in data:
|
| 107 |
+
for name, depth in zip(data['names'], data['depths']):
|
| 108 |
+
preds[str(name)] = depth
|
| 109 |
+
elif 'depth' in data:
|
| 110 |
+
preds[fp.name[:-len('.npz')]] = data['depth']
|
| 111 |
+
else:
|
| 112 |
+
raise ValueError(f'无法识别的 NPZ 格式: {fp}')
|
| 113 |
+
return preds
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
def main():
|
| 117 |
+
parser = argparse.ArgumentParser(description='NYU 654 相对深度评测')
|
| 118 |
+
parser.add_argument('--mat', required=True)
|
| 119 |
+
parser.add_argument('--split', required=True, help='split.json')
|
| 120 |
+
parser.add_argument('--pred', nargs='+', required=True, help='名称=NPZ目录,第一个为基准')
|
| 121 |
+
parser.add_argument('--out', required=True, help='结果目录')
|
| 122 |
+
parser.add_argument('--align', choices=['disparity', 'depth'], default='disparity')
|
| 123 |
+
parser.add_argument('--max-depth', type=float, default=10.0)
|
| 124 |
+
args = parser.parse_args()
|
| 125 |
+
|
| 126 |
+
pred_sets = []
|
| 127 |
+
for item in args.pred:
|
| 128 |
+
name, _, path = item.partition('=')
|
| 129 |
+
if not path:
|
| 130 |
+
raise ValueError(f'--pred 需要 名称=路径 格式: {item}')
|
| 131 |
+
pred_sets.append((name, load_predictions(path)))
|
| 132 |
+
|
| 133 |
+
split = json.load(open(args.split))
|
| 134 |
+
out_dir = Path(args.out)
|
| 135 |
+
out_dir.mkdir(parents=True, exist_ok=True)
|
| 136 |
+
|
| 137 |
+
per_image = {name: [] for name, _ in pred_sets}
|
| 138 |
+
failed = {name: [] for name, _ in pred_sets}
|
| 139 |
+
with h5py.File(args.mat, 'r') as f:
|
| 140 |
+
for record in tqdm(split, desc=f'eval({args.align})', ncols=80):
|
| 141 |
+
i = record['mat_index']
|
| 142 |
+
gt = np.asarray(f['depths'][i], dtype=np.float32)
|
| 143 |
+
raw = np.asarray(f['rawDepths'][i], dtype=np.float32)
|
| 144 |
+
mask = (raw > 0) & (gt > 0) & (gt <= args.max_depth)
|
| 145 |
+
|
| 146 |
+
for name, preds in pred_sets:
|
| 147 |
+
if record['file'] not in preds:
|
| 148 |
+
raise KeyError(f'{name} 缺少预测: {record["file"]}')
|
| 149 |
+
pred = np.asarray(preds[record['file']], dtype=np.float32)
|
| 150 |
+
if pred.ndim == 3:
|
| 151 |
+
pred = pred[0]
|
| 152 |
+
pred = cv2.resize(pred, (gt.shape[1], gt.shape[0]), interpolation=cv2.INTER_LINEAR)
|
| 153 |
+
try:
|
| 154 |
+
m, a, b, clipped = compute_metrics(pred, gt, mask, args.align, args.max_depth)
|
| 155 |
+
except ValueError as error:
|
| 156 |
+
failed[name].append({'file': record['file'], 'reason': str(error)})
|
| 157 |
+
continue
|
| 158 |
+
m.update({'file': record['file'], 'scale': a, 'shift': b,
|
| 159 |
+
'clipped_frac': clipped, 'valid_pixels': int(mask.sum())})
|
| 160 |
+
per_image[name].append(m)
|
| 161 |
+
|
| 162 |
+
results = {}
|
| 163 |
+
for name, _ in pred_sets:
|
| 164 |
+
rows = per_image[name]
|
| 165 |
+
results[name] = {k: float(np.mean([r[k] for r in rows])) for k in METRICS}
|
| 166 |
+
results[name]['p90_absrel'] = float(np.percentile([r['AbsRel'] for r in rows], 90))
|
| 167 |
+
results[name]['clipped_images'] = int(sum(r['clipped_frac'] > 0 for r in rows))
|
| 168 |
+
results[name]['clipped_pixel_frac'] = float(np.mean([r['clipped_frac'] for r in rows]))
|
| 169 |
+
results[name]['evaluated_images'] = len(rows)
|
| 170 |
+
results[name]['failed_images'] = len(failed[name])
|
| 171 |
+
|
| 172 |
+
base = pred_sets[0][0]
|
| 173 |
+
drops = {}
|
| 174 |
+
for name, _ in pred_sets[1:]:
|
| 175 |
+
drops[name] = {}
|
| 176 |
+
for k in METRICS:
|
| 177 |
+
b, v = results[base][k], results[name][k]
|
| 178 |
+
drops[name][k] = float((v - b) / b * 100)
|
| 179 |
+
b, v = results[base]['p90_absrel'], results[name]['p90_absrel']
|
| 180 |
+
drops[name]['p90_absrel'] = float((v - b) / b * 100)
|
| 181 |
+
|
| 182 |
+
summary = {
|
| 183 |
+
'dataset': 'nyu_depth_v2_labeled / eigen test split',
|
| 184 |
+
'num_images': len(split),
|
| 185 |
+
'align': args.align,
|
| 186 |
+
'max_depth': args.max_depth,
|
| 187 |
+
'eigen_crop': False,
|
| 188 |
+
'results': results,
|
| 189 |
+
'baseline': base,
|
| 190 |
+
'drop_percent': drops,
|
| 191 |
+
'failed': failed,
|
| 192 |
+
'per_image': {name: per_image[name] for name, _ in pred_sets},
|
| 193 |
+
}
|
| 194 |
+
with open(out_dir / 'metrics.json', 'w') as fp:
|
| 195 |
+
json.dump(summary, fp, indent=2)
|
| 196 |
+
|
| 197 |
+
print(f'\n{"指标":<10}' + ''.join(f'{n:>14}' for n, _ in pred_sets))
|
| 198 |
+
for k in METRICS + ('p90_absrel',):
|
| 199 |
+
print(f'{k:<10}' + ''.join(f'{results[n][k]:>14.4f}' for n, _ in pred_sets))
|
| 200 |
+
for name, d in drops.items():
|
| 201 |
+
print(f'\n{name} 相对 {base} 掉点 (%): ' +
|
| 202 |
+
', '.join(f'{k}={v:+.2f}' for k, v in d.items()))
|
| 203 |
+
for name, _ in pred_sets:
|
| 204 |
+
if failed[name]:
|
| 205 |
+
print(f'\n{name}: {len(failed[name])} 张对齐失败(方向反了或输出异常),未计入均值')
|
| 206 |
+
print(f'\n结果已保存: {out_dir / "metrics.json"}')
|
| 207 |
+
|
| 208 |
+
|
| 209 |
+
if __name__ == '__main__':
|
| 210 |
+
main()
|
src/tool/export_nyu654_test.py
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# -*- coding: utf-8 -*-
|
| 3 |
+
"""
|
| 4 |
+
从 nyu_depth_v2_labeled.mat 导出官方 Eigen test split 的 654 张 RGB 图片。
|
| 5 |
+
|
| 6 |
+
划分来源:BTS 仓库 utils/splits.mat 的 testNdxs(1-based,共 654 个),
|
| 7 |
+
与 train_test_inputs/nyudepthv2_test_files_with_gt.txt 的 654 行逐场景帧数一致,
|
| 8 |
+
已核对无重叠、并集覆盖 1..1449。
|
| 9 |
+
|
| 10 |
+
只导出 RGB,GT 不落盘:评测脚本按 split.json 里的 mat_index 直接从 mat 读取,
|
| 11 |
+
避免重复存储。
|
| 12 |
+
|
| 13 |
+
用法:
|
| 14 |
+
# 导出 654 张到 datasets/nyu654_test/
|
| 15 |
+
../../venv/bin/python src/tool/export_nyu654_test.py \
|
| 16 |
+
--mat ../../datasets/nyu_depth_v2_labeled.mat \
|
| 17 |
+
--splits datasets/nyu654_test/splits.mat \
|
| 18 |
+
--out-dir datasets/nyu654_test
|
| 19 |
+
|
| 20 |
+
输出:
|
| 21 |
+
<out-dir>/images/{序号:04d}_{scene}.png
|
| 22 |
+
<out-dir>/split.json [{mat_index, scene, file}, ...]
|
| 23 |
+
"""
|
| 24 |
+
|
| 25 |
+
import argparse
|
| 26 |
+
import json
|
| 27 |
+
from pathlib import Path
|
| 28 |
+
|
| 29 |
+
import cv2
|
| 30 |
+
import h5py
|
| 31 |
+
import numpy as np
|
| 32 |
+
import scipy.io as sio
|
| 33 |
+
from tqdm import tqdm
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def decode_cell(f, ref):
|
| 37 |
+
return ''.join(chr(c) for c in np.asarray(f[ref]).ravel())
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def main():
|
| 41 |
+
parser = argparse.ArgumentParser(description='导出 NYU Depth V2 Eigen test split 654 张')
|
| 42 |
+
parser.add_argument('--mat', required=True, help='nyu_depth_v2_labeled.mat')
|
| 43 |
+
parser.add_argument('--splits', required=True, help='含 testNdxs 的 splits.mat')
|
| 44 |
+
parser.add_argument('--out-dir', required=True, help='输出目录')
|
| 45 |
+
args = parser.parse_args()
|
| 46 |
+
|
| 47 |
+
out_dir = Path(args.out_dir)
|
| 48 |
+
image_dir = out_dir / 'images'
|
| 49 |
+
image_dir.mkdir(parents=True, exist_ok=True)
|
| 50 |
+
|
| 51 |
+
ndxs = sio.loadmat(args.splits)['testNdxs'].ravel().astype(int)
|
| 52 |
+
if len(ndxs) != 654:
|
| 53 |
+
raise ValueError(f'testNdxs 应为 654 个,实际 {len(ndxs)}')
|
| 54 |
+
|
| 55 |
+
records = []
|
| 56 |
+
with h5py.File(args.mat, 'r') as f:
|
| 57 |
+
for seq, mat_index in enumerate(tqdm(ndxs, desc='export', ncols=80)):
|
| 58 |
+
i = mat_index - 1 # matlab 1-based
|
| 59 |
+
scene = decode_cell(f, f['scenes'][0, i])
|
| 60 |
+
rgb = np.asarray(f['images'][i]).transpose(1, 2, 0) # CHW -> HWC
|
| 61 |
+
name = f'{seq:04d}_{scene}.png'
|
| 62 |
+
if not cv2.imwrite(str(image_dir / name), cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)):
|
| 63 |
+
raise RuntimeError(f'写入失败: {image_dir / name}')
|
| 64 |
+
records.append({'mat_index': int(i), 'scene': scene, 'file': name})
|
| 65 |
+
|
| 66 |
+
with open(out_dir / 'split.json', 'w') as fp:
|
| 67 |
+
json.dump(records, fp, indent=2)
|
| 68 |
+
print(f'已导出 {len(records)} 张到 {image_dir},划分文件 {out_dir / "split.json"}')
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
if __name__ == '__main__':
|
| 72 |
+
main()
|
src/tool/export_nyu_train795.py
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# -*- coding: utf-8 -*-
|
| 3 |
+
"""
|
| 4 |
+
从 nyu_depth_v2_labeled.mat 导出 795 张训练集(trainNdxs)图片。
|
| 5 |
+
|
| 6 |
+
划分来源与 export_nyu654_test.py 相同:BTS utils/splits.mat。
|
| 7 |
+
trainNdxs 与 testNdxs(654)无重叠、并集覆盖 1..1449,可以与测试集严格隔离,
|
| 8 |
+
用于量化校准或训练。
|
| 9 |
+
|
| 10 |
+
只导出 RGB;GT 仍从 mat 按 split.json 的 mat_index 读取。
|
| 11 |
+
|
| 12 |
+
用法:
|
| 13 |
+
../../venv/bin/python src/tool/export_nyu_train795.py \
|
| 14 |
+
--mat ../../datasets/nyu_depth_v2_labeled.mat \
|
| 15 |
+
--splits datasets/nyu654_test/splits.mat \
|
| 16 |
+
--out-dir datasets/nyu_train795
|
| 17 |
+
"""
|
| 18 |
+
|
| 19 |
+
import argparse
|
| 20 |
+
import json
|
| 21 |
+
from pathlib import Path
|
| 22 |
+
|
| 23 |
+
import cv2
|
| 24 |
+
import h5py
|
| 25 |
+
import numpy as np
|
| 26 |
+
import scipy.io as sio
|
| 27 |
+
from tqdm import tqdm
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def decode_cell(f, ref):
|
| 31 |
+
return ''.join(chr(c) for c in np.asarray(f[ref]).ravel())
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def main():
|
| 35 |
+
parser = argparse.ArgumentParser(description='导出 NYU Depth V2 训练集 795 张')
|
| 36 |
+
parser.add_argument('--mat', required=True, help='nyu_depth_v2_labeled.mat')
|
| 37 |
+
parser.add_argument('--splits', required=True, help='含 trainNdxs 的 splits.mat')
|
| 38 |
+
parser.add_argument('--out-dir', required=True, help='输出目录')
|
| 39 |
+
args = parser.parse_args()
|
| 40 |
+
|
| 41 |
+
image_dir = Path(args.out_dir) / 'images'
|
| 42 |
+
image_dir.mkdir(parents=True, exist_ok=True)
|
| 43 |
+
|
| 44 |
+
ndxs = sio.loadmat(args.splits)['trainNdxs'].ravel().astype(int)
|
| 45 |
+
if len(ndxs) != 795:
|
| 46 |
+
raise ValueError(f'trainNdxs 应为 795 个,实际 {len(ndxs)}')
|
| 47 |
+
|
| 48 |
+
records = []
|
| 49 |
+
with h5py.File(args.mat, 'r') as f:
|
| 50 |
+
for seq, mat_index in enumerate(tqdm(ndxs, desc='export', ncols=80)):
|
| 51 |
+
i = mat_index - 1
|
| 52 |
+
scene = decode_cell(f, f['scenes'][0, i])
|
| 53 |
+
rgb = np.asarray(f['images'][i]).transpose(1, 2, 0)
|
| 54 |
+
name = f'{seq:04d}_{scene}.png'
|
| 55 |
+
if not cv2.imwrite(str(image_dir / name), cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)):
|
| 56 |
+
raise RuntimeError(f'写入失败: {image_dir / name}')
|
| 57 |
+
records.append({'mat_index': int(i), 'scene': scene, 'file': name})
|
| 58 |
+
|
| 59 |
+
with open(Path(args.out_dir) / 'split.json', 'w') as fp:
|
| 60 |
+
json.dump(records, fp, indent=2)
|
| 61 |
+
print(f'已导出 {len(records)} 张到 {image_dir}')
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
if __name__ == '__main__':
|
| 65 |
+
main()
|
src/tool/inspect_nyu_depth_v2.py
ADDED
|
@@ -0,0 +1,201 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# -*- coding: utf-8 -*-
|
| 3 |
+
"""
|
| 4 |
+
NYU Depth V2 (labeled) 数据集查看与导出工具。
|
| 5 |
+
|
| 6 |
+
nyu_depth_v2_labeled.mat 是 MATLAB v7.3 (HDF5) 格式,scipy.io.loadmat 无法读取,
|
| 7 |
+
这里统一用 h5py 按需读取,避免一次性载入 2.8GB。
|
| 8 |
+
|
| 9 |
+
数据内容(N = 1449,图像为竖版 640x480,H=640, W=480,RGB 与深度已对齐):
|
| 10 |
+
images [N, 3, 640, 480] uint8 RGB
|
| 11 |
+
depths [N, 640, 480] float32 已填补深度,单位米
|
| 12 |
+
rawDepths [N, 640, 480] float32 原始深度,0 表示缺失
|
| 13 |
+
labels [N, 640, 480] uint16 语义标签,0=未标注,1..894 对应 names
|
| 14 |
+
instances [N, 640, 480] uint8 实例标签,0=无实例
|
| 15 |
+
names [1, 894] cell 类别名(1-based 下标即 label id)
|
| 16 |
+
namesToIds containers.Map MATLAB Map,h5py 无法直接解析
|
| 17 |
+
scenes [1, N] cell 场景名,如 kitchen_0004
|
| 18 |
+
sceneTypes [1, N] cell 场景类型,如 kitchen
|
| 19 |
+
rawRgbFilenames [1, N] cell 原始 RGB 文件名
|
| 20 |
+
rawDepthFilenames [1, N] cell 原始深度文件名
|
| 21 |
+
accelData [4, N] float32 加速度计数据
|
| 22 |
+
|
| 23 |
+
用法(在 hs/DepthART 目录下执行):
|
| 24 |
+
# 打印结构、场景/类别统计和深度数值范围(默认抽样 step=50)
|
| 25 |
+
../../venv/bin/python src/tool/inspect_nyu_depth_v2.py --mat ../../datasets/nyu_depth_v2_labeled.mat
|
| 26 |
+
|
| 27 |
+
# 全量扫描统计(较慢)
|
| 28 |
+
... --mat ../../datasets/nyu_depth_v2_labeled.mat --full-stats
|
| 29 |
+
|
| 30 |
+
# 列出全部 894 个类别名
|
| 31 |
+
... --mat ... --list-classes
|
| 32 |
+
|
| 33 |
+
# 单样本导出:rgb.png / depth.npy / depth_vis.png / raw_depth.npy
|
| 34 |
+
... --mat ... --index 0 --out-dir results/nyu_demo --with-labels
|
| 35 |
+
|
| 36 |
+
# 批量导出,例如某场景全部帧
|
| 37 |
+
... --mat ... --export-dir results/nyu_export --scene kitchen_0004 --with-labels
|
| 38 |
+
|
| 39 |
+
# 批量导出前 50 张
|
| 40 |
+
... --mat ... --export-dir results/nyu_export --count 50
|
| 41 |
+
"""
|
| 42 |
+
|
| 43 |
+
import argparse
|
| 44 |
+
import collections
|
| 45 |
+
from pathlib import Path
|
| 46 |
+
|
| 47 |
+
import cv2
|
| 48 |
+
import h5py
|
| 49 |
+
import numpy as np
|
| 50 |
+
from tqdm import tqdm
|
| 51 |
+
|
| 52 |
+
# 深度可视化统一范围(米),超出部分截断
|
| 53 |
+
DEPTH_VIS_RANGE = (0.5, 10.0)
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def decode_cell(f, ref):
|
| 57 |
+
"""把 MATLAB cell 中的 char 数组解码为 str。"""
|
| 58 |
+
return ''.join(chr(c) for c in np.asarray(f[ref]).ravel())
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def read_cell_list(dset):
|
| 62 |
+
return [decode_cell(dset.file, dset[0, i]) for i in range(dset.shape[1])]
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def print_info(f, stride, full_stats):
|
| 66 |
+
print('== 变量 ==')
|
| 67 |
+
for k in f.keys():
|
| 68 |
+
if k.startswith('#'):
|
| 69 |
+
continue
|
| 70 |
+
o = f[k]
|
| 71 |
+
line = f'{k:20s} shape={str(o.shape):24s} dtype={o.dtype}'
|
| 72 |
+
if o.compression:
|
| 73 |
+
line += f' compression={o.compression}'
|
| 74 |
+
print(line)
|
| 75 |
+
|
| 76 |
+
n = f['depths'].shape[0]
|
| 77 |
+
h, w = f['depths'].shape[1:]
|
| 78 |
+
names = read_cell_list(f['names'])
|
| 79 |
+
scenes = read_cell_list(f['scenes'])
|
| 80 |
+
scene_types = read_cell_list(f['sceneTypes'])
|
| 81 |
+
|
| 82 |
+
print('\n== 规模 ==')
|
| 83 |
+
print(f'样本数 {n}')
|
| 84 |
+
print(f'分辨率 W={w}, H={h}(data layout 为 [N, C, H, W])')
|
| 85 |
+
print(f'场景数 {len(set(scenes))}')
|
| 86 |
+
print(f'场景类型数 {len(set(scene_types))}')
|
| 87 |
+
print(f'语义类别数 {len(names)}(label id 1..{len(names)},0 为未标注)')
|
| 88 |
+
print('场景类型分布 ' + ', '.join(f'{k}:{v}' for k, v in
|
| 89 |
+
collections.Counter(scene_types).most_common()))
|
| 90 |
+
|
| 91 |
+
idx = range(0, n) if full_stats else range(0, n, stride)
|
| 92 |
+
idx = list(idx)
|
| 93 |
+
print(f'\n== 数值统计({"全量" if full_stats else f"抽样 {len(idx)} 张, step={stride}"}) ==')
|
| 94 |
+
d_min, d_max = np.inf, -np.inf
|
| 95 |
+
raw_missing = 0
|
| 96 |
+
raw_total = 0
|
| 97 |
+
cls_count = collections.Counter()
|
| 98 |
+
inst_max = 0
|
| 99 |
+
for i in tqdm(idx, desc='scan', ncols=80):
|
| 100 |
+
d = f['depths'][i]
|
| 101 |
+
r = f['rawDepths'][i]
|
| 102 |
+
d_min = min(d_min, float(d.min()))
|
| 103 |
+
d_max = max(d_max, float(d.max()))
|
| 104 |
+
raw_missing += int((r == 0).sum())
|
| 105 |
+
raw_total += r.size
|
| 106 |
+
for c in np.unique(f['labels'][i]):
|
| 107 |
+
if c > 0:
|
| 108 |
+
cls_count[int(c)] += 1
|
| 109 |
+
inst_max = max(inst_max, int(f['instances'][i].max()))
|
| 110 |
+
|
| 111 |
+
print(f'depths 范围 [{d_min:.4f}, {d_max:.4f}] m')
|
| 112 |
+
print(f'rawDepths 缺失 {raw_missing / raw_total * 100:.2f}% (值为 0)')
|
| 113 |
+
print(f'出现类别数 {len(cls_count)} / {len(names)}')
|
| 114 |
+
print(f'instances 最大 {inst_max}')
|
| 115 |
+
print('\n最常见类别 ' + ', '.join(
|
| 116 |
+
f'{names[c - 1]}({cnt})' for c, cnt in cls_count.most_common(15)))
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
def depth_to_vis(depth, raw=False):
|
| 120 |
+
lo, hi = DEPTH_VIS_RANGE
|
| 121 |
+
norm = np.clip((depth - lo) / (hi - lo), 0.0, 1.0)
|
| 122 |
+
vis = cv2.applyColorMap((norm * 255).astype(np.uint8), cv2.COLORMAP_TURBO)
|
| 123 |
+
if raw:
|
| 124 |
+
vis[depth <= 0] = 0
|
| 125 |
+
return vis
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
def export_sample(f, index, out_dir, with_labels):
|
| 129 |
+
entry = out_dir / f'{index:05d}_{decode_cell(f, f["scenes"][0, index])}'
|
| 130 |
+
entry.mkdir(parents=True, exist_ok=True)
|
| 131 |
+
|
| 132 |
+
rgb = np.asarray(f['images'][index]).transpose(1, 2, 0) # CHW -> HWC, RGB
|
| 133 |
+
depth = np.asarray(f['depths'][index], dtype=np.float32)
|
| 134 |
+
raw = np.asarray(f['rawDepths'][index], dtype=np.float32)
|
| 135 |
+
|
| 136 |
+
cv2.imwrite(str(entry / 'rgb.png'), cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR))
|
| 137 |
+
cv2.imwrite(str(entry / 'depth_vis.png'), depth_to_vis(depth))
|
| 138 |
+
cv2.imwrite(str(entry / 'raw_depth_vis.png'), depth_to_vis(raw, raw=True))
|
| 139 |
+
np.save(entry / 'depth.npy', depth)
|
| 140 |
+
np.save(entry / 'raw_depth.npy', raw)
|
| 141 |
+
|
| 142 |
+
if with_labels:
|
| 143 |
+
labels = np.asarray(f['labels'][index])
|
| 144 |
+
np.save(entry / 'labels.npy', labels)
|
| 145 |
+
# 语义标签着色:按 label id 的哈希色,0 保持黑色
|
| 146 |
+
rng = np.random.default_rng(0)
|
| 147 |
+
lut = rng.integers(40, 255, size=(65536, 3), dtype=np.uint8)
|
| 148 |
+
lut[0] = 0
|
| 149 |
+
cv2.imwrite(str(entry / 'labels_vis.png'), lut[labels])
|
| 150 |
+
|
| 151 |
+
return entry
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
def main():
|
| 155 |
+
parser = argparse.ArgumentParser(description='查看/导出 NYU Depth V2 labeled 数据集')
|
| 156 |
+
parser.add_argument('--mat', required=True, help='nyu_depth_v2_labeled.mat 路径')
|
| 157 |
+
parser.add_argument('--list-classes', action='store_true', help='打印全部类别名及 id')
|
| 158 |
+
parser.add_argument('--index', type=int, help='导出单个样本的下标')
|
| 159 |
+
parser.add_argument('--out-dir', help='--index 模式的输出目录')
|
| 160 |
+
parser.add_argument('--export-dir', help='批量导出目录')
|
| 161 |
+
parser.add_argument('--count', type=int, help='批量导出数量,默认全部')
|
| 162 |
+
parser.add_argument('--step', type=int, default=1, help='批量导出间隔,默认 1')
|
| 163 |
+
parser.add_argument('--scene', help='只导出指定场景,如 kitchen_0004')
|
| 164 |
+
parser.add_argument('--with-labels', action='store_true', help='同时导出语义标签')
|
| 165 |
+
parser.add_argument('--full-stats', action='store_true', help='统计时扫描全部样本')
|
| 166 |
+
parser.add_argument('--stats-stride', type=int, default=50, help='抽样统计间隔,默认 50')
|
| 167 |
+
args = parser.parse_args()
|
| 168 |
+
|
| 169 |
+
with h5py.File(args.mat, 'r') as f:
|
| 170 |
+
if args.list_classes:
|
| 171 |
+
names = read_cell_list(f['names'])
|
| 172 |
+
for i, name in enumerate(names, 1):
|
| 173 |
+
print(f'{i:4d} {name}')
|
| 174 |
+
return
|
| 175 |
+
|
| 176 |
+
if args.index is not None:
|
| 177 |
+
out_dir = Path(args.out_dir or '.')
|
| 178 |
+
entry = export_sample(f, args.index, out_dir, args.with_labels)
|
| 179 |
+
print(f'已导出 {entry}')
|
| 180 |
+
return
|
| 181 |
+
|
| 182 |
+
if args.export_dir:
|
| 183 |
+
n = f['depths'].shape[0]
|
| 184 |
+
indices = list(range(0, n, args.step))
|
| 185 |
+
if args.scene:
|
| 186 |
+
scenes = read_cell_list(f['scenes'])
|
| 187 |
+
indices = [i for i in indices if scenes[i] == args.scene]
|
| 188 |
+
print(f'场景 {args.scene} 命中 {len(indices)} 张')
|
| 189 |
+
if args.count:
|
| 190 |
+
indices = indices[:args.count]
|
| 191 |
+
out_dir = Path(args.export_dir)
|
| 192 |
+
for i in tqdm(indices, desc='export', ncols=80):
|
| 193 |
+
export_sample(f, i, out_dir, args.with_labels)
|
| 194 |
+
print(f'已导出 {len(indices)} 张到 {out_dir}')
|
| 195 |
+
return
|
| 196 |
+
|
| 197 |
+
print_info(f, args.stats_stride, args.full_stats)
|
| 198 |
+
|
| 199 |
+
|
| 200 |
+
if __name__ == '__main__':
|
| 201 |
+
main()
|
src/tool/make_nyu_calibration.py
ADDED
|
@@ -0,0 +1,112 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
# -*- coding: utf-8 -*-
|
| 3 |
+
"""
|
| 4 |
+
从 NYU 训练集 795 张中按场景均匀抽取量化校准图片。
|
| 5 |
+
|
| 6 |
+
抽样规则:
|
| 7 |
+
- 按场景类型(scene 去掉 _0001 后缀,如 kitchen、bedroom)分组;
|
| 8 |
+
- 每个场景至少 1 张,其余名额按场景图片数比例分配(最大余数法);
|
| 9 |
+
- 场景内按等距索引选取,结果确定可复现。
|
| 10 |
+
|
| 11 |
+
输出:
|
| 12 |
+
<out-dir>/images/*.png 校准图片
|
| 13 |
+
<out-dir>/selection.json 每张图的来源 mat_index / scene / 源文件
|
| 14 |
+
<out-dir>/<name>.tar 给 pulsar2 的校准包,内部为 <name>/xxx.png
|
| 15 |
+
|
| 16 |
+
用法:
|
| 17 |
+
../../venv/bin/python src/tool/make_nyu_calibration.py \
|
| 18 |
+
--source-dir datasets/nyu_train795 --count 48 \
|
| 19 |
+
--out-dir datasets/nyu_calibration_48
|
| 20 |
+
"""
|
| 21 |
+
|
| 22 |
+
import argparse
|
| 23 |
+
import json
|
| 24 |
+
import shutil
|
| 25 |
+
import subprocess
|
| 26 |
+
from collections import defaultdict
|
| 27 |
+
from pathlib import Path
|
| 28 |
+
|
| 29 |
+
import cv2
|
| 30 |
+
import numpy as np
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def scene_type(scene):
|
| 34 |
+
return scene.rsplit('_', 1)[0]
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def allocate(groups, count):
|
| 38 |
+
"""最大余数法分配名额,每个场景至少 1 张。"""
|
| 39 |
+
total = sum(len(v) for v in groups.values())
|
| 40 |
+
if count < len(groups):
|
| 41 |
+
raise ValueError(f'抽取数量 {count} 少于场景数 {len(groups)},无法保证每场景 1 张')
|
| 42 |
+
quota = {k: 1 for k in groups}
|
| 43 |
+
left = count - len(groups)
|
| 44 |
+
exact = {k: left * len(v) / total for k, v in groups.items()}
|
| 45 |
+
for k, v in exact.items():
|
| 46 |
+
quota[k] += int(np.floor(v))
|
| 47 |
+
remain = count - sum(quota.values())
|
| 48 |
+
order = sorted(exact, key=lambda k: exact[k] - np.floor(exact[k]), reverse=True)
|
| 49 |
+
for k in order[:remain]:
|
| 50 |
+
quota[k] += 1
|
| 51 |
+
return quota
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def main():
|
| 55 |
+
parser = argparse.ArgumentParser(description='按场景均匀抽取 NYU 校准集')
|
| 56 |
+
parser.add_argument('--source-dir', required=True, help='export_nyu_train795.py 的输出目录')
|
| 57 |
+
parser.add_argument('--count', type=int, default=48, help='抽取张数,默认 48')
|
| 58 |
+
parser.add_argument('--out-dir', required=True, help='校准集输出目录')
|
| 59 |
+
parser.add_argument('--name', help='tar 包内目录名,默认取 out-dir 名')
|
| 60 |
+
parser.add_argument('--ext', choices=('png', 'jpg'), default='png',
|
| 61 |
+
help='校准图片格式,默认 png;pulsar2 读取异常时可改 jpg')
|
| 62 |
+
args = parser.parse_args()
|
| 63 |
+
|
| 64 |
+
source = Path(args.source_dir)
|
| 65 |
+
split = json.load(open(source / 'split.json'))
|
| 66 |
+
groups = defaultdict(list)
|
| 67 |
+
for record in split:
|
| 68 |
+
groups[scene_type(record['scene'])].append(record)
|
| 69 |
+
|
| 70 |
+
quota = allocate(groups, args.count)
|
| 71 |
+
selected = []
|
| 72 |
+
for scene, records in sorted(groups.items()):
|
| 73 |
+
records = sorted(records, key=lambda r: r['file'])
|
| 74 |
+
idx = np.linspace(0, len(records) - 1, quota[scene]).round().astype(int)
|
| 75 |
+
selected.extend(records[i] for i in sorted(set(idx.tolist())))
|
| 76 |
+
|
| 77 |
+
if len(selected) != args.count:
|
| 78 |
+
raise RuntimeError(f'实际选出 {len(selected)} 张,与目标 {args.count} 不符')
|
| 79 |
+
|
| 80 |
+
out_dir = Path(args.out_dir)
|
| 81 |
+
image_dir = out_dir / 'images'
|
| 82 |
+
image_dir.mkdir(parents=True, exist_ok=True)
|
| 83 |
+
for record in selected:
|
| 84 |
+
src = source / 'images' / record['file']
|
| 85 |
+
if args.ext == 'png':
|
| 86 |
+
dst = image_dir / record['file']
|
| 87 |
+
shutil.copy2(src, dst)
|
| 88 |
+
else:
|
| 89 |
+
dst = image_dir / (Path(record['file']).stem + '.jpg')
|
| 90 |
+
image = cv2.imread(str(src), cv2.IMREAD_COLOR)
|
| 91 |
+
if image is None:
|
| 92 |
+
raise RuntimeError(f'读取失败: {src}')
|
| 93 |
+
cv2.imwrite(str(dst), image, [cv2.IMWRITE_JPEG_QUALITY, 95])
|
| 94 |
+
|
| 95 |
+
with open(out_dir / 'selection.json', 'w') as fp:
|
| 96 |
+
json.dump(selected, fp, indent=2)
|
| 97 |
+
|
| 98 |
+
name = args.name or out_dir.name
|
| 99 |
+
tar_path = out_dir / f'{name}.tar'
|
| 100 |
+
subprocess.run(
|
| 101 |
+
['tar', '-cf', str(tar_path), '--transform', f's,^,{name}/,', '-C', str(image_dir)]
|
| 102 |
+
+ sorted(p.name for p in image_dir.iterdir()),
|
| 103 |
+
check=True)
|
| 104 |
+
|
| 105 |
+
print(f'已抽取 {len(selected)} 张,覆盖 {len(groups)} 个场景类型')
|
| 106 |
+
for scene in sorted(groups):
|
| 107 |
+
print(f' {scene:<16} {quota[scene]:>3} 张 / 共 {len(groups[scene])}')
|
| 108 |
+
print(f'校准包: {tar_path}')
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
if __name__ == '__main__':
|
| 112 |
+
main()
|