PotterWhite commited on
Commit
249ed54
·
1 Parent(s): 1bec881

chore: README improve

Browse files

README.md changed from en -> zh
add frontmatter within it

modified: README.md

Files changed (1) hide show
  1. README.md +270 -246
README.md CHANGED
@@ -1,149 +1,171 @@
1
- # MODNet Model Artifact Registry
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
 
3
- > **Purpose**: Comprehensive catalog of MODNet checkpoints, ONNX models, and training artifacts
4
- >
5
- > **Maintainer**: PotterWhite
6
- > **Last Updated**: 2026-03-31
7
- > **License**: MIT
 
 
8
 
9
  ---
10
 
11
- ## 📋 Table of Contents
12
 
13
- 1. [Official Pretrained Models](#official-pretrained-models)
14
- 2. [Fine-tuned Models (Photographic Dataset)](#fine-tuned-models-photographic-dataset)
15
- 3. [ONNX Model Variants](#onnx-model-variants)
16
- 4. [Directory Structure](#directory-structure)
17
- 5. [Generation & Deployment Guide](#generation--deployment-guide)
 
 
 
18
 
19
  ---
20
 
21
- ## 1. Official Pretrained Models
22
 
23
- ### 1.1 Photographic Portrait Matting
24
 
25
- **File**: `photographic/modnet_photographic_portrait_matting.ckpt`
26
 
27
  ```
28
- Original MODNet checkpoint trained on portrait matting dataset
29
- - Source: Author's Google Drive (ZHKKKe/MODNet)
30
- - Format: PyTorch .ckpt (state_dict)
31
- - Architecture: MODNet with IBNorm + InstanceNormalization
32
- - Input Size: 512×512
33
- - Purpose: Baseline reference for fine-tuning experiments
34
- - Status: Production baseline
35
  ```
36
 
37
- ### 1.2 Webcam Portrait Matting
38
 
39
- **File**: `modnet_webcam_portrait_matting.ckpt`
40
 
41
  ```
42
- MODNet checkpoint optimized for webcam real-time matting
43
- - Source: Author's Google Drive
44
- - Format: PyTorch .ckpt (state_dict)
45
- - Architecture: MODNet with IBNorm + InstanceNormalization
46
- - Input Size: 384×384 (lower latency)
47
- - Purpose: Real-time video / streaming applications
48
- - Status: Available, not actively used in current pipeline
49
  ```
50
 
51
- ### 1.3 MobileNetV2 Human Segmentation
52
 
53
- **File**: `mobilenetv2_human_seg.ckpt`
54
 
55
  ```
56
- Auxiliary segmentation model for preprocessing
57
- - Source: Author's Google Drive
58
- - Format: PyTorch .ckpt
59
- - Purpose: Optional preprocessing stage (not currently deployed)
60
- - Status: Available for reference
61
  ```
62
 
63
  ---
64
 
65
- ## 2. Fine-tuned Models (Photographic Dataset)
66
 
67
- ### 2.1 Pure Batch Normalization Variant
68
 
69
- **Training Run**: Block 1.2 Fine-tuning (2026-03-19 ~ 2026-03-19)
70
 
71
- #### Summary
72
 
73
  ```
74
- Fine-tuned MODNet-BN on P3M-10k photographic dataset
75
- - Replaced all IBNorm + InstanceNormalization with pure BatchNorm2d
76
- - 15-epoch supervised training with learning rate schedule
77
- - Best model achieved: Val L1 Loss 0.0062
78
  ```
79
 
80
- #### Training Configuration
81
-
82
- | Parameter | Value |
83
- |-----------|-------|
84
- | Dataset | P3M-10k (Photographic subset) |
85
- | Train Samples | 9,421 |
86
- | Val Samples | 500 |
87
- | Batch Size | 8 |
88
- | Epochs | 15 |
89
- | Learning Rate (Initial) | 0.01 |
90
- | LR Schedule | StepLR: γ=0.1 @ epoch 5, 10 |
91
- | Input Size | 512×512 |
92
- | Optimizer | Adam (β₁=0.9, β₂=0.999) |
93
- | Loss Function | L1 (MAE) on alpha matte |
94
- | Device | NVIDIA A100 (CUDA 11.8) |
95
- | Training Time | ~4 hours |
96
- | Timestamp | 2026-03-19 15:40:18 |
97
-
98
- #### Artifacts Generated
99
 
100
  ```
101
  photographic/finetune/
102
  ├── checkpoints/
103
- │ ├── modnet_bn_best.ckpt # ★ Best model (Val L1: 0.0062)
104
  │ ├── modnet_bn_epoch_01.ckpt
105
  │ ├── modnet_bn_epoch_02.ckpt
106
- │ ├── ... (epochs 3-14 omitted)
107
  │ └── modnet_bn_epoch_15.ckpt
108
  ├── logs/
109
- │ └── block1_2_training_20260319_154018.log # Training log (detailed)
110
  ├── onnx/
111
- │ └── modnet_bn_best_pureBN.onnx # ★ ONNX export (see §3.3)
112
  └── output/
113
- ├── epoch_01_val.png # Validation preview (epoch 1)
114
  ├── epoch_02_val.png
115
- ├── ... (epochs 3-14 omitted)
116
- └── epoch_15_val.png # Final validation visualization
117
  ```
118
 
119
- #### Validation Loss Curve
120
 
121
  ```
122
- Epoch | Val L1 Loss | Improvement
123
  ------|-------------|-------------------
124
- 1 | 0.0264 | Δ = -0.0202 (new best)
125
- 2 | 0.0175 | Δ = -0.0089 (new best)
126
- 3 | 0.0121 | Δ = -0.0054 (new best)
127
- 4 | 0.0098 | Δ = -0.0023 (new best)
128
- 5 | 0.0089 | Δ = -0.0009 (new best)
129
- 6 | 0.0081 | Δ = -0.0008 (new best)
130
- 7 | 0.0076 | Δ = -0.0005 (new best)
131
- 8 | 0.0074 | Δ = -0.0002 (new best)
132
- 9 | 0.0072 | Δ = -0.0002 (new best)
133
- 10 | 0.0070 | Δ = -0.0002 (new best)
134
- 11 | 0.0068 | Δ = -0.0002 (new best)
135
- 12 | 0.0066 | Δ = -0.0002 (new best)
136
- 13 | 0.0065 | Δ = -0.0001 (new best)
137
- 14 | 0.0063 | Δ = -0.0002 (new best)
138
- 15 | 0.0062 | Δ = -0.0001 (final)
139
-
140
- Converged after epoch 5 (LR schedule kick-in), steady improvement
141
  ```
142
 
143
- #### How to Use
144
 
145
- ```bash
146
- # PyTorch inference
147
  import torch
148
  from modnet import MODNet
149
 
@@ -152,161 +174,161 @@ model = MODNet()
152
  model.load_state_dict(checkpoint)
153
  model.eval()
154
 
155
- # Or ONNX inference (recommended for deployment)
156
  import onnxruntime
157
  sess = onnxruntime.InferenceSession('photographic/finetune/onnx/modnet_bn_best_pureBN.onnx')
158
  ```
159
 
160
  ---
161
 
162
- ## 3. ONNX Model Variants
163
 
164
- ### 3.1 Official Original (Photographic)
165
 
166
- **File**: `photographic/modnet_photographic_portrait_matting.onnx`
167
 
168
  ```
169
- Direct ONNX export from official checkpoint
170
- - Source: Author's Google Drive
171
- - Format: ONNX opset 11
172
- - Contains: InstanceNormalization operations
173
- - Input: [1, 3, 512, 512] (float32, [-1, 1] normalized)
174
- - Output: [1, 1, 512, 512] (float32, [0, 1] range)
175
- - Status: Reference for comparison
176
- - Note: InstanceNormalization CPU fallback on NPU, **not recommended for edge deployment**
177
  ```
178
 
179
- ### 3.2 Folded Variant (Anti-fusion)
180
 
181
- **File**: `photographic/modnet_photographic_portrait_matting_in_folded.onnx`
182
 
183
  ```
184
- InstanceNormalization folded out via anti-fusion method
185
- - Optimizer: PotterWhite (potter_white@outlook.com)
186
- - Date: 2026-03-11 16:11
187
- - Method: Expand InstanceNorm into arithmetic primitives
188
  - Var(x) = E[x²] − (E[x])²
189
- - Prevents RKNN compiler from reconstructing InstanceNormalization
190
- - Forces NPU to execute on CPU (negative effect)
191
- - Status: ⚠️ Experimental, not recommended
192
- - Analysis: Defeats the optimization purpose
193
  ```
194
 
195
- ### 3.3 Pure Batch Normalization (ONNX Export)
196
 
197
- **File**: `photographic/finetune/onnx/modnet_bn_best_pureBN.onnx`
198
 
199
  ```
200
- RECOMMENDED for deployment
201
-
202
- ONNX export from modnet_bn_best.ckpt (fine-tuned model)
203
- - Source: PyTorch fine-tuning run (epoch 15)
204
- - Export Date: 2026-03-31 16:15
205
- - Format: ONNX opset 11
206
- - Architecture: Pure BatchNormalization (no InstanceNorm)
207
- - Input: [1, 3, 512, 512] (float32, [-1, 1] normalized)
208
- - Output: [1, 1, 512, 512] (float32, [0, 1] range)
209
- - File Size: 25 MB
210
- - Status: Production ready for C++ inference
211
-
212
- Why Preferred:
213
- No InstanceNormalization → Better NPU scheduling
214
- All ops: Conv2d, BatchNorm2d, ReLU, etc. (hardware-friendly)
215
- Improved numerical precision on fixed-point inference
216
- Faster compilation on RKNN toolchain
217
- Better convergence than IBNorm variant
218
-
219
- Tested On:
220
- - ONNX Runtime 1.16.3 (CPU, x86_64)
221
- - ONNX Runtime 1.16.3 (aarch64, simulated)
222
- - RKNN toolchain v2.3.2 (compile-stage verification)
223
  ```
224
 
225
- #### Validation Against Reference
226
 
227
  ```
228
- Golden Test Vector: green-fall-girl-point-to.png (1803×1019)
229
- - Python inference output: py_08_inference-Output.bin ✓
230
- - C++ inference output: cpp_08_inference-Output.bin (pending C++ build)
231
- - Expected match: Pixel-wise L∞ error < 1e-5 (float32 precision)
232
  ```
233
 
234
  ---
235
 
236
- ## 4. Directory Structure
237
 
238
  ```
239
  MODNet/
240
 
241
- ├── README.md ← You are here
242
 
243
- ├── [Official Models - Root Level]
244
- │ ├── mobilenetv2_human_seg.ckpt (backup, not active)
245
- │ └── modnet_webcam_portrait_matting.ckpt (reference, 384×384)
246
 
247
- └── photographic/ ← ★ Active deployment variant
248
 
249
- ├── README.md (historical, superseded)
250
 
251
- ├── [Official Baseline]
252
  │ ├── modnet_photographic_portrait_matting.ckpt (1.8 GB)
253
  │ ├── modnet_photographic_portrait_matting.onnx (26 MB, InstanceNorm)
254
  │ └── modnet_photographic_portrait_matting_in_folded.onnx (26 MB, folded)
255
 
256
- └── finetune/ ← ★ Active training output
257
 
258
- ├── checkpoints/ (PyTorch artifacts)
259
- │ ├── modnet_bn_best.ckpt ★ (1.8 GB, best model)
260
  │ ├── modnet_bn_epoch_01.ckpt
261
  │ ├── modnet_bn_epoch_02.ckpt
262
- │ ├── ... (epochs 3-14)
263
  │ └── modnet_bn_epoch_15.ckpt
264
 
265
- ├── onnx/ (Deployment)
266
- │ └── modnet_bn_best_pureBN.onnx ★ (25 MB, RECOMMENDED)
267
 
268
- ├── logs/ (Metadata)
269
  │ └── block1_2_training_20260319_154018.log
270
 
271
- └── output/ (Validation visualization)
272
  ├── epoch_01_val.png
273
  ├── epoch_02_val.png
274
- ├── ... (epochs 3-14)
275
  └── epoch_15_val.png
276
  ```
277
 
278
  ---
279
 
280
- ## 5. Generation & Deployment Guide
281
 
282
- ### 5.1 How This ONNX Was Generated
283
 
284
  ```python
285
- # Step 1: Train fine-tuned checkpoint
286
  # $ cd helmsman.git/
287
  # $ python3 third-party/scripts/modnet/train_modnet_block1_2.py
288
- # → Output: photographic/finetune/checkpoints/modnet_bn_best.ckpt
289
 
290
- # Step 2: Export to ONNX (Pure-BN architecture)
291
  import torch
292
  import onnx
293
- from modnet import MODNet # Pure-BN version
294
 
295
  checkpoint = torch.load('checkpoints/modnet_bn_best.ckpt')
296
  model = MODNet()
297
  model.load_state_dict(checkpoint)
298
  model.eval()
299
 
300
- # Dummy input
301
  dummy_input = torch.randn(1, 3, 512, 512)
302
 
303
- # Export with dynamic axes
304
  torch.onnx.export(
305
- model, dummy_input,
306
  'onnx/modnet_bn_best_pureBN.onnx',
307
  export_params=True,
308
  opset_version=11,
309
- do_constant_folding=False, # Keep BN params visible
310
  input_names=['input'],
311
  output_names=['output'],
312
  dynamic_axes={
@@ -315,124 +337,126 @@ torch.onnx.export(
315
  }
316
  )
317
 
318
- # Step 3: Verify ONNX model
319
  onnx_model = onnx.load('onnx/modnet_bn_best_pureBN.onnx')
320
  onnx.checker.check_model(onnx_model)
321
- print("✓ ONNX model validated")
322
  ```
323
 
324
- ### 5.2 C++ Inference Deployment
325
 
326
  ```bash
327
- # Build C++ inference engine
328
  cd helmsman.git/
329
- ./helmsman prepare # Install Python deps, MODNet submodule
330
- ./helmsman build cpp cb native # Clean build for native x86_64
331
 
332
- # Run inference
333
  ./install/native/release/bin/Helmsman_Matting_Client \
334
  <input_image> \
335
  photographic/finetune/onnx/modnet_bn_best_pureBN.onnx \
336
  <output_dir>
337
 
338
- # Verify against Python golden
339
  python3 tools/MODNet/verify_golden_tensor.py
340
  ```
341
 
342
- ### 5.3 Deployment Checklist
343
 
344
- - [ ] ONNX model validated with `onnx.checker.check_model()`
345
- - [ ] C++ build passes golden tensor verification
346
- - [ ] Python vs C++ inference outputs match (L∞ error < 1e-5)
347
- - [ ] Edge device (RK3588S) cross-compile tested
348
- - [ ] Latency benchmark: <100ms per inference (512×512 input)
349
 
350
  ---
351
 
352
- ## 6. Quick Reference
353
 
354
- | Model | File | Size | Purpose | Status |
355
- |-------|------|------|---------|--------|
356
- | **Official Photographic** | `photographic/modnet_photographic_portrait_matting.ckpt` | 1.8 GB | Baseline reference | ✓ Reference |
357
- | **Official ONNX** | `photographic/modnet_photographic_portrait_matting.onnx` | 26 MB | InstanceNorm variant | ⚠️ Not recommended |
358
- | **Fine-tuned (Best)** | `photographic/finetune/checkpoints/modnet_bn_best.ckpt` | 1.8 GB | PyTorch deployment | ✓ Production |
359
- | **Fine-tuned ONNX** | `photographic/finetune/onnx/modnet_bn_best_pureBN.onnx` | 25 MB | C++/RKNN deployment | ★ **RECOMMENDED** |
360
- | **Webcam Model** | `modnet_webcam_portrait_matting.ckpt` | 1.8 GB | Real-time streaming | ✓ Available |
361
 
362
  ---
363
 
364
- ## 7. RobustVideoMatting (RVM) Models
365
 
366
- ### 7.1 ONNX Models
367
 
368
- Located in `RobustVideoMatting/onnx/`.
369
 
370
- | Model | File | Size | Backbone | Refiner | Source |
371
- |-------|------|------|----------|---------|--------|
372
- | **MobileNetV3 FP32** | `rvm_mobilenetv3_fp32.onnx` | 14.3MB | MobileNetV3 | No | PeterL1n/RobustVideoMatting v1.0.0 |
373
- | **MobileNetV3 FP16** | `rvm_mobilenetv3_fp16.onnx` | 7.2MB | MobileNetV3 | No | FP16 export |
374
- | **MobileNetV3 FP32 (no refiner)** | `rvm_mobilenetv3_fp32_no_refiner.onnx` | 14.3MB | MobileNetV3 | No | Stripped refiner ops |
375
- | **ResNet50 FP32** | `rvm_resnet50_fp32.onnx` | ~90MB | ResNet50 | No | PeterL1n v1.0.0 |
376
- | **ResNet50 FP16** | `rvm_resnet50_fp16.onnx` | ~45MB | ResNet50 | No | FP16 export |
377
 
378
- **Key model properties** (MobileNetV3):
379
- - Opset 12, IR version 6, 353 nodes
380
- - Zero InstanceNorm nodes (NPU-friendly)
381
- - No refiner ops (DeepGuidedFilterRefiner not included)
382
- - 6 inputs: `src [1,3,H,W]`, `r1i~r4i` (ConvGRU states), `downsample_ratio [1]`
383
- - After ArcFoundry `fold_constant_inputs`: 5 inputs (downsample_ratio baked as constant)
384
 
385
- ### 7.2 RKNN Models
386
 
387
- Located in `RobustVideoMatting/rknn/`.
388
 
389
- | Model | File | Size | Resolution | Precision | dsr | Status | PKB Ref |
390
- |-------|------|------|-----------|-----------|-----|--------|---------|
391
  | **256×256 FP16** | `rvm_mobilenetv3_fp16_256x256_rk3588_fp16_v20260416.rknn` | 9.0MB | 256×256 | FP16 | 0.25 | Phase-1 验证用 | §3 |
392
- | **288×512 FP16** | `rvm_mobilenetv3_fp16_288x512_rk3588_fp16_v20260416.rknn` | 9.3MB | 288×512 | FP16 | 0.25 | 早期 board 实验 | §3 |
393
  | **1080p FP16 (dsr=0.25)** | `rvm_mobilenetv3_fp16_1080x1920_rk3588_fp16.rknn` | 12MB | 1080×1920 | FP16 | 0.25 | **当前默认模型** | §12-§13, §20 |
394
  | **1080p FP16 (dsr=0.5)** | `rvm_mobilenetv3_fp16_1080x1920_0.5-dsr_rk3588_fp16.rknn` | 16MB | 1080×1920 | FP16 | 0.5 | **最佳质量模型** | §21 |
395
  | **256×256 INT8** | `rvm_mobilenetv3_int8_256x256_rk3588_int8_v20260416.rknn` | 5.5MB | 256×256 | INT8 | 0.25 | Phase-4 量化实验 | §4 |
396
  | **288×512 INT8** | `rvm_mobilenetv3_int8_288x512_rk3588_int8_v20260416.rknn` | 5.7MB | 288×512 | INT8 | 0.25 | Phase-4 量化实验 | §4 |
397
 
398
- **Conversion tool**: [ArcFoundry](https://github.com/PotterWhite/ArcFoundry.git) v0.14.0
399
- - Config directory: `ArcFoundry.git/configs/rvm/`
400
- - Target platform: RK3588
401
- - Normalization: `mean=[0,0,0], std=[255,255,255]` (RKNN runtime applies `/255.0`)
402
 
403
- ### 7.3 Quality Summary (s14/s20/s21 board experiments)
404
 
405
- | Model | dsr | mean_diff vs PyTorch | bg_ratio | Speed (infer+composite) | Notes |
406
- |-------|-----|---------------------|----------|------------------------|-------|
407
- | 1080p FP16 (s14) | 0.25 | 2.30 | 86.6% | ~485ms | NCHW→NHWC transpose fix |
408
  | 1080p FP16 (s20) | 0.25 | 2.30 | 86.6% | ~485ms | r2o stride padding: OK |
409
- | 1080p FP16 (s21) | 0.5 | **2.18** | **86.5%** | ~691ms | **Best quality** |
 
 
 
 
410
 
411
- - PyTorch baseline bg_ratio: 86.6% (dsr=0.25), 86.5% (dsr=0.5)
412
- - All models tested on `dance.mp4` (363 frames, 1920×1080)
413
- - Board: RK3588S, NPU0=40%, NPU1&2=10%
414
 
415
- ### 7.4 Recommended Model Selection
 
 
 
 
416
 
417
- | Use Case | Recommended Model | Reason |
418
- |----------|------------------|--------|
419
- | **Quality-first** | `rvm_mobilenetv3_fp16_1080x1920_0.5-dsr_rk3588_fp16.rknn` | mean_diff=2.18, best alpha edges |
420
- | **Speed-first** | `rvm_mobilenetv3_fp16_1080x1920_rk3588_fp16.rknn` | 1.43× faster than dsr=0.5 |
421
- | **Low memory** | `rvm_mobilenetv3_fp16_288x512_rk3588_fp16_v20260416.rknn` | Smallest r-state footprint |
422
 
423
- ## 8. Related Documentation
424
 
425
- - **Training Script**: `helmsman.git/third-party/scripts/modnet/train_modnet_block1_2.py`
426
- - **ONNX Export Script**: `helmsman.git/third-party/scripts/modnet/onnx/export_onnx_pureBN.py`
427
- - **C++ Inference**: `helmsman.git/runtime/cpp/apps/matting/client/`
428
- - **Python Golden Reference**: `helmsman.git/third-party/scripts/modnet/onnx/generate_golden_files.py`
429
- - **Verification**: `helmsman.git/tools/MODNet/verify_golden_tensor.py`
430
- - **RVM PKB**: `/volumes_pkb_helmsman/model/round2-rvm/log-MR2-P5-rknn-inference.md`
431
- - **RVM ArcFoundry configs**: `ArcFoundry.git/configs/rvm/`
432
 
433
  ---
434
 
435
- ## Appendix: Training Log Summary
436
 
437
  ```
438
  [Config] Device: cuda
@@ -454,6 +478,6 @@ Overfitting: ✓ No significant degradation, clean convergence
454
 
455
  ---
456
 
457
- **Document Version**: 1.0
458
- **Last Updated**: 2026-03-31 by Claude Code (AI Agent)
459
- **Commit History**: Will be tracked via Git commit message
 
1
+ ---
2
+ language:
3
+ - en
4
+ - zh
5
+ tags:
6
+ - image-matting
7
+ - portrait-segmentation
8
+ - modnet
9
+ - onnx
10
+ - rknn
11
+ - rk3588
12
+ - video-matting
13
+ - robust-video-matting
14
+ license: mit
15
+ library_name: pytorch
16
+ base_model: ZHKKKe/MODNet
17
+ pipeline_tag: image-segmentation
18
+ ---
19
 
20
+ # MODNet 模型制品仓库 (Model Artifact Registry)
21
+
22
+ > **用途**:集中管理 MODNet 检查点(Checkpoint)、ONNX 模型及训练制品(Training Artifacts)
23
+ >
24
+ > **维护者**:PotterWhite
25
+ > **最后更新**:2026-03-31
26
+ > **许可证(License)**:MIT
27
 
28
  ---
29
 
30
+ ## 目录(Table of Contents
31
 
32
+ 1. [官方预训练模型(Official Pretrained Models](#1-官方预训练模型official-pretrained-models)
33
+ 2. [微调模型(Fine-tuned Models)—— 摄影数据集](#2-微调模型fine-tuned-models-摄影数据集)
34
+ 3. [ONNX 模型变体(ONNX Model Variants](#3-onnx-模型变体onnx-model-variants)
35
+ 4. [目录结构(Directory Structure](#4-目录结构directory-structure)
36
+ 5. [生成与部署指南(Generation & Deployment Guide](#5-生成与部署指南generation--deployment-guide)
37
+ 6. [速查表(Quick Reference)](#6-速查表quick-reference)
38
+ 7. [RobustVideoMatting(RVM)模型](#7-robustvideoMattingrvm模型)
39
+ 8. [相关文档(Related Documentation)](#8-相关文档related-documentation)
40
 
41
  ---
42
 
43
+ ## 1. 官方预训练模型(Official Pretrained Models
44
 
45
+ ### 1.1 摄影人像抠图(Photographic Portrait Matting
46
 
47
+ **文件**`photographic/modnet_photographic_portrait_matting.ckpt`
48
 
49
  ```
50
+ 原始 MODNet 检查点,在人像抠图数据集上训练
51
+ - 来源:作者 Google DriveZHKKKe/MODNet
52
+ - 格式:PyTorch .ckptstate_dict
53
+ - 架构:MODNet + IBNorm + InstanceNormalization
54
+ - 输入尺寸:512×512
55
+ - 用途:微调实验的基线参考
56
+ - 状态:生产基线
57
  ```
58
 
59
+ ### 1.2 摄像头人像抠图(Webcam Portrait Matting
60
 
61
+ **文件**`modnet_webcam_portrait_matting.ckpt`
62
 
63
  ```
64
+ 针对摄像头实时抠图优化的 MODNet 检查点
65
+ - 来源:作者 Google Drive
66
+ - 格式:PyTorch .ckptstate_dict
67
+ - 架构:MODNet + IBNorm + InstanceNormalization
68
+ - 输入尺寸:384×384(更低延迟)
69
+ - 用途:实时视频/直播场景
70
+ - 状态:可用,当前管线未使用
71
  ```
72
 
73
+ ### 1.3 MobileNetV2 人体分割(Human Segmentation
74
 
75
+ **文件**`mobilenetv2_human_seg.ckpt`
76
 
77
  ```
78
+ 辅助分割模型,用于预处理阶段
79
+ - 来源:作者 Google Drive
80
+ - 格式:PyTorch .ckpt
81
+ - 用途:可选预处理阶段(当前未部署)
82
+ - 状态:可用作参考
83
  ```
84
 
85
  ---
86
 
87
+ ## 2. 微调模型(Fine-tuned Models)—— 摄影数据集
88
 
89
+ ### 2.1 纯批归一化变体(Pure Batch Normalization Variant
90
 
91
+ **训练轮次**Block 1.2 微调(2026-03-19 ~ 2026-03-19
92
 
93
+ #### 概要
94
 
95
  ```
96
+ P3M-10k 摄影数据集上微调 MODNet-BN
97
+ - 将所有 IBNorm + InstanceNormalization 替换为纯 BatchNorm2d
98
+ - 15 Epoch 的监督训练,含学习率调度(Learning Rate Schedule)
99
+ - 最佳模型:验证集 L1 Loss 0.0062
100
  ```
101
 
102
+ #### 训练配置
103
+
104
+ | 参数 | |
105
+ |------|-----|
106
+ | 数据集(Dataset | P3M-10kPhotographic 子集) |
107
+ | 训练样本数 | 9,421 |
108
+ | 验证样本数 | 500 |
109
+ | 批大小(Batch Size | 8 |
110
+ | 轮次(Epochs | 15 |
111
+ | 初始学习率(Learning Rate | 0.01 |
112
+ | 学习率调度 | StepLRγ=0.1 @ epoch 5, 10 |
113
+ | 输入尺寸 | 512×512 |
114
+ | 优化器(Optimizer | Adam (β₁=0.9, β₂=0.999) |
115
+ | 损失函数(Loss Function | L1MAE),作用于 alpha 遮罩 |
116
+ | 设备(Device | NVIDIA A100 (CUDA 11.8) |
117
+ | 训练时长 | ~4 小时 |
118
+ | 时间戳 | 2026-03-19 15:40:18 |
119
+
120
+ #### 生成的制品(Artifacts
121
 
122
  ```
123
  photographic/finetune/
124
  ├── checkpoints/
125
+ │ ├── modnet_bn_best.ckpt # ★ 最佳模型(Val L1: 0.0062
126
  │ ├── modnet_bn_epoch_01.ckpt
127
  │ ├── modnet_bn_epoch_02.ckpt
128
+ │ ├── ...(epoch 3-14 省略)
129
  │ └── modnet_bn_epoch_15.ckpt
130
  ├── logs/
131
+ │ └── block1_2_training_20260319_154018.log # 训练日志(详细)
132
  ├── onnx/
133
+ │ └── modnet_bn_best_pureBN.onnx # ★ ONNX 导出(见 §3.3
134
  └── output/
135
+ ├── epoch_01_val.png # 验证预览(第 1 轮)
136
  ├── epoch_02_val.png
137
+ ├── ...(epoch 3-14 省略)
138
+ └── epoch_15_val.png # 最终验证可视化
139
  ```
140
 
141
+ #### 验证损失曲线(Validation Loss Curve
142
 
143
  ```
144
+ Epoch | Val L1 Loss | 改进幅度
145
  ------|-------------|-------------------
146
+ 1 | 0.0264 | Δ = -0.0202(新最佳)
147
+ 2 | 0.0175 | Δ = -0.0089(新最佳)
148
+ 3 | 0.0121 | Δ = -0.0054(新最佳)
149
+ 4 | 0.0098 | Δ = -0.0023(新最佳)
150
+ 5 | 0.0089 | Δ = -0.0009(新最佳)
151
+ 6 | 0.0081 | Δ = -0.0008(新最佳)
152
+ 7 | 0.0076 | Δ = -0.0005(新最佳)
153
+ 8 | 0.0074 | Δ = -0.0002(新最佳)
154
+ 9 | 0.0072 | Δ = -0.0002(新最佳)
155
+ 10 | 0.0070 | Δ = -0.0002(新最佳)
156
+ 11 | 0.0068 | Δ = -0.0002(新最佳)
157
+ 12 | 0.0066 | Δ = -0.0002(新最佳)
158
+ 13 | 0.0065 | Δ = -0.0001(新最佳)
159
+ 14 | 0.0063 | Δ = -0.0002(新最佳)
160
+ 15 | 0.0062 | Δ = -0.0001(最终)
161
+
162
+ 5 轮后收敛(学习率调度生效),持续稳步改进
163
  ```
164
 
165
+ #### 使用方法
166
 
167
+ ```python
168
+ # PyTorch 推理
169
  import torch
170
  from modnet import MODNet
171
 
 
174
  model.load_state_dict(checkpoint)
175
  model.eval()
176
 
177
+ # 或使用 ONNX 推理(推荐用于部署)
178
  import onnxruntime
179
  sess = onnxruntime.InferenceSession('photographic/finetune/onnx/modnet_bn_best_pureBN.onnx')
180
  ```
181
 
182
  ---
183
 
184
+ ## 3. ONNX 模型变体(ONNX Model Variants
185
 
186
+ ### 3.1 官方原始版本(Photographic
187
 
188
+ **文件**`photographic/modnet_photographic_portrait_matting.onnx`
189
 
190
  ```
191
+ 从官方检查点直接导出的 ONNX
192
+ - 来源:作者 Google Drive
193
+ - 格式:ONNX opset 11
194
+ - 包含:InstanceNormalization 算子
195
+ - 输入:[1, 3, 512, 512]float32[-1, 1] 归一化)
196
+ - 输出:[1, 1, 512, 512]float32[0, 1] 范围)
197
+ - 状态:对比参考
198
+ - 注意:InstanceNormalization NPU 上会回退到 CPU,**不推荐用于边缘部署**
199
  ```
200
 
201
+ ### 3.2 折叠变体(Folded VariantAnti-fusion
202
 
203
+ **文件**`photographic/modnet_photographic_portrait_matting_in_folded.onnx`
204
 
205
  ```
206
+ 通过 anti-fusion 方法展开 InstanceNormalization
207
+ - 优化者:PotterWhite (potter_white@outlook.com)
208
+ - 日期:2026-03-11 16:11
209
+ - 方法:将 InstanceNorm 展开为算术原语
210
  - Var(x) = E[x²] − (E[x])²
211
+ - 防止 RKNN 编译器重新识别 InstanceNormalization
212
+ - 强制在 CPU 上执行(负面效果)
213
+ - 状态:⚠️ 实验性,不推荐
214
+ - 分析:违背了优化初衷
215
  ```
216
 
217
+ ### 3.3 纯批归一化版本(ONNX 导出)
218
 
219
+ **文件**`photographic/finetune/onnx/modnet_bn_best_pureBN.onnx`
220
 
221
  ```
222
+ 推荐用于部署
223
+
224
+ modnet_bn_best.ckpt(微调模型)导出的 ONNX
225
+ - 来源:PyTorch 微调训练(第 15 轮)
226
+ - 导出日期:2026-03-31 16:15
227
+ - 格式:ONNX opset 11
228
+ - 架构:纯 BatchNormalization(无 InstanceNorm
229
+ - 输入:[1, 3, 512, 512]float32[-1, 1] 归一化)
230
+ - 输出:[1, 1, 512, 512]float32[0, 1] 范围)
231
+ - 文件大小:25 MB
232
+ - 状态:可用于 C++ 推理的生产版本
233
+
234
+ 推荐理由:
235
+ InstanceNormalization → 更好的 NPU 调度
236
+ 全部算子:Conv2d, BatchNorm2d, ReLU 等(硬件友好)
237
+ 定点推理下数值精度更优
238
+ ✓ RKNN 工具链编译更快
239
+ IBNorm 变体收敛更好
240
+
241
+ 已测试环境:
242
+ - ONNX Runtime 1.16.3CPU, x86_64
243
+ - ONNX Runtime 1.16.3aarch64, 模拟环境)
244
+ - RKNN 工具链 v2.3.2(编译阶段验证)
245
  ```
246
 
247
+ #### 验证结果(对比参考)
248
 
249
  ```
250
+ 黄金测试向量(Golden Test Vector):green-fall-girl-point-to.png (1803×1019)
251
+ - Python 推理输出:py_08_inference-Output.bin ✓
252
+ - C++ 推理输出:cpp_08_inference-Output.bin(待 C++ 构建)
253
+ - 预期匹配:像素级 L∞ 误差 < 1e-5float32 精度)
254
  ```
255
 
256
  ---
257
 
258
+ ## 4. 目录结构(Directory Structure
259
 
260
  ```
261
  MODNet/
262
 
263
+ ├── README.md ← 当前文件
264
 
265
+ ├── [官方模型 - 根目录]
266
+ │ ├── mobilenetv2_human_seg.ckpt (备份,非活跃)
267
+ │ └── modnet_webcam_portrait_matting.ckpt (参考用,384×384
268
 
269
+ └── photographic/ ← ★ 活跃部署变体
270
 
271
+ ├── README.md (历史文件,已被取代)
272
 
273
+ ├── [官方基线]
274
  │ ├── modnet_photographic_portrait_matting.ckpt (1.8 GB)
275
  │ ├── modnet_photographic_portrait_matting.onnx (26 MB, InstanceNorm)
276
  │ └── modnet_photographic_portrait_matting_in_folded.onnx (26 MB, folded)
277
 
278
+ └── finetune/ ← ★ 活跃训练输出
279
 
280
+ ├── checkpoints/ PyTorch 制品)
281
+ │ ├── modnet_bn_best.ckpt ★ (1.8 GB, 最佳模型)
282
  │ ├── modnet_bn_epoch_01.ckpt
283
  │ ├── modnet_bn_epoch_02.ckpt
284
+ │ ├── ... (epoch 3-14)
285
  │ └── modnet_bn_epoch_15.ckpt
286
 
287
+ ├── onnx/ (部署用)
288
+ │ └── modnet_bn_best_pureBN.onnx ★ (25 MB, 推荐)
289
 
290
+ ├── logs/ (元数据)
291
  │ └── block1_2_training_20260319_154018.log
292
 
293
+ └── output/ (验证可视化)
294
  ├── epoch_01_val.png
295
  ├── epoch_02_val.png
296
+ ├── ... (epoch 3-14)
297
  └── epoch_15_val.png
298
  ```
299
 
300
  ---
301
 
302
+ ## 5. 生成与部署指南(Generation & Deployment Guide
303
 
304
+ ### 5.1 ONNX 生成方法
305
 
306
  ```python
307
+ # 1 步:训练微调检查点
308
  # $ cd helmsman.git/
309
  # $ python3 third-party/scripts/modnet/train_modnet_block1_2.py
310
+ # → 输出:photographic/finetune/checkpoints/modnet_bn_best.ckpt
311
 
312
+ # 2 步:导出为 ONNXPure-BN 架构)
313
  import torch
314
  import onnx
315
+ from modnet import MODNet # Pure-BN 版本
316
 
317
  checkpoint = torch.load('checkpoints/modnet_bn_best.ckpt')
318
  model = MODNet()
319
  model.load_state_dict(checkpoint)
320
  model.eval()
321
 
322
+ # 虚拟输入(Dummy Input)
323
  dummy_input = torch.randn(1, 3, 512, 512)
324
 
325
+ # 导出,支持动态轴(Dynamic Axes)
326
  torch.onnx.export(
327
+ model, dummy_input,
328
  'onnx/modnet_bn_best_pureBN.onnx',
329
  export_params=True,
330
  opset_version=11,
331
+ do_constant_folding=False, # 保留 BN 参数可见
332
  input_names=['input'],
333
  output_names=['output'],
334
  dynamic_axes={
 
337
  }
338
  )
339
 
340
+ # 3 步:验证 ONNX 模型
341
  onnx_model = onnx.load('onnx/modnet_bn_best_pureBN.onnx')
342
  onnx.checker.check_model(onnx_model)
343
+ print("✓ ONNX 模型验证通过")
344
  ```
345
 
346
+ ### 5.2 C++ 推理部署
347
 
348
  ```bash
349
+ # 构建 C++ 推理引擎
350
  cd helmsman.git/
351
+ ./helmsman prepare # 安装 Python 依赖、MODNet 子模块
352
+ ./helmsman build cpp cb native # 清理构建(x86_64 原生)
353
 
354
+ # 运行推理
355
  ./install/native/release/bin/Helmsman_Matting_Client \
356
  <input_image> \
357
  photographic/finetune/onnx/modnet_bn_best_pureBN.onnx \
358
  <output_dir>
359
 
360
+ # 对比 Python 黄金参考
361
  python3 tools/MODNet/verify_golden_tensor.py
362
  ```
363
 
364
+ ### 5.3 部署清单(Deployment Checklist
365
 
366
+ - [ ] ONNX 模型通过 `onnx.checker.check_model()` 验证
367
+ - [ ] C++ 构建通过黄金张量验证(Golden Tensor Verification)
368
+ - [ ] Python C++ 推理输出匹配(L∞ 误差 < 1e-5
369
+ - [ ] 边缘设备(RK3588S)交叉编译测试通过
370
+ - [ ] 延迟基准测试:每次推理 < 100ms(512×512 输入)
371
 
372
  ---
373
 
374
+ ## 6. 速查表(Quick Reference
375
 
376
+ | 模型 | 文件 | 大小 | 用途 | 状态 |
377
+ |------|------|------|------|------|
378
+ | **官方摄影模型** | `photographic/modnet_photographic_portrait_matting.ckpt` | 1.8 GB | 基线参考 | ✓ 参考 |
379
+ | **官方 ONNX** | `photographic/modnet_photographic_portrait_matting.onnx` | 26 MB | InstanceNorm 变体 | ⚠️ 不推荐 |
380
+ | **微调最佳模型** | `photographic/finetune/checkpoints/modnet_bn_best.ckpt` | 1.8 GB | PyTorch 部署 | ✓ 生产 |
381
+ | **微调 ONNX** | `photographic/finetune/onnx/modnet_bn_best_pureBN.onnx` | 25 MB | C++/RKNN 部署 | ★ **推荐** |
382
+ | **摄像头模型** | `modnet_webcam_portrait_matting.ckpt` | 1.8 GB | 实时流媒体 | ✓ 可用 |
383
 
384
  ---
385
 
386
+ ## 7. RobustVideoMattingRVM)模型
387
 
388
+ ### 7.1 ONNX 模型
389
 
390
+ 位于 `RobustVideoMatting/onnx/`
391
 
392
+ | 模型 | 文件 | 大小 | 骨干网络(Backbone | 精炼器(Refiner | 来源 |
393
+ |------|------|------|----------|---------|------|
394
+ | **MobileNetV3 FP32** | `rvm_mobilenetv3_fp32.onnx` | 14.3MB | MobileNetV3 | | PeterL1n/RobustVideoMatting v1.0.0 |
395
+ | **MobileNetV3 FP16** | `rvm_mobilenetv3_fp16.onnx` | 7.2MB | MobileNetV3 | | FP16 导出 |
396
+ | **MobileNetV3 FP32(无精炼器)** | `rvm_mobilenetv3_fp32_no_refiner.onnx` | 14.3MB | MobileNetV3 | | 移除 refiner 算子 |
397
+ | **ResNet50 FP32** | `rvm_resnet50_fp32.onnx` | ~90MB | ResNet50 | | PeterL1n v1.0.0 |
398
+ | **ResNet50 FP16** | `rvm_resnet50_fp16.onnx` | ~45MB | ResNet50 | | FP16 导出 |
399
 
400
+ **关键模型属性**MobileNetV3):
401
+ - Opset 12IR version 6353 个节点
402
+ - 零个 InstanceNorm 节点(NPU 友好)
403
+ - 无精炼器算子(DeepGuidedFilterRefiner 未包含)
404
+ - 6 个输入:`src [1,3,H,W]``r1i~r4i`ConvGRU 状态)、`downsample_ratio [1]`
405
+ - ArcFoundry `fold_constant_inputs` 处理后:5 个输入(downsample_ratio 烘焙为常量)
406
 
407
+ ### 7.2 RKNN 模型
408
 
409
+ 位于 `RobustVideoMatting/rknn/`
410
 
411
+ | 模型 | 文件 | 大小 | 分辨率 | 精度 | dsr | 状态 | PKB 参考 |
412
+ |------|------|------|--------|------|-----|------|----------|
413
  | **256×256 FP16** | `rvm_mobilenetv3_fp16_256x256_rk3588_fp16_v20260416.rknn` | 9.0MB | 256×256 | FP16 | 0.25 | Phase-1 验证用 | §3 |
414
+ | **288×512 FP16** | `rvm_mobilenetv3_fp16_288x512_rk3588_fp16_v20260416.rknn` | 9.3MB | 288×512 | FP16 | 0.25 | 早期板端实验 | §3 |
415
  | **1080p FP16 (dsr=0.25)** | `rvm_mobilenetv3_fp16_1080x1920_rk3588_fp16.rknn` | 12MB | 1080×1920 | FP16 | 0.25 | **当前默认模型** | §12-§13, §20 |
416
  | **1080p FP16 (dsr=0.5)** | `rvm_mobilenetv3_fp16_1080x1920_0.5-dsr_rk3588_fp16.rknn` | 16MB | 1080×1920 | FP16 | 0.5 | **最佳质量模型** | §21 |
417
  | **256×256 INT8** | `rvm_mobilenetv3_int8_256x256_rk3588_int8_v20260416.rknn` | 5.5MB | 256×256 | INT8 | 0.25 | Phase-4 量化实验 | §4 |
418
  | **288×512 INT8** | `rvm_mobilenetv3_int8_288x512_rk3588_int8_v20260416.rknn` | 5.7MB | 288×512 | INT8 | 0.25 | Phase-4 量化实验 | §4 |
419
 
420
+ **转换工具**[ArcFoundry](https://github.com/PotterWhite/ArcFoundry.git) v0.14.0
421
+ - 配置目录:`ArcFoundry.git/configs/rvm/`
422
+ - 目标平台:RK3588
423
+ - 归一化:`mean=[0,0,0], std=[255,255,255]`RKNN 运行时执行 `/255.0`
424
 
425
+ ### 7.3 质量总结(s14/s20/s21 板端实验)
426
 
427
+ | 模型 | dsr | PyTorch 的 mean_diff | bg_ratio | 速度(推理+合成) | 备注 |
428
+ |------|-----|------------------------|----------|------------------|------|
429
+ | 1080p FP16 (s14) | 0.25 | 2.30 | 86.6% | ~485ms | NCHW→NHWC 转置修复 |
430
  | 1080p FP16 (s20) | 0.25 | 2.30 | 86.6% | ~485ms | r2o stride padding: OK |
431
+ | 1080p FP16 (s21) | 0.5 | **2.18** | **86.5%** | ~691ms | **最佳质量** |
432
+
433
+ - PyTorch 基线 bg_ratio:86.6%(dsr=0.25)、86.5%(dsr=0.5)
434
+ - 所有模型在 `dance.mp4` 上测试(363 帧,1920×1080)
435
+ - 板端:RK3588S,NPU0=40%,NPU1&2=10%
436
 
437
+ ### 7.4 推荐模型选择
 
 
438
 
439
+ | 使用场景 | 推荐模型 | 理由 |
440
+ |----------|---------|------|
441
+ | **质量优先** | `rvm_mobilenetv3_fp16_1080x1920_0.5-dsr_rk3588_fp16.rknn` | mean_diff=2.18,最佳 alpha 边缘 |
442
+ | **速度优先** | `rvm_mobilenetv3_fp16_1080x1920_rk3588_fp16.rknn` | 比 dsr=0.5 快 1.43 倍 |
443
+ | **低内存** | `rvm_mobilenetv3_fp16_288x512_rk3588_fp16_v20260416.rknn` | 最小 r-state 占用 |
444
 
445
+ ---
 
 
 
 
446
 
447
+ ## 8. 相关文档(Related Documentation
448
 
449
+ - **训练脚本**`helmsman.git/third-party/scripts/modnet/train_modnet_block1_2.py`
450
+ - **ONNX 导出脚本**`helmsman.git/third-party/scripts/modnet/onnx/export_onnx_pureBN.py`
451
+ - **C++ 推理**`helmsman.git/runtime/cpp/apps/matting/client/`
452
+ - **Python 黄金参考**`helmsman.git/third-party/scripts/modnet/onnx/generate_golden_files.py`
453
+ - **验证工具**`helmsman.git/tools/MODNet/verify_golden_tensor.py`
454
+ - **RVM PKB**`/volumes_pkb_helmsman/model/round2-rvm/log-MR2-P5-rknn-inference.md`
455
+ - **RVM ArcFoundry 配置**`ArcFoundry.git/configs/rvm/`
456
 
457
  ---
458
 
459
+ ## 附录:训练日志摘要
460
 
461
  ```
462
  [Config] Device: cuda
 
478
 
479
  ---
480
 
481
+ **文档版本**1.1
482
+ **最后更新**2026-05-06 by Claude Code (AI Agent)
483
+ **提交历史**:通过 Git commit message 追踪