PotterWhite Claude Opus 4.7 commited on
Commit
3d6f7a4
·
1 Parent(s): 0455bad

docs: add RVM model registry (ONNX + RKNN) with quality summary

Browse files

- §7.1: 5 ONNX variants (MobileNetV3/ResNet50, FP32/FP16, with/without refiner)
- §7.2: 6 RKNN models (256/288/1080p, FP16/INT8, dsr=0.25/0.5)
- §7.3: Quality summary from s14/s20/s21 board experiments
- §7.4: Recommended model selection guide

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Files changed (1) hide show
  1. README.md +62 -1
README.md CHANGED
@@ -361,13 +361,74 @@ python3 tools/MODNet/verify_golden_tensor.py
361
 
362
  ---
363
 
364
- ## 7. Related Documentation
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
365
 
366
  - **Training Script**: `helmsman.git/third-party/scripts/modnet/train_modnet_block1_2.py`
367
  - **ONNX Export Script**: `helmsman.git/third-party/scripts/modnet/onnx/export_onnx_pureBN.py`
368
  - **C++ Inference**: `helmsman.git/runtime/cpp/apps/matting/client/`
369
  - **Python Golden Reference**: `helmsman.git/third-party/scripts/modnet/onnx/generate_golden_files.py`
370
  - **Verification**: `helmsman.git/tools/MODNet/verify_golden_tensor.py`
 
 
371
 
372
  ---
373
 
 
361
 
362
  ---
363
 
364
+ ## 7. RobustVideoMatting (RVM) Models
365
+
366
+ ### 7.1 ONNX Models
367
+
368
+ Located in `RobustVideoMatting/onnx/`.
369
+
370
+ | Model | File | Size | Backbone | Refiner | Source |
371
+ |-------|------|------|----------|---------|--------|
372
+ | **MobileNetV3 FP32** | `rvm_mobilenetv3_fp32.onnx` | 14.3MB | MobileNetV3 | No | PeterL1n/RobustVideoMatting v1.0.0 |
373
+ | **MobileNetV3 FP16** | `rvm_mobilenetv3_fp16.onnx` | 7.2MB | MobileNetV3 | No | FP16 export |
374
+ | **MobileNetV3 FP32 (no refiner)** | `rvm_mobilenetv3_fp32_no_refiner.onnx` | 14.3MB | MobileNetV3 | No | Stripped refiner ops |
375
+ | **ResNet50 FP32** | `rvm_resnet50_fp32.onnx` | ~90MB | ResNet50 | No | PeterL1n v1.0.0 |
376
+ | **ResNet50 FP16** | `rvm_resnet50_fp16.onnx` | ~45MB | ResNet50 | No | FP16 export |
377
+
378
+ **Key model properties** (MobileNetV3):
379
+ - Opset 12, IR version 6, 353 nodes
380
+ - Zero InstanceNorm nodes (NPU-friendly)
381
+ - No refiner ops (DeepGuidedFilterRefiner not included)
382
+ - 6 inputs: `src [1,3,H,W]`, `r1i~r4i` (ConvGRU states), `downsample_ratio [1]`
383
+ - After ArcFoundry `fold_constant_inputs`: 5 inputs (downsample_ratio baked as constant)
384
+
385
+ ### 7.2 RKNN Models
386
+
387
+ Located in `RobustVideoMatting/rknn/`.
388
+
389
+ | Model | File | Size | Resolution | Precision | dsr | Status | PKB Ref |
390
+ |-------|------|------|-----------|-----------|-----|--------|---------|
391
+ | **256×256 FP16** | `rvm_mobilenetv3_fp16_256x256_rk3588_fp16_v20260416.rknn` | 9.0MB | 256×256 | FP16 | 0.25 | Phase-1 验证用 | §3 |
392
+ | **288×512 FP16** | `rvm_mobilenetv3_fp16_288x512_rk3588_fp16_v20260416.rknn` | 9.3MB | 288×512 | FP16 | 0.25 | 早期 board 实验 | §3 |
393
+ | **1080p FP16 (dsr=0.25)** | `rvm_mobilenetv3_fp16_1080x1920_rk3588_fp16.rknn` | 12MB | 1080×1920 | FP16 | 0.25 | **当前默认模型** | §12-§13, §20 |
394
+ | **1080p FP16 (dsr=0.5)** | `rvm_mobilenetv3_fp16_1080x1920_0.5-dsr_rk3588_fp16.rknn` | 16MB | 1080×1920 | FP16 | 0.5 | **最佳质量模型** | §21 |
395
+ | **256×256 INT8** | `rvm_mobilenetv3_int8_256x256_rk3588_int8_v20260416.rknn` | 5.5MB | 256×256 | INT8 | 0.25 | Phase-4 量化实验 | §4 |
396
+ | **288×512 INT8** | `rvm_mobilenetv3_int8_288x512_rk3588_int8_v20260416.rknn` | 5.7MB | 288×512 | INT8 | 0.25 | Phase-4 量化实验 | §4 |
397
+
398
+ **Conversion tool**: [ArcFoundry](https://github.com/PotterWhite/ArcFoundry.git) v0.14.0
399
+ - Config directory: `ArcFoundry.git/configs/rvm/`
400
+ - Target platform: RK3588
401
+ - Normalization: `mean=[0,0,0], std=[255,255,255]` (RKNN runtime applies `/255.0`)
402
+
403
+ ### 7.3 Quality Summary (s14/s20/s21 board experiments)
404
+
405
+ | Model | dsr | mean_diff vs PyTorch | bg_ratio | Speed (infer+composite) | Notes |
406
+ |-------|-----|---------------------|----------|------------------------|-------|
407
+ | 1080p FP16 (s14) | 0.25 | 2.30 | 86.6% | ~485ms | NCHW→NHWC transpose fix |
408
+ | 1080p FP16 (s20) | 0.25 | 2.30 | 86.6% | ~485ms | r2o stride padding: OK |
409
+ | 1080p FP16 (s21) | 0.5 | **2.18** | **86.5%** | ~691ms | **Best quality** |
410
+
411
+ - PyTorch baseline bg_ratio: 86.6% (dsr=0.25), 86.5% (dsr=0.5)
412
+ - All models tested on `dance.mp4` (363 frames, 1920×1080)
413
+ - Board: RK3588S, NPU0=40%, NPU1&2=10%
414
+
415
+ ### 7.4 Recommended Model Selection
416
+
417
+ | Use Case | Recommended Model | Reason |
418
+ |----------|------------------|--------|
419
+ | **Quality-first** | `rvm_mobilenetv3_fp16_1080x1920_0.5-dsr_rk3588_fp16.rknn` | mean_diff=2.18, best alpha edges |
420
+ | **Speed-first** | `rvm_mobilenetv3_fp16_1080x1920_rk3588_fp16.rknn` | 1.43× faster than dsr=0.5 |
421
+ | **Low memory** | `rvm_mobilenetv3_fp16_288x512_rk3588_fp16_v20260416.rknn` | Smallest r-state footprint |
422
+
423
+ ## 8. Related Documentation
424
 
425
  - **Training Script**: `helmsman.git/third-party/scripts/modnet/train_modnet_block1_2.py`
426
  - **ONNX Export Script**: `helmsman.git/third-party/scripts/modnet/onnx/export_onnx_pureBN.py`
427
  - **C++ Inference**: `helmsman.git/runtime/cpp/apps/matting/client/`
428
  - **Python Golden Reference**: `helmsman.git/third-party/scripts/modnet/onnx/generate_golden_files.py`
429
  - **Verification**: `helmsman.git/tools/MODNet/verify_golden_tensor.py`
430
+ - **RVM PKB**: `/volumes_pkb_helmsman/model/round2-rvm/log-MR2-P5-rknn-inference.md`
431
+ - **RVM ArcFoundry configs**: `ArcFoundry.git/configs/rvm/`
432
 
433
  ---
434