Commit ·
3d6f7a4
1
Parent(s): 0455bad
docs: add RVM model registry (ONNX + RKNN) with quality summary
Browse files- §7.1: 5 ONNX variants (MobileNetV3/ResNet50, FP32/FP16, with/without refiner)
- §7.2: 6 RKNN models (256/288/1080p, FP16/INT8, dsr=0.25/0.5)
- §7.3: Quality summary from s14/s20/s21 board experiments
- §7.4: Recommended model selection guide
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
README.md
CHANGED
|
@@ -361,13 +361,74 @@ python3 tools/MODNet/verify_golden_tensor.py
|
|
| 361 |
|
| 362 |
---
|
| 363 |
|
| 364 |
-
## 7.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 365 |
|
| 366 |
- **Training Script**: `helmsman.git/third-party/scripts/modnet/train_modnet_block1_2.py`
|
| 367 |
- **ONNX Export Script**: `helmsman.git/third-party/scripts/modnet/onnx/export_onnx_pureBN.py`
|
| 368 |
- **C++ Inference**: `helmsman.git/runtime/cpp/apps/matting/client/`
|
| 369 |
- **Python Golden Reference**: `helmsman.git/third-party/scripts/modnet/onnx/generate_golden_files.py`
|
| 370 |
- **Verification**: `helmsman.git/tools/MODNet/verify_golden_tensor.py`
|
|
|
|
|
|
|
| 371 |
|
| 372 |
---
|
| 373 |
|
|
|
|
| 361 |
|
| 362 |
---
|
| 363 |
|
| 364 |
+
## 7. RobustVideoMatting (RVM) Models
|
| 365 |
+
|
| 366 |
+
### 7.1 ONNX Models
|
| 367 |
+
|
| 368 |
+
Located in `RobustVideoMatting/onnx/`.
|
| 369 |
+
|
| 370 |
+
| Model | File | Size | Backbone | Refiner | Source |
|
| 371 |
+
|-------|------|------|----------|---------|--------|
|
| 372 |
+
| **MobileNetV3 FP32** | `rvm_mobilenetv3_fp32.onnx` | 14.3MB | MobileNetV3 | No | PeterL1n/RobustVideoMatting v1.0.0 |
|
| 373 |
+
| **MobileNetV3 FP16** | `rvm_mobilenetv3_fp16.onnx` | 7.2MB | MobileNetV3 | No | FP16 export |
|
| 374 |
+
| **MobileNetV3 FP32 (no refiner)** | `rvm_mobilenetv3_fp32_no_refiner.onnx` | 14.3MB | MobileNetV3 | No | Stripped refiner ops |
|
| 375 |
+
| **ResNet50 FP32** | `rvm_resnet50_fp32.onnx` | ~90MB | ResNet50 | No | PeterL1n v1.0.0 |
|
| 376 |
+
| **ResNet50 FP16** | `rvm_resnet50_fp16.onnx` | ~45MB | ResNet50 | No | FP16 export |
|
| 377 |
+
|
| 378 |
+
**Key model properties** (MobileNetV3):
|
| 379 |
+
- Opset 12, IR version 6, 353 nodes
|
| 380 |
+
- Zero InstanceNorm nodes (NPU-friendly)
|
| 381 |
+
- No refiner ops (DeepGuidedFilterRefiner not included)
|
| 382 |
+
- 6 inputs: `src [1,3,H,W]`, `r1i~r4i` (ConvGRU states), `downsample_ratio [1]`
|
| 383 |
+
- After ArcFoundry `fold_constant_inputs`: 5 inputs (downsample_ratio baked as constant)
|
| 384 |
+
|
| 385 |
+
### 7.2 RKNN Models
|
| 386 |
+
|
| 387 |
+
Located in `RobustVideoMatting/rknn/`.
|
| 388 |
+
|
| 389 |
+
| Model | File | Size | Resolution | Precision | dsr | Status | PKB Ref |
|
| 390 |
+
|-------|------|------|-----------|-----------|-----|--------|---------|
|
| 391 |
+
| **256×256 FP16** | `rvm_mobilenetv3_fp16_256x256_rk3588_fp16_v20260416.rknn` | 9.0MB | 256×256 | FP16 | 0.25 | Phase-1 验证用 | §3 |
|
| 392 |
+
| **288×512 FP16** | `rvm_mobilenetv3_fp16_288x512_rk3588_fp16_v20260416.rknn` | 9.3MB | 288×512 | FP16 | 0.25 | 早期 board 实验 | §3 |
|
| 393 |
+
| **1080p FP16 (dsr=0.25)** | `rvm_mobilenetv3_fp16_1080x1920_rk3588_fp16.rknn` | 12MB | 1080×1920 | FP16 | 0.25 | **当前默认模型** | §12-§13, §20 |
|
| 394 |
+
| **1080p FP16 (dsr=0.5)** | `rvm_mobilenetv3_fp16_1080x1920_0.5-dsr_rk3588_fp16.rknn` | 16MB | 1080×1920 | FP16 | 0.5 | **最佳质量模型** | §21 |
|
| 395 |
+
| **256×256 INT8** | `rvm_mobilenetv3_int8_256x256_rk3588_int8_v20260416.rknn` | 5.5MB | 256×256 | INT8 | 0.25 | Phase-4 量化实验 | §4 |
|
| 396 |
+
| **288×512 INT8** | `rvm_mobilenetv3_int8_288x512_rk3588_int8_v20260416.rknn` | 5.7MB | 288×512 | INT8 | 0.25 | Phase-4 量化实验 | §4 |
|
| 397 |
+
|
| 398 |
+
**Conversion tool**: [ArcFoundry](https://github.com/PotterWhite/ArcFoundry.git) v0.14.0
|
| 399 |
+
- Config directory: `ArcFoundry.git/configs/rvm/`
|
| 400 |
+
- Target platform: RK3588
|
| 401 |
+
- Normalization: `mean=[0,0,0], std=[255,255,255]` (RKNN runtime applies `/255.0`)
|
| 402 |
+
|
| 403 |
+
### 7.3 Quality Summary (s14/s20/s21 board experiments)
|
| 404 |
+
|
| 405 |
+
| Model | dsr | mean_diff vs PyTorch | bg_ratio | Speed (infer+composite) | Notes |
|
| 406 |
+
|-------|-----|---------------------|----------|------------------------|-------|
|
| 407 |
+
| 1080p FP16 (s14) | 0.25 | 2.30 | 86.6% | ~485ms | NCHW→NHWC transpose fix |
|
| 408 |
+
| 1080p FP16 (s20) | 0.25 | 2.30 | 86.6% | ~485ms | r2o stride padding: OK |
|
| 409 |
+
| 1080p FP16 (s21) | 0.5 | **2.18** | **86.5%** | ~691ms | **Best quality** |
|
| 410 |
+
|
| 411 |
+
- PyTorch baseline bg_ratio: 86.6% (dsr=0.25), 86.5% (dsr=0.5)
|
| 412 |
+
- All models tested on `dance.mp4` (363 frames, 1920×1080)
|
| 413 |
+
- Board: RK3588S, NPU0=40%, NPU1&2=10%
|
| 414 |
+
|
| 415 |
+
### 7.4 Recommended Model Selection
|
| 416 |
+
|
| 417 |
+
| Use Case | Recommended Model | Reason |
|
| 418 |
+
|----------|------------------|--------|
|
| 419 |
+
| **Quality-first** | `rvm_mobilenetv3_fp16_1080x1920_0.5-dsr_rk3588_fp16.rknn` | mean_diff=2.18, best alpha edges |
|
| 420 |
+
| **Speed-first** | `rvm_mobilenetv3_fp16_1080x1920_rk3588_fp16.rknn` | 1.43× faster than dsr=0.5 |
|
| 421 |
+
| **Low memory** | `rvm_mobilenetv3_fp16_288x512_rk3588_fp16_v20260416.rknn` | Smallest r-state footprint |
|
| 422 |
+
|
| 423 |
+
## 8. Related Documentation
|
| 424 |
|
| 425 |
- **Training Script**: `helmsman.git/third-party/scripts/modnet/train_modnet_block1_2.py`
|
| 426 |
- **ONNX Export Script**: `helmsman.git/third-party/scripts/modnet/onnx/export_onnx_pureBN.py`
|
| 427 |
- **C++ Inference**: `helmsman.git/runtime/cpp/apps/matting/client/`
|
| 428 |
- **Python Golden Reference**: `helmsman.git/third-party/scripts/modnet/onnx/generate_golden_files.py`
|
| 429 |
- **Verification**: `helmsman.git/tools/MODNet/verify_golden_tensor.py`
|
| 430 |
+
- **RVM PKB**: `/volumes_pkb_helmsman/model/round2-rvm/log-MR2-P5-rknn-inference.md`
|
| 431 |
+
- **RVM ArcFoundry configs**: `ArcFoundry.git/configs/rvm/`
|
| 432 |
|
| 433 |
---
|
| 434 |
|