HelloSun commited on
Commit
68a4935
·
verified ·
1 Parent(s): 6a459ad

Add public-domain NASA video sample: LR vs PyTorch/FP16/INT8/INT4 4x outputs, side-by-side grid, per-frame timings and metrics

Browse files
.gitattributes CHANGED
@@ -1,6 +1,7 @@
1
  *.bin filter=lfs diff=lfs merge=lfs -text
2
  *.xml filter=lfs diff=lfs merge=lfs -text
3
  *.onnx filter=lfs diff=lfs merge=lfs -text
 
4
  input/nasa_bald_eagle_KSC-20230425-PH-JBS02_0003.jpg filter=lfs diff=lfs merge=lfs -text
5
  inputs/00003.png filter=lfs diff=lfs merge=lfs -text
6
  samples/eagle_4x/00_input_lr_x4.png filter=lfs diff=lfs merge=lfs -text
@@ -13,3 +14,12 @@ samples/eagle_4x/openvino_int4_mixed_w4a16.jpg filter=lfs diff=lfs merge=lfs -te
13
  samples/eagle_4x/openvino_int4_w4a16.jpg filter=lfs diff=lfs merge=lfs -text
14
  samples/eagle_4x/openvino_int4_w4a8.jpg filter=lfs diff=lfs merge=lfs -text
15
  samples/eagle_4x/openvino_int8_w8a8.jpg filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
1
  *.bin filter=lfs diff=lfs merge=lfs -text
2
  *.xml filter=lfs diff=lfs merge=lfs -text
3
  *.onnx filter=lfs diff=lfs merge=lfs -text
4
+ input/nasa_ICON_airglow_640x360_63s-66s.mp4 filter=lfs diff=lfs merge=lfs -text
5
  input/nasa_bald_eagle_KSC-20230425-PH-JBS02_0003.jpg filter=lfs diff=lfs merge=lfs -text
6
  inputs/00003.png filter=lfs diff=lfs merge=lfs -text
7
  samples/eagle_4x/00_input_lr_x4.png filter=lfs diff=lfs merge=lfs -text
 
14
  samples/eagle_4x/openvino_int4_w4a16.jpg filter=lfs diff=lfs merge=lfs -text
15
  samples/eagle_4x/openvino_int4_w4a8.jpg filter=lfs diff=lfs merge=lfs -text
16
  samples/eagle_4x/openvino_int8_w8a8.jpg filter=lfs diff=lfs merge=lfs -text
17
+ samples/video_airglow/00_source_lr.mp4 filter=lfs diff=lfs merge=lfs -text
18
+ samples/video_airglow/10_pytorch_fp32_4x.mp4 filter=lfs diff=lfs merge=lfs -text
19
+ samples/video_airglow/11_openvino_fp16_4x.mp4 filter=lfs diff=lfs merge=lfs -text
20
+ samples/video_airglow/12_openvino_int8_w8a8_4x.mp4 filter=lfs diff=lfs merge=lfs -text
21
+ samples/video_airglow/13_openvino_int4_mixed_4x.mp4 filter=lfs diff=lfs merge=lfs -text
22
+ samples/video_airglow/14_openvino_int4_4x.mp4 filter=lfs diff=lfs merge=lfs -text
23
+ samples/video_airglow/20_side_by_side_1920x780.mp4 filter=lfs diff=lfs merge=lfs -text
24
+ samples/video_airglow/21_poster_frame.jpg filter=lfs diff=lfs merge=lfs -text
25
+ samples/video_airglow/22_poster_detail_zoom.jpg filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -102,6 +102,36 @@ Real-ESRGAN 是純卷積的深層殘差網路(23 個 RRDB,每層輸出 `x*0.
102
 
103
  ---
104
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
105
  ## 2. 安裝
106
 
107
  ```bash
@@ -132,7 +162,8 @@ cv2.imwrite('output.png', sr)
132
  print('output:', sr.shape, 'scale:', scale)
133
  ```
134
 
135
- `inputs/` 內附 3 張測試圖(來自 Real-ESRGAN 官方 inputs),可直接試跑:
 
136
 
137
  ```bash
138
  python realesrgan_ov.py -i inputs/00003.png -o results/ -m int4-mixed -t 128
@@ -241,4 +272,5 @@ PY
241
  * 量化:[OpenVINO NNCF](https://github.com/openvinotoolkit/nncf)
242
  * 本 repo 的推論程式、轉換腳本與本說明文檔為本專案新增。
243
  * 樣品照片:NASA Kennedy Space Center(公有領域),見 `samples/eagle_4x/README.md` 的來源與授權說明
 
244
  * License:Apache-2.0(見 `LICENSE`)。
 
102
 
103
  ---
104
 
105
+ ## 1.6 影片對比(放大前 vs 各個量化版本)
106
+
107
+ 除了靜態圖,repo 也放了一段**公有領域**的 NASA 影片(ISS 氣輝夜景,`GSFC_20181022_ICON_m12902_Airglow`),
108
+ 用**原始 PyTorch 模型**與**所有 OpenVINO 版本**做 640×360 → 2560×1440 的 4 倍放大,逐影格記錄時間與畫質。
109
+
110
+ ![video poster](samples/video_airglow/21_poster_frame.jpg)
111
+
112
+ 城市燈光帶的 **1:1 放大**比較(最能看出量化差別:左上 bicubic、右上 PyTorch FP32、中上 FP16、
113
+ 左下 INT8、下中 INT4 混合、右下 純 INT4):
114
+
115
+ ![video zoom](samples/video_airglow/22_poster_detail_zoom.jpg)
116
+
117
+ | 模型 | 大小 | 45 影格總時間 | ms/影格 | 張/秒 | vs PyTorch PSNR ↑ | SSIM ↑ | 加速比 ↑ | 影片 |
118
+ |---|---|---|---|---|---|---|---|---|
119
+ | `pytorch_fp32`(原始) | 67.0 MB | 663.5 s | 14745 | 0.07 | 基準 | 1.0000 | 1.0× | [`10`](samples/video_airglow/10_pytorch_fp32_4x.mp4) |
120
+ | `openvino_fp16` | 35.1 MB | 211.5 s | 4700 | 0.21 | **45.41 dB** | **0.9927** | 3.1× | [`11`](samples/video_airglow/11_openvino_fp16_4x.mp4) |
121
+ | `openvino_int8_w8a8` | 19.5 MB | **124.8 s** | **2773** | **0.36** | 36.90 dB | 0.9284 | **5.3×** | [`12`](samples/video_airglow/12_openvino_int8_w8a8_4x.mp4) |
122
+ | `openvino_int4_mixed_w4a16` | 15.2 MB | 214.6 s | 4769 | 0.21 | 31.31 dB | 0.9344 | 3.1× | [`13`](samples/video_airglow/13_openvino_int4_mixed_4x.mp4) |
123
+ | `openvino_int4_w4a16` | 11.0 MB | 246.8 s | 5485 | 0.18 | 25.40 dB | 0.7894 | 2.7× | [`14`](samples/video_airglow/14_openvino_int4_4x.mp4) |
124
+
125
+ * 放大前(輸入):[`00_source_lr.mp4`](samples/video_airglow/00_source_lr.mp4) 
126
+ 六方並排比較:[`20_side_by_side_1920x780.mp4`](samples/video_airglow/20_side_by_side_1920x780.mp4)
127
+ * 完整說明、來源與授權、重現方式:**[`samples/video_airglow/README.md`](samples/video_airglow/README.md)**
128
+
129
+ 重點結論:**INT8 (W8A8) 在 CPU 上比 PyTorch FP32 快 5.3×** 且畫質僅 36.9 dB / SSIM 0.93;
130
+ **INT4 在 CPU 上不會更快**(CPU plugin 執行時會把 4-bit 權重解壓回 FP32/FP16),
131
+ 它的優勢是模型只有 11 MB;若要在 4-bit 尺寸下兼顧畫質,請用 `int4-mixed`。
132
+
133
+ ---
134
+
135
  ## 2. 安裝
136
 
137
  ```bash
 
162
  print('output:', sr.shape, 'scale:', scale)
163
  ```
164
 
165
+ `inputs/` 內附 3 張測試圖(來自 Real-ESRGAN 官方 inputs),`samples/` 內附 1 張公有領域照片與
166
+ 1 段公有領域影片(都已用所有模型跑過),可直接試跑:
167
 
168
  ```bash
169
  python realesrgan_ov.py -i inputs/00003.png -o results/ -m int4-mixed -t 128
 
272
  * 量化:[OpenVINO NNCF](https://github.com/openvinotoolkit/nncf)
273
  * 本 repo 的推論程式、轉換腳本與本說明文檔為本專案新增。
274
  * 樣品照片:NASA Kennedy Space Center(公有領域),見 `samples/eagle_4x/README.md` 的來源與授權說明
275
+ * 樣品影片:NASA/Goddard ICON 任務 ISS 氣輝畫面(公有領域),見 `samples/video_airglow/README.md`
276
  * License:Apache-2.0(見 `LICENSE`)。
input/nasa_ICON_airglow_640x360_63s-66s.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:74cfe01f6dc03aa321484e0823fa70a5128c93799ae94add6bbb66f60cf89d16
3
+ size 933609
samples/video_airglow/00_source_lr.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0bf16c8a6d168ec1476301edbe927b570a9d9cfde5db46f68f89f4549252ad02
3
+ size 405138
samples/video_airglow/10_pytorch_fp32_4x.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b171def478b0db0ef242e98823ed36c381158fd90014b3b2d8220f059aa936a
3
+ size 787181
samples/video_airglow/11_openvino_fp16_4x.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1a2b42cc77346c1a85f4253429ae28b717bd007e85db52c62cf407709d28a476
3
+ size 779152
samples/video_airglow/12_openvino_int8_w8a8_4x.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:51423cc8a28d08065102961f6ba68c8a2297e08f618561899c18b43d713369c0
3
+ size 927349
samples/video_airglow/13_openvino_int4_mixed_4x.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cb7d64813c412ffaf931b8bc03c347ecd7bfb0ce84857332e3846b8b7b84d3a3
3
+ size 847136
samples/video_airglow/14_openvino_int4_4x.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:97b5dee515699f5c5c3425118ff9121161e79c144c06867328cdabf47d0683e8
3
+ size 1301821
samples/video_airglow/20_side_by_side_1920x780.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b11618fa15f5688d9c6bdcbcb596628dbc1c9922224aad0264c3f6e59315381d
3
+ size 417982
samples/video_airglow/21_poster_frame.jpg ADDED

Git LFS Details

  • SHA256: 6f807a991524839dfdf86e3a9e507d59de7a1505fe31480cb78a882dd988c9a9
  • Pointer size: 131 Bytes
  • Size of remote file: 365 kB
samples/video_airglow/22_poster_detail_zoom.jpg ADDED

Git LFS Details

  • SHA256: d85ae2b872fcd3fe92a8ee0c5d33424fe91da08d0e94fe525dce1cc3b8bc88d1
  • Pointer size: 131 Bytes
  • Size of remote file: 206 kB
samples/video_airglow/README.md ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 影片對比:ISS 氣輝(Airglow)夜景
2
+
3
+ 一段**公有領域**的 NASA 影片片段,餵給「原始 PyTorch 模型」與「所有 OpenVINO 量化版本」,
4
+ 比較 **640×360 → 4 倍(2560×1440)** 的放大結果、畫質與速度。
5
+
6
+ ## 來源與授權
7
+
8
+ | 項目 | 內容 |
9
+ |---|---|
10
+ | 影片 | NASA/Goddard,*The Secrets behind Earth's Multi-colored Glow*(ICON 任務,2018-10-22 攝於 ISS) |
11
+ | NASA ID | `GSFC_20181022_ICON_m12902_Airglow` |
12
+ | 來源頁面 | https://images.nasa.gov/details/GSFC_20181022_ICON_m12902_Airglow |
13
+ | 原始檔 | `GSFC_20181022_ICON_m12902_Airglow~small.mp4`(640×360, 30 fps, 194 s, 35 MB) |
14
+ | 本 repo 內的輸入 | `input/nasa_ICON_airglow_640x360_63s-66s.mp4`(1.5 s 片段,約 0.9 MB) |
15
+ | 授權 | **公有領域(Public Domain)** — NASA 媒體一般不受版權保護 |
16
+
17
+ 裁切指令(已內建在 `tools/make_video_sample.py` 的說明中):
18
+
19
+ ```bash
20
+ ffmpeg -ss 63 -t 1.5 -i GSFC_20181022_ICON_m12902_Airglow~small.mp4 \
21
+ -an -c:v libx264 -crf 20 -pix_fmt yuv420p input/nasa_ICON_airglow_640x360_63s-66s.mp4
22
+ ```
23
+
24
+ 選這段的理由:銀河、星空、氣輝帶、城市燈光與 ISS 桁架結構同時入鏡,
25
+ 高頻(星星/燈點)與低頻(大面積暗部)都有,最能看出各種量化的差別。
26
+
27
+ ## 檔案
28
+
29
+ | 檔案 | 說明 | 大小 |
30
+ |---|---|---|
31
+ | `00_source_lr.mp4` | **放大前**:模型輸入 640×360 @30fps(45 影格) | 0.4 MB |
32
+ | `10_pytorch_fp32_4x.mp4` | **原始 PyTorch FP32** 4 倍放大(2560×1440) | 0.8 MB |
33
+ | `11_openvino_fp16_4x.mp4` | OpenVINO FP16 | 0.8 MB |
34
+ | `12_openvino_int8_w8a8_4x.mp4` | OpenVINO INT8(W8A8) | 0.9 MB |
35
+ | `13_openvino_int4_mixed_4x.mp4` | OpenVINO INT4 混合精度(W4/W8) | 0.8 MB |
36
+ | `14_openvino_int4_4x.mp4` | OpenVINO 純 INT4(W4A16) | 1.3 MB |
37
+ | `20_side_by_side_1920x780.mp4` | 2×3 並排比較(bicubic / PyTorch / FP16 / INT8 / INT4 mixed / INT4),含標籤 | 0.4 MB |
38
+ | `21_poster_frame.jpg` | 並排比較影片的第 23 影格 | 0.4 MB |
39
+ | `22_poster_detail_zoom.jpg` | 同一段落在城市燈光帶的 **1:1 放大**比較(最能看出差別) | 0.2 MB |
40
+ | `metrics.json` | 機讀指標(時間、PSNR、SSIM) | – |
41
+
42
+ ## 實測結果
43
+
44
+ 測試設定:tile=128 / tile_pad=16,45 影格逐frame 量測(僅計推論時間,不含編碼),
45
+ Xeon Platinum 8559C(16 vCPU)、OpenVINO 2026.4.1。
46
+
47
+ | 模型 | 模型大小 | 總時間 | ms/影格 | 張/秒 | vs PyTorch PSNR ↑ | SSIM ↑ | 加速比 ↑ |
48
+ |---|---|---|---|---|---|---|---|
49
+ | `pytorch_fp32`(原始) | 67.0 MB | 663.5 s | 14745 | 0.07 | 基準 | 1.0000 | 1.0× |
50
+ | `openvino_fp16` | 35.1 MB | 211.5 s | 4700 | 0.21 | **45.41 dB** | **0.9927** | 3.1× |
51
+ | `openvino_int8_w8a8` | 19.5 MB | **124.8 s** | **2773** | **0.36** | 36.90 dB | 0.9284 | **5.3×** |
52
+ | `openvino_int4_mixed_w4a16` | 15.2 MB | 214.6 s | 4769 | 0.21 | 31.31 dB | 0.9344 | 3.1× |
53
+ | `openvino_int4_w4a16` | 11.0 MB | 246.8 s | 5485 | 0.18 | 25.40 dB | 0.7894 | 2.7× |
54
+
55
+ 輸入吞吐量(640×360 影格):INT8 約 0.08 MPix/s、其餘約 0.04–0.05 MPix/s。
56
+
57
+ ### 結論
58
+
59
+ 1. **INT8 (W8A8) 是這個網路最快的版本**:比原始 PyTorch FP32 快 **5.3×**,畫質仍有 36.9 dB / SSIM 0.93,
60
+ 在夜景這種暗部多、細節密的素材上肉眼幾乎看不出差異。
61
+ 2. **FP16 品質最好**:45.4 dB / SSIM 0.993,與 PyTorch 幾乎一致,但只有 3.1× 加速。
62
+ 3. **INT4 在 CPU 上不會更快**(2.7×,還比 INT8 慢),因為 CPU plugin 會在執行時把 `u4` 權重解壓回
63
+ FP32/FP16 再卷積;INT4 的價值在**模型體積只有 11 MB(PyTorch 的 1/6)**,適合記憶體受限或邊緣裝置。
64
+ 4. **純 INT4 的畫質代價在動態畫面特別明顯**:星星點會消失、暗部出現色偏與色塊(見 `22_poster_detail_zoom.jpg` 右下)。
65
+ `int4-mixed` 把敏感層留 INT8 後,SSIM 從 0.79 拉回 0.93,是 INT4 尺寸下最實用的選擇。
66
+ 5. 這支影片的 PSNR 比靜態圖測試高(INT4 25.4 dB vs 18.0 dB),因為夜景暗部平坦、沒有大量高頻邊緣;
67
+ 量化誤差在這種素材上比較不容易被放大。
68
+
69
+ ## 重現
70
+
71
+ ```bash
72
+ pip install torch onnx nncf openvino opencv-python numpy
73
+ mkdir -p weights && curl -L -o weights/RealESRGAN_x4plus.pth \
74
+ https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth
75
+ python tools/make_video_sample.py # 全部重跑(約 25 分鐘,本機 16 vCPU)
76
+ python tools/make_video_sample.py --grid-only # 只重做並排比較影片
77
+ python tools/make_video_sample.py --poster-only # 只重做 1:1 細節比較圖
78
+ ```
samples/video_airglow/metrics.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "clip": {
3
+ "source": "nasa_ICON_airglow_640x360_63s-66s.mp4",
4
+ "frames": 45,
5
+ "fps": 30,
6
+ "input": "640x360",
7
+ "output": "2560x1440",
8
+ "tile": 128,
9
+ "tile_pad": 16,
10
+ "seconds": [
11
+ "0",
12
+ "1.5"
13
+ ]
14
+ },
15
+ "results": {
16
+ "pytorch_fp32": {
17
+ "file": "10_pytorch_fp32_4x.mp4",
18
+ "seconds_total": 663.54,
19
+ "ms_per_frame": 14745.4,
20
+ "fps": 0.07,
21
+ "input_mpix_per_s": 0.02
22
+ },
23
+ "openvino_fp16": {
24
+ "file": "11_openvino_fp16_4x.mp4",
25
+ "seconds_total": 211.5,
26
+ "ms_per_frame": 4700.0,
27
+ "fps": 0.21,
28
+ "input_mpix_per_s": 0.05,
29
+ "psnr_vs_pytorch": 45.41,
30
+ "ssim_vs_pytorch": 0.9927
31
+ },
32
+ "openvino_int8_w8a8": {
33
+ "file": "12_openvino_int8_w8a8_4x.mp4",
34
+ "seconds_total": 124.78,
35
+ "ms_per_frame": 2772.9,
36
+ "fps": 0.36,
37
+ "input_mpix_per_s": 0.08,
38
+ "psnr_vs_pytorch": 36.9,
39
+ "ssim_vs_pytorch": 0.9284
40
+ },
41
+ "openvino_int4_mixed_w4a16": {
42
+ "file": "13_openvino_int4_mixed_4x.mp4",
43
+ "seconds_total": 214.59,
44
+ "ms_per_frame": 4768.6,
45
+ "fps": 0.21,
46
+ "input_mpix_per_s": 0.05,
47
+ "psnr_vs_pytorch": 31.31,
48
+ "ssim_vs_pytorch": 0.9344
49
+ },
50
+ "openvino_int4_w4a16": {
51
+ "file": "14_openvino_int4_4x.mp4",
52
+ "seconds_total": 246.8,
53
+ "ms_per_frame": 5484.5,
54
+ "fps": 0.18,
55
+ "input_mpix_per_s": 0.04,
56
+ "psnr_vs_pytorch": 25.4,
57
+ "ssim_vs_pytorch": 0.7894
58
+ }
59
+ }
60
+ }
tools/make_video_sample.py ADDED
@@ -0,0 +1,252 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Build the `samples/video_airglow/` gallery: one public-domain NASA clip processed by
3
+ the original PyTorch model and by every OpenVINO variant (INT4 / INT8 / FP16).
4
+
5
+ Source: NASA/Goddard "The Secrets behind Earth's Multi-colored Glow" (ICON mission),
6
+ airglow filmed from the ISS - NASA media is public domain.
7
+ The clip (`input/nasa_ICON_airglow_640x360_63s-66s.mp4`, 640x360) is the LOW RESOLUTION
8
+ input; every model upscales it to 2560x1440.
9
+
10
+ Per model the script writes a 4x mp4, records the wall-clock time of every frame and
11
+ computes PSNR/SSIM against the PyTorch FP32 reference. Frames are streamed (never all
12
+ kept in RAM): the reference is stored JPEG-compressed and the side-by-side grid is
13
+ assembled in a second pass by decoding the encoded videos.
14
+ """
15
+ import json
16
+ import os
17
+ import pathlib
18
+ import subprocess
19
+ import sys
20
+ import time
21
+
22
+ import cv2
23
+ import numpy as np
24
+
25
+ HERE = pathlib.Path(__file__).resolve().parent
26
+ ROOT = HERE.parent
27
+ sys.path.insert(0, str(ROOT))
28
+ from realesrgan_ov import RealESRGAN # noqa: E402
29
+
30
+ # The clip is cut from the NASA video asset (see samples/video_airglow/README.md):
31
+ # ffmpeg -ss 63 -t 3 -i GSFC_20181022_ICON_m12902_Airglow~small.mp4 -an -c:v libx264 ...
32
+ SRC = ROOT / 'input' / 'nasa_ICON_airglow_640x360_63s-66s.mp4'
33
+ OUT = ROOT / 'samples' / 'video_airglow'
34
+ SS, DUR = '0', '1.5' # clip start (s) / duration (s)
35
+ FPS = 30
36
+ TILE, TILE_PAD = 128, 16
37
+ GRID = '20_side_by_side_1920x780.mp4'
38
+ POSTER = '21_poster_frame.jpg'
39
+ PANELS = [('00_source_lr.mp4', 'input (LR) bicubic x4'),
40
+ ('10_pytorch_fp32_4x.mp4', 'PyTorch FP32'),
41
+ ('11_openvino_fp16_4x.mp4', 'OV FP16'),
42
+ ('12_openvino_int8_w8a8_4x.mp4', 'OV INT8 W8A8'),
43
+ ('13_openvino_int4_mixed_4x.mp4', 'OV INT4 mixed'),
44
+ ('14_openvino_int4_4x.mp4', 'OV INT4 (W4A16)')]
45
+
46
+
47
+ def psnr(a, b, peak=255.0):
48
+ mse = np.mean((a.astype(np.float64) - b.astype(np.float64)) ** 2)
49
+ return 99.0 if mse == 0 else 10 * np.log10(peak ** 2 / mse)
50
+
51
+
52
+ def ssim(a, b):
53
+ a = cv2.cvtColor(a, cv2.COLOR_BGR2GRAY).astype(np.float64) / 255.
54
+ b = cv2.cvtColor(b, cv2.COLOR_BGR2GRAY).astype(np.float64) / 255.
55
+ c1, c2 = 0.01 ** 2, 0.03 ** 2
56
+ mu_a, mu_b = cv2.GaussianBlur(a, (11, 11), 1.5), cv2.GaussianBlur(b, (11, 11), 1.5)
57
+ sa = cv2.GaussianBlur(a * a, (11, 11), 1.5) - mu_a ** 2
58
+ sb = cv2.GaussianBlur(b * b, (11, 11), 1.5) - mu_b ** 2
59
+ sab = cv2.GaussianBlur(a * b, (11, 11), 1.5) - mu_a * mu_b
60
+ return (((2 * mu_a * mu_b + c1) * (2 * sab + c2)) / ((mu_a ** 2 + mu_b ** 2 + c1) * (sa + sb + c2))).mean()
61
+
62
+
63
+ class Writer:
64
+ """Raw BGR frames -> H.264 mp4 through ffmpeg."""
65
+
66
+ def __init__(self, path, size, fps=FPS, crf=26):
67
+ self.w, self.h = size
68
+ cmd = ['ffmpeg', '-hide_banner', '-loglevel', 'error', '-y', '-f', 'rawvideo', '-pix_fmt', 'bgr24',
69
+ '-s', f'{self.w}x{self.h}', '-r', str(fps), '-i', '-', '-c:v', 'libx264', '-crf', str(crf),
70
+ '-preset', 'veryfast', '-pix_fmt', 'yuv420p', '-movflags', '+faststart', str(path)]
71
+ self.proc = subprocess.Popen(cmd, stdin=subprocess.PIPE)
72
+
73
+ def write(self, frame):
74
+ self.proc.stdin.write(np.ascontiguousarray(frame, dtype=np.uint8).tobytes())
75
+
76
+ def close(self):
77
+ self.proc.stdin.close()
78
+ self.proc.wait()
79
+
80
+
81
+ def cut_clip():
82
+ OUT.mkdir(parents=True, exist_ok=True)
83
+ dst = OUT / '00_source_lr.mp4'
84
+ if not SRC.exists():
85
+ raise SystemExit(f'source clip not found: {SRC}')
86
+ subprocess.run(['ffmpeg', '-hide_banner', '-loglevel', 'error', '-y', '-ss', SS, '-t', DUR, '-i', str(SRC),
87
+ '-an', '-c:v', 'libx264', '-crf', '20', '-preset', 'veryfast', '-pix_fmt', 'yuv420p',
88
+ '-movflags', '+faststart', str(dst)], check=True)
89
+ return dst
90
+
91
+
92
+ def build_torch_runner():
93
+ """Original PyTorch model behind the same API/tiling as realesrgan_ov."""
94
+ import torch
95
+ from rrdbnet_arch import RRDBNet
96
+ weights = os.environ.get('REALESRGAN_PTH', str(ROOT / 'weights' / 'RealESRGAN_x4plus.pth'))
97
+ if not os.path.exists(weights):
98
+ raise SystemExit(f'PyTorch checkpoint not found: {weights}\n'
99
+ ' curl -L -o weights/RealESRGAN_x4plus.pth \\\n'
100
+ ' https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth')
101
+ ckpt = torch.load(weights, map_location='cpu', weights_only=True)
102
+ net = RRDBNet(3, 3, 4, 64, 23, 32)
103
+ net.load_state_dict(ckpt['params_ema'] if 'params_ema' in ckpt else ckpt['params'], strict=True)
104
+ net.eval()
105
+
106
+ class Shim: # mimics ov.CompiledModel: compiled([NCHW float32]) -> [(1, C, H, W)]
107
+ def __call__(self, inputs):
108
+ with torch.inference_mode():
109
+ return [net(torch.from_numpy(inputs[0])).numpy()]
110
+
111
+ runner = RealESRGAN.__new__(RealESRGAN) # reuse _forward_tiled without compiling a model
112
+ runner.compiled = Shim()
113
+ runner.scale = 4
114
+ return runner
115
+
116
+
117
+ LABEL_H = 30
118
+
119
+
120
+ def build_zoom_poster(frame_idx=22, crop=(150, 1010, 640, 360)):
121
+ """2x3 poster with a 1:1 zoom on the city-lights strip of every variant."""
122
+ x, y, cw, ch = crop
123
+ colors = [(235, 235, 235), (255, 255, 255), (255, 255, 255),
124
+ (235, 235, 235), (235, 235, 235), (235, 235, 235)]
125
+ tiles = []
126
+ for panel in range(len(PANELS)):
127
+ cap = cv2.VideoCapture(str(OUT / PANELS[panel][0]))
128
+ cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx)
129
+ ok, img = cap.read()
130
+ cap.release()
131
+ if not ok:
132
+ raise SystemExit(f'cannot read frame {frame_idx} of {PANELS[panel][0]}')
133
+ if panel == 0: # the LR source: show its bicubic x4 rendering, same geometry as the others
134
+ img = cv2.resize(img, (img.shape[1] * 4, img.shape[0] * 4), interpolation=cv2.INTER_CUBIC)
135
+ tiles.append(label(img[y:y + ch, x:x + cw], PANELS[panel][1] + ' (1:1 zoom)', colors[panel]))
136
+ grid = np.vstack([np.hstack(tiles[:3]), np.hstack(tiles[3:])])
137
+ cv2.imwrite(str(OUT / '22_poster_detail_zoom.jpg'), grid, [cv2.IMWRITE_JPEG_QUALITY, 93])
138
+ print('zoom poster written', flush=True)
139
+
140
+
141
+ def label(img, text, bar_color=(255, 255, 255)):
142
+ bar = np.full((LABEL_H, img.shape[1], 3), bar_color, np.uint8)
143
+ cv2.putText(bar, text, (8, 21), cv2.FONT_HERSHEY_SIMPLEX, 0.65, (0, 0, 0), 2, cv2.LINE_AA)
144
+ return np.vstack([bar, img])
145
+
146
+
147
+ def main():
148
+ clip = cut_clip()
149
+ cap = cv2.VideoCapture(str(clip))
150
+ ok, f = cap.read()
151
+ cap.release()
152
+ w, h = f.shape[1], f.shape[0]
153
+ n = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
154
+ cap = cv2.VideoCapture(str(clip))
155
+ frames = []
156
+ while True:
157
+ ok, f = cap.read()
158
+ if not ok:
159
+ break
160
+ frames.append(f)
161
+ cap.release()
162
+ h, w = frames[0].shape[:2]
163
+ n = len(frames)
164
+ print(f'clip: {n} frames @ {FPS} fps, {w}x{h} -> 4x = {w * 4}x{h * 4}, tile={TILE}', flush=True)
165
+
166
+ def make_grid():
167
+ caps = [cv2.VideoCapture(str(OUT / name)) for name, _ in PANELS]
168
+ writer = Writer(OUT / GRID, (w * 3, (h + LABEL_H) * 2), crf=24)
169
+ poster = None
170
+ i = 0
171
+ while True:
172
+ imgs = []
173
+ for cap in caps:
174
+ ok, img = cap.read()
175
+ if not ok:
176
+ imgs = []
177
+ break
178
+ imgs.append(cv2.resize(img, (w, h), interpolation=cv2.INTER_AREA))
179
+ if not imgs:
180
+ break
181
+ row1 = np.hstack([label(imgs[k], PANELS[k][1]) for k in (0, 1, 2)])
182
+ row2 = np.hstack([label(imgs[k], PANELS[k][1]) for k in (3, 4, 5)])
183
+ grid = np.vstack([row1, row2])
184
+ writer.write(grid)
185
+ if i == n // 2:
186
+ poster = grid
187
+ i += 1
188
+ for cap in caps:
189
+ cap.release()
190
+ writer.close()
191
+ cv2.imwrite(str(OUT / POSTER), poster, [cv2.IMWRITE_JPEG_QUALITY, 93])
192
+ print(f'grid video: {i} frames', flush=True)
193
+
194
+ if '--grid-only' in sys.argv: # rebuild the comparison grid from the existing videos
195
+ make_grid()
196
+ return
197
+ if '--poster-only' in sys.argv:
198
+ build_zoom_poster()
199
+ return
200
+
201
+ results = {}
202
+
203
+ def process(name, xml, precision, out_name, reference_jpeg=None):
204
+ runner = build_torch_runner() if xml is None else RealESRGAN(
205
+ model=str(ROOT / xml), precision=precision or 'f32')
206
+ writer = Writer(OUT / out_name, (w * 4, h * 4))
207
+ times, ps, ss, jpegs = [], [], [], []
208
+ for i, frame in enumerate(frames):
209
+ t0 = time.time()
210
+ out = runner.enhance(frame, tile=TILE, tile_pad=TILE_PAD)[0]
211
+ times.append(time.time() - t0)
212
+ writer.write(out)
213
+ if reference_jpeg is not None:
214
+ ref = cv2.imdecode(reference_jpeg[i], cv2.IMREAD_COLOR)
215
+ ps.append(psnr(ref, out))
216
+ ss.append(ssim(ref, out))
217
+ else:
218
+ jpegs.append(cv2.imencode('.jpg', out, [cv2.IMWRITE_JPEG_QUALITY, 95])[1])
219
+ if (i + 1) % 20 == 0:
220
+ print(f' {name}: {i + 1}/{n} frames, {np.mean(times):.2f} s/frame', flush=True)
221
+ writer.close()
222
+ total = sum(times)
223
+ row = {'file': out_name, 'seconds_total': round(total, 2), 'ms_per_frame': round(total / n * 1000, 1),
224
+ 'fps': round(n / total, 2), 'input_mpix_per_s': round(w * h / 1e6 * n / total, 2)}
225
+ if ps:
226
+ row['psnr_vs_pytorch'] = round(float(np.mean(ps)), 2)
227
+ row['ssim_vs_pytorch'] = round(float(np.mean(ss)), 4)
228
+ results[name] = row
229
+ print(name, row, flush=True)
230
+ del runner
231
+ return jpegs
232
+
233
+ ref_jpeg = process('pytorch_fp32', None, None, PANELS[1][0])
234
+ variants = [('openvino_fp16', 'models/realesrgan_x4plus_fp16.xml', 'f32', PANELS[2][0]),
235
+ ('openvino_int8_w8a8', 'models/realesrgan_x4plus_int8_w8a8.xml', None, PANELS[3][0]),
236
+ ('openvino_int4_mixed_w4a16', 'models/realesrgan_x4plus_int4_mixed_w4a16.xml', 'f32', PANELS[4][0]),
237
+ ('openvino_int4_w4a16', 'models/realesrgan_x4plus_int4_w4a16.xml', 'f32', PANELS[5][0])]
238
+ for name, xml, prec, out_name in variants:
239
+ process(name, xml, prec, out_name, reference_jpeg=ref_jpeg)
240
+
241
+ make_grid()
242
+ build_zoom_poster()
243
+ (OUT / 'metrics.json').write_text(json.dumps(
244
+ {'clip': {'source': SRC.name, 'frames': n, 'fps': FPS, 'input': f'{w}x{h}',
245
+ 'output': f'{w * 4}x{h * 4}', 'tile': TILE, 'tile_pad': TILE_PAD,
246
+ 'seconds': [SS, DUR]},
247
+ 'results': results}, indent=2))
248
+ print(json.dumps(results, indent=2))
249
+
250
+
251
+ if __name__ == '__main__':
252
+ main()