alexandrealvaro commited on
Commit
f8f8cdb
·
verified ·
1 Parent(s): 243fd1b

docs: rewrite README as spec (drop devlog narrative, focus on what each variant is)

Browse files
Files changed (1) hide show
  1. README.md +62 -95
README.md CHANGED
@@ -15,122 +15,116 @@ tags:
15
  pipeline_tag: image-segmentation
16
  ---
17
 
18
- # CorridorKey — runtime model variants
19
 
20
- Pre-compiled inference variants of the **CorridorKey** chroma-keying network,
21
- re-exported from the upstream training checkpoints to the formats the
22
- [CorridorKey Runtime](https://github.com/alexandremendoncaalvaro/CorridorKey-Runtime)
23
- loads at runtime (ONNX for TensorRT-RTX EP / DirectML, Torch-TensorRT
24
- TorchScript engines for the in-process Windows blue path, MLX for Apple
25
- Silicon, plus the auxiliary tracker fixtures).
26
 
27
  ## Upstream sources
28
 
29
- The training checkpoints (`.pth`, base `.safetensors`) are not duplicated
30
- here they live at the canonical locations published by the model author:
31
 
32
- | Variant | Upstream repo |
33
  |---|---|
34
- | Green (CorridorKey v1.0) | [`nikopueringer/CorridorKey_v1.0`](https://huggingface.co/nikopueringer/CorridorKey_v1.0) |
35
- | Blue (CorridorKeyBlue 1.0) | [`nikopueringer/CorridorKeyBlue_1.0`](https://huggingface.co/nikopueringer/CorridorKeyBlue_1.0) |
36
 
37
- License (both upstreams + this repo): **CC BY-NC-SA 4.0**.
38
 
39
- ## What this repo publishes
40
 
41
- Everything below is generated from the upstream checkpoints by the
42
- CorridorKey Runtime build pipeline. Re-exports are deterministic; if
43
- upstream re-trains, regenerate the artefacts and bump the runtime
44
- version that consumes them.
45
 
46
- ### ONNX FP16 green ladder (Windows / Linux GPU via TensorRT-RTX EP, DirectML)
47
-
48
- | File | Resolution | Approx size |
49
  |---|---|---|
50
  | `onnx/fp16/corridorkey_fp16_512.onnx` | 512 × 512 | ~145 MB |
51
  | `onnx/fp16/corridorkey_fp16_1024.onnx` | 1024 × 1024 | ~156 MB |
52
  | `onnx/fp16/corridorkey_fp16_1536.onnx` | 1536 × 1536 | ~174 MB |
53
  | `onnx/fp16/corridorkey_fp16_2048.onnx` | 2048 × 2048 | ~200 MB |
54
 
55
- ### ONNX FP16 context cache — TensorRT-RTX EP pre-compiled
56
 
57
- Optional companion artefacts paired by resolution. Shipping these lets
58
- the runtime skip first-launch TensorRT compile (10–30 s on RTX 4090) and
59
- go straight to a warm session.
60
 
61
- | File | Resolution | Approx size |
62
  |---|---|---|
63
  | `onnx/fp16_ctx/corridorkey_fp16_512_ctx.onnx` | 512 × 512 | ~167 MB |
64
  | `onnx/fp16_ctx/corridorkey_fp16_1024_ctx.onnx` | 1024 × 1024 | ~178 MB |
65
  | `onnx/fp16_ctx/corridorkey_fp16_1536_ctx.onnx` | 1536 × 1536 | ~197 MB |
66
  | `onnx/fp16_ctx/corridorkey_fp16_2048_ctx.onnx` | 2048 × 2048 | ~223 MB |
67
 
68
- ### ONNX INT8 — research / browser experiment only
69
 
70
- Quantised models retained for the browser-based demo. **Not used by the
71
- production OFX plugin**; the Windows RTX render path uses FP16 + TRT-RTX
72
- EP exclusively.
73
 
74
- | File | Resolution | Approx size |
75
  |---|---|---|
76
  | `onnx/int8/corridorkey_int8_512.onnx` | 512 × 512 | ~79 MB |
77
  | `onnx/int8/corridorkey_int8_768.onnx` | 768 × 768 | ~88 MB |
78
  | `onnx/int8/corridorkey_int8_1024.onnx` | 1024 × 1024 | ~101 MB |
79
 
80
- ### Torch-TensorRT — green ladder (Windows blue-pack runtime, in-process)
81
 
82
  Pre-compiled TorchScript engines (Torch-TensorRT 2.8.0 + TensorRT 10.12,
83
- CUDA 12.8). Loaded by the runtime via `torch::jit::load` when the
84
- session is bound to `Backend::TorchTRT` and the curated TorchTRT runtime
85
- DLLs (libtorch + nvinfer + cuDNN) are resident next to the install.
86
 
87
- | File | Resolution | Precision | Approx size |
88
  |---|---|---|---|
89
  | `torchtrt/fp16/corridorkey_torchtrt_fp16_512.ts` | 512 × 512 | FP16 | ~408 MB |
90
  | `torchtrt/fp16/corridorkey_torchtrt_fp16_1024.ts` | 1024 × 1024 | FP16 | ~430 MB |
91
  | `torchtrt/fp16/corridorkey_torchtrt_fp16_1536.ts` | 1536 × 1536 | FP16 | ~502 MB |
92
  | `torchtrt/fp16/corridorkey_torchtrt_fp16_2048.ts` | 2048 × 2048 | FP16 | ~558 MB |
93
 
94
- ### Torch-TensorRT — blue ladder (CorridorKeyBlue, dedicated)
 
 
95
 
96
- Sprint 1 added a dedicated path for blue-screen plates so the runtime
97
- no longer relies on the green-domain canonicalisation workaround. The
98
- 1536 / 2048 rungs use FP32 because Sprint 0 measured FP16 NaNs at those
99
- graph sizes for the blue checkpoint.
100
 
101
- | File | Resolution | Precision | Approx size |
102
  |---|---|---|---|
103
- | `torchtrt/fp16-blue/corridorkey_blue_torchtrt_fp16_512.ts` | 512 × 512 | FP16 | _pending compile_ |
104
  | `torchtrt/fp16-blue/corridorkey_blue_torchtrt_fp16_1024.ts` | 1024 × 1024 | FP16 | ~258 MB |
105
  | `torchtrt/fp32-blue/corridorkey_blue_torchtrt_fp32_1536.ts` | 1536 × 1536 | FP32 | ~488 MB |
106
- | `torchtrt/fp32-blue/corridorkey_blue_torchtrt_fp32_2048.ts` | 2048 × 2048 | FP32 | _pending compile_ |
107
 
108
- ### MLX — Apple Silicon
109
 
110
- Native Apple Silicon weights and per-resolution pre-compiled bridges,
111
- loaded by the runtime when `Backend::MLX` is selected on macOS.
 
 
112
 
113
- | File | Description | Approx size |
114
  |---|---|---|
115
- | `mlx/corridorkey_mlx.safetensors` | Re-exported base weights for MLX layout | ~399 MB |
116
  | `mlx/corridorkey_mlx_bridge_512.mlxfn` | 512 × 512 bridge | ~289 MB |
117
  | `mlx/corridorkey_mlx_bridge_768.mlxfn` | 768 × 768 bridge | ~298 MB |
118
  | `mlx/corridorkey_mlx_bridge_1024.mlxfn` | 1024 × 1024 bridge | ~311 MB |
119
  | `mlx/corridorkey_mlx_bridge_1536.mlxfn` | 1536 × 1536 bridge | ~348 MB |
120
  | `mlx/corridorkey_mlx_bridge_2048.mlxfn` | 2048 × 2048 bridge | ~399 MB |
121
 
122
- The `.safetensors` here is **re-exported** for the MLX framework's
123
- expected layout — it differs byte-wise from the upstream
124
- `nikopueringer/CorridorKey_v1.0.safetensors` and is required by the MLX
125
- bridges below it.
126
 
127
- ### Hint trackers auxiliary fixtures
 
 
 
128
 
129
- Tracker models the runtime calls during alpha-hint generation. Exported
130
- from the upstream MobileSAM and Cutie projects under their own licences
131
- (see model cards on Hugging Face).
132
-
133
- | File | Approx size |
134
  |---|---|
135
  | `hint/mobilesam_image_encoder_fp16.onnx` | ~14 MB |
136
  | `hint/mobilesam_prompt_decoder_fp16.onnx` | ~8 MB |
@@ -138,52 +132,25 @@ from the upstream MobileSAM and Cutie projects under their own licences
138
  | `hint/cutie_init_memory_fp16.onnx` | ~1 MB |
139
  | `hint/cutie_tracking_memory_fp16.onnx` | ~1 MB |
140
 
141
- ### Releases
142
-
143
- Standalone CorridorKey Runtime installer artefacts (Windows OFX plugin
144
- bundle for DaVinci Resolve / Foundry Nuke). Each subdirectory under
145
- `releases/` is a single tagged build of the
146
- [CorridorKey-Runtime repo](https://github.com/alexandremendoncaalvaro/CorridorKey-Runtime).
147
-
148
- ## What this repo deliberately does NOT publish
149
 
150
- - **Training checkpoints (`.pth`, base `.safetensors`)** fetch directly
151
- from the `nikopueringer/*` upstream repos linked above. Removing these
152
- duplicates avoids drift if the author republishes a checkpoint.
153
- - **Curated runtime DLL bundles** (libtorch / nvinfer / cuDNN for the
154
- Windows blue pack) — those ship inside the OFX installer artefact,
155
- not as a model file.
156
 
157
  ## Usage
158
 
159
  ```bash
160
- # Single artefact
161
  hf download alexandrealvaro/CorridorKey onnx/fp16/corridorkey_fp16_1024.onnx
162
 
163
- # All ONNX FP16 green
164
  hf download alexandrealvaro/CorridorKey --include "onnx/fp16/*.onnx"
165
 
166
- # All Torch-TRT (green + blue)
167
  hf download alexandrealvaro/CorridorKey --include "torchtrt/**/*.ts"
168
 
169
  # Direct HTTP
170
- # https://huggingface.co/alexandrealvaro/CorridorKey/resolve/main/onnx/fp16/corridorkey_fp16_1024.onnx
171
  ```
172
-
173
- The CorridorKey Runtime ships a wrapper that pulls the right subset for
174
- each platform: `scripts/fetch_models.ps1 -Profile windows-rtx`,
175
- `-Profile windows-rtx-blue`, `-Profile apple`, etc.
176
-
177
- ## Runtime
178
-
179
- Models above are consumed by
180
- [CorridorKey Runtime](https://github.com/alexandremendoncaalvaro/CorridorKey-Runtime),
181
- shipped as:
182
-
183
- - OFX plugin for DaVinci Resolve and Foundry Nuke (Windows RTX, macOS Apple Silicon)
184
- - CLI for offline batch processing
185
- - Browser demo (uses the INT8 ONNX rung above)
186
-
187
- ## License
188
-
189
- CC BY-NC-SA 4.0 (matches upstream).
 
15
  pipeline_tag: image-segmentation
16
  ---
17
 
18
+ # CorridorKey — model variants
19
 
20
+ Ready-to-load inference variants of the CorridorKey chroma-keying network
21
+ in the formats common runtimes consume: ONNX (FP16 + INT8), Torch-TensorRT
22
+ TorchScript engines, and MLX safetensors + bridges for Apple Silicon.
23
+
24
+ License: **CC BY-NC-SA 4.0**.
 
25
 
26
  ## Upstream sources
27
 
28
+ Training checkpoints (`.pth`, base `.safetensors`) are not republished
29
+ here; fetch them from the original repos:
30
 
31
+ | Variant | Upstream |
32
  |---|---|
33
+ | Green | [`nikopueringer/CorridorKey_v1.0`](https://huggingface.co/nikopueringer/CorridorKey_v1.0) |
34
+ | Blue | [`nikopueringer/CorridorKeyBlue_1.0`](https://huggingface.co/nikopueringer/CorridorKeyBlue_1.0) |
35
 
36
+ Everything in this repo is derived from those checkpoints.
37
 
38
+ ## ONNX FP16
39
 
40
+ Static-shape models for ONNX Runtime (TensorRT, DirectML, CUDA, CPU EPs).
 
 
 
41
 
42
+ | File | Resolution | Size |
 
 
43
  |---|---|---|
44
  | `onnx/fp16/corridorkey_fp16_512.onnx` | 512 × 512 | ~145 MB |
45
  | `onnx/fp16/corridorkey_fp16_1024.onnx` | 1024 × 1024 | ~156 MB |
46
  | `onnx/fp16/corridorkey_fp16_1536.onnx` | 1536 × 1536 | ~174 MB |
47
  | `onnx/fp16/corridorkey_fp16_2048.onnx` | 2048 × 2048 | ~200 MB |
48
 
49
+ ## ONNX FP16 context cache
50
 
51
+ Companion `_ctx.onnx` files for ONNX Runtime's TensorRT EP. When paired
52
+ with the matching FP16 model, the EP loads a pre-built TensorRT engine
53
+ instead of compiling on first run. Pair by resolution.
54
 
55
+ | File | Resolution | Size |
56
  |---|---|---|
57
  | `onnx/fp16_ctx/corridorkey_fp16_512_ctx.onnx` | 512 × 512 | ~167 MB |
58
  | `onnx/fp16_ctx/corridorkey_fp16_1024_ctx.onnx` | 1024 × 1024 | ~178 MB |
59
  | `onnx/fp16_ctx/corridorkey_fp16_1536_ctx.onnx` | 1536 × 1536 | ~197 MB |
60
  | `onnx/fp16_ctx/corridorkey_fp16_2048_ctx.onnx` | 2048 × 2048 | ~223 MB |
61
 
62
+ ## ONNX INT8
63
 
64
+ Quantised FP16 INT8 models. Lower memory footprint, lower precision
65
+ than FP16; useful where GPU memory or model size matters more than
66
+ edge fidelity (browsers, lower-tier hardware).
67
 
68
+ | File | Resolution | Size |
69
  |---|---|---|
70
  | `onnx/int8/corridorkey_int8_512.onnx` | 512 × 512 | ~79 MB |
71
  | `onnx/int8/corridorkey_int8_768.onnx` | 768 × 768 | ~88 MB |
72
  | `onnx/int8/corridorkey_int8_1024.onnx` | 1024 × 1024 | ~101 MB |
73
 
74
+ ## Torch-TensorRT — green
75
 
76
  Pre-compiled TorchScript engines (Torch-TensorRT 2.8.0 + TensorRT 10.12,
77
+ CUDA 12.8 toolchain). Loaded via `torch::jit::load`. The runtime
78
+ process must have the matching `torch_tensorrt` runtime DLLs reachable
79
+ to the OS loader.
80
 
81
+ | File | Resolution | Precision | Size |
82
  |---|---|---|---|
83
  | `torchtrt/fp16/corridorkey_torchtrt_fp16_512.ts` | 512 × 512 | FP16 | ~408 MB |
84
  | `torchtrt/fp16/corridorkey_torchtrt_fp16_1024.ts` | 1024 × 1024 | FP16 | ~430 MB |
85
  | `torchtrt/fp16/corridorkey_torchtrt_fp16_1536.ts` | 1536 × 1536 | FP16 | ~502 MB |
86
  | `torchtrt/fp16/corridorkey_torchtrt_fp16_2048.ts` | 2048 × 2048 | FP16 | ~558 MB |
87
 
88
+ ## Torch-TensorRT — blue
89
+
90
+ Same toolchain as green, dedicated to the CorridorKeyBlue checkpoint.
91
 
92
+ **Precision split:** FP16 for 512 / 1024, FP32 for 1536 / 2048. The
93
+ blue checkpoint produces NaN activations at higher resolutions when
94
+ compiled at FP16; FP32 above 1024 is the workaround until the
95
+ checkpoint is retrained.
96
 
97
+ | File | Resolution | Precision | Size |
98
  |---|---|---|---|
99
+ | `torchtrt/fp16-blue/corridorkey_blue_torchtrt_fp16_512.ts` | 512 × 512 | FP16 | _not yet published_ |
100
  | `torchtrt/fp16-blue/corridorkey_blue_torchtrt_fp16_1024.ts` | 1024 × 1024 | FP16 | ~258 MB |
101
  | `torchtrt/fp32-blue/corridorkey_blue_torchtrt_fp32_1536.ts` | 1536 × 1536 | FP32 | ~488 MB |
102
+ | `torchtrt/fp32-blue/corridorkey_blue_torchtrt_fp32_2048.ts` | 2048 × 2048 | FP32 | _not yet published_ |
103
 
104
+ ## MLX — Apple Silicon
105
 
106
+ Safetensors weights for the MLX framework plus per-resolution
107
+ pre-compiled bridges. The `.safetensors` here is re-exported for the
108
+ MLX layout and differs byte-wise from the upstream
109
+ `CorridorKey_v1.0.safetensors`; use it together with the bridges below.
110
 
111
+ | File | Description | Size |
112
  |---|---|---|
113
+ | `mlx/corridorkey_mlx.safetensors` | Re-exported base weights | ~399 MB |
114
  | `mlx/corridorkey_mlx_bridge_512.mlxfn` | 512 × 512 bridge | ~289 MB |
115
  | `mlx/corridorkey_mlx_bridge_768.mlxfn` | 768 × 768 bridge | ~298 MB |
116
  | `mlx/corridorkey_mlx_bridge_1024.mlxfn` | 1024 × 1024 bridge | ~311 MB |
117
  | `mlx/corridorkey_mlx_bridge_1536.mlxfn` | 1536 × 1536 bridge | ~348 MB |
118
  | `mlx/corridorkey_mlx_bridge_2048.mlxfn` | 2048 × 2048 bridge | ~399 MB |
119
 
120
+ ## Hint trackers
 
 
 
121
 
122
+ Auxiliary models used by alpha-hint generation pipelines (interactive
123
+ matte refinement, segment-then-track flows). Exported from
124
+ [MobileSAM](https://github.com/ChaoningZhang/MobileSAM) and
125
+ [Cutie](https://github.com/hkchengrex/Cutie) under their own licences.
126
 
127
+ | File | Size |
 
 
 
 
128
  |---|---|
129
  | `hint/mobilesam_image_encoder_fp16.onnx` | ~14 MB |
130
  | `hint/mobilesam_prompt_decoder_fp16.onnx` | ~8 MB |
 
132
  | `hint/cutie_init_memory_fp16.onnx` | ~1 MB |
133
  | `hint/cutie_tracking_memory_fp16.onnx` | ~1 MB |
134
 
135
+ ## Releases
 
 
 
 
 
 
 
136
 
137
+ `releases/v*/` contains tagged builds of the
138
+ [CorridorKey Runtime](https://github.com/alexandremendoncaalvaro/CorridorKey-Runtime)
139
+ OFX plugin and CLI for Windows, packaged with the model subset each
140
+ build expects.
 
 
141
 
142
  ## Usage
143
 
144
  ```bash
145
+ # Single file
146
  hf download alexandrealvaro/CorridorKey onnx/fp16/corridorkey_fp16_1024.onnx
147
 
148
+ # All ONNX FP16
149
  hf download alexandrealvaro/CorridorKey --include "onnx/fp16/*.onnx"
150
 
151
+ # All Torch-TensorRT (green + blue)
152
  hf download alexandrealvaro/CorridorKey --include "torchtrt/**/*.ts"
153
 
154
  # Direct HTTP
155
+ # https://huggingface.co/alexandrealvaro/CorridorKey/resolve/main/<path>
156
  ```