|
Download README.md from mantelmoritz/CorridorKey-Realtime-EdgeRefiner: direct link, hf CLI and curl.
- Browser
- Download file 12.6 kB
-
https://huggingface.co/mantelmoritz/CorridorKey-Realtime-EdgeRefiner/resolve/main/README.md
- Command line
-
hf download hf://mantelmoritz/CorridorKey-Realtime-EdgeRefiner/README.md
-
curl -L -o README.md https://huggingface.co/mantelmoritz/CorridorKey-Realtime-EdgeRefiner/resolve/main/README.md
12.6 kB
| license: other | |
| license_name: corridor-key-licence | |
| license_link: LICENSE | |
| library_name: pytorch | |
| pipeline_tag: image-segmentation | |
| tags: | |
| - matting | |
| - alpha-matting | |
| - green-screen | |
| - chroma-key | |
| - keying | |
| - real-time | |
| - obs | |
| - corridorkey | |
| datasets: | |
| - mantelmoritz/CorridorKey-Synthetic-Streamers | |
| # CorridorKey Real-Time Edge Refiner | |
| A small CNN (40k parameters in v15) that restores native-resolution hair and edge detail when | |
| [CorridorKey 2.0](https://huggingface.co/nikopueringer/CorridorKey2.0) runs at reduced resolution for live | |
| green-screen keying, for example a 4K or 1080p facecam in OBS at 30 to 60 fps. | |
|  | |
| <sub>Footage: CorridorKey 1.0 Quality Benchmark (Niko Pueringer / Corridor Digital, Apache-2.0). Real-time | |
| pipeline: CorridorKey 2.0 Stage 1 on a 512x512 person crop, then Edge Refiner v15.</sub> | |
| ## What it is for | |
| At full 4K, CorridorKey 2.0 is an offline model. For live video the network has to run on a small grid, such as a | |
| 512x512 crop around the person or 512x256 for the whole frame, and its matte then has to be scaled back up to the | |
| camera resolution. Bilinear upsampling makes the edge blurry and blocky, with a green or dark halo. A colour guided | |
| filter recovers some structure, but it smears hair, leaves fringes and fails where foreground and screen have | |
| similar brightness. | |
| The refiner looks at the native-resolution pixels and has learned how CorridorKey 2.0 keys them at full | |
| resolution. It predicts corrections for alpha and for the foreground colour (despill and unmixing) at every native | |
| pixel near the edge. | |
|  | |
| Top row: composite over a dark slate. Bottom row: alpha. All crops are 1:1 native pixels, enlarged with nearest | |
| neighbour. "CorridorKey 2.0 offline" is the full model at native resolution, which is what the refiner was trained | |
| to match. | |
| ## How it works | |
| The refiner is fully convolutional: 11 input channels, a 3x3 input convolution, four residual dilated 3x3 blocks | |
| (dilations 1, 2, 4, 1) with GELU, and a 1x1 output convolution. v15 has 32 channels, v12 and v10w have 24 and v9b | |
| has 16. The receptive field is 9 pixels in each direction. Per native pixel it receives: | |
| | Channels | Input | | |
| |---|---| | |
| | 0 to 2 | `I`: native RGB frame (sRGB) | | |
| | 3 to 5 | `B`: smooth background estimate (sRGB), hole-filled from pixels CorridorKey calls background | | |
| | 6 to 8 | `F`: CorridorKey foreground transferred onto the native frame (`I + upsample(F_low - I_low)`) | | |
| | 9 | `a_up`: CorridorKey alpha, bilinear upsampled | | |
| | 10 | `a_gf`: alpha of a colour guided filter driven by CorridorKey's low-resolution alpha | | |
| It outputs residuals on top of `a_gf` (alpha) and `F` (foreground). The result is only used inside an uncertainty | |
| band around CorridorKey's edges. Outside the band the output is the colour-guided upsampling of CorridorKey's matte, | |
| which is CorridorKey's own 0 or 1 in solid foreground and clean background. The band is weighted by nearby | |
| foreground, so the refiner stays out of dark or noisy screen areas. | |
| In the real-time engine the refiner only runs on 32x32 tiles that touch the band, up to a fixed tile budget, most | |
| uncertain tiles first. The tile count decides the cost, not the frame resolution. In the facecam profile, raising | |
| the budget from 256 to 512 tiles adds about 1.8 ms per frame on an RTX 5070 Ti. | |
| `corridorkey_refiner.py` in this repository is a standalone full-frame reference implementation of the same maths. | |
| On a test frame it matches the tiled engine to a mean absolute difference of 0.0001 in alpha. | |
| ## Files | |
| | File | Description | | |
| |---|---| | |
| | `corridorkey_edge_refiner_v15.safetensors` | recommended (October 2026), 32 channels, 40,324 parameters | | |
| | `corridorkey_edge_refiner_v12.safetensors` | previous version, 24 channels, 23,332 parameters, about 0.1 to 0.4 ms cheaper per frame | | |
| | `corridorkey_edge_refiner_v10w.safetensors` | older version, 24 channels | | |
| | `corridorkey_edge_refiner_v9b.safetensors` | 16 channels, 10,948 parameters | | |
| | `corridorkey_refiner.py` | model definition, loader, input construction and full-frame `refine()` | | |
| | `LICENSE` | Corridor Key Licence, Version 2.0 | | |
| The channel count is stored in the safetensors metadata, so `load_refiner` reads every version. | |
| ## Usage | |
| You need CorridorKey 2.0 (code and weights from https://huggingface.co/nikopueringer/CorridorKey2.0, not included | |
| here) to produce the low-resolution matte. Then: | |
| ```python | |
| import torch | |
| from corridorkey_refiner import load_refiner, refine | |
| net = load_refiner("corridorkey_edge_refiner_v15.safetensors") # fp16, CUDA | |
| # rgb: (1, 3, H, W) native frame, sRGB in [0, 1] | |
| # rgb_low: (1, 3, h, w) the (crop of the) frame as CorridorKey saw it | |
| # alpha_low: (1, 1, h', w') CorridorKey alpha at its output resolution (e.g. Stage-1 aux head, 256x256) | |
| # fg_low: (1, 3, h', w') CorridorKey straight foreground, sRGB | |
| alpha, fg = refine(net, rgb, rgb_low, alpha_low, fg_low) # native resolution | |
| ``` | |
| If you key a crop, pass the native-resolution crop as `rgb` and paste the result back. Apply despill and alpha | |
| contrast after the refiner, as you would for CorridorKey's own output. | |
| ONNX graphs of the refiner (fixed tile batches of 256, 384, 512 and 768) are in | |
| [mantelmoritz/CorridorKey-ONNX-Graphs](https://huggingface.co/mantelmoritz/CorridorKey-ONNX-Graphs). | |
| ## Results | |
| Edge error is the mean absolute alpha error inside the edge band (x1000), measured against CorridorKey 2.0 offline | |
| at native resolution. Flicker is the error of the frame-to-frame alpha change in the edge band (x1000). Lower is | |
| better. Each segment has 32 consecutive frames, and none of these clips were used for training. | |
| - Real stock footage: 6 Pexels and Mixkit green-screen clips (4K, 4K DCI, 1080p, 720p) with curly, wavy and long | |
| hair, headphones, a headset and glasses. | |
| - Real 4K, hair and Real 4K, transparency: the two clips of the CorridorKey 1.0 Quality Benchmark, 2 segments | |
| each. The druid has curly hair, a beard and fur; the clown has tulle and water glasses. | |
| - Generated streamers: the 8 series-3 test clips of version 2 of the dataset. They are deliberately hard, with | |
| props in front of the person, partly covered screens and people leaving the frame. | |
| Person crop, 512x512 (CorridorKey 2.0 Stage 1 on a tracked square around the person, 512 tiles): | |
| | Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) | | |
| |---|---|---|---|---|---| | |
| | Bilinear | 121.3 | 91.2 | 80.1 | 134.4 | 75.3 | | |
| | Colour guided filter | 51.2 | 45.3 | 64.1 | 103.8 | 33.1 | | |
| | Edge Refiner v12 | 29.2 | 28.5 | 61.7 | 82.9 | 19.7 | | |
| | Edge Refiner v15 | 28.5 | 27.8 | 58.7 | 82.7 | 19.2 | | |
| Full frame, 512x256 (the facecam profile for 60 fps, 384 tiles): | |
| | Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) | | |
| |---|---|---|---|---|---| | |
| | Bilinear | 174.7 | 117.8 | 106.7 | 187.8 | 91.5 | | |
| | Colour guided filter | 61.3 | 53.4 | 76.4 | 137.5 | 34.9 | | |
| | Edge Refiner v12 | 34.8 | 38.7 | 70.9 | 115.9 | 22.2 | | |
| | Edge Refiner v15 | 34.2 | 38.4 | 68.7 | 116.3 | 21.7 | | |
| End-to-end GPU time per frame on an RTX 5070 Ti, CorridorKey 2.0 Stage 1 included, without video I/O: about | |
| 16.5 ms for the person crop and 12 to 13 ms for the full frame. v15 costs 0.1 to 0.4 ms more than v12. | |
| Against the guided filter, v15 has 44 % less edge error on the real stock clips with the person crop, 39 % less on | |
| the druid and 42 % less flicker. Against v12, v15 has 1 to 5 % less edge error on the real clips and about 2 % less | |
| flicker. On the generated streamers both are level within 0.4 %. The largest gains are on hair. On | |
| semi-transparent material such as the clown's tulle the result is limited by what the low-resolution network sees. | |
| In a matched comparison (same patches and steps, only the width changed), going from 24 to 32 channels lowered the | |
| edge error on all six of our test sets in both profiles. 48 channels gained nothing more and cost 6 to 7 ms per frame. | |
|  | |
| ## Training | |
| - Target: CorridorKey 2.0 offline at native resolution, alpha and foreground. The refiner is a student of | |
| CorridorKey. It cannot be sharper than its teacher and it inherits the teacher's mistakes. | |
| - Inputs during training are exactly what the real-time engine produces at run time. The training patches were cut | |
| by running the real-time pipeline (512x512 person crop and 512x256 full frame, Stage 1 only) over every training | |
| clip and computing the 11 input channels, mostly from the edge band. | |
| - Data: | |
| - 32 segments of real stock green-screen footage (Pexels, Mixkit) | |
| - the same frames re-composited from the teacher's foreground and alpha over procedural green and teal screens, | |
| also passed through a camera model with sensor noise, Bayer demosaicing, 4:2:0 chroma, JPEG and exposure errors | |
| - the 207 training clips of version 2 of the | |
| [CorridorKey Synthetic Streamers](https://huggingface.co/datasets/mantelmoritz/CorridorKey-Synthetic-Streamers) | |
| dataset. Every clip was reviewed by eye, broken labels were removed, and everything in front of the screen | |
| (mugs, microphones, a second person) counts as foreground. | |
| Real footage is sampled 3 times as often as its share. Loss on alpha, alpha gradients, composite and foreground. | |
| v15: a random 90,000 patches held in memory, 96,000 steps. v12: 110,000 patches, 96,000 steps. | |
| - Tried and dropped: removing screen colour from the foreground before re-compositing (it makes the synthetic frames | |
| cleaner than real camera frames; edge error rose by 3 to 5 %), 25 % blue synthetic screens (worse on every test | |
| set) and 48 channels (see above). | |
| - The evaluation clips were never used for training. | |
| ## Limitations | |
|  | |
| - The teacher is the ceiling. Where CorridorKey 2.0 is soft at native resolution, the refiner is soft too. | |
| - It can only refine what the low-resolution network found. Single strands far outside CorridorKey's | |
| low-resolution matte lie outside the refiner band and are lost, as in the image above. Semi-transparent objects | |
| (glass, tulle, water) depend almost entirely on the low-resolution network. | |
| - It was trained for green, teal and blue-green screens with people in front, mostly facecam and upper-body | |
| framing. It is not a general background-removal model. | |
| - The numbers above compare against CorridorKey 2.0, not against hand-made ground truth. | |
| ## Licence | |
| Corridor Key Licence, Version 2.0 (CC BY-NC-SA 4.0 with Additional Terms by Corridor Digital LLC). The full text is | |
| in [`LICENSE`](LICENSE). | |
| The refiner was trained on the outputs of CorridorKey 2.0 and only works together with it, so it is released as a | |
| CorridorKey derivative under the same licence and keeps the CorridorKey name (Additional Terms §4, §5). In short: | |
| - It is free of charge. You may not sell it, repackage it as a standalone or competing product, or offer it as a | |
| paid API or inference service. | |
| - Derivatives must be released under the same licence, free of charge, and keep the "CorridorKey" name. | |
| - Images and video you produce with it are yours. | |
| - Commercial software integrations need a separate agreement with Corridor Digital (contact@corridordigital.com). | |
| Organisations above US $20M revenue need CorridorKey plugin seats (https://corridorkey.com). | |
| ### Attribution and modifications | |
| - CorridorKey 2.0 © Corridor Digital LLC. The model was designed and trained by Zachary Harris in collaboration | |
| with Niko Pueringer and Corridor Digital. https://huggingface.co/nikopueringer/CorridorKey2.0, | |
| https://corridorkey.com | |
| - CorridorKey 2.0 builds on frozen foundation models under their own licences (MoGe-2 by Microsoft, RVM by Google | |
| DeepMind, optional C-RADIO by NVIDIA). None of them, and no CorridorKey weights, are contained in this repository. | |
| - Changes relative to CorridorKey 2.0: this repository adds a new, separately trained edge-refinement network and | |
| its input pipeline. The CorridorKey 2.0 weights and code are not modified or redistributed. | |
| - Example footage: CorridorKey 1.0 Quality Benchmark | |
| (https://huggingface.co/datasets/nikopueringer/CorridorKey1.0_QualityBenchmark, Apache-2.0), Pexels (Pexels | |
| licence), and the CorridorKey Synthetic Streamers dataset. | |
| - This is an independent community project. It is not affiliated with, sponsored by or endorsed by Corridor | |
| Digital LLC. | |