README.md

---
license: other
license_name: corridor-key-licence
license_link: LICENSE
library_name: pytorch
pipeline_tag: image-segmentation
tags:
  - matting
  - alpha-matting
  - green-screen
  - chroma-key
  - keying
  - real-time
  - obs
  - corridorkey
datasets:
  - mantelmoritz/CorridorKey-Synthetic-Streamers
---

# CorridorKey Real-Time Edge Refiner

A small CNN (40k parameters in v15) that restores native-resolution hair and edge detail when
[CorridorKey 2.0](https://huggingface.co/nikopueringer/CorridorKey2.0) runs at reduced resolution for live
green-screen keying, for example a 4K or 1080p facecam in OBS at 30 to 60 fps.

![Input and real-time result on real 4K green-screen footage](images/hero_druid_4k.png)

<sub>Footage: CorridorKey 1.0 Quality Benchmark (Niko Pueringer / Corridor Digital, Apache-2.0). Real-time
pipeline: CorridorKey 2.0 Stage 1 on a 512x512 person crop, then Edge Refiner v15.</sub>

## What it is for

At full 4K, CorridorKey 2.0 is an offline model. For live video the network has to run on a small grid, such as a
512x512 crop around the person or 512x256 for the whole frame, and its matte then has to be scaled back up to the
camera resolution. Bilinear upsampling makes the edge blurry and blocky, with a green or dark halo. A colour guided
filter recovers some structure, but it smears hair, leaves fringes and fails where foreground and screen have
similar brightness.

The refiner looks at the native-resolution pixels and has learned how CorridorKey 2.0 keys them at full
resolution. It predicts corrections for alpha and for the foreground colour (despill and unmixing) at every native
pixel near the edge.

![1:1 native-pixel crops on real 4K footage](images/compare_druid_4k.png)

Top row: composite over a dark slate. Bottom row: alpha. All crops are 1:1 native pixels, enlarged with nearest
neighbour. "CorridorKey 2.0 offline" is the full model at native resolution, which is what the refiner was trained
to match.

## How it works

The refiner is fully convolutional: 11 input channels, a 3x3 input convolution, four residual dilated 3x3 blocks
(dilations 1, 2, 4, 1) with GELU, and a 1x1 output convolution. v15 has 32 channels, v12 and v10w have 24 and v9b
has 16. The receptive field is 9 pixels in each direction. Per native pixel it receives:

| Channels | Input |
|---|---|
| 0 to 2 | `I`: native RGB frame (sRGB) |
| 3 to 5 | `B`: smooth background estimate (sRGB), hole-filled from pixels CorridorKey calls background |
| 6 to 8 | `F`: CorridorKey foreground transferred onto the native frame (`I + upsample(F_low - I_low)`) |
| 9 | `a_up`: CorridorKey alpha, bilinear upsampled |
| 10 | `a_gf`: alpha of a colour guided filter driven by CorridorKey's low-resolution alpha |

It outputs residuals on top of `a_gf` (alpha) and `F` (foreground). The result is only used inside an uncertainty
band around CorridorKey's edges. Outside the band the output is the colour-guided upsampling of CorridorKey's matte,
which is CorridorKey's own 0 or 1 in solid foreground and clean background. The band is weighted by nearby
foreground, so the refiner stays out of dark or noisy screen areas.

In the real-time engine the refiner only runs on 32x32 tiles that touch the band, up to a fixed tile budget, most
uncertain tiles first. The tile count decides the cost, not the frame resolution. In the facecam profile, raising
the budget from 256 to 512 tiles adds about 1.8 ms per frame on an RTX 5070 Ti.

`corridorkey_refiner.py` in this repository is a standalone full-frame reference implementation of the same maths.
On a test frame it matches the tiled engine to a mean absolute difference of 0.0001 in alpha.

## Files

| File | Description |
|---|---|
| `corridorkey_edge_refiner_v15.safetensors` | recommended (October 2026), 32 channels, 40,324 parameters |
| `corridorkey_edge_refiner_v12.safetensors` | previous version, 24 channels, 23,332 parameters, about 0.1 to 0.4 ms cheaper per frame |
| `corridorkey_edge_refiner_v10w.safetensors` | older version, 24 channels |
| `corridorkey_edge_refiner_v9b.safetensors` | 16 channels, 10,948 parameters |
| `corridorkey_refiner.py` | model definition, loader, input construction and full-frame `refine()` |
| `LICENSE` | Corridor Key Licence, Version 2.0 |

The channel count is stored in the safetensors metadata, so `load_refiner` reads every version.

## Usage

You need CorridorKey 2.0 (code and weights from https://huggingface.co/nikopueringer/CorridorKey2.0, not included
here) to produce the low-resolution matte. Then:

```python
import torch
from corridorkey_refiner import load_refiner, refine

net = load_refiner("corridorkey_edge_refiner_v15.safetensors")    # fp16, CUDA

# rgb:       (1, 3, H, W)   native frame, sRGB in [0, 1]
# rgb_low:   (1, 3, h, w)   the (crop of the) frame as CorridorKey saw it
# alpha_low: (1, 1, h', w') CorridorKey alpha at its output resolution (e.g. Stage-1 aux head, 256x256)
# fg_low:    (1, 3, h', w') CorridorKey straight foreground, sRGB
alpha, fg = refine(net, rgb, rgb_low, alpha_low, fg_low)           # native resolution
```

If you key a crop, pass the native-resolution crop as `rgb` and paste the result back. Apply despill and alpha
contrast after the refiner, as you would for CorridorKey's own output.

ONNX graphs of the refiner (fixed tile batches of 256, 384, 512 and 768) are in
[mantelmoritz/CorridorKey-ONNX-Graphs](https://huggingface.co/mantelmoritz/CorridorKey-ONNX-Graphs).

## Results

Edge error is the mean absolute alpha error inside the edge band (x1000), measured against CorridorKey 2.0 offline
at native resolution. Flicker is the error of the frame-to-frame alpha change in the edge band (x1000). Lower is
better. Each segment has 32 consecutive frames, and none of these clips were used for training.

- Real stock footage: 6 Pexels and Mixkit green-screen clips (4K, 4K DCI, 1080p, 720p) with curly, wavy and long
  hair, headphones, a headset and glasses.
- Real 4K, hair and Real 4K, transparency: the two clips of the CorridorKey 1.0 Quality Benchmark, 2 segments
  each. The druid has curly hair, a beard and fur; the clown has tulle and water glasses.
- Generated streamers: the 8 series-3 test clips of version 2 of the dataset. They are deliberately hard, with
  props in front of the person, partly covered screens and people leaving the frame.

Person crop, 512x512 (CorridorKey 2.0 Stage 1 on a tracked square around the person, 512 tiles):

| Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) |
|---|---|---|---|---|---|
| Bilinear | 121.3 | 91.2 | 80.1 | 134.4 | 75.3 |
| Colour guided filter | 51.2 | 45.3 | 64.1 | 103.8 | 33.1 |
| Edge Refiner v12 | 29.2 | 28.5 | 61.7 | 82.9 | 19.7 |
| Edge Refiner v15 | 28.5 | 27.8 | 58.7 | 82.7 | 19.2 |

Full frame, 512x256 (the facecam profile for 60 fps, 384 tiles):

| Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) |
|---|---|---|---|---|---|
| Bilinear | 174.7 | 117.8 | 106.7 | 187.8 | 91.5 |
| Colour guided filter | 61.3 | 53.4 | 76.4 | 137.5 | 34.9 |
| Edge Refiner v12 | 34.8 | 38.7 | 70.9 | 115.9 | 22.2 |
| Edge Refiner v15 | 34.2 | 38.4 | 68.7 | 116.3 | 21.7 |

End-to-end GPU time per frame on an RTX 5070 Ti, CorridorKey 2.0 Stage 1 included, without video I/O: about
16.5 ms for the person crop and 12 to 13 ms for the full frame. v15 costs 0.1 to 0.4 ms more than v12.

Against the guided filter, v15 has 44 % less edge error on the real stock clips with the person crop, 39 % less on
the druid and 42 % less flicker. Against v12, v15 has 1 to 5 % less edge error on the real clips and about 2 % less
flicker. On the generated streamers both are level within 0.4 %. The largest gains are on hair. On
semi-transparent material such as the clown's tulle the result is limited by what the low-resolution network sees.

In a matched comparison (same patches and steps, only the width changed), going from 24 to 32 channels lowered the
edge error on all six of our test sets in both profiles. 48 channels gained nothing more and cost 6 to 7 ms per frame.

![Generated streamer clip from the test split, 1:1 crops](images/compare_generated_streamer.png)

## Training

- Target: CorridorKey 2.0 offline at native resolution, alpha and foreground. The refiner is a student of
  CorridorKey. It cannot be sharper than its teacher and it inherits the teacher's mistakes.
- Inputs during training are exactly what the real-time engine produces at run time. The training patches were cut
  by running the real-time pipeline (512x512 person crop and 512x256 full frame, Stage 1 only) over every training
  clip and computing the 11 input channels, mostly from the edge band.
- Data:
  - 32 segments of real stock green-screen footage (Pexels, Mixkit)
  - the same frames re-composited from the teacher's foreground and alpha over procedural green and teal screens,
    also passed through a camera model with sensor noise, Bayer demosaicing, 4:2:0 chroma, JPEG and exposure errors
  - the 207 training clips of version 2 of the
    [CorridorKey Synthetic Streamers](https://huggingface.co/datasets/mantelmoritz/CorridorKey-Synthetic-Streamers)
    dataset. Every clip was reviewed by eye, broken labels were removed, and everything in front of the screen
    (mugs, microphones, a second person) counts as foreground.

  Real footage is sampled 3 times as often as its share. Loss on alpha, alpha gradients, composite and foreground.
  v15: a random 90,000 patches held in memory, 96,000 steps. v12: 110,000 patches, 96,000 steps.
- Tried and dropped: removing screen colour from the foreground before re-compositing (it makes the synthetic frames
  cleaner than real camera frames; edge error rose by 3 to 5 %), 25 % blue synthetic screens (worse on every test
  set) and 48 channels (see above).
- The evaluation clips were never used for training.

## Limitations

![Limitation: free-flying strands far from the head](images/compare_facecam_1080.png)

- The teacher is the ceiling. Where CorridorKey 2.0 is soft at native resolution, the refiner is soft too.
- It can only refine what the low-resolution network found. Single strands far outside CorridorKey's
  low-resolution matte lie outside the refiner band and are lost, as in the image above. Semi-transparent objects
  (glass, tulle, water) depend almost entirely on the low-resolution network.
- It was trained for green, teal and blue-green screens with people in front, mostly facecam and upper-body
  framing. It is not a general background-removal model.
- The numbers above compare against CorridorKey 2.0, not against hand-made ground truth.

## Licence

Corridor Key Licence, Version 2.0 (CC BY-NC-SA 4.0 with Additional Terms by Corridor Digital LLC). The full text is
in [`LICENSE`](LICENSE).

The refiner was trained on the outputs of CorridorKey 2.0 and only works together with it, so it is released as a
CorridorKey derivative under the same licence and keeps the CorridorKey name (Additional Terms §4, §5). In short:

- It is free of charge. You may not sell it, repackage it as a standalone or competing product, or offer it as a
  paid API or inference service.
- Derivatives must be released under the same licence, free of charge, and keep the "CorridorKey" name.
- Images and video you produce with it are yours.
- Commercial software integrations need a separate agreement with Corridor Digital (contact@corridordigital.com).
  Organisations above US $20M revenue need CorridorKey plugin seats (https://corridorkey.com).

### Attribution and modifications

- CorridorKey 2.0 © Corridor Digital LLC. The model was designed and trained by Zachary Harris in collaboration
  with Niko Pueringer and Corridor Digital. https://huggingface.co/nikopueringer/CorridorKey2.0,
  https://corridorkey.com
- CorridorKey 2.0 builds on frozen foundation models under their own licences (MoGe-2 by Microsoft, RVM by Google
  DeepMind, optional C-RADIO by NVIDIA). None of them, and no CorridorKey weights, are contained in this repository.
- Changes relative to CorridorKey 2.0: this repository adds a new, separately trained edge-refinement network and
  its input pipeline. The CorridorKey 2.0 weights and code are not modified or redistributed.
- Example footage: CorridorKey 1.0 Quality Benchmark
  (https://huggingface.co/datasets/nikopueringer/CorridorKey1.0_QualityBenchmark, Apache-2.0), Pexels (Pexels
  licence), and the CorridorKey Synthetic Streamers dataset.
- This is an independent community project. It is not affiliated with, sponsored by or endorsed by Corridor
  Digital LLC.