ControlNet Preprocessors (ONNX) β Modular & Unified I/O Ecosystem
This repository provides pre-converted ONNX models for controlnet_aux preprocessors.
Unlike standalone model dumps, this collection is designed as a modular pipeline ecosystem with unified Input/Output specifications. Complex pre-processing, test-time augmentations, and post-processing filters (such as safe_step, scribble, depth-to-normal conversion, etc.) are completely encapsulated inside ONNX graphs.
This allows you to run the entire ControlNet preprocessing pipeline in non-Python environments (C#, C++, Rust, Web, etc.) or on hardware accelerators (NPU/DirectML/CUDA) by simply chaining ONNX outputs directly into the next ONNX inputs.
π Key Features
- Unified Base I/O: Standard models accept raw
uint8BGR images directly from OpenCV and output normalizedfloat32RGB tensors ready for ControlNet. - Modular Chaining: Post-processing filters (
filter_*.onnx) and multi-stage pipelines (ZoeDepth, MiDaS Normal) share compatible tensor formats (float32 [N, 3, H, W]), enabling zero-copy/minimal-overhead chaining. - Verified Precision: Validated against original
controlnet_auxoutputs usingpytest, achieving >97% pixel match rate. - Fast: Up to 13x faster on CPU and 90x faster on NPU compared to the original Python implementation.
π Core I/O Specification
Standard Single-Stage Models
(HED, PidiNet, TEED, Lineart, Lineart Coarse, Lineart Anime, NormalBae, LeReS)
- Input:
image(uint8[Batch, Height, Width, 3]) β BGR ([0 - 255], directly compatible with OpenCV). - Output:
output(float32[Batch, 3, Height, Width]) β RGB ([0.0 - 1.0], ready for ControlNet / Filters).
π Multi-Stage Pipelines & Special I/O
1. ZoeDepth Pipeline (3-Stage Split)
Split into 3 modules to prevent memory spikes during flip-augmentation and optimize NPU execution:
zoe_preprocessor.onnx- Input:
image(uint8 [N, H, W, 3]BGR) - Outputs:
normalized_image(float32 [N, 3, H, W]),normalized_image_flipped(float32 [N, 3, H, W])
- Input:
zoe_inference.onnx(Run twice: once for normal, once for flipped)- Input:
normalized_image(float32 [N, 3, H, W]) - Output:
depth(float32 [N, 1, H, W])
- Input:
zoe_postprocessor.onnx- Inputs:
depth(float32 [N, 1, H, W]),depth_flipped(float32 [N, 1, H, W]) - Output:
output(float32 [N, 3, H, W]RGB[0.0 - 1.0])
- Inputs:
2. MiDaS Depth & Normal Pipeline
midas.onnx outputs both the normalized depth map and the raw depth tensor, allowing you to generate a Surface Normal map without re-running inference:
midas.onnx- Input:
image(uint8 [N, H, W, 3]BGR) - Outputs:
output(float32 [N, 3, H, W]RGB[0.0 - 1.0]) β Standard Depth Mapraw_depth(float32 [N, H, W]) β Unnormalized depth for Normal calculation
- Input:
midas_postprocessor_normal.onnx- Inputs:
raw_depth(float32 [N, H, W])a(float32 [1]) β Normal norm parameter (default:np.pi * 2.0)th_bg(float32 [1]) β Background removal threshold (default:0.1)
- Output:
output(float32 [N, 3, H, W]RGB[0.0 - 1.0]) β Normal Map
- Inputs:
3. Post-Processing Filters (filter_*.onnx)
All filter models accept the standard output tensor from any detector above and return a filtered tensor in the exact same format, allowing arbitrary chaining.
- Common Input:
normalized_image(float32 [N, 3, H, W]RGB[0.0 - 1.0]) - Common Output:
output(float32 [N, 3, H, W]RGB[0.0 - 1.0])
| Filter File | Target Models | Additional Inputs | Description |
|---|---|---|---|
filter_gamma_corrected.onnx |
ZoeDepth | None | Applies gamma correction to depth maps. |
filter_pidi.onnx |
PidiNet | None | Applies PidiNet-specific edge threshold filtering (edge > 0.5). |
filter_safe_step.onnx |
HED, PidiNet, TEED | step (int64 [1]) |
Quantizes/clips edge contrast cleanly (safe=True). |
filter_scribble.onnx |
HED, PidiNet | None | Applies Non-Maximum Suppression (NMS) for scribble-style thin lines. |
filter_remove_background.onnx |
LeReS | removal_threshold (int64 [1]) |
Removes far background based on depth threshold [0-100]. |
filter_remove_near.onnx |
LeReS | removal_threshold (int64 [1]) |
Removes near foreground based on depth threshold [0-100]. |
π¦ Model Inventory & Licenses
While the wrapper scripts and conversion code are licensed under Apache 2.0, the weights embedded in each ONNX file retain the license of their respective original authors.
| Model / Component | Task | Original License | Commercial Use |
|---|---|---|---|
hed.onnx |
Edge Detection | Apache 2.0 | β Allowed |
teed.onnx |
Edge Detection | MIT | β Allowed |
midas.onnx & midas_postprocessor_normal.onnx |
Depth / Normal | MIT | β Allowed |
zoe_*.onnx (Pre / Infer / Post) |
Depth Estimation | MIT | β Allowed |
normalbae.onnx |
Normal Map | MIT | β Allowed |
lineart.onnx |
Lineart | MIT | β Allowed |
lineart_coarse.onnx |
Lineart (Coarse) | MIT | β Allowed |
lineart_anime.onnx |
Lineart (Anime) | MIT | β Allowed |
filter_*.onnx (All 6 filters) |
Post-processing | Apache 2.0 | β Allowed |
pidi.onnx |
Edge Detection | β οΈ OTHER | β Non-Commercial |
leres.onnx |
Depth Estimation | β οΈ Adobe Research License | β Non-Commercial |
LeReS: Not hosted on Hugging Face due to the original Adobe Research License (no redistribution allowed). Please use the conversion script (export_leres.py) in the GitHub repository to generate the ONNX file locally.
β‘ Benchmark Results
Tested on Snapdragon X Elite (CPU & Hexagon NPU) with 512x512 input resolution (average of 5 images):
CPU Inference (CPUExecutionProvider)
| Model | Original (controlnet_aux) |
ONNX Runtime (CPU) | Speedup |
|---|---|---|---|
| HED | 2509.32 ms | 183.43 ms | 13.68x faster |
| NormalBae | 5977.35 ms | 487.33 ms | 12.27x faster |
| MiDaS | 8279.16 ms | 731.03 ms | 11.33x faster |
| PidiNet | 1782.00 ms | 167.53 ms | 10.64x faster |
| LeReS | 5363.05 ms | 690.56 ms | 7.77x faster |
| Lineart (Coarse) | 3278.46 ms | 459.72 ms | 7.13x faster |
| Lineart | 3250.57 ms | 457.40 ms | 7.11x faster |
| ZoeDepth (Total) | 32620.80 ms | 5717.00 ms | 5.71x faster |
| Lineart Anime | 841.17 ms | 165.95 ms | 5.07x faster |
| TEED | 205.23 ms | 95.33 ms | 2.15x faster |
[WIP] NPU Inference (Pre-compiled QNN / FP16)
| Model | Original (controlnet_aux) |
ONNX Runtime (NPU) | Speedup |
|---|---|---|---|
| HED | 2390.74 ms | 26.49 ms | 90.26x faster |
| PidiNet | 1321.31 ms | 43.40 ms | 30.45x faster |
π Quick Start (Chaining Example in Python)
import cv2
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
repo_id = "Buuta/controlnet_aux_onnx"
# 1. Download base detector and post-processing filter
hed_path = hf_hub_download(repo_id=repo_id, filename="hed.onnx")
scribble_path = hf_hub_download(repo_id=repo_id, filename="filter_scribble.onnx")
# 2. Prepare OpenCV BGR image
img = cv2.imread("input.png")
input_tensor = cv2.resize(img, (512, 512))[np.newaxis, ...] # uint8 [1, 512, 512, 3]
# 3. Run HED -> Chain directly into Scribble (NMS) Filter
hed_sess = ort.InferenceSession(hed_path, providers=["CPUExecutionProvider"])
scribble_sess = ort.InferenceSession(scribble_path, providers=["CPUExecutionProvider"])
hed_out = hed_sess.run(["output"], {"image": input_tensor})[0] # float32 [1, 3, 512, 512]
scribble_out = scribble_sess.run(["output"], {"normalized_image": hed_out})[0] # float32 [1, 3, 512, 512]
# 4. Save result
out_img = (np.transpose(scribble_out[0], (1, 2, 0))[:, :, ::-1] * 255.0).clip(0, 255).astype(np.uint8)
cv2.imwrite("output_hed_scribble.png", out_img)
π οΈ Source Code & C# Integration
For conversion scripts, pytest test suites, and C# (OpenCvSharp4 + OnnxRuntime) pipeline examples, please visit the GitHub repository:
Model tree for Buuta/controlnet_aux_onnx
Base model
lllyasviel/Annotators