ControlNet Preprocessors (ONNX) β€” Modular & Unified I/O Ecosystem

This repository provides pre-converted ONNX models for controlnet_aux preprocessors.
Unlike standalone model dumps, this collection is designed as a modular pipeline ecosystem with unified Input/Output specifications. Complex pre-processing, test-time augmentations, and post-processing filters (such as safe_step, scribble, depth-to-normal conversion, etc.) are completely encapsulated inside ONNX graphs.

This allows you to run the entire ControlNet preprocessing pipeline in non-Python environments (C#, C++, Rust, Web, etc.) or on hardware accelerators (NPU/DirectML/CUDA) by simply chaining ONNX outputs directly into the next ONNX inputs.

🌟 Key Features

  • Unified Base I/O: Standard models accept raw uint8 BGR images directly from OpenCV and output normalized float32 RGB tensors ready for ControlNet.
  • Modular Chaining: Post-processing filters (filter_*.onnx) and multi-stage pipelines (ZoeDepth, MiDaS Normal) share compatible tensor formats (float32 [N, 3, H, W]), enabling zero-copy/minimal-overhead chaining.
  • Verified Precision: Validated against original controlnet_aux outputs using pytest, achieving >97% pixel match rate.
  • Fast: Up to 13x faster on CPU and 90x faster on NPU compared to the original Python implementation.

πŸ”Œ Core I/O Specification

Standard Single-Stage Models

(HED, PidiNet, TEED, Lineart, Lineart Coarse, Lineart Anime, NormalBae, LeReS)

  • Input: image (uint8 [Batch, Height, Width, 3]) β€” BGR ([0 - 255], directly compatible with OpenCV).
  • Output: output (float32 [Batch, 3, Height, Width]) β€” RGB ([0.0 - 1.0], ready for ControlNet / Filters).

πŸ”— Multi-Stage Pipelines & Special I/O

1. ZoeDepth Pipeline (3-Stage Split)

Split into 3 modules to prevent memory spikes during flip-augmentation and optimize NPU execution:

  1. zoe_preprocessor.onnx
    • Input: image (uint8 [N, H, W, 3] BGR)
    • Outputs: normalized_image (float32 [N, 3, H, W]), normalized_image_flipped (float32 [N, 3, H, W])
  2. zoe_inference.onnx (Run twice: once for normal, once for flipped)
    • Input: normalized_image (float32 [N, 3, H, W])
    • Output: depth (float32 [N, 1, H, W])
  3. zoe_postprocessor.onnx
    • Inputs: depth (float32 [N, 1, H, W]), depth_flipped (float32 [N, 1, H, W])
    • Output: output (float32 [N, 3, H, W] RGB [0.0 - 1.0])

2. MiDaS Depth & Normal Pipeline

midas.onnx outputs both the normalized depth map and the raw depth tensor, allowing you to generate a Surface Normal map without re-running inference:

  • midas.onnx
    • Input: image (uint8 [N, H, W, 3] BGR)
    • Outputs:
      • output (float32 [N, 3, H, W] RGB [0.0 - 1.0]) β€” Standard Depth Map
      • raw_depth (float32 [N, H, W]) β€” Unnormalized depth for Normal calculation
  • midas_postprocessor_normal.onnx
    • Inputs:
      • raw_depth (float32 [N, H, W])
      • a (float32 [1]) β€” Normal norm parameter (default: np.pi * 2.0)
      • th_bg (float32 [1]) β€” Background removal threshold (default: 0.1)
    • Output: output (float32 [N, 3, H, W] RGB [0.0 - 1.0]) β€” Normal Map

3. Post-Processing Filters (filter_*.onnx)

All filter models accept the standard output tensor from any detector above and return a filtered tensor in the exact same format, allowing arbitrary chaining.

  • Common Input: normalized_image (float32 [N, 3, H, W] RGB [0.0 - 1.0])
  • Common Output: output (float32 [N, 3, H, W] RGB [0.0 - 1.0])
Filter File Target Models Additional Inputs Description
filter_gamma_corrected.onnx ZoeDepth None Applies gamma correction to depth maps.
filter_pidi.onnx PidiNet None Applies PidiNet-specific edge threshold filtering (edge > 0.5).
filter_safe_step.onnx HED, PidiNet, TEED step (int64 [1]) Quantizes/clips edge contrast cleanly (safe=True).
filter_scribble.onnx HED, PidiNet None Applies Non-Maximum Suppression (NMS) for scribble-style thin lines.
filter_remove_background.onnx LeReS removal_threshold (int64 [1]) Removes far background based on depth threshold [0-100].
filter_remove_near.onnx LeReS removal_threshold (int64 [1]) Removes near foreground based on depth threshold [0-100].

πŸ“¦ Model Inventory & Licenses

While the wrapper scripts and conversion code are licensed under Apache 2.0, the weights embedded in each ONNX file retain the license of their respective original authors.

Model / Component Task Original License Commercial Use
hed.onnx Edge Detection Apache 2.0 βœ… Allowed
teed.onnx Edge Detection MIT βœ… Allowed
midas.onnx & midas_postprocessor_normal.onnx Depth / Normal MIT βœ… Allowed
zoe_*.onnx (Pre / Infer / Post) Depth Estimation MIT βœ… Allowed
normalbae.onnx Normal Map MIT βœ… Allowed
lineart.onnx Lineart MIT βœ… Allowed
lineart_coarse.onnx Lineart (Coarse) MIT βœ… Allowed
lineart_anime.onnx Lineart (Anime) MIT βœ… Allowed
filter_*.onnx (All 6 filters) Post-processing Apache 2.0 βœ… Allowed
pidi.onnx Edge Detection ⚠️ OTHER ❌ Non-Commercial
leres.onnx Depth Estimation ⚠️ Adobe Research License ❌ Non-Commercial

LeReS: Not hosted on Hugging Face due to the original Adobe Research License (no redistribution allowed). Please use the conversion script (export_leres.py) in the GitHub repository to generate the ONNX file locally.


⚑ Benchmark Results

Tested on Snapdragon X Elite (CPU & Hexagon NPU) with 512x512 input resolution (average of 5 images):

CPU Inference (CPUExecutionProvider)

Model Original (controlnet_aux) ONNX Runtime (CPU) Speedup
HED 2509.32 ms 183.43 ms 13.68x faster
NormalBae 5977.35 ms 487.33 ms 12.27x faster
MiDaS 8279.16 ms 731.03 ms 11.33x faster
PidiNet 1782.00 ms 167.53 ms 10.64x faster
LeReS 5363.05 ms 690.56 ms 7.77x faster
Lineart (Coarse) 3278.46 ms 459.72 ms 7.13x faster
Lineart 3250.57 ms 457.40 ms 7.11x faster
ZoeDepth (Total) 32620.80 ms 5717.00 ms 5.71x faster
Lineart Anime 841.17 ms 165.95 ms 5.07x faster
TEED 205.23 ms 95.33 ms 2.15x faster

[WIP] NPU Inference (Pre-compiled QNN / FP16)

Model Original (controlnet_aux) ONNX Runtime (NPU) Speedup
HED 2390.74 ms 26.49 ms 90.26x faster
PidiNet 1321.31 ms 43.40 ms 30.45x faster

πŸš€ Quick Start (Chaining Example in Python)

import cv2
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download

repo_id = "Buuta/controlnet_aux_onnx"

# 1. Download base detector and post-processing filter
hed_path = hf_hub_download(repo_id=repo_id, filename="hed.onnx")
scribble_path = hf_hub_download(repo_id=repo_id, filename="filter_scribble.onnx")

# 2. Prepare OpenCV BGR image
img = cv2.imread("input.png")
input_tensor = cv2.resize(img, (512, 512))[np.newaxis, ...]  # uint8 [1, 512, 512, 3]

# 3. Run HED -> Chain directly into Scribble (NMS) Filter
hed_sess = ort.InferenceSession(hed_path, providers=["CPUExecutionProvider"])
scribble_sess = ort.InferenceSession(scribble_path, providers=["CPUExecutionProvider"])

hed_out = hed_sess.run(["output"], {"image": input_tensor})[0]          # float32 [1, 3, 512, 512]
scribble_out = scribble_sess.run(["output"], {"normalized_image": hed_out})[0] # float32 [1, 3, 512, 512]

# 4. Save result
out_img = (np.transpose(scribble_out[0], (1, 2, 0))[:, :, ::-1] * 255.0).clip(0, 255).astype(np.uint8)
cv2.imwrite("output_hed_scribble.png", out_img)

πŸ› οΈ Source Code & C# Integration

For conversion scripts, pytest test suites, and C# (OpenCvSharp4 + OnnxRuntime) pipeline examples, please visit the GitHub repository:

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Buuta/controlnet_aux_onnx

Adapter
(1)
this model