How much video memory is required for operation, and does it support ControlNet and IP Adapter
How much video memory is required for operation, and does it support ControlNet and IP Adapter
The examples with low VRAM under usage requires about 16 GB of VRAM. I have not tested controlnet or IP adapters for it. Someone released a version that runs on 12 GB VRAM too.
I've recently put some effort into training a ControlNet for this, and setting up an inference pipeline using CFG. I think it's working properly now...
Training Repo: https://github.com/NeuralVFX/LibreFLUX-ControlNet
Example Model: https://huggingface.co/neuralvfx/LibreFlux-ControlNet
If I quantize both the Transformer and ControlNet with optimum.quanto, VRAM usage goes down to 16GB while generating a 1024x1024 image.
Wow! Awesome!
In case anyone's still interested in an IP Adapter for this, I took some time to finetune one. It's packaged together with my ControlNet in a pipeline ( and ComfyUI nodes ).
- ComfyUI Nodes: https://github.com/NeuralVFX/LibreFLUX-ComfyUI
- Combined Model: https://huggingface.co/neuralvfx/LibreFlux-IP-Adapter-ControlNet
- IP Adapter Training Repo: https://github.com/NeuralVFX/LibreFLUX-IP-Adapter
It works with about 35GB VRAM if you enable quantization and cpu offloading.