|
|
--- |
|
|
license: |
|
|
- apache-2.0 |
|
|
- other |
|
|
license_name: flux-1-dev-non-commercial-license |
|
|
license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md |
|
|
library_name: diffusers |
|
|
pipeline_tag: text-to-image |
|
|
datasets: |
|
|
- SA1B |
|
|
- opendiffusionai/laion2b-squareish-1024px |
|
|
base_model: |
|
|
- jimmycarter/LibreFLUX |
|
|
--- |
|
|
# LibreFLUX-IP-Adapter |
|
|
 |
|
|
|
|
|
This model/pipeline is the product of my [LibreFlux IP-Adapter training repo](https://github.com/NeuralVFX/LibreFLUX-IP-Adapter), which uses [LibreFLUX](https://huggingface.co/jimmycarter/LibreFLUX) as the underlying Transformer model. The IP Adapter and Attention Wrapper design is roughly based on the [InstantX IP Adapter](https://huggingface.co/InstantX/FLUX.1-dev-IP-Adapter/) |
|
|
|
|
|
I used transfer learning, to fintune the InstantX weights until they worked with LibreFlux and attention masking. For the dataset, I trained on [laion2b-squareish-1024px](https://huggingface.co/datasets/opendiffusionai/laion2b-squareish-1024px/) for roughly 20,000 iterations. |
|
|
|
|
|
# How does this relate to LibreFLUX? |
|
|
- Base model is [LibreFLUX](https://huggingface.co/jimmycarter/LibreFLUX) |
|
|
- Trained in same non-distilled fashion |
|
|
- Uses Attention Masking |
|
|
- Uses CFG during Inference |
|
|
# Fun Facts |
|
|
- Fine tuned from these weights: [https://huggingface.co/InstantX/FLUX.1-dev-IP-Adapter/](https://huggingface.co/InstantX/FLUX.1-dev-IP-Adapter/) |
|
|
- Trained on the [laion2b-squareish-1024px Dataset](https://huggingface.co/datasets/opendiffusionai/laion2b-squareish-1024px/) |
|
|
- Trained using this repo: [https://github.com/NeuralVFX/LibreFLUX-IP-Adapter](https://github.com/NeuralVFX/LibreFLUX-IP-Adapter) |
|
|
- Transformer model used: [https://huggingface.co/jimmycarter/LibreFlux](https://huggingface.co/jimmycarter/LibreFlux) |
|
|
- Inference code roughly adapted from: [https://github.com/bghira/SimpleTuner](https://github.com/bghira/SimpleTuner) |
|
|
|
|
|
# Compatibility |
|
|
```py |
|
|
pip install -U diffusers==0.35.2 |
|
|
pip install -U transformers==4.57.1 |
|
|
``` |
|
|
|
|
|
Low VRAM: |
|
|
```py |
|
|
pip install optimum.quanto |
|
|
``` |
|
|
|
|
|
|
|
|
# Load Pipeline |
|
|
```py |
|
|
import torch |
|
|
from diffusers import DiffusionPipeline |
|
|
from huggingface_hub import hf_hub_download |
|
|
|
|
|
model_id = "neuralvfx/LibreFlux-IP-Adapter" |
|
|
device = "cuda" if torch.cuda.is_available() else "cpu" |
|
|
|
|
|
dtype = torch.bfloat16 if device == "cuda" else torch.float32 |
|
|
|
|
|
pipe = DiffusionPipeline.from_pretrained( |
|
|
model_id, |
|
|
custom_pipeline=model_id, |
|
|
trust_remote_code=True, |
|
|
torch_dtype=dtype, |
|
|
safety_checker=None |
|
|
) |
|
|
|
|
|
# Optional way to download the weights |
|
|
hf_hub_download(repo_id="neuralvfx/LibreFlux-IP-Adapter", |
|
|
filename="ip_adapter.pt", |
|
|
local_dir=".", |
|
|
local_dir_use_symlinks=False) |
|
|
|
|
|
pipe.load_ip_adapter('ip_adapter.pt') |
|
|
|
|
|
pipe.to(device) |
|
|
``` |
|
|
|
|
|
# Inference |
|
|
```py |
|
|
from PIL import Image |
|
|
from torchvision.transforms import ToTensor |
|
|
|
|
|
# Optional way to download test IP Adapter Image |
|
|
hf_hub_download(repo_id="neuralvfx/LibreFlux-IP-Adapter", |
|
|
filename="examples/david.jpg", |
|
|
local_dir=".", |
|
|
local_dir_use_symlinks=False) |
|
|
|
|
|
# Load IP Adapter Image |
|
|
ip_image = Image.open("examples/david.jpg").convert("RGB") |
|
|
ip_image = ip_image.resize((512, 512)) |
|
|
|
|
|
prompt = "george washington" |
|
|
negative_prompt = "blurry, low quality" |
|
|
|
|
|
generator = torch.Generator(device="cuda").manual_seed(1995) |
|
|
|
|
|
images = pipe( |
|
|
prompt=prompt, |
|
|
negative_prompt=negative_prompt, |
|
|
return_dict=False, |
|
|
ip_adapter_image=ip_image, |
|
|
ip_adapter_scale=1.0, |
|
|
height=512, |
|
|
width=512, |
|
|
num_inference_steps=75, |
|
|
generator=generator |
|
|
)[0][0] |
|
|
``` |
|
|
|
|
|
# Load Pipeline ( Low VRAM ) |
|
|
```py |
|
|
import torch |
|
|
from huggingface_hub import hf_hub_download |
|
|
from diffusers import DiffusionPipeline |
|
|
from optimum.quanto import freeze, quantize, qint8 |
|
|
|
|
|
model_id = "neuralvfx/LibreFlux-IP-Adapter" |
|
|
|
|
|
device = "cuda" if torch.cuda.is_available() else "cpu" |
|
|
dtype = torch.bfloat16 if device == "cuda" else torch.float32 |
|
|
|
|
|
pipe = DiffusionPipeline.from_pretrained( |
|
|
model_id, |
|
|
custom_pipeline=model_id, |
|
|
trust_remote_code=True, |
|
|
torch_dtype=dtype, |
|
|
safety_checker=None |
|
|
) |
|
|
|
|
|
# Optional way to download the weights |
|
|
hf_hub_download(repo_id="neuralvfx/LibreFlux-IP-Adapter", |
|
|
filename="ip_adapter.pt", |
|
|
local_dir=".", |
|
|
local_dir_use_symlinks=False) |
|
|
|
|
|
# Load the IP Adapter First |
|
|
pipe.load_ip_adapter('ip_adapter.pt') |
|
|
|
|
|
# Quantize and Freeze |
|
|
quantize( |
|
|
pipe.transformer, |
|
|
weights=qint8, |
|
|
exclude=[ |
|
|
"*.norm", "*.norm1", "*.norm2", "*.norm2_context", |
|
|
"proj_out", "x_embedder", "norm_out", "context_embedder", |
|
|
], |
|
|
) |
|
|
|
|
|
quantize( |
|
|
pipe.ip_adapter, |
|
|
weights=qint8, |
|
|
exclude=[ |
|
|
"*.norm", "*.norm1", "*.norm2", "*.norm2_context", |
|
|
"proj_out", "x_embedder", "norm_out", "context_embedder", |
|
|
], |
|
|
) |
|
|
freeze(pipe.transformer) |
|
|
freeze(pipe.ip_adapter) |
|
|
|
|
|
# Enable Model Offloading |
|
|
pipe.enable_model_cpu_offload() |
|
|
``` |
|
|
|
