Instructions to use XLabs-AI/flux-ip-adapter-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use XLabs-AI/flux-ip-adapter-v2 with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("XLabs-AI/flux-ip-adapter-v2", dtype=torch.bfloat16, device_map="cuda") prompt = "Turn this cat into a dog" input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png") image = pipe(image=input_image, prompt=prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Bug about ipadapter v2
RuntimeError: Error(s) in loading state_dict for ImageProjModel:
size mismatch for proj.weight: copying a param with shape torch.Size([65536, 768]) from checkpoint, the shape in current model is torch.Size([16384, 768]).
size mismatch for proj.bias: copying a param with shape torch.Size([65536]) from checkpoint, the shape in current model is torch.Size([16384]).
src/flux/xflux_pipeline.py, line 77:
self.improj = ImageProjModel(4096, 768, 4)
same error
I updated the Comfyui version to the latest and also updated the XLABS plugin to the latest, which solved the problem
same error
please update comfy nodes, it'd solve the problem you mentioned
The problem says that model graph expect 4 visual tokens (was in previous version) now we have 16, update nodes to support both ip-adapters
RuntimeError: Error(s) in loading state_dict for ImageProjModel:
size mismatch for proj.weight: copying a param with shape torch.Size([65536, 768]) from checkpoint, the shape in current model is torch.Size([16384, 768]).
size mismatch for proj.bias: copying a param with shape torch.Size([65536]) from checkpoint, the shape in current model is torch.Size([16384]).src/flux/xflux_pipeline.py, line 77:
self.improj = ImageProjModel(4096, 768, 4)