Create README.md

README.md

@@ -0,0 +1,132 @@
+---
+license: 
+- apache-2.0
+library_name: diffusers
+pipeline_tag: text-to-image
+datasets:
+- SA1B
+base_model: 
+- jimmycarter/LibreFLUX
+- InstantX/FLUX.1-dev-IP-Adapter
+---
+# LibreFLUX-IP-Adapter
+![Example: Control image vs result](examples/matrix_edge.png)
+
+This model/pipeline is the product of my LibreFlux IP-Adapter training repo, which uses LibreFLUX as the underlying Transformer model. The Adapter design is roughly based on InstantX, and I was able to actualy fine-tune there weights to work with LibreFlux. For the dataset, I trained on laion2b-squareish-1024px for 20,000 iterations.
+
+# How does this relate to LibreFLUX?
+- Base model is [LibreFLUX](https://huggingface.co/jimmycarter/LibreFLUX)
+- Trained in same non-distilled fashion 
+- Uses Attention Masking
+- Uses CFG during Inference
+
+
+# Fun Facts
+- Trained on the [laion2b-squareish-1024px Dataset](https://huggingface.co/datasets/opendiffusionai/laion2b-squareish-1024px/)
+- Trained using this repo: [https://github.com/NeuralVFX/LibreFLUX-IP-Adapter](https://github.com/NeuralVFX/LibreFLUX-IP-Adapter)
+- Transformer model used: [https://huggingface.co/jimmycarter/LibreFlux](https://huggingface.co/jimmycarter/LibreFlux)
+- Inference code roughly adapted from: [https://github.com/bghira/SimpleTuner](https://github.com/bghira/SimpleTuner)
+  
+# Compatibility
+```py
+pip install -U diffusers==0.35.2
+pip install -U transformers==4.57.1
+# Compatibility
+```py
+pip install -U diffusers==0.35.2
+pip install -U transformers==4.57.1
+```
+Low VRAM:
+```py
+pip install optimum-quanto
+```
+# Load Pipeline
+```py
+import torch
+from diffusers import DiffusionPipeline
+
+model_id = "neuralvfx/LibreFlux-IP-Adapter" 
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+dtype  = torch.bfloat16 if device == "cuda" else torch.float32
+
+pipe = DiffusionPipeline.from_pretrained(
+    model_id,
+    custom_pipeline=model_id,
+    trust_remote_code=True,   
+    torch_dtype=dtype,
+    safety_checker=None        
+)
+
+pipe.load_ip_adapter('ip_adapter.pt')
+
+pipe.to(device)
+```
+
+# Inference
+```py
+from PIL import Image
+from torchvision.transforms import ToTensor
+
+# Load IP Adapter Image
+ip_image = Image.open("examples/david.png").convert("RGB")
+ip_image = cond.resize((512, 512))
+
+prompt = "george washington"
+negative_prompt = "blurry, low quality"
+
+generator = torch.Generator(device="cuda").manual_seed(1995)
+
+images = pipe(
+  prompt=prompt,
+  negative_prompt=negative_prompt,
+  return_dict=False,
+  ip_adapter_image=ip_image, 
+  ip_adapter_scale=1.0,
+  height=512, 
+  width=512,  
+  num_inference_steps=75, 
+  generator=generator
+)
+```
+# Load Pipeline ( Low VRAM )
+```py
+import torch
+from diffusers import DiffusionPipeline
+from optimum.quanto import freeze, quantize, qint8
+
+model_id = "neuralvfx/LibreFlux-IP-Adapter" 
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype  = torch.bfloat16 if device == "cuda" else torch.float32
+
+pipe = DiffusionPipeline.from_pretrained(
+    model_id,
+    custom_pipeline=model_id,
+    trust_remote_code=True,    
+    torch_dtype=dtype,
+    safety_checker=None         
+)
+pipe.load_ip_adapter('ip_adapter.pt')
+
+quantize(
+    pipe.transformer,
+    weights=qint8,
+    exclude=[
+        "*.norm", "*.norm1", "*.norm2", "*.norm2_context",
+        "proj_out", "x_embedder", "norm_out", "context_embedder",
+    ],
+)
+
+quantize(
+    pipe.controlnet,
+    weights=qint8,
+    exclude=[
+        "*.norm", "*.norm1", "*.norm2", "*.norm2_context",
+        "proj_out", "x_embedder", "norm_out", "context_embedder",
+    ],
+)
+freeze(pipe.transformer)
+freeze(pipe.controlnet)
+
+pipe.enable_model_cpu_offload()
+```