| | --- |
| | base_model: |
| | - stable-diffusion-v1-5/stable-diffusion-v1-5 |
| | datasets: |
| | - opendiffusionai/cc12m-small-squarish-simple |
| | --- |
| | |
| | # What is this? |
| | This is an initial version of Stable Diffusion 1.5 base model, with its noise scheduler/prediction replaced with |
| | FlowMatchEulerDiscrete |
| |
|
| | This model probably has a buncha low quality stuff in it. Base model SD might give better output in many reguards. |
| | The reason this model exists is to allow other people to take advantage of FlowMatch for their own finetunes |
| | and other experiments. |
| |
|
| | For that reason, this is a FULL FP32 precision model. But the sample code below loads it as bf16. |
| |
|
| | # Usage note |
| |
|
| | Original diffusers module for stable_diffusion has a hardcode that stops this working. I have submitted a patch |
| | that was accepted.. but as far as I know, it has not been added to an official release yet. So, |
| | "diffusers 0.34.0" wont work with it. |
| | That means that to use this, you currently need to either use my tweaked code, [imgsample-hacked.py](imgsample-hacked.py) |
| | or manually add in the main git version to use this. |
| | eg: |
| | |
| | pip install git+https://github.com/huggingface/diffusers |
| | |
| | |
| | You should then be able to do the typical diffusers code. For example: |
| | |
| | from diffusers import DiffusionPipeline |
| | import torch.nn as nn, torch, types |
| | import os,sys |
| | MODEL="opendiffusionai/sd-flow-alpha" |
| | pipe = DiffusionPipeline.from_pretrained( |
| | MODEL, use_safetensors=True, |
| | safety_checker=None, requires_safety_checker=False, |
| | torch_dtype=torch.bfloat16, |
| | ) |
| | pipe.enable_sequential_cpu_offload() |
| | prompt="Some pretty photo of something" |
| | images = pipe(prompt, num_inference_steps=args.steps, generator=generator).images |
| | for i,image in enumerate(images): |
| | fname=f"{OUTDIR}/sample{i}.png" |
| | print(f"saving to {fname}") |
| | image.save(fname) |
| | |
| | ## ComfyUI note |
| |
|
| | From the author: |
| |
|
| | >It works fine in comfy, just load the unet with the load diffusion model node and hook it to a |
| | > ModelSamplingSD3 node. |
| | > |
| | >For the clip/vae you can just use the one from the SD1.5 checkpoint." |
| |
|
| |
|
| | # Making your own FlowMatch model |
| |
|
| | Doing the training itself, did not take that long. |
| | Writing [my own functional training code](https://ppbrown@github.com/ppbrown/ai-training), and trying various pathways to find what works, took WEEKS. |
| |
|
| | That, and putting together a 40k clean |
| | [ALL-SQUARE IMAGE DATASET](https://huggingface.co/datasets/opendiffusionai/cc12m-small-squarish-simple) |
| |
|
| | If you wanted to recreate your own from scratch, here's the details from one of my runs: |
| | (This only takes a few hours to complete, on a 4090) |
| |
|
| | First, download the sd base model in diffusers format, and hand-edit the [model_config.json](model_config.json) and |
| | [scheduler/scheduler_config.json](scheduler/scheduler_config.json) file. |
| | (I was going to detail it here, but... just copy/look at the files in this repo. I linked them, after all!) |
| |
|
| | (Batchsize 40, accum=1 for all) |
| |
|
| | * time blocks only, 1e-5, 350 steps (result very murky here, thats expected) |
| | * up.0 and up.1, 1e-6, 75 steps |
| | * mid, 1e-6, 60 steps |
| | * up.2, 1e-6, 160 steps |
| | * up.3, 1e-6, 120 steps |
| |
|
| | ## Sampling |
| | During the first phase, maybe sample every 50 steps. |
| | After the first phase, you'll want to take samples every 10 steps. Make sure you use MULTIPLE samples, |
| | and ideally of different types. You should have at least one "single token" prompt, and then a few more |
| | complex ones. |
| |
|
| |
|
| |  |
