| | --- |
| | license: apache-2.0 |
| | base_model: |
| | - lodestones/Chroma1-Flash |
| | base_model_relation: quantized |
| | language: |
| | - en |
| | pipeline_tag: text-to-image |
| | library_name: diffusers |
| | --- |
| | ## Update: I have uploaded an updated version of this model, that should further reduce disk size and VRAM usage by ~82 MB. This is because I missed out on compressing a small portion of the model (the `distilled_guidance_layer.layers`) in my original upload. There is <u>no need to download again</u> if you are not having any issues with the older version. |
| |
|
| | Being a distilled model, this model requires different parameters to run optimally compared to the undistilled Chroma1-HD version. However, after quite a while of testing, I am unable to determine what settings to use. The [model card](https://huggingface.co/lodestones/Chroma1-Flash) is blank, but the commit message by the author says "use heun 8 steps CFG=1". Sadly, trying to use `HeunDiscreteScheduler` or `FlowMatchHeunDiscreteScheduler` with this model causes the pipeline to fail, presumably due to [this issue](https://github.com/huggingface/diffusers/issues/9971) with the `diffusers` library. |
| |
|
| |
|
| | For more information (including how to compress models yourself), check out https://huggingface.co/DFloat11 and https://github.com/LeanModels/DFloat11 |
| |
|
| | This is my first time using DF11 to compress a model outside the Flux architecture. The process for compressing Flux-based models is much more straightforward as compared to other architectures because the compression code requires a `pattern_dict` as input, but the original [example code](https://github.com/LeanModels/DFloat11/tree/master/examples/compress_flux1) only provides it for Flux, which meant I had to learn the notation myself and modify it to fit other models. At least Chroma is just a pruned version of Flux, so it was relatively simple to derive the correct `pattern_dict` this time. Do let me know if you run into any problems. |
| |
|
| | This is the `pattern_dict` I used for compression: |
| | ```python |
| | pattern_dict = { |
| | r"distilled_guidance_layer\.layers\.\d+": ( |
| | "linear_1", |
| | "linear_2" |
| | ), |
| | r"transformer_blocks\.\d+": ( |
| | "attn.to_q", |
| | "attn.to_k", |
| | "attn.to_v", |
| | "attn.add_k_proj", |
| | "attn.add_v_proj", |
| | "attn.add_q_proj", |
| | "attn.to_out.0", |
| | "attn.to_add_out", |
| | "ff.net.0.proj", |
| | "ff.net.2", |
| | "ff_context.net.0.proj", |
| | "ff_context.net.2", |
| | ), |
| | r"single_transformer_blocks\.\d+": ( |
| | "proj_mlp", |
| | "proj_out", |
| | "attn.to_q", |
| | "attn.to_k", |
| | "attn.to_v", |
| | ), |
| | } |
| | ``` |
| |
|
| | ### How to Use |
| |
|
| | #### `diffusers` |
| |
|
| | 1. Install the DFloat11 pip package *(installs the CUDA kernel automatically; requires a CUDA-compatible GPU and PyTorch installed)*: |
| |
|
| | ```bash |
| | pip install dfloat11[cuda12] |
| | # or if you have CUDA version 11: |
| | # pip install dfloat11[cuda11] |
| | ``` |
| | 2. To use the DFloat11 model, run the following example code in Python: |
| | ```python |
| | import torch |
| | from diffusers import ChromaPipeline, ChromaTransformer2DModel |
| | from dfloat11 import DFloat11Model |
| | from transformers.modeling_utils import no_init_weights |
| | with no_init_weights(): |
| | transformer = ChromaTransformer2DModel.from_config( |
| | ChromaTransformer2DModel.load_config( |
| | "lodestones/Chroma1-Flash", |
| | subfolder="transformer" |
| | ), |
| | torch_dtype=torch.bfloat16 |
| | ).to(torch.bfloat16) |
| | |
| | pipe = ChromaPipeline.from_pretrained( |
| | "lodestones/Chroma1-Flash", |
| | transformer=transformer, |
| | torch_dtype=torch.bfloat16 |
| | ) |
| | DFloat11Model.from_pretrained("mingyi456/Chroma1-Flash-DF11", device='cpu', bfloat16_model=pipe.transformer) |
| | pipe.enable_model_cpu_offload() |
| | prompt = "A high-fashion close-up portrait of a blonde woman in clear sunglasses. The image uses a bold teal and red color split for dramatic lighting. The background is a simple teal-green. The photo is sharp and well-composed, and is designed for viewing with anaglyph 3D glasses for optimal effect. It looks professionally done." |
| | negative_prompt = "low quality, ugly, unfinished, out of focus, deformed, disfigure, blurry, smudged, restricted palette, flat colors" |
| | |
| | # Call the pipeline with your own parameters, I am not sure what are the optimal settings for this model in `diffusers` |
| | |
| | ``` |
| | #### ComfyUI |
| | Refer to this [model](https://huggingface.co/mingyi456/Chroma1-Flash-DF11-ComfyUI) page instead, and follow the instructions there. |
