File size: 5,304 Bytes
1ad4b94 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 | ---
base_model: OpenMOSS-Team/MOSS-Image-Base
base_model_relation: adapter
language:
- en
- zh
license: apache-2.0
library_name: peft
pipeline_tag: text-to-image
tags:
- moss-image
- diffusers
- lora
- text-to-image
---
# moss_pht_lora
A photo-oriented LoRA (rank 128) for
[OpenMOSS-Team/MOSS-Image-Base](https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base).
It adapts only the transformer; the text encoder (Qwen3.5-9B), VAE (FLUX.2 small
decoder), tokenizer, scheduler and inference pipeline are the base release's,
unchanged. Load those from their official repositories, or swap in any
compatible ones (for example a quantized text encoder).
| File | Format |
| --- | --- |
| `transformer/` | PEFT LoRA adapter for the Diffusers `MOSSImageTransformer` |
| `loras/moss_pht_lora_bf16.safetensors` | ComfyUI LoRA (kohya layout on the release's module names) |
Both files hold the same weights.
## Diffusers
```bash
pip install "diffusers>=0.38" "transformers>=5.3" "accelerate>=1.10" "safetensors>=0.6" "peft>=0.21" "Pillow>=10"
```
```python
import torch
from diffusers import AutoencoderKLFlux2
from diffusers.utils.dynamic_modules_utils import get_class_from_dynamic_module
from peft import PeftModel
from transformers import AutoModel, AutoTokenizer
BASE = "OpenMOSS-Team/MOSS-Image-Base"
def moss_class(name):
return get_class_from_dynamic_module(
BASE, "pipeline_moss_image.py", class_name=name, trust_remote_code=True
)
transformer = moss_class("MOSSImageTransformer").from_pretrained(
BASE, subfolder="transformer", torch_dtype=torch.bfloat16
)
transformer = PeftModel.from_pretrained(
transformer, "data-archetype/moss_pht_lora", subfolder="transformer"
).get_base_model()
pipe = moss_class("MOSSImagePipeline")(
transformer=transformer,
scheduler=moss_class("MOSSImageScheduler").from_pretrained(BASE, subfolder="scheduler"),
text_encoder=AutoModel.from_pretrained(BASE, subfolder="text_encoder", dtype=torch.bfloat16),
tokenizer=AutoTokenizer.from_pretrained(BASE, subfolder="tokenizer"),
vae=AutoencoderKLFlux2.from_pretrained(BASE, subfolder="vae", torch_dtype=torch.float32),
)
pipe.to("cuda") # or pipe.enable_model_cpu_offload() on a smaller GPU
image = pipe(
"A candid street photograph of a cyclist crossing a rain-soaked intersection at dusk",
height=1024,
width=1024,
num_inference_steps=50,
guidance_scale=3.0,
uncond_skip_layers=[22],
generator=torch.Generator("cpu").manual_seed(42),
).images[0]
```
The adapter is applied unmerged, at strength 1. Sampling settings, prompt
formats (natural language, tags and JSON layouts) and requirements are the base
model's; see its [model card](https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base)
and [user guide](https://github.com/OpenMOSS/MOSS-Image/blob/main/USERGUIDE.md).
## ComfyUI
Use ComfyUI's native MOSS-Image support with the
[Comfy-Org/MOSS-Image](https://huggingface.co/Comfy-Org/MOSS-Image) base model,
text encoder and VAE, and add the LoRA after Load Diffusion Model.
| File | Put in | Loader |
| --- | --- | --- |
| `loras/moss_pht_lora_bf16.safetensors` (this repo) | `models/loras/` | LoraLoaderModelOnly |
| `diffusion_models/moss_image_base_bf16.safetensors`, from Comfy-Org/MOSS-Image | `models/diffusion_models/` | Load Diffusion Model |
| `text_encoders/qwen3.5_9b_bf16.safetensors` (or its int8 / w4a8 variants), from Comfy-Org/MOSS-Image | `models/text_encoders/` | Load CLIP, type `moss_image` |
| `vae/flux2-vae_small.safetensors`, from Comfy-Org/MOSS-Image | `models/vae/` | Load VAE |
## Recommended: a DMD LoRA
Like the base checkpoint, generations with the standard multi-step sampler can be
a bit unstable. Using a DMD LoRA distilled from MOSS-Image-Base is recommended: it
sharpens the output distribution and stabilises generations.
## Training
| Setting | Value |
| --- | --- |
| Base | MOSS-Image-Base, frozen (bf16); text encoder and VAE frozen |
| Adapted modules | `q`, `k`, `v`, `o`, `ffn_up_gate` and `ffn_down` of all 42 transformer blocks; the text embedder is not adapted |
| LoRA | Rank 128, alpha 128 (scale 1), A Kaiming-uniform and B zero at initialization, no dropout; `ffn_up_gate` carries one rank-128 pair for each of its up and gate halves |
| Steps | 5,500 at 32 images per step |
| Resolution | About 1 megapixel, aspect-ratio buckets |
| Text | Captions 80% natural language, 20% JSON layouts (MOSS-Image's layout prompt format); prompts up to 2,048 tokens; 10% of captions dropped to the empty prompt |
| Objective | The base's flow-matching objective; noise levels from a truncated logit-normal (μ 1.09, σ 0.95, truncated at 0.95), plus 1% uniform and 1% pure noise |
| Optimizer | AdamW on the LoRA factors (fp32), β = (0.9, 0.9975), ε 1e-8 |
| Learning rate | 1e-4 |
| Schedule | 150-step linear warmup, constant to step 4,500, then a 1,000-step 1 − √ cooldown to 0 |
| Weight decay | 0.01 (decoupled) on all LoRA factors |
| Gradient clipping | Global norm 1.0 |
| Released weights | EMA of the LoRA factors (decay 0.999, fp32, updated every step) at step 5,500, stored in bfloat16 |
## License
The weights are released under the base model's license, Apache 2.0
([LICENSE](LICENSE), copied from OpenMOSS-Team/MOSS-Image-Base).
We thank the OpenMOSS team for MOSS-Image.
|