OzzyGT/MOSS_Image_SFT_diffusers

MOSS-Image SFT

This is OpenMOSS-Team/MOSS-Image-SFT converted to the standard diffusers layout, so it runs as a Modular Diffusers pipeline with the custom blocks below instead of the custom pipeline shipped in the original repository. The weights are the original bf16 ones.

Note: You need diffusers from main (0.41.0.dev0 or later).

Usage

You can find ready-to-use scripts in the diffusers-recipes repository.

MOSS-Image runs as a Modular Diffusers pipeline built from the MOSS-Image custom blocks, which load the transformer code shipped in this repository (hence trust_remote_code and trust_components_code). The VAE runs in float32, like the original.

Note: both flags run Python code downloaded from the Hub: trust_remote_code=True runs the code of the blocks repository, and trust_components_code=True runs the code shipped in the components_repo. Only use them with repositories you trust, and review their code before passing different ones.

With every component on the GPU, generating at 2048x1152 needs more than 32 GB of VRAM. For smaller GPUs, see the offloading recipes in diffusers-recipes or use one of the SDNQ quantized versions.

For prompting, recommended settings and the layout format, see the original model card and the user guide. The sample below uses a layout prompt: boxes are [y_min, x_min, y_max, x_max] in pixels with the origin at the center of the canvas.

Note: the samples use 28 steps, which is also the default of the custom blocks, because it already gives good results in about half the time. The official recommendation is 50 steps or more, so raise num_inference_steps if you want the best quality.

Sample image

The image above was generated with the following prompt and settings (seed 7):

import json

import torch
from diffusers import ModularPipelineBlocks


blocks = ModularPipelineBlocks.from_pretrained(
    "OzzyGT/moss_image_custom_blocks",
    trust_remote_code=True,
    components_repo="OzzyGT/MOSS_Image_SFT_diffusers",
    trust_components_code=True,
)
pipe = blocks.init_pipeline()
pipe.load_components(dtype={"default": torch.bfloat16, "vae": torch.float32})
pipe.to("cuda")

layout = {
    "summary": {
        "meta": ["photo"],
        "canvas": "single",
        "caption": (
            "A photograph of the inside of a small bicycle repair workshop, shot on a Sony A7 IV with a 35mm f/1.8 "
            "lens at f/2.8, 1/125 s, ISO 1600, handheld at eye level, wide 16:9 frame. Low late-afternoon sunlight "
            "comes through a large front window on the right in hard shafts with dust in the air, the rest of the "
            "room is lit by bare fluorescent tubes on the ceiling. Shallow depth of field with the back wall slightly "
            "out of focus, visible sensor grain. A red road bicycle is clamped in a repair stand in the center while "
            "an adult mechanic in a grease-stained navy work apron adjusts its rear derailleur. A painted wooden sign "
            'on the back wall reads "SPOKE & CHAIN". A pegboard of hand tools hangs on the left wall above an A-frame '
            "chalkboard with a price list, and an adult black cat sleeps curled up on a metal stool by the window. "
            "Oil stains on a grey concrete floor, spare tires hanging from hooks, scuffed white walls."
        ),
    },
    "items": [
        {
            "label": "text",
            "caption": "A long painted wooden sign on the back wall, dark green with cream serif capital letters.",
            "text": "SPOKE & CHAIN",
            "bbox": [-496, -464, -346, 236],
        },
        {
            "label": "object",
            "caption": "A perforated pegboard with wrenches, screwdrivers, tire levers and a chain whip on hooks.",
            "bbox": [-396, -984, 124, -544],
        },
        {
            "label": "object",
            "caption": "A red steel road bicycle with drop handlebars, clamped by its seat post in a grey repair stand.",
            "bbox": [-196, -504, 329, 371],
        },
        {
            "label": "object",
            "caption": (
                "An adult man in his forties with short grey hair, a grey t-shirt and a grease-stained navy work "
                "apron, standing and turning a small screwdriver on the rear derailleur."
            ),
            "bbox": [-371, 396, 504, 736],
        },
        {
            "label": "text",
            "caption": "A black A-frame chalkboard on the floor with a handwritten price list in white chalk.",
            "text": "FLAT REPAIR $12\nFULL TUNE-UP $45\nOPEN 8-6",
            "bbox": [184, -964, 524, -604],
        },
        {
            "label": "object",
            "caption": "An adult black cat sleeping curled up on the seat of a metal shop stool in a patch of sunlight.",
            "bbox": [104, 766, 209, 981],
        },
    ],
    "bbox_coord_space": "abs_centralized",
    "bbox_format": "yxyx",
}
prompt = json.dumps(layout, ensure_ascii=False)

image = pipe(
    prompt=prompt,
    num_inference_steps=28,
    height=1152,
    width=2048,
    generator=torch.Generator("cuda").manual_seed(7),
    output="images",
)[0]
image.save("moss_image.png")

License

This model is released under the Apache-2.0 license, the same as the original.