--- license: other license_name: qwen-research license_link: LICENSE base_model: Qwen/Qwen-Image-2.1 base_model_relation: adapter library_name: diffusers pipeline_tag: image-to-image tags: - diffusers - lora - image-to-image - image-editing - character-swap - few-step - qwen-image --- # Viggle Image Swap **Built with Qwen.** A LoRA adapter for [Qwen/Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1). It replaces people in a picture with characters from character sheets. Each person is replaced whole: face, hair, body and clothes. The pose, expression, framing, background and lighting of the picture stay as they were. It samples in 6 steps without classifier-free guidance, or, with `--turbo`, in 2 steps handed over to [Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) for 6 more: sharper detail. ![Two dancers recast as playing-card royals](examples/card-royals.jpg) ## Examples Each row is: the character sheet(s), the source picture, the output. All are made with `inference/swap.py` at its defaults (seed 0, tone fix on); the only inputs are the sheets, the source and the short `WHO` text shown. The top image replaces two people at once: `--char queen.png woman --char king.png man --left 1 2`. ![Dancer recast as an astronaut](examples/astronaut.jpg) One person, letterboxed source: `--char astronaut.png dancer` ![Ballroom couple recast as flamingos](examples/flamingos.jpg) Two people in close contact, both replaced: `--char flavio.png man --char fleur.png woman --left 1 2` ![Selfie recast as a 1920s aviatrix](examples/aviatrix.jpg) A close-up selfie: `--char odessa.png woman` ![Man recast as a pineapple mascot](examples/pineapple-mascot.jpg) A full-body mascot costume; the prop in his hand stays: `--char pete.png man` | file | what it is | |---|---| | `viggle-image-swap-v0.1-lora.safetensors` | the LoRA (rank 256, alpha 256), bf16, diffusers key format | | `inference/swap.py` | command-line inference: builds the prompt, orders the images, samples, tone fix | | `requirements.txt` | the tested package versions | ## Inputs - **The source picture**: any size and aspect. The output is 1 MP at the source's aspect, with sides multiples of 32. - **One character sheet per person to replace**: a 1536×1024 image. The left third is a portrait; the right two thirds is a full-body three-view (front, side, back) on a plain background. A photo reference can be used by pasting a face photo over the sheet's portrait cell (left third, rows 128–896, 512×768). A bare photo on its own is not supported. Image order: the sheets first (image 1, 2, ...), the source last. Each image is shrunk to at most 1 MP. ## Prompt One sentence per character, then one sentence that names what to keep. With one sheet (image 1) and the source as image 2: ``` Replace the man in a green shirt in image 2 with the character from image 1. Keep everything else exactly as in image 2: the pose, expression, framing, background and lighting. ``` With two people in the picture and only one replaced, say where the person is and that the other stays: ``` Replace the woman on the left in image 2 with the character from image 1. Keep everything else exactly as in image 2: the pose, expression, framing, background and lighting, and leave the other person unchanged. ``` Positions are `on the left` / `on the right`, `in the middle` (three people), or `2nd from the left` and so on. `inference/swap.py` writes these prompts for you (`--left`, `--kept`). ## Install ```bash pip install -U torch torchvision "transformers==5.17.0" accelerate safetensors "peft==0.18.0" pillow numpy pip install "git+https://github.com/huggingface/diffusers.git@80c7ed262aeffbeb43ef13ae04baeb9b84515a69" ``` `QwenImage21Pipeline` is pinned to that diffusers commit, which is what this adapter was tested on. ## Usage Command line: ```bash python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --out out.png # two people replaced, one kept: places are 1-based, counted left to right among all three python inference/swap.py --src scene.png --char a.png "woman in red" --char b.png "man in a cap" --left 1 3 --kept 1 python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --dry-run # prompt, sizes, sigmas; no GPU python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --turbo # sharper: 2 + 6 Turbo steps ``` Python: ```python import torch from diffusers import QwenImage21Pipeline, FlowMatchEulerDiscreteScheduler from PIL import Image sched = FlowMatchEulerDiscreteScheduler.from_pretrained("Qwen/Qwen-Image-2.1", subfolder="scheduler") sched = FlowMatchEulerDiscreteScheduler.from_config(sched.config, shift_terminal=None) pipe = QwenImage21Pipeline.from_pretrained("Qwen/Qwen-Image-2.1", scheduler=sched, dtype=torch.bfloat16) pipe.load_lora_weights("Viggle/Viggle-Image-Swap", weight_name="viggle-image-swap-v0.1-lora.safetensors") pipe.to("cuda") sheet, src = Image.open("sheet.png").convert("RGB"), Image.open("scene.png").convert("RGB") # each at most 1 MP prompt = ("Replace the man in a green shirt in image 2 with the character from image 1. Keep everything else exactly as in " "image 2: the pose, expression, framing, background and lighting.") out = pipe(prompt=prompt, image=[sheet, src], num_inference_steps=6, sigmas=[1.0, 0.9375, 0.875, 0.75, 0.5, 0.25], true_cfg_scale=1.0, output_resolution=1024, generator=torch.Generator("cpu").manual_seed(0)).images[0] out.save("out.png") ``` The `sigmas` are passed raw; the pipeline adds its resolution-dependent shift. `shift_terminal=None` is required. `swap.py` also replaces the vision encoder's bf16 Conv3d patch embedding with the equal matmul, which saves about 30 s per input image; the output is bitwise the same. ### Sharper output: `--turbo` `--turbo` runs the whole sample on Qwen-Image-2.1-Turbo's scheduler and its 8 `sample_sigmas` (from its `model_index.json`). The LoRA model takes the first 2 steps, which settle who is in the picture; then the transformer is swapped for Qwen-Image-2.1-Turbo's, which runs the other 6 and draws finer detail. The KV cache is off, since the two transformers cannot share one. `--turbo-model` points at another copy of Qwen-Image-2.1-Turbo. In Python, the same is: ```python from diffusers import QwenImage21Transformer2DModel turbo = QwenImage21Transformer2DModel.from_pretrained("Qwen/Qwen-Image-2.1-Turbo", subfolder="transformer", torch_dtype=torch.bfloat16).to("cuda") pipe.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained("Qwen/Qwen-Image-2.1-Turbo", subfolder="scheduler") sigmas = QwenImage21Pipeline.load_config("Qwen/Qwen-Image-2.1-Turbo")["sample_sigmas"] def handoff(p, i, t, kw): # after the LoRA model's 2nd step if i == 1: p.transformer = turbo return kw out = pipe(prompt=prompt, image=[sheet, src], num_inference_steps=len(sigmas), sigmas=sigmas, true_cfg_scale=1.0, output_resolution=1024, generator=torch.Generator("cpu").manual_seed(0), callback_on_step_end=handoff, use_kv_cache=False).images[0] ``` With only 2 steps of its own the LoRA model hands over a rough picture, so Turbo sometimes pulls back parts of the source person: their sunglasses, beard or clothes, and now and then the whole person. Without `--turbo` that happens less, at a softer finish. Try another `--seed`, or drop `--turbo`, when it does. ### Tone fix The output can be 1–4 levels darker or lighter than the source over the whole frame. `swap.py` corrects this by default: per colour channel, it subtracts the mean offset from the source over the pixels that did not change (the background). It does nothing when under 30 % of the frame is unchanged. `--no-tone-fix` turns it off. ## Cost Measured on one B200 with the models loaded, a 1344×768 output and one sheet, bf16: about 2.2 s per image and 46 GB peak; with `--turbo`, about 4.1 s and 55 GB (51.3 GiB) peak, the second transformer included. ## Limitations - The person to replace is chosen by the prompt. With several people, give the position. - Sometimes the new character keeps the source person's clothes. - The pose or framing of the new character can drift from the source person's. - The output is 1 MP. - `--turbo` can bring back the source person's accessories, clothes or, rarely, the whole person (see above). ## Example footage The source pictures in `examples/` are frames of third-party footage, © their respective owners, credited by YouTube channel and video ID. The character sheets are original. | example | source | YouTube video | |---|---|---| | card-royals | West Coast Swing Online | `nRdO0HS-D5E` | | astronaut | Chris King | `A37myLTm2Vo` | | flamingos | jtballroom | `GAZ2LeNX3BU` | | aviatrix | Alecia Renece The Artist | `rHDTkqDqBp0` | | pineapple-mascot | Massimo Nocerino | `W1P5qiqaqCs` | The video at the top (`examples/viggle-image-swap.mp4`) uses frames of films, advertisements and other third-party footage, © their respective owners. Some of its character sheets depict real public figures; they are made for this research demonstration and imply no endorsement. ## License The adapter is a derivative of Qwen-Image-2.1 and is released under the [Qwen RESEARCH LICENSE AGREEMENT](LICENSE): research and evaluation use only. Commercial use needs a separate licence from the Qwen team (model-business@notice.qwencloud.com). `--turbo` also loads Qwen/Qwen-Image-2.1-Turbo, under the same licence. See [NOTICE](NOTICE). The inference code in `inference/` is under the Apache License 2.0 ([LICENSE-CODE](LICENSE-CODE)).