Add model card and metadata for Rethinking Global Text Conditioning

README.md

@@ -1,3 +1,103 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+library_name: diffusers
+pipeline_tag: text-to-image
+base_model: stabilityai/stable-diffusion-3.5-large
+tags:
+- lora
+- stable-diffusion-3
+- text-to-image
+---
+
+# Rethinking Global Text Conditioning in Diffusion Transformers
+
+This repository contains the DMD2 LoRA adapter for Stable Diffusion 3.5 Large, as presented in the paper [Rethinking Global Text Conditioning in Diffusion Transformers](https://huggingface.co/papers/2602.09268).
+
+## Introduction
+
+Diffusion transformers typically incorporate textual information via attention layers and a modulation mechanism using a pooled text embedding. Our analysis shows that, in its conventional usage, the pooled embedding contributes little to overall performance. However, we reveal that the pooled embedding can provide significant gains when used as **guidance**—enabling controllable shifts toward more desirable properties like complexity, realism, and better hand rendering.
+
+This approach is training-free, simple to implement, and can be applied to various tasks including text-to-image/video generation and image editing.
+
+*   **GitHub Repository:** [quickjkee/modulation-guidance](https://github.com/quickjkee/modulation-guidance)
+*   **Paper:** [arXiv:2602.09268](https://huggingface.co/papers/2602.09268)
+
+## Usage
+
+This model is a LoRA adapter designed to work with the `diffusers` library. Below is an example of how to use it with **SD3.5-Large DMD2** and the proposed modulation guidance.
+
+*Note: This usage requires the helper functions `encode_prompt` and `forward_modulation_guidance` from the [official repository](https://github.com/quickjkee/modulation-guidance).*
+
+```python
+import torch
+import types
+from diffusers import StableDiffusion3Pipeline
+from functools import partial
+from models.sd35 import encode_prompt, forward_modulation_guidance
+from peft import PeftModel
+
+# Load base model with custom pipeline
+pipe = StableDiffusion3Pipeline.from_pretrained(
+    "stabilityai/stable-diffusion-3.5-large",
+    torch_dtype=torch.float16,
+    custom_pipeline='quickjkee/swd_pipeline'
+)
+pipe = pipe.to("cuda")
+
+# Load this LoRA adapter
+lora_path = 'yresearch/stable-diffusion-3.5-large-dmd2'
+pipe.transformer = PeftModel.from_pretrained(
+    pipe.transformer,
+    lora_path,
+).to("cuda")
+
+# Define hyperparameters
+prompt = "a cardboard spaceship"
+prompt_positive = "Extremely complex, the highest quality"
+prompt_negative = "very simple, no details at all"
+w = 2
+start_layer = 2
+
+# Get pooled CLIP embeddings
+clip_positive = encode_prompt(pipe=pipe, prompt=prompt_positive)
+clip_negative = encode_prompt(pipe=pipe, prompt=prompt_negative)
+
+# Apply modulation guidance
+forward_modulation_guidance = partial(
+    forward_modulation_guidance, 
+    pooled_projections_1=clip_positive, 
+    pooled_projections_0=clip_negative,
+    w=w, 
+    start_layer=start_layer
+)
+pipe.transformer.forward = types.MethodType(forward_modulation_guidance, pipe.transformer)
+
+# Run generation
+seed = 0
+sigmas = [1.0000, 0.9454, 0.8959, 0.7904, 0.7371, 0.6022, 0.0000]
+scales = [128, 128, 128, 128, 128, 128]
+image = pipe(
+    [prompt] * 1,
+    sigmas=torch.tensor(sigmas).to('cuda'),
+    timesteps=torch.tensor(sigmas[:-1]).to('cuda') * 1000,
+    scales=scales,
+    guidance_scale=0.0,
+    height=int(scales[0] * 8),
+    width=int(scales[0] * 8),
+    generator=torch.Generator("cpu").manual_seed(seed),
+    output_type='pil'
+).images[0]
+
+image.show()
+```
+
+## Citation
+
+```bibtex
+@article{starodubcev2026rethinking,
+  title={Rethinking Global Text Conditioning in Diffusion Transformers},
+  author={Starodubcev, Nikita and Pakhomov, Daniil and Wu, Zongze and Drobyshevskiy, Ilya and Liu, Yuchen and Wang, Zhonghao and Zhou, Yuqian and Lin, Zhe and Baranchuk, Dmitry},
+  journal={arXiv preprint arXiv:2602.09268},
+  year={2026}
+}
+```