| --- |
| license: apache-2.0 |
| base_model: Efficient-Large-Model/Sana_600M_512px_diffusers |
| library_name: peft |
| datasets: |
| - leffff/diffusion-alignment-workshop |
| tags: |
| - diffusion |
| - alignment |
| - rlhf |
| - text-rendering |
| - sana |
| - lora |
| - text-to-image |
| --- |
| |
| # Diffusion Alignment Workshop: Sana-600M aligned for text rendering |
|
|
| LoRA adapters that align [Sana-600M-512px](https://huggingface.co/Efficient-Large-Model/Sana_600M_512px_diffusers) to **render the text asked for in the prompt** (`a chalkboard with the word "Recycling" written on it`), trained with six methods, plus two more reward experts and their on-policy-distilled merge. Every adapter was trained for **5 minutes on one H100** with the workshop notebook. Data: [`leffff/diffusion-alignment-workshop`](https://huggingface.co/datasets/leffff/diffusion-alignment-workshop). |
|
|
| Rewards: **OCR** = 1 − (character edits to find the target text in the [Qwen3-VL-2B](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) transcription) / its length; **PickScore**; **aesthetic** (LAION predictor v2); **CLIP score** (held out, never trained on). Evaluation: 64 held-out prompts × 2 seeds, 20 Euler steps, CFG 4.5. |
|
|
| | file | method | data | training (5 min) | OCR ↑ | PickScore | aesthetic | CLIP | |
| |---|---|---|---|---|---|---|---| |
| | — | Sana-600M (reference) | — | — | 0.569 | 22.41 | 5.05 | 32.2 | |
| | `adapter_sft.pt` | SFT on the best of 5 samples (OCR ≥ 0.5) | offline | 1018 it × 32, lr 3e-4 | 0.712 | 22.45 | 4.91 | 33.3 | |
| | `adapter_rwr.pt` | reward-weighted regression, group softmax τ = 0.1 | offline | 1052 it × 6 prompts × 5, lr 3e-4 | 0.696 | 22.42 | 4.95 | 33.0 | |
| | `adapter_dpo.pt` | Diffusion-DPO, best vs worst of 5, β = 500 | offline | 463 it × 32 pairs, lr 3e-4 | 0.677 | 22.51 | 4.99 | 32.9 | |
| | `adapter_online_dpo.pt` | online Diffusion-DPO, β = 500, 4 updates / rollout | online | 40 it × 16 prompts × 4, lr 1e-3 | 0.612 | 22.40 | 5.04 | 32.5 | |
| | `adapter_grpo.pt` | Flow-GRPO, SDE window of 2 steps, branching groups | online | 28 it × 8 prompts × 8, lr 1e-3 | 0.599 | 22.40 | 5.03 | 32.3 | |
| | `adapter_nft.pt` | **DiffusionNFT** (OCR expert), β = 1 | online | 38 it × 8 prompts × 8, lr 1e-3 | **0.746** | 22.35 | 4.79 | 32.3 | |
| | `adapter_nft_pick.pt` | DiffusionNFT on PickScore (expert) | online | 40 it × 8 × 8, lr 1e-3 | 0.687 | 22.54 | 4.95 | 32.6 | |
| | `adapter_nft_aes.pt` | DiffusionNFT on the aesthetic score (expert, **reward-hacked**: drops the text) | online | 39 it × 8 × 8, lr 1e-3 | 0.259 | 21.88 | 5.56 | 29.9 | |
| | `adapter_opd.pt` | **OPD**: on-policy distillation of $v_\text{ref}+\sum_k(v_k-v_\text{ref})$ over the three experts | online, no reward | 101 it × 32, lr 1e-4 | 0.642 | 22.52 | 5.01 | 32.9 | |
|
|
| For comparison, merging the three experts without training: LoRA sum 0.558 / 22.32 / 5.13 / 31.3, LoRA mean 0.581 / 22.42 / 5.07 / 32.2, and the composed teacher (4 networks per step) 0.620 / 22.49 / 5.04 / 32.6. The OPD student matches the teacher at one network per step. |
|
|
| Run-to-run spread at 5 minutes (OCR, several seeds): online DPO 0.646 ± 0.03, GRPO 0.632 ± 0.02, DiffusionNFT 0.736 ± 0.01. The adapters here are from the single reference run of the notebook. |
|
|
|  |
|
|
|  |
|
|
| ## Format |
|
|
| Each `.pt` file is a dict produced by the notebook's `save_adapter`: a peft LoRA state dict (`get_peft_model_state_dict`, fp32, r = 32, alpha = 32) for the targets `to_q, to_k, to_v, to_out.0, conv_inverted, conv_point, proj_out` of `SanaTransformer2DModel`. All adapters are used **with CFG 4.5** (the policy is the guided velocity). |
|
|
| ## Usage (in the workshop notebook) |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| import shutil |
| for m in ["sft", "rwr", "dpo", "online_dpo", "grpo", "nft", "nft_pick", "nft_aes", "opd"]: |
| shutil.copy(hf_hub_download("leffff/diffusion-alignment-workshop-adapters", f"adapter_{m}.pt"), f"{CACHE_DIR}/adapter_{m}.pt") |
| load_adapter(m) |
| evaluate("DiffusionNFT", pi("nft")) |
| ``` |
|
|
