File size: 4,012 Bytes
35368ea | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | ---
license: apache-2.0
base_model: Efficient-Large-Model/Sana_600M_512px_diffusers
library_name: peft
datasets:
- leffff/diffusion-alignment-workshop
tags:
- diffusion
- alignment
- rlhf
- text-rendering
- sana
- lora
- text-to-image
---
# Diffusion Alignment Workshop: Sana-600M aligned for text rendering
LoRA adapters that align [Sana-600M-512px](https://huggingface.co/Efficient-Large-Model/Sana_600M_512px_diffusers) to **render the text asked for in the prompt** (`a chalkboard with the word "Recycling" written on it`), trained with six methods, plus two more reward experts and their on-policy-distilled merge. Every adapter was trained for **5 minutes on one H100** with the workshop notebook. Data: [`leffff/diffusion-alignment-workshop`](https://huggingface.co/datasets/leffff/diffusion-alignment-workshop).
Rewards: **OCR** = 1 − (character edits to find the target text in the [Qwen3-VL-2B](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) transcription) / its length; **PickScore**; **aesthetic** (LAION predictor v2); **CLIP score** (held out, never trained on). Evaluation: 64 held-out prompts × 2 seeds, 20 Euler steps, CFG 4.5.
| file | method | data | training (5 min) | OCR ↑ | PickScore | aesthetic | CLIP |
|---|---|---|---|---|---|---|---|
| — | Sana-600M (reference) | — | — | 0.569 | 22.41 | 5.05 | 32.2 |
| `adapter_sft.pt` | SFT on the best of 5 samples (OCR ≥ 0.5) | offline | 1018 it × 32, lr 3e-4 | 0.712 | 22.45 | 4.91 | 33.3 |
| `adapter_rwr.pt` | reward-weighted regression, group softmax τ = 0.1 | offline | 1052 it × 6 prompts × 5, lr 3e-4 | 0.696 | 22.42 | 4.95 | 33.0 |
| `adapter_dpo.pt` | Diffusion-DPO, best vs worst of 5, β = 500 | offline | 463 it × 32 pairs, lr 3e-4 | 0.677 | 22.51 | 4.99 | 32.9 |
| `adapter_online_dpo.pt` | online Diffusion-DPO, β = 500, 4 updates / rollout | online | 40 it × 16 prompts × 4, lr 1e-3 | 0.612 | 22.40 | 5.04 | 32.5 |
| `adapter_grpo.pt` | Flow-GRPO, SDE window of 2 steps, branching groups | online | 28 it × 8 prompts × 8, lr 1e-3 | 0.599 | 22.40 | 5.03 | 32.3 |
| `adapter_nft.pt` | **DiffusionNFT** (OCR expert), β = 1 | online | 38 it × 8 prompts × 8, lr 1e-3 | **0.746** | 22.35 | 4.79 | 32.3 |
| `adapter_nft_pick.pt` | DiffusionNFT on PickScore (expert) | online | 40 it × 8 × 8, lr 1e-3 | 0.687 | 22.54 | 4.95 | 32.6 |
| `adapter_nft_aes.pt` | DiffusionNFT on the aesthetic score (expert, **reward-hacked**: drops the text) | online | 39 it × 8 × 8, lr 1e-3 | 0.259 | 21.88 | 5.56 | 29.9 |
| `adapter_opd.pt` | **OPD**: on-policy distillation of $v_\text{ref}+\sum_k(v_k-v_\text{ref})$ over the three experts | online, no reward | 101 it × 32, lr 1e-4 | 0.642 | 22.52 | 5.01 | 32.9 |
For comparison, merging the three experts without training: LoRA sum 0.558 / 22.32 / 5.13 / 31.3, LoRA mean 0.581 / 22.42 / 5.07 / 32.2, and the composed teacher (4 networks per step) 0.620 / 22.49 / 5.04 / 32.6. The OPD student matches the teacher at one network per step.
Run-to-run spread at 5 minutes (OCR, several seeds): online DPO 0.646 ± 0.03, GRPO 0.632 ± 0.02, DiffusionNFT 0.736 ± 0.01. The adapters here are from the single reference run of the notebook.
## Format
Each `.pt` file is a dict produced by the notebook's `save_adapter`: a peft LoRA state dict (`get_peft_model_state_dict`, fp32, r = 32, alpha = 32) for the targets `to_q, to_k, to_v, to_out.0, conv_inverted, conv_point, proj_out` of `SanaTransformer2DModel`. All adapters are used **with CFG 4.5** (the policy is the guided velocity).
## Usage (in the workshop notebook)
```python
from huggingface_hub import hf_hub_download
import shutil
for m in ["sft", "rwr", "dpo", "online_dpo", "grpo", "nft", "nft_pick", "nft_aes", "opd"]:
shutil.copy(hf_hub_download("leffff/diffusion-alignment-workshop-adapters", f"adapter_{m}.pt"), f"{CACHE_DIR}/adapter_{m}.pt")
load_adapter(m)
evaluate("DiffusionNFT", pi("nft"))
```
|