LoRA adapters of the diffusion alignment workshop (reference run, H100, 5 min each)

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+samples/dataset.png filter=lfs diff=lfs merge=lfs -text
+samples/experts_and_merges.png filter=lfs diff=lfs merge=lfs -text
+samples/ocr_methods.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,57 @@
+---
+license: apache-2.0
+base_model: Efficient-Large-Model/Sana_600M_512px_diffusers
+library_name: peft
+datasets:
+- leffff/diffusion-alignment-workshop
+tags:
+- diffusion
+- alignment
+- rlhf
+- text-rendering
+- sana
+- lora
+- text-to-image
+---
+
+# Diffusion Alignment Workshop: Sana-600M aligned for text rendering
+
+LoRA adapters that align [Sana-600M-512px](https://huggingface.co/Efficient-Large-Model/Sana_600M_512px_diffusers) to **render the text asked for in the prompt** (`a chalkboard with the word "Recycling" written on it`), trained with six methods, plus two more reward experts and their on-policy-distilled merge. Every adapter was trained for **5 minutes on one H100** with the workshop notebook. Data: [`leffff/diffusion-alignment-workshop`](https://huggingface.co/datasets/leffff/diffusion-alignment-workshop).
+
+Rewards: **OCR** = 1 − (character edits to find the target text in the [Qwen3-VL-2B](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) transcription) / its length; **PickScore**; **aesthetic** (LAION predictor v2); **CLIP score** (held out, never trained on). Evaluation: 64 held-out prompts × 2 seeds, 20 Euler steps, CFG 4.5.
+
+| file | method | data | training (5 min) | OCR ↑ | PickScore | aesthetic | CLIP |
+|---|---|---|---|---|---|---|---|
+| — | Sana-600M (reference) | — | — | 0.569 | 22.41 | 5.05 | 32.2 |
+| `adapter_sft.pt` | SFT on the best of 5 samples (OCR ≥ 0.5) | offline | 1018 it × 32, lr 3e-4 | 0.712 | 22.45 | 4.91 | 33.3 |
+| `adapter_rwr.pt` | reward-weighted regression, group softmax τ = 0.1 | offline | 1052 it × 6 prompts × 5, lr 3e-4 | 0.696 | 22.42 | 4.95 | 33.0 |
+| `adapter_dpo.pt` | Diffusion-DPO, best vs worst of 5, β = 500 | offline | 463 it × 32 pairs, lr 3e-4 | 0.677 | 22.51 | 4.99 | 32.9 |
+| `adapter_online_dpo.pt` | online Diffusion-DPO, β = 500, 4 updates / rollout | online | 40 it × 16 prompts × 4, lr 1e-3 | 0.612 | 22.40 | 5.04 | 32.5 |
+| `adapter_grpo.pt` | Flow-GRPO, SDE window of 2 steps, branching groups | online | 28 it × 8 prompts × 8, lr 1e-3 | 0.599 | 22.40 | 5.03 | 32.3 |
+| `adapter_nft.pt` | **DiffusionNFT** (OCR expert), β = 1 | online | 38 it × 8 prompts × 8, lr 1e-3 | **0.746** | 22.35 | 4.79 | 32.3 |
+| `adapter_nft_pick.pt` | DiffusionNFT on PickScore (expert) | online | 40 it × 8 × 8, lr 1e-3 | 0.687 | 22.54 | 4.95 | 32.6 |
+| `adapter_nft_aes.pt` | DiffusionNFT on the aesthetic score (expert, **reward-hacked**: drops the text) | online | 39 it × 8 × 8, lr 1e-3 | 0.259 | 21.88 | 5.56 | 29.9 |
+| `adapter_opd.pt` | **OPD**: on-policy distillation of $v_\text{ref}+\sum_k(v_k-v_\text{ref})$ over the three experts | online, no reward | 101 it × 32, lr 1e-4 | 0.642 | 22.52 | 5.01 | 32.9 |
+
+For comparison, merging the three experts without training: LoRA sum 0.558 / 22.32 / 5.13 / 31.3, LoRA mean 0.581 / 22.42 / 5.07 / 32.2, and the composed teacher (4 networks per step) 0.620 / 22.49 / 5.04 / 32.6. The OPD student matches the teacher at one network per step.
+
+Run-to-run spread at 5 minutes (OCR, several seeds): online DPO 0.646 ± 0.03, GRPO 0.632 ± 0.02, DiffusionNFT 0.736 ± 0.01. The adapters here are from the single reference run of the notebook.
+
+![OCR methods](samples/ocr_methods.png)
+
+![experts and merges](samples/experts_and_merges.png)
+
+## Format
+
+Each `.pt` file is a dict produced by the notebook's `save_adapter`: a peft LoRA state dict (`get_peft_model_state_dict`, fp32, r = 32, alpha = 32) for the targets `to_q, to_k, to_v, to_out.0, conv_inverted, conv_point, proj_out` of `SanaTransformer2DModel`. All adapters are used **with CFG 4.5** (the policy is the guided velocity).
+
+## Usage (in the workshop notebook)
+
+```python
+from huggingface_hub import hf_hub_download
+import shutil
+for m in ["sft", "rwr", "dpo", "online_dpo", "grpo", "nft", "nft_pick", "nft_aes", "opd"]:
+    shutil.copy(hf_hub_download("leffff/diffusion-alignment-workshop-adapters", f"adapter_{m}.pt"), f"{CACHE_DIR}/adapter_{m}.pt")
+    load_adapter(m)
+evaluate("DiffusionNFT", pi("nft"))
+```

adapter_dpo.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc60c685bbe713b5afaa9377dbecbfd5903d6a03fad5052871fe4675e5a47a56
+size 105622075

adapter_grpo.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d52a0cb0101393dff5462eda475fe288652f6662e85da8d051de2411609650fc
+size 105622643

adapter_nft.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7d4f1c3454061ef7c273c845e2c6451942c5eceec657e2023d6f47955524acf
+size 105622075

adapter_nft_aes.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb9ce88224abfb2813114634666490e446c8a910d6fd0340abc5cd973cb549d5
+size 105624347

adapter_nft_pick.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb0759b4e51367c7a06641eb0032fe168c4e7fb5815acf2282377ab6851ca3f0
+size 105624915

adapter_online_dpo.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:319ec3664e835e932a319f7a995c4ac1e1930487ac148e044188c81064ac9b17
+size 105626051

adapter_opd.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c102853660970403b484a8a894521f52eaa0cceb815411049e0e8ce998f66ca9
+size 105622075

adapter_rwr.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a3fab18c2551d0d18d9ac44d35b809a4535dfe990af3a9b56cea56a158f2999
+size 105622075

adapter_sft.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6f3a573048b035b18f9637c49116fb35c758a8f4abef4abdbe17f380f11de2f
+size 105622075