| --- |
| license: apache-2.0 |
| base_model: stabilityai/stable-diffusion-3.5-medium |
| library_name: peft |
| tags: |
| - lora |
| - on-policy-distillation |
| --- |
| |
| # Text-rendering distillation — PDM student (λ = 2) |
|
|
| The **Positive-Direction Matching** student from the text-rendering half of our study of |
| classifier-free guidance in on-policy distillation: a single OCR teacher distilled into a |
| student along the student's own denoising trajectories, on SD3.5-Medium. |
|
|
| ``` |
| pdm_lambda2.safetensors 71.7 MB LoRA over a frozen stabilityai/stable-diffusion-3.5-medium |
| adapter_config.json r=32, alpha=64, attention q/k/v + output projections |
| ``` |
|
|
| ## What this setting is for |
|
|
| It is the **benign half** of the paper's "when does NBA emerge?" comparison, and the result is |
| that PDM and the naive objective perform the same. |
|
|
| Teacher and student here share one base model, one LoRA shape and one guidance scale, and the |
| teacher's negative branch is the plain unconditional branch — nothing the student cannot |
| represent (`c⁻_T = c⁻_S = ∅`). Under that shared negative conditioning both branch errors fall |
| together for every objective, so naive's under-identification never gets travelled. The |
| contrasting privileged-conditioning setting is where the two objectives separate. |
|
|
| ## Results |
|
|
| OCR reward (%, ↑) on the 1 018-prompt held-out set, 40 denoising steps, trained at γ = 4.5: |
|
|
| | Method | γ=1 | γ=2 | γ=3 | γ=4 | γ=4.5 | |
| |---|---|---|---|---|---| |
| | Teacher | 75.24 | 92.91 | 94.16 | 94.43 | 94.09 | |
| | OPD (Naive) | 73.87 | 92.72 | 94.52 | 94.43 | 94.11 | |
| | **OPD (PDM, this)** | **74.48** | **93.73** | **93.99** | **94.02** | **94.38** | |
|
|
| Both students track the teacher across the sweep, including the drop at γ = 1 — that drop is |
| inherited from the teacher rather than introduced by distillation. |
|
|
| Reward is `R_OCR = 1 − min(d, |s|) / |s|`, where `s` is the target string, `ŝ` is what |
| PaddleOCR reads back, both lowercased with spaces stripped, and `d = 0` if `s` occurs inside `ŝ` |
| and `Lev(ŝ, s)` otherwise. Note the substring short-circuit: rendering the target inside otherwise |
| unrelated text still scores 1.0, so this is a legibility score rather than a plain edit distance. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | objective | PDM, `train.opd_loss = "pdm"`, `train.pdm_lambda = 2.0` | |
| | guidance | γ = 4.5, student and teacher | |
| | sampler | flow (ODE), `noise_level = 0` ⇒ mean matching; 10 steps train / 40 eval | |
| | partial trajectory | `timestep_fraction = 0.99` ⇒ K = 9 of 10 transitions supervised | |
| | optimiser | AdamW, constant lr 3e-4, grad clip 1.0, EMA decay 0.9 every 8 steps | |
| | batch | 8 GPUs × 3 prompts × 3 accumulated ⇒ 72 trajectories per update | |
| | steps | 1000 updates, seed 42 | |
| | teacher | upstream's released OCR teacher, [`quanhaol/DiffusionOPD`](https://huggingface.co/quanhaol/DiffusionOPD) | |
|
|
| The teacher is **not** included here — fetch it from upstream. (The PEFT checkpoint this was |
| extracted from carried a copy of it; we verified it is byte-identical to the released one and |
| dropped it.) |
|
|
| ## Use |
|
|
| The code lives in the text-rendering repo, a fork of |
| [`ali-vilab/DiffusionOPD`](https://github.com/ali-vilab/DiffusionOPD): |
|
|
| ```bash |
| torchrun --nproc_per_node=8 scripts/evaluation.py \ |
| --checkpoint_path <dir containing lora/> \ |
| --model_type sd3 --dataset ocr \ |
| --guidance_scale 4.5 --num_inference_steps 40 --mixed_precision fp16 \ |
| --output_dir evaluation_output/pdm_lambda2 |
| ``` |
|
|
| `evaluation.py` expects the adapter at `<checkpoint_path>/lora/adapter_model.safetensors`, so |
| place `pdm_lambda2.safetensors` and `adapter_config.json` in a `lora/` subdirectory, renaming the |
| weights to `adapter_model.safetensors`. |
|
|
| ## Licence |
|
|
| Apache-2.0. SD3.5-Medium and the upstream teacher carry their own terms. |
|
|
