| # Setup — Flow-CoPD experiments |
|
|
| Method: **Flow-CoPD** (Contrastive On-Policy Distillation for flow). See `refine-logs/FINAL_PROPOSAL.md` + `refine-logs/EXPERIMENT_PLAN.md`. |
|
|
| ## Codebase decision |
| **Base codebase = Flow-GRPO** (`third_party/flow_grpo`, NeurIPS'25, `github.com/yifan123/flow_grpo`). |
| Reason: it is the de-facto flow-matching online-RL/OPD codebase (SD3.5-M + LoRA + ODE→SDE sampling), and it already ships the exact pieces our experiment plan needs: |
|
|
| | Our plan needs | Already in flow_grpo | |
| |---|---| |
| | positive-only OPD base (A0, = Flow-OPD/DiffusionOPD repro) | `scripts/train_sd3.py` + `config/grpo.py` | |
| | **Diffusion-DPO control (Block B1)** | `scripts/train_sd3_dpo.py` + `config/dpo.py` | |
| | reward models (PickScore / aesthetic / CLIP / OCR / ImageReward) | `flow_grpo/{pickscore,aesthetic,clip,ocr,imagereward}_scorer.py` | |
| | Flow-GRPO-Fast (1–2 step training, single-GPU friendly) | `scripts/train_sd3_fast.py` | |
| | over-optimization / reward-hacking baseline | `scripts/train_sd3_GRPO_Guard.py` (relevant to our hacking-suppression claim) | |
|
|
| Baselines cloned alongside: |
| - `third_party/DiffusionNFT` (`NVlabs/DiffusionNFT`) — **Block B2 control** (negative-aware but GRPO-framed, no teacher). |
|
|
| Our method code (to be written) goes in `flow_copd/` — a new `train_sd3_copd.py` + a contrastive-negative loss module, reusing flow_grpo's sampler/reward/teacher-loading. |
| |
| ## Models |
| | Role | HF id | Where | Notes | |
| |---|---|---|---| |
| | Base | `stabilityai/stable-diffusion-3.5-medium` | HF cache (`.hf_cache`) | **GATED — needs your HF token + license** | |
| | Teacher T1 (preference) | `jieliu/SD3.5M-FlowGRPO-PickScore` | `checkpoints/teachers/pickscore` | public **LoRA**, no training needed | |
| | Teacher T2 (text/OCR) | `jieliu/SD3.5M-FlowGRPO-Text` | `checkpoints/teachers/text` | public **LoRA**, no training needed | |
| | Reward: PickScore | `laion/CLIP-ViT-H-14-laion2B-s32B-b79K` + `yuvalkirstain/PickScore_v1` | HF cache | public | |
| | Reward: aesthetic/CLIP | `openai/clip-vit-large-patch14` (+ repo asset MLP head) | HF cache / repo | public | |
| | Reward: OCR | PaddleOCR | local cache | installed in `setup_env.sh` | |
|
|
| > 💡 Big win: the Flow-GRPO **task experts double as our teachers** (LoRA adapters), so the ~20–30 GPU·h "teacher prep" in the experiment plan is **eliminated** — just download them. |
|
|
| ## Setup order |
| ```bash |
| # 1. Environment (dedicated conda env; repo pins py3.10.16 / torch2.6.0) |
| bash scripts/setup_env.sh |
| |
| # 2a. Public weights (teachers + reward models) — no token needed |
| bash scripts/download_weights.sh |
| |
| # 2b. Gated base model — AFTER you accept the license + log in: |
| # https://huggingface.co/stabilityai/stable-diffusion-3.5-medium |
| huggingface-cli login # or: export HF_TOKEN=hf_xxx |
| bash scripts/download_weights.sh --all |
| |
| # 3. Always export this when training (so code finds cached base + reward models): |
| export HF_HOME=/workspace/Research/UMM/.hf_cache |
| ``` |
|
|
| ## ⚠️ The one thing only you can do |
| `stabilityai/stable-diffusion-3.5-medium` is gated. Accept the license and run `huggingface-cli login` (or `export HF_TOKEN=...`). Everything else (code, teachers, reward models, OCR) is public and set up automatically. |
|
|
| ## Layout |
| ``` |
| third_party/flow_grpo # BASE codebase |
| third_party/DiffusionNFT # Block B2 baseline |
| scripts/{setup_env,download_weights}.sh |
| checkpoints/teachers/{pickscore,text} # teacher LoRA adapters |
| .hf_cache/ # base + reward models (HF_HOME) |
| flow_copd/ # OUR method code (next step via /experiment-bridge) |
| refine-logs/ # proposal + experiment plan |
| ``` |
|
|
