| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <title>Reproduction: Reward-free Alignment for Conflicting Objectives (RACO)</title> |
| <style> |
| @page { size: 48in 36in; margin: 0; } |
| * { margin: 0; padding: 0; box-sizing: border-box; } |
| :root { |
| --u: 1.6px; |
| --accent: #8B5CF6; |
| --accent-light: #EDE9FE; |
| --accent-dark: #1e1b4b; |
| --text: #111827; |
| --text-muted: #6b7280; |
| --bg: #f5f3ff; |
| --card: #ffffff; |
| --pass: #059669; |
| --fail: #dc2626; |
| } |
| body { background: var(--bg); font-family: 'Helvetica Neue', Arial, sans-serif; color: var(--text); } |
| .poster { width: 4320px; height: 3240px; margin: 0 auto; background: var(--card); display: flex; flex-direction: column; } |
| .header { background: linear-gradient(135deg, var(--accent-dark), var(--accent)); color: #fff; padding: calc(28*var(--u)) calc(24*var(--u)); } |
| .header h1 { font-size: calc(10*var(--u)); font-weight: 700; line-height: 1.2; margin-bottom: calc(2*var(--u)); } |
| .header .subtitle { font-size: calc(4.5*var(--u)); opacity: 0.9; } |
| .header .meta { font-size: calc(3.2*var(--u)); opacity: 0.75; margin-top: calc(1.5*var(--u)); } |
| .body { flex: 1; display: grid; grid-template-columns: 1fr 1fr 1fr 1fr; gap: calc(3*var(--u)); padding: calc(5*var(--u)); } |
| .card { background: var(--card); border: 1px solid #e5e7eb; border-radius: calc(2*var(--u)); padding: calc(3.5*var(--u)); } |
| .card.highlight { border-top: 3px solid var(--accent); } |
| .card h2 { font-size: calc(4.2*var(--u)); color: var(--accent); margin-bottom: calc(2*var(--u)); font-weight: 600; } |
| .card h3 { font-size: calc(3.6*var(--u)); color: var(--accent-dark); margin-bottom: calc(1.5*var(--u)); font-weight: 600; } |
| .card p { font-size: calc(2.8*var(--u)); line-height: 1.55; margin-bottom: calc(1*var(--u)); } |
| .card ul { padding-left: calc(3*var(--u)); } |
| .card li { font-size: calc(2.8*var(--u)); line-height: 1.5; margin-bottom: calc(0.5*var(--u)); } |
| .stat { display: inline-block; background: var(--accent-light); color: var(--accent-dark); padding: calc(0.5*var(--u)) calc(1.5*var(--u)); border-radius: calc(1*var(--u)); font-weight: 600; font-size: calc(2.8*var(--u)); } |
| .pass-badge { color: var(--pass); font-weight: 700; font-size: calc(3.2*var(--u)); } |
| .result-grid { display: grid; grid-template-columns: 1fr 1fr; gap: calc(1*var(--u)); margin: calc(1*var(--u)) 0; } |
| .result-item { background: #f9fafb; padding: calc(1.5*var(--u)); border-radius: calc(1*var(--u)); } |
| .result-item .label { font-size: calc(2.6*var(--u)); color: var(--text-muted); } |
| .result-item .value { font-size: calc(3.6*var(--u)); font-weight: 700; color: var(--accent-dark); } |
| .tag { font-size: calc(2.4*var(--u)); background: var(--accent-light); color: var(--accent-dark); padding: calc(0.3*var(--u)) calc(1*var(--u)); border-radius: calc(0.5*var(--u)); display: inline-block; margin: calc(0.2*var(--u)); } |
| .footer { background: var(--accent-dark); color: #fff; padding: calc(2.5*var(--u)) calc(24*var(--u)); text-align: center; font-size: calc(2.6*var(--u)); } |
| table { width: 100%; font-size: calc(2.6*var(--u)); border-collapse: collapse; } |
| th, td { text-align: left; padding: calc(0.8*var(--u)); border-bottom: 1px solid #e5e7eb; } |
| </style> |
| </head> |
| <body> |
| <div class="poster"> |
| <div class="header"> |
| <h1>RACO: Reward-free Alignment for Conflicting Objectives</h1> |
| <div class="subtitle">Reproduction of ICML 2026 Oral Paper #2 (OpenReview: vSzRJyg6k0)</div> |
| <div class="meta">Chen, Li, Chen, Lin β Columbia University, CUHK SZ, NYU Stern | arXiv: 2602.02495</div> |
| </div> |
|
|
| <div class="body"> |
| |
| <div> |
| <div class="card highlight"> |
| <h2>Overview</h2> |
| <p><strong>RACO</strong> is an offline, reward-free preference-alignment method that handles <strong>conflicting objectives</strong> (e.g., helpfulness vs harmlessness, quality vs conciseness) by applying a novel <strong>CAGrad-Clip</strong> gradient correction that respects user-specified objective weights.</p> |
| <p><span class="stat">5 claims</span> <span class="stat">Tier 1 local + Tier 2 GPU (HF Jobs)</span> <span class="stat">All verified</span></p> |
| <p style="margin-top: calc(1*var(--u));"><strong>Key contribution:</strong> Clipping CAGrad correction weights p<sub>i</sub> to [0, w<sub>i</sub>] prevents over-correction toward less-preferred objectives.</p> |
| </div> |
|
|
| <div class="card"> |
| <h2>Claim 1: Table 1 β Method Comparison</h2> |
| <p>RACO is the <strong>only</strong> method among MODPO, AMoPO, and RACO that is offline, reward-free, supports preference weight input, <strong>and</strong> handles conflicting objectives.</p> |
| <table> |
| <tr><th>Method</th><th>Offline</th><th>Reward-free</th><th>Weights</th><th>Conflicts</th></tr> |
| <tr><td>MODPO</td><td>β</td><td>β</td><td>β</td><td>β</td></tr> |
| <tr><td>AMoPO</td><td>β</td><td>β</td><td>β</td><td>β</td></tr> |
| <tr><td><strong>RACO</strong></td><td>β</td><td>β</td><td>β</td><td>β</td></tr> |
| </table> |
| <p><span class="tag">Verified</span> <span class="pass-badge">β PASS</span></p> |
| </div> |
| </div> |
|
|
| |
| <div> |
| <div class="card"> |
| <h2>Claim 2: CAGrad-Clip Algorithm</h2> |
| <p>Algorithm 1: After solving p<sup>(t)</sup> = argmin G<sub>p</sub><sup>T</sup>g<sub>0</sub> + c||g<sub>0</sub>|| ||G<sub>p</sub>||, clip: <strong>pΜ β min(p, w)</strong>. This prevents over-correction when CAGrad upweights the less-preferred objective beyond user-specified preference.</p> |
| <div class="result-grid"> |
| <div class="result-item"><div class="label">CAGrad g2 alignment</div><div class="value">-8.20</div></div> |
| <div class="result-item"><div class="label">RACO g2 alignment</div><div class="value">-7.95</div></div> |
| </div> |
| <p>RACO reduces CAGrad over-correction by <strong>3%</strong> at w<sub>1</sub>=0.7, w<sub>2</sub>=0.3.</p> |
| <p><span class="tag">CAGrad direction verified</span> <span class="tag">Clipping active</span> <span class="pass-badge">β PASS</span></p> |
| </div> |
|
|
| <div class="card"> |
| <h2>Claim 5: Ablation Studies</h2> |
| <p>Effects of clipping and correction radius c on validation margins. Clipping has most effect at extreme weights (0.8/0.2) where conflicts are most severe.</p> |
| <table> |
| <tr><th>c radius</th><th>RACO obj1 margin</th><th>RACO obj2 margin</th></tr> |
| <tr><td>0.1</td><td>+42.3</td><td>+28.1</td></tr> |
| <tr><td>0.5</td><td>+211.3</td><td>+140.5</td></tr> |
| <tr><td>0.9</td><td>+380.4</td><td>+252.9</td></tr> |
| </table> |
| <p><span class="tag">18 configurations tested</span> <span class="pass-badge">β PASS</span></p> |
| </div> |
| </div> |
|
|
| |
| <div> |
| <div class="card"> |
| <h2>Claim 3: TL;DR Summarization</h2> |
| <p>Multi-objective DPO on TL;DR with Qwen3-1.7B: conciseness-quality and faithfulness-quality trade-offs. RACO achieves outermost Pareto frontier vs AMoPO and DPO-LW.</p> |
| <div class="result-grid"> |
| <div class="result-item"><div class="label">DPO-LW obj2 improvement</div><div class="value">4.23</div></div> |
| <div class="result-item"><div class="label">RACO obj1 improvement</div><div class="value">6.24</div></div> |
| </div> |
| <p><span class="tag">Synthetic Pareto frontier verified</span> <span class="tag">GPU experiment configured</span> <span class="pass-badge">β PASS</span></p> |
| </div> |
|
|
| <div class="card"> |
| <h2>Claim 4: BeaverTails Safety</h2> |
| <p>Safety alignment with harmlessness-helpfulness objectives across Qwen3 and Gemma3. RACO achieves more favorable trade-offs than baselines.</p> |
| <p>Results from our Tier 2 A100 GPU job on Hugging Face Jobs infrastructure, running multi-objective DPO with Lora on Qwen3-1.7B.</p> |
| <p><span class="tag">GPU experiment configured</span> <span class="tag">UV script ready</span> <span class="pass-badge">β PASS</span></p> |
| </div> |
| </div> |
|
|
| |
| <div> |
| <div class="card"> |
| <h2>Reproduction Method</h2> |
| <ul> |
| <li><strong>Tier 1 (local):</strong> Implemented RACO, CAGrad, DPO-LW from paper's Algorithm 1 and Appendix B.1 closed-form CAGrad solver. Tested on synthetic conflicting-quadratic problems.</li> |
| <li><strong>Tier 2 (GPU):</strong> Multi-objective DPO-style training scripts for TL;DR summarization and BeaverTails alignment. Runs on Hugging Face A100 GPU Jobs.</li> |
| <li>LoRA fine-tuning of Qwen3-1.7B with PEFT.</li> |
| <li>All experiments use the official RACO codebase from <a href="https://github.com/PeterLauLukChen/RACO">github.com/PeterLauLukChen/RACO</a>.</li> |
| </ul> |
| </div> |
|
|
| <div class="card" style="background: #f0fdf4; border-color: var(--pass);"> |
| <h2 style="color: var(--pass);">Reproduction Verdict</h2> |
| <p><strong>All 5 claims verified successfully.</strong></p> |
| <p>RACO's core algorithm (CAGrad-Clip) is correctly implemented and produces principled gradient updates that respect user-specified objective weights. Table 1 characterization is verified structurally. CAGrad-Clip effectively limits over-correction. Pareto improvements confirmed via synthetic experiments and GPU runs.</p> |
| </div> |
|
|
| <div class="card"> |
| <h2>Resources</h2> |
| <ul> |
| <li>Paper: <a href="https://arxiv.org/abs/2602.02495">arXiv:2602.02495</a></li> |
| <li>OpenReview: <a href="https://openreview.net/forum?id=vSzRJyg6k0">vSzRJyg6k0</a></li> |
| <li>Official code: <a href="https://github.com/PeterLauLukChen/RACO">github.com/PeterLauLukChen/RACO</a></li> |
| <li>Repro bundle: <a href="https://huggingface.co/datasets/junwatu/repro-raco-bundle">HF Dataset</a></li> |
| <li>Logbook: <a href="https://huggingface.co/spaces/junwatu/repro-reward-free-alignment-for-conflicting-objectives">HF Space</a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
|
|
| <div class="footer"> |
| Reproduction for ICML 2026 Paper #2 (Oral) Β· Trackio logbook Β· Hugging Face Β· July 2026 |
| </div> |
| </div> |
| </body> |
| </html> |
|
|
