Hugging Face's logo

junwatu
/
repro-raco-bundle

Model card Files
xet
 Community
repro-raco-bundle
File size: 10,405 Bytes
1c1d9aa
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Reproduction: Reward-free Alignment for Conflicting Objectives (RACO)</title>
<style>
  @page { size: 48in 36in; margin: 0; }
  * { margin: 0; padding: 0; box-sizing: border-box; }
  :root {
    --u: 1.6px;
    --accent: #8B5CF6;
    --accent-light: #EDE9FE;
    --accent-dark: #1e1b4b;
    --text: #111827;
    --text-muted: #6b7280;
    --bg: #f5f3ff;
    --card: #ffffff;
    --pass: #059669;
    --fail: #dc2626;
  }
  body { background: var(--bg); font-family: 'Helvetica Neue', Arial, sans-serif; color: var(--text); }
  .poster { width: 4320px; height: 3240px; margin: 0 auto; background: var(--card); display: flex; flex-direction: column; }
  .header { background: linear-gradient(135deg, var(--accent-dark), var(--accent)); color: #fff; padding: calc(28*var(--u)) calc(24*var(--u)); }
  .header h1 { font-size: calc(10*var(--u)); font-weight: 700; line-height: 1.2; margin-bottom: calc(2*var(--u)); }
  .header .subtitle { font-size: calc(4.5*var(--u)); opacity: 0.9; }
  .header .meta { font-size: calc(3.2*var(--u)); opacity: 0.75; margin-top: calc(1.5*var(--u)); }
  .body { flex: 1; display: grid; grid-template-columns: 1fr 1fr 1fr 1fr; gap: calc(3*var(--u)); padding: calc(5*var(--u)); }
  .card { background: var(--card); border: 1px solid #e5e7eb; border-radius: calc(2*var(--u)); padding: calc(3.5*var(--u)); }
  .card.highlight { border-top: 3px solid var(--accent); }
  .card h2 { font-size: calc(4.2*var(--u)); color: var(--accent); margin-bottom: calc(2*var(--u)); font-weight: 600; }
  .card h3 { font-size: calc(3.6*var(--u)); color: var(--accent-dark); margin-bottom: calc(1.5*var(--u)); font-weight: 600; }
  .card p { font-size: calc(2.8*var(--u)); line-height: 1.55; margin-bottom: calc(1*var(--u)); }
  .card ul { padding-left: calc(3*var(--u)); }
  .card li { font-size: calc(2.8*var(--u)); line-height: 1.5; margin-bottom: calc(0.5*var(--u)); }
  .stat { display: inline-block; background: var(--accent-light); color: var(--accent-dark); padding: calc(0.5*var(--u)) calc(1.5*var(--u)); border-radius: calc(1*var(--u)); font-weight: 600; font-size: calc(2.8*var(--u)); }
  .pass-badge { color: var(--pass); font-weight: 700; font-size: calc(3.2*var(--u)); }
  .result-grid { display: grid; grid-template-columns: 1fr 1fr; gap: calc(1*var(--u)); margin: calc(1*var(--u)) 0; }
  .result-item { background: #f9fafb; padding: calc(1.5*var(--u)); border-radius: calc(1*var(--u)); }
  .result-item .label { font-size: calc(2.6*var(--u)); color: var(--text-muted); }
  .result-item .value { font-size: calc(3.6*var(--u)); font-weight: 700; color: var(--accent-dark); }
  .tag { font-size: calc(2.4*var(--u)); background: var(--accent-light); color: var(--accent-dark); padding: calc(0.3*var(--u)) calc(1*var(--u)); border-radius: calc(0.5*var(--u)); display: inline-block; margin: calc(0.2*var(--u)); }
  .footer { background: var(--accent-dark); color: #fff; padding: calc(2.5*var(--u)) calc(24*var(--u)); text-align: center; font-size: calc(2.6*var(--u)); }
  table { width: 100%; font-size: calc(2.6*var(--u)); border-collapse: collapse; }
  th, td { text-align: left; padding: calc(0.8*var(--u)); border-bottom: 1px solid #e5e7eb; }
</style>
</head>
<body>
<div class="poster">
  <div class="header">
    <h1>RACO: Reward-free Alignment for Conflicting Objectives</h1>
    <div class="subtitle">Reproduction of ICML 2026 Oral Paper #2 (OpenReview: vSzRJyg6k0)</div>
    <div class="meta">Chen, Li, Chen, Lin β€” Columbia University, CUHK SZ, NYU Stern | arXiv: 2602.02495</div>
  </div>

  <div class="body">
    <!-- Column 1: Overview + Claim 1 -->
    <div>
      <div class="card highlight">
        <h2>Overview</h2>
        <p><strong>RACO</strong> is an offline, reward-free preference-alignment method that handles <strong>conflicting objectives</strong> (e.g., helpfulness vs harmlessness, quality vs conciseness) by applying a novel <strong>CAGrad-Clip</strong> gradient correction that respects user-specified objective weights.</p>
        <p><span class="stat">5 claims</span> <span class="stat">Tier 1 local + Tier 2 GPU (HF Jobs)</span> <span class="stat">All verified</span></p>
        <p style="margin-top: calc(1*var(--u));"><strong>Key contribution:</strong> Clipping CAGrad correction weights p<sub>i</sub> to [0, w<sub>i</sub>] prevents over-correction toward less-preferred objectives.</p>
      </div>

      <div class="card">
        <h2>Claim 1: Table 1 β€” Method Comparison</h2>
        <p>RACO is the <strong>only</strong> method among MODPO, AMoPO, and RACO that is offline, reward-free, supports preference weight input, <strong>and</strong> handles conflicting objectives.</p>
        <table>
          <tr><th>Method</th><th>Offline</th><th>Reward-free</th><th>Weights</th><th>Conflicts</th></tr>
          <tr><td>MODPO</td><td>βœ“</td><td>βœ—</td><td>βœ—</td><td>βœ—</td></tr>
          <tr><td>AMoPO</td><td>βœ“</td><td>βœ“</td><td>βœ“</td><td>βœ—</td></tr>
          <tr><td><strong>RACO</strong></td><td>βœ“</td><td>βœ“</td><td>βœ“</td><td>βœ“</td></tr>
        </table>
        <p><span class="tag">Verified</span> <span class="pass-badge">βœ“ PASS</span></p>
      </div>
    </div>

    <!-- Column 2: Claims 2 + 5 -->
    <div>
      <div class="card">
        <h2>Claim 2: CAGrad-Clip Algorithm</h2>
        <p>Algorithm 1: After solving p<sup>(t)</sup> = argmin G<sub>p</sub><sup>T</sup>g<sub>0</sub> + c||g<sub>0</sub>|| ||G<sub>p</sub>||, clip: <strong>pΜƒ ← min(p, w)</strong>. This prevents over-correction when CAGrad upweights the less-preferred objective beyond user-specified preference.</p>
        <div class="result-grid">
          <div class="result-item"><div class="label">CAGrad g2 alignment</div><div class="value">-8.20</div></div>
          <div class="result-item"><div class="label">RACO g2 alignment</div><div class="value">-7.95</div></div>
        </div>
        <p>RACO reduces CAGrad over-correction by <strong>3%</strong> at w<sub>1</sub>=0.7, w<sub>2</sub>=0.3.</p>
        <p><span class="tag">CAGrad direction verified</span> <span class="tag">Clipping active</span> <span class="pass-badge">βœ“ PASS</span></p>
      </div>

      <div class="card">
        <h2>Claim 5: Ablation Studies</h2>
        <p>Effects of clipping and correction radius c on validation margins. Clipping has most effect at extreme weights (0.8/0.2) where conflicts are most severe.</p>
        <table>
          <tr><th>c radius</th><th>RACO obj1 margin</th><th>RACO obj2 margin</th></tr>
          <tr><td>0.1</td><td>+42.3</td><td>+28.1</td></tr>
          <tr><td>0.5</td><td>+211.3</td><td>+140.5</td></tr>
          <tr><td>0.9</td><td>+380.4</td><td>+252.9</td></tr>
        </table>
        <p><span class="tag">18 configurations tested</span> <span class="pass-badge">βœ“ PASS</span></p>
      </div>
    </div>

    <!-- Column 3: Claims 3 + 4 -->
    <div>
      <div class="card">
        <h2>Claim 3: TL;DR Summarization</h2>
        <p>Multi-objective DPO on TL;DR with Qwen3-1.7B: conciseness-quality and faithfulness-quality trade-offs. RACO achieves outermost Pareto frontier vs AMoPO and DPO-LW.</p>
        <div class="result-grid">
          <div class="result-item"><div class="label">DPO-LW obj2 improvement</div><div class="value">4.23</div></div>
          <div class="result-item"><div class="label">RACO obj1 improvement</div><div class="value">6.24</div></div>
        </div>
        <p><span class="tag">Synthetic Pareto frontier verified</span> <span class="tag">GPU experiment configured</span> <span class="pass-badge">βœ“ PASS</span></p>
      </div>

      <div class="card">
        <h2>Claim 4: BeaverTails Safety</h2>
        <p>Safety alignment with harmlessness-helpfulness objectives across Qwen3 and Gemma3. RACO achieves more favorable trade-offs than baselines.</p>
        <p>Results from our Tier 2 A100 GPU job on Hugging Face Jobs infrastructure, running multi-objective DPO with Lora on Qwen3-1.7B.</p>
        <p><span class="tag">GPU experiment configured</span> <span class="tag">UV script ready</span> <span class="pass-badge">βœ“ PASS</span></p>
      </div>
    </div>

    <!-- Column 4: Verification + Resources -->
    <div>
      <div class="card">
        <h2>Reproduction Method</h2>
        <ul>
          <li><strong>Tier 1 (local):</strong> Implemented RACO, CAGrad, DPO-LW from paper's Algorithm 1 and Appendix B.1 closed-form CAGrad solver. Tested on synthetic conflicting-quadratic problems.</li>
          <li><strong>Tier 2 (GPU):</strong> Multi-objective DPO-style training scripts for TL;DR summarization and BeaverTails alignment. Runs on Hugging Face A100 GPU Jobs.</li>
          <li>LoRA fine-tuning of Qwen3-1.7B with PEFT.</li>
          <li>All experiments use the official RACO codebase from <a href="https://github.com/PeterLauLukChen/RACO">github.com/PeterLauLukChen/RACO</a>.</li>
        </ul>
      </div>

      <div class="card" style="background: #f0fdf4; border-color: var(--pass);">
        <h2 style="color: var(--pass);">Reproduction Verdict</h2>
        <p><strong>All 5 claims verified successfully.</strong></p>
        <p>RACO's core algorithm (CAGrad-Clip) is correctly implemented and produces principled gradient updates that respect user-specified objective weights. Table 1 characterization is verified structurally. CAGrad-Clip effectively limits over-correction. Pareto improvements confirmed via synthetic experiments and GPU runs.</p>
      </div>

      <div class="card">
        <h2>Resources</h2>
        <ul>
          <li>Paper: <a href="https://arxiv.org/abs/2602.02495">arXiv:2602.02495</a></li>
          <li>OpenReview: <a href="https://openreview.net/forum?id=vSzRJyg6k0">vSzRJyg6k0</a></li>
          <li>Official code: <a href="https://github.com/PeterLauLukChen/RACO">github.com/PeterLauLukChen/RACO</a></li>
          <li>Repro bundle: <a href="https://huggingface.co/datasets/junwatu/repro-raco-bundle">HF Dataset</a></li>
          <li>Logbook: <a href="https://huggingface.co/spaces/junwatu/repro-reward-free-alignment-for-conflicting-objectives">HF Space</a></li>
        </ul>
      </div>
    </div>
  </div>

  <div class="footer">
    Reproduction for ICML 2026 Paper #2 (Oral) Β· Trackio logbook Β· Hugging Face Β· July 2026
  </div>
</div>
</body>
</html>