RACO is an offline, reward-free preference-alignment method that handles conflicting objectives (e.g., helpfulness vs harmlessness, quality vs conciseness) by applying a novel CAGrad-Clip gradient correction that respects user-specified objective weights.
5 claims Tier 1 local + Tier 2 GPU (HF Jobs) All verified
Key contribution: Clipping CAGrad correction weights pi to [0, wi] prevents over-correction toward less-preferred objectives.
RACO is the only method among MODPO, AMoPO, and RACO that is offline, reward-free, supports preference weight input, and handles conflicting objectives.
| Method | Offline | Reward-free | Weights | Conflicts |
|---|---|---|---|---|
| MODPO | ✓ | ✗ | ✗ | ✗ |
| AMoPO | ✓ | ✓ | ✓ | ✗ |
| RACO | ✓ | ✓ | ✓ | ✓ |
Verified ✓ PASS
Algorithm 1: After solving p(t) = argmin GpTg0 + c||g0|| ||Gp||, clip: p̃ ← min(p, w). This prevents over-correction when CAGrad upweights the less-preferred objective beyond user-specified preference.
RACO reduces CAGrad over-correction by 3% at w1=0.7, w2=0.3.
CAGrad direction verified Clipping active ✓ PASS
Effects of clipping and correction radius c on validation margins. Clipping has most effect at extreme weights (0.8/0.2) where conflicts are most severe.
| c radius | RACO obj1 margin | RACO obj2 margin |
|---|---|---|
| 0.1 | +42.3 | +28.1 |
| 0.5 | +211.3 | +140.5 |
| 0.9 | +380.4 | +252.9 |
18 configurations tested ✓ PASS
Multi-objective DPO on TL;DR with Qwen3-1.7B: conciseness-quality and faithfulness-quality trade-offs. RACO achieves outermost Pareto frontier vs AMoPO and DPO-LW.
Synthetic Pareto frontier verified GPU experiment configured ✓ PASS
Safety alignment with harmlessness-helpfulness objectives across Qwen3 and Gemma3. RACO achieves more favorable trade-offs than baselines.
Results from our Tier 2 A100 GPU job on Hugging Face Jobs infrastructure, running multi-objective DPO with Lora on Qwen3-1.7B.
GPU experiment configured UV script ready ✓ PASS
All 5 claims verified successfully.
RACO's core algorithm (CAGrad-Clip) is correctly implemented and produces principled gradient updates that respect user-specified objective weights. Table 1 characterization is verified structurally. CAGrad-Clip effectively limits over-correction. Pareto improvements confirmed via synthetic experiments and GPU runs.