| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <meta name="generator" content="posterly"> |
| <title>CSPO Reproduction — ICML 2026</title> |
| <script>window.MathJax={tex:{inlineMath:[['$','$'],['\\(','\\)']],displayMath:[['$$','$$'],['\\[','\\]']],packages:{'[+]':['ams']}},svg:{fontCache:'global'}};</script> |
| <script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-svg.js"></script> |
| <style> |
| @page { size: 60in 36in; margin: 0; } |
| :root { |
| --accent: #2D5F8B; --accent-deep: #1F4566; --accent-light: #E8F1F8; |
| --accent-soft: #D7E5F0; --accent-ink: #FFFFFF; |
| --emph: #C9A24A; --emph-soft: #FFF7E0; --emph-ink: #14314A; |
| --text-primary: #1A1A1A; --text-secondary: #555555; --text-muted: #888888; |
| --bg-page: #F6F2F0; --bg-card: #FFFFFF; --bg-strip: #F0EDEA; |
| --border: #DDD8D4; --shadow: rgba(0,0,0,0.08); |
| --u: 1.6px; --fs-title: calc(48 * var(--u)); --fs-authors: calc(28 * var(--u)); |
| --fs-h1: calc(32 * var(--u)); --fs-h2: calc(26 * var(--u)); --fs-body: calc(20 * var(--u)); |
| --fs-small: calc(17 * var(--u)); --fs-code: calc(16 * var(--u)); |
| --col-gap: calc(20 * var(--u)); --card-pad: calc(16 * var(--u)); |
| --page-margin: calc(30 * var(--u)); |
| } |
| * { margin: 0; padding: 0; box-sizing: border-box; } |
| body { background: var(--bg-page); font-family: 'Helvetica Neue', Arial, sans-serif; color: var(--text-primary); } |
| .poster { width: calc(3600 * var(--u)); height: calc(2160 * var(--u)); margin: 0 auto; background: var(--bg-page); display: flex; flex-direction: column; } |
| .header { background: var(--accent); color: white; padding: calc(24 * var(--u)) calc(30 * var(--u)); text-align: center; } |
| .header h1 { font-size: var(--fs-title); font-weight: 700; margin-bottom: calc(8 * var(--u)); } |
| .header .authors { font-size: var(--fs-authors); opacity: 0.9; } |
| .header .venue { font-size: var(--fs-h2); opacity: 0.8; margin-top: calc(6 * var(--u)); } |
| .columns { display: flex; flex: 1; gap: var(--col-gap); padding: var(--col-gap) var(--page-margin); } |
| .col { flex: 1; display: flex; flex-direction: column; gap: var(--col-gap); } |
| .card { background: var(--bg-card); border-radius: calc(6 * var(--u)); padding: var(--card-pad); box-shadow: 0 calc(2 * var(--u)) calc(8 * var(--u)) var(--shadow); } |
| .card h2 { font-size: var(--fs-h1); color: var(--accent); border-bottom: calc(3 * var(--u)) solid var(--accent); padding-bottom: calc(6 * var(--u)); margin-bottom: calc(10 * var(--u)); } |
| .card h3 { font-size: var(--fs-h2); color: var(--accent-deep); margin: calc(8 * var(--u)) 0 calc(4 * var(--u)); } |
| .card p, .card li { font-size: var(--fs-body); line-height: 1.4; margin-bottom: calc(4 * var(--u)); } |
| .card ul { padding-left: calc(20 * var(--u)); } |
| .card .highlight { background: var(--emph-soft); padding: calc(4 * var(--u)) calc(8 * var(--u)); border-radius: calc(4 * var(--u)); font-weight: 600; } |
| .card .tag { display: inline-block; background: var(--accent-light); color: var(--accent-deep); padding: calc(2 * var(--u)) calc(8 * var(--u)); border-radius: calc(4 * var(--u)); font-size: var(--fs-small); margin: calc(2 * var(--u)); } |
| .footer { background: var(--accent-deep); color: white; text-align: center; padding: calc(12 * var(--u)); font-size: var(--fs-small); } |
| table { width: 100%; border-collapse: collapse; font-size: var(--fs-small); margin: calc(6 * var(--u)) 0; } |
| th { background: var(--accent); color: white; padding: calc(4 * var(--u)) calc(6 * var(--u)); text-align: left; } |
| td { padding: calc(4 * var(--u)) calc(6 * var(--u)); border-bottom: 1px solid var(--border); } |
| .ours { background: var(--emph-soft); font-weight: 600; } |
| </style> |
| </head> |
| <body> |
| <div class="poster"> |
| <div class="header"> |
| <h1>CSPO: Constraint-Sensitive Policy Optimization for Safe RL</h1> |
| <div class="authors">Ayoub Belouadah, Sylvain Kubler, Yves Le Traon</div> |
| <div class="venue">ICML 2026 Spotlight — Reproduction Report</div> |
| </div> |
| <div class="columns"> |
| <div class="col"> |
| <div class="card"> |
| <h2>What is CSPO?</h2> |
| <p>First-order primal-dual safe RL algorithm that incorporates <strong>local constraint sensitivity</strong> into policy updates.</p> |
| <p>Key idea: scale constraint correction by $w_k = 1/\|\nabla g(\theta_k)\|^2$, derived from shortest signed distance to safety boundary.</p> |
| <p>Effective multiplier: $\lambda_{\text{eff}} = \lambda + \alpha w_k [g(\theta)]_+$</p> |
| <p>Flat gradients → larger $w$ → stronger correction (fast recovery)<br>Steep gradients → smaller $w$ → cautious correction (avoid overshoot)</p> |
| </div> |
| <div class="card"> |
| <h2>Claim 1: Weight Derivation</h2> |
| <p><span class="tag">Numerical Audit</span> <span class="tag">Supported</span></p> |
| <p>Verified minimal-norm update: $\Delta\theta^* = -\frac{g(\theta_k)}{\|\nabla g(\theta_k)\|^2}\nabla g(\theta_k)$</p> |
| <p>Shortest signed distance: $\|\Delta\theta^*\| = |g(\theta_k)|/\|\nabla g(\theta_k)\|$</p> |
| <p>Code at <code>cspo.py:56-90</code> matches paper Eq. (12) exactly.</p> |
| </div> |
| <div class="card"> |
| <h2>Claim 2: Theory</h2> |
| <p><span class="tag">Numerical Audit</span> <span class="tag">Supported</span></p> |
| <p>KKT equivalence: $q_k(\theta)$ vanishes at feasible points and boundary → same solution set.</p> |
| <p>Convergence: $O(L^3 G^2 \lambda_{\max}^2 / \varepsilon^6)$ — consistent with nonconvex-concave minimax theory.</p> |
| </div> |
| </div> |
| <div class="col"> |
| <div class="card"> |
| <h2>Claim 3: Experimental Setup</h2> |
| <p><span class="tag">Code Audit</span> <span class="tag">Supported</span></p> |
| <p><strong>9 tasks</strong> (5 locomotion + 4 navigation) from Safety Gymnasium</p> |
| <p><strong>12 baselines</strong>: PPO-Lag, CPPO-PID, CPO, PCPO, C-TRPO, FOCOPS, CUP, P3O, IPO, EPO, APPO, CSPO</p> |
| <p>All confirmed in official codebase at <code>github.com/serval-uni-lu/CSPO</code></p> |
| </div> |
| <div class="card"> |
| <h2>Claim 4: PointGoal Results</h2> |
| <p><span class="tag">Partially Verified</span></p> |
| <table> |
| <tr><th>Method</th><th>Return</th><th>Cost</th></tr> |
| <tr class="ours"><td>CSPO</td><td>23.79 ± 0.75</td><td>≤25 ✓</td></tr> |
| <tr><td>PPO-Lag</td><td>21.78 ± 2.38</td><td>≤25 ✓</td></tr> |
| <tr><td>APPO</td><td>22.14 ± 1.02</td><td>≤25 ✓</td></tr> |
| <tr><td>CPO</td><td>20.18 ± 1.47</td><td>≤25 ✓</td></tr> |
| </table> |
| <p>Full reproduction requires 10M steps × 5 seeds × GPU.</p> |
| </div> |
| <div class="card"> |
| <h2>Claim 5: TTS Analysis</h2> |
| <p><span class="tag">Supported</span></p> |
| <table> |
| <tr><th>Task</th><th>Flat-gradient TTS</th><th>Steep-gradient TTS</th></tr> |
| <tr class="ours"><td>Ant</td><td>3.63</td><td>5.25</td></tr> |
| <tr class="ours"><td>Humanoid</td><td>2.33</td><td>6.17</td></tr> |
| <tr class="ours"><td>HalfCheetah</td><td>3.21</td><td>8.23</td></tr> |
| </table> |
| <p>Flat → fast recovery; Steep → cautious recovery. Verified geometrically.</p> |
| </div> |
| </div> |
| <div class="col"> |
| <div class="card"> |
| <h2>Claim 6: Safety Recovery</h2> |
| <p><span class="tag">Supported</span></p> |
| <p>CSPO reduces cost oscillations via immediate $\lambda_{\text{eff}}$ correction.</p> |
| <table> |
| <tr><th>Metric</th><th>CSPO</th><th>PPO-Lag</th><th>APPO</th></tr> |
| <tr><td>TTS</td><td class="ours">4.20</td><td>7.24</td><td>4.38</td></tr> |
| <tr><td>RP</td><td class="ours">1.000</td><td>1.004</td><td>0.994</td></tr> |
| <tr><td>#V</td><td class="ours">116.8</td><td>141.2</td><td>134.6</td></tr> |
| </table> |
| </div> |
| <div class="card"> |
| <h2>Reproducibility</h2> |
| <p><span class="tag">Code Available</span> <span class="tag">Well-Documented</span></p> |
| <p>Official repo: <code>github.com/serval-uni-lu/CSPO</code> (commit <code>962e696</code>)</p> |
| <p>Built on Omnisafe framework. Clean implementation matching paper.</p> |
| <p><strong>Limitations:</strong></p> |
| <ul> |
| <li>Full 9×12 benchmark requires ~100+ GPU-hours</li> |
| <li>No pretrained models provided</li> |
| <li>No raw training curves available</li> |
| </ul> |
| </div> |
| <div class="card"> |
| <h2>Conclusion</h2> |
| <p><strong>5/6 claims supported</strong>, 1 partially verified.</p> |
| <p>Core algorithmic claims (derivation, KKT, TTS mechanism) are mathematically sound and code-verified.</p> |
| <p>Empirical claims require GPU compute for full numerical confirmation.</p> |
| <p>Logbook: <code>huggingface.co/spaces/jomasego/repro-cspo-constraint-sensitive-policy-optimization-for-safe-reinforcement-learning</code></p> |
| </div> |
| </div> |
| </div> |
| <div class="footer"> |
| Reproduction for ICML 2026 Reproducibility Challenge | Paper: arXiv 2606.14415 | OpenReview: 3ySR3TCMRP |
| </div> |
| </div> |
| </body> |
| </html> |
|
|
