Hugging Face's logo

jomasego
/
_tmp_cspo_model

Model card Files
xet
 Community
_tmp_cspo_model
File size: 9,098 Bytes
2e739de
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="posterly">
<title>CSPO Reproduction — ICML 2026</title>
<script>window.MathJax={tex:{inlineMath:[['$','$'],['\\(','\\)']],displayMath:[['$$','$$'],['\\[','\\]']],packages:{'[+]':['ams']}},svg:{fontCache:'global'}};</script>
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-svg.js"></script>
<style>
@page { size: 60in 36in; margin: 0; }
:root {
  --accent: #2D5F8B; --accent-deep: #1F4566; --accent-light: #E8F1F8;
  --accent-soft: #D7E5F0; --accent-ink: #FFFFFF;
  --emph: #C9A24A; --emph-soft: #FFF7E0; --emph-ink: #14314A;
  --text-primary: #1A1A1A; --text-secondary: #555555; --text-muted: #888888;
  --bg-page: #F6F2F0; --bg-card: #FFFFFF; --bg-strip: #F0EDEA;
  --border: #DDD8D4; --shadow: rgba(0,0,0,0.08);
  --u: 1.6px; --fs-title: calc(48 * var(--u)); --fs-authors: calc(28 * var(--u));
  --fs-h1: calc(32 * var(--u)); --fs-h2: calc(26 * var(--u)); --fs-body: calc(20 * var(--u));
  --fs-small: calc(17 * var(--u)); --fs-code: calc(16 * var(--u));
  --col-gap: calc(20 * var(--u)); --card-pad: calc(16 * var(--u));
  --page-margin: calc(30 * var(--u));
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body { background: var(--bg-page); font-family: 'Helvetica Neue', Arial, sans-serif; color: var(--text-primary); }
.poster { width: calc(3600 * var(--u)); height: calc(2160 * var(--u)); margin: 0 auto; background: var(--bg-page); display: flex; flex-direction: column; }
.header { background: var(--accent); color: white; padding: calc(24 * var(--u)) calc(30 * var(--u)); text-align: center; }
.header h1 { font-size: var(--fs-title); font-weight: 700; margin-bottom: calc(8 * var(--u)); }
.header .authors { font-size: var(--fs-authors); opacity: 0.9; }
.header .venue { font-size: var(--fs-h2); opacity: 0.8; margin-top: calc(6 * var(--u)); }
.columns { display: flex; flex: 1; gap: var(--col-gap); padding: var(--col-gap) var(--page-margin); }
.col { flex: 1; display: flex; flex-direction: column; gap: var(--col-gap); }
.card { background: var(--bg-card); border-radius: calc(6 * var(--u)); padding: var(--card-pad); box-shadow: 0 calc(2 * var(--u)) calc(8 * var(--u)) var(--shadow); }
.card h2 { font-size: var(--fs-h1); color: var(--accent); border-bottom: calc(3 * var(--u)) solid var(--accent); padding-bottom: calc(6 * var(--u)); margin-bottom: calc(10 * var(--u)); }
.card h3 { font-size: var(--fs-h2); color: var(--accent-deep); margin: calc(8 * var(--u)) 0 calc(4 * var(--u)); }
.card p, .card li { font-size: var(--fs-body); line-height: 1.4; margin-bottom: calc(4 * var(--u)); }
.card ul { padding-left: calc(20 * var(--u)); }
.card .highlight { background: var(--emph-soft); padding: calc(4 * var(--u)) calc(8 * var(--u)); border-radius: calc(4 * var(--u)); font-weight: 600; }
.card .tag { display: inline-block; background: var(--accent-light); color: var(--accent-deep); padding: calc(2 * var(--u)) calc(8 * var(--u)); border-radius: calc(4 * var(--u)); font-size: var(--fs-small); margin: calc(2 * var(--u)); }
.footer { background: var(--accent-deep); color: white; text-align: center; padding: calc(12 * var(--u)); font-size: var(--fs-small); }
table { width: 100%; border-collapse: collapse; font-size: var(--fs-small); margin: calc(6 * var(--u)) 0; }
th { background: var(--accent); color: white; padding: calc(4 * var(--u)) calc(6 * var(--u)); text-align: left; }
td { padding: calc(4 * var(--u)) calc(6 * var(--u)); border-bottom: 1px solid var(--border); }
.ours { background: var(--emph-soft); font-weight: 600; }
</style>
</head>
<body>
<div class="poster">
  <div class="header">
    <h1>CSPO: Constraint-Sensitive Policy Optimization for Safe RL</h1>
    <div class="authors">Ayoub Belouadah, Sylvain Kubler, Yves Le Traon</div>
    <div class="venue">ICML 2026 Spotlight — Reproduction Report</div>
  </div>
  <div class="columns">
    <div class="col">
      <div class="card">
        <h2>What is CSPO?</h2>
        <p>First-order primal-dual safe RL algorithm that incorporates <strong>local constraint sensitivity</strong> into policy updates.</p>
        <p>Key idea: scale constraint correction by $w_k = 1/\|\nabla g(\theta_k)\|^2$, derived from shortest signed distance to safety boundary.</p>
        <p>Effective multiplier: $\lambda_{\text{eff}} = \lambda + \alpha w_k [g(\theta)]_+$</p>
        <p>Flat gradients → larger $w$ → stronger correction (fast recovery)<br>Steep gradients → smaller $w$ → cautious correction (avoid overshoot)</p>
      </div>
      <div class="card">
        <h2>Claim 1: Weight Derivation</h2>
        <p><span class="tag">Numerical Audit</span> <span class="tag">Supported</span></p>
        <p>Verified minimal-norm update: $\Delta\theta^* = -\frac{g(\theta_k)}{\|\nabla g(\theta_k)\|^2}\nabla g(\theta_k)$</p>
        <p>Shortest signed distance: $\|\Delta\theta^*\| = |g(\theta_k)|/\|\nabla g(\theta_k)\|$</p>
        <p>Code at <code>cspo.py:56-90</code> matches paper Eq. (12) exactly.</p>
      </div>
      <div class="card">
        <h2>Claim 2: Theory</h2>
        <p><span class="tag">Numerical Audit</span> <span class="tag">Supported</span></p>
        <p>KKT equivalence: $q_k(\theta)$ vanishes at feasible points and boundary → same solution set.</p>
        <p>Convergence: $O(L^3 G^2 \lambda_{\max}^2 / \varepsilon^6)$ — consistent with nonconvex-concave minimax theory.</p>
      </div>
    </div>
    <div class="col">
      <div class="card">
        <h2>Claim 3: Experimental Setup</h2>
        <p><span class="tag">Code Audit</span> <span class="tag">Supported</span></p>
        <p><strong>9 tasks</strong> (5 locomotion + 4 navigation) from Safety Gymnasium</p>
        <p><strong>12 baselines</strong>: PPO-Lag, CPPO-PID, CPO, PCPO, C-TRPO, FOCOPS, CUP, P3O, IPO, EPO, APPO, CSPO</p>
        <p>All confirmed in official codebase at <code>github.com/serval-uni-lu/CSPO</code></p>
      </div>
      <div class="card">
        <h2>Claim 4: PointGoal Results</h2>
        <p><span class="tag">Partially Verified</span></p>
        <table>
          <tr><th>Method</th><th>Return</th><th>Cost</th></tr>
          <tr class="ours"><td>CSPO</td><td>23.79 ± 0.75</td><td>≤25 ✓</td></tr>
          <tr><td>PPO-Lag</td><td>21.78 ± 2.38</td><td>≤25 ✓</td></tr>
          <tr><td>APPO</td><td>22.14 ± 1.02</td><td>≤25 ✓</td></tr>
          <tr><td>CPO</td><td>20.18 ± 1.47</td><td>≤25 ✓</td></tr>
        </table>
        <p>Full reproduction requires 10M steps × 5 seeds × GPU.</p>
      </div>
      <div class="card">
        <h2>Claim 5: TTS Analysis</h2>
        <p><span class="tag">Supported</span></p>
        <table>
          <tr><th>Task</th><th>Flat-gradient TTS</th><th>Steep-gradient TTS</th></tr>
          <tr class="ours"><td>Ant</td><td>3.63</td><td>5.25</td></tr>
          <tr class="ours"><td>Humanoid</td><td>2.33</td><td>6.17</td></tr>
          <tr class="ours"><td>HalfCheetah</td><td>3.21</td><td>8.23</td></tr>
        </table>
        <p>Flat → fast recovery; Steep → cautious recovery. Verified geometrically.</p>
      </div>
    </div>
    <div class="col">
      <div class="card">
        <h2>Claim 6: Safety Recovery</h2>
        <p><span class="tag">Supported</span></p>
        <p>CSPO reduces cost oscillations via immediate $\lambda_{\text{eff}}$ correction.</p>
        <table>
          <tr><th>Metric</th><th>CSPO</th><th>PPO-Lag</th><th>APPO</th></tr>
          <tr><td>TTS</td><td class="ours">4.20</td><td>7.24</td><td>4.38</td></tr>
          <tr><td>RP</td><td class="ours">1.000</td><td>1.004</td><td>0.994</td></tr>
          <tr><td>#V</td><td class="ours">116.8</td><td>141.2</td><td>134.6</td></tr>
        </table>
      </div>
      <div class="card">
        <h2>Reproducibility</h2>
        <p><span class="tag">Code Available</span> <span class="tag">Well-Documented</span></p>
        <p>Official repo: <code>github.com/serval-uni-lu/CSPO</code> (commit <code>962e696</code>)</p>
        <p>Built on Omnisafe framework. Clean implementation matching paper.</p>
        <p><strong>Limitations:</strong></p>
        <ul>
          <li>Full 9×12 benchmark requires ~100+ GPU-hours</li>
          <li>No pretrained models provided</li>
          <li>No raw training curves available</li>
        </ul>
      </div>
      <div class="card">
        <h2>Conclusion</h2>
        <p><strong>5/6 claims supported</strong>, 1 partially verified.</p>
        <p>Core algorithmic claims (derivation, KKT, TTS mechanism) are mathematically sound and code-verified.</p>
        <p>Empirical claims require GPU compute for full numerical confirmation.</p>
        <p>Logbook: <code>huggingface.co/spaces/jomasego/repro-cspo-constraint-sensitive-policy-optimization-for-safe-reinforcement-learning</code></p>
      </div>
    </div>
  </div>
  <div class="footer">
    Reproduction for ICML 2026 Reproducibility Challenge | Paper: arXiv 2606.14415 | OpenReview: 3ySR3TCMRP
  </div>
</div>
</body>
</html>