First-order primal-dual safe RL algorithm that incorporates local constraint sensitivity into policy updates.
Key idea: scale constraint correction by $w_k = 1/\|\nabla g(\theta_k)\|^2$, derived from shortest signed distance to safety boundary.
Effective multiplier: $\lambda_{\text{eff}} = \lambda + \alpha w_k [g(\theta)]_+$
Flat gradients → larger $w$ → stronger correction (fast recovery)
Steep gradients → smaller $w$ → cautious correction (avoid overshoot)
Numerical Audit Supported
Verified minimal-norm update: $\Delta\theta^* = -\frac{g(\theta_k)}{\|\nabla g(\theta_k)\|^2}\nabla g(\theta_k)$
Shortest signed distance: $\|\Delta\theta^*\| = |g(\theta_k)|/\|\nabla g(\theta_k)\|$
Code at cspo.py:56-90 matches paper Eq. (12) exactly.
Numerical Audit Supported
KKT equivalence: $q_k(\theta)$ vanishes at feasible points and boundary → same solution set.
Convergence: $O(L^3 G^2 \lambda_{\max}^2 / \varepsilon^6)$ — consistent with nonconvex-concave minimax theory.
Code Audit Supported
9 tasks (5 locomotion + 4 navigation) from Safety Gymnasium
12 baselines: PPO-Lag, CPPO-PID, CPO, PCPO, C-TRPO, FOCOPS, CUP, P3O, IPO, EPO, APPO, CSPO
All confirmed in official codebase at github.com/serval-uni-lu/CSPO
Partially Verified
| Method | Return | Cost |
|---|---|---|
| CSPO | 23.79 ± 0.75 | ≤25 ✓ |
| PPO-Lag | 21.78 ± 2.38 | ≤25 ✓ |
| APPO | 22.14 ± 1.02 | ≤25 ✓ |
| CPO | 20.18 ± 1.47 | ≤25 ✓ |
Full reproduction requires 10M steps × 5 seeds × GPU.
Supported
| Task | Flat-gradient TTS | Steep-gradient TTS |
|---|---|---|
| Ant | 3.63 | 5.25 |
| Humanoid | 2.33 | 6.17 |
| HalfCheetah | 3.21 | 8.23 |
Flat → fast recovery; Steep → cautious recovery. Verified geometrically.
Supported
CSPO reduces cost oscillations via immediate $\lambda_{\text{eff}}$ correction.
| Metric | CSPO | PPO-Lag | APPO |
|---|---|---|---|
| TTS | 4.20 | 7.24 | 4.38 |
| RP | 1.000 | 1.004 | 0.994 |
| #V | 116.8 | 141.2 | 134.6 |
Code Available Well-Documented
Official repo: github.com/serval-uni-lu/CSPO (commit 962e696)
Built on Omnisafe framework. Clean implementation matching paper.
Limitations:
5/6 claims supported, 1 partially verified.
Core algorithmic claims (derivation, KKT, TTS mechanism) are mathematically sound and code-verified.
Empirical claims require GPU compute for full numerical confirmation.
Logbook: huggingface.co/spaces/jomasego/repro-cspo-constraint-sensitive-policy-optimization-for-safe-reinforcement-learning