File size: 1,916 Bytes
2e739de
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
## Numerical audit of Claim 1

**Claim**: CSPO incorporates local constraint sensitivity into policy updates by scaling the constraint correction using $w_k = 1/\|\nabla g(\theta_k)\|^2$, derived from the shortest signed distance $g(\theta)/\|\nabla g(\theta)\|$ to the linearized safety boundary (Section 3).

**Verification**: We performed an independent numerical audit of the mathematical derivation:

1. **Minimal-norm update** (Eq. 9-10): Verified that $\Delta\theta^* = -g(\theta_k)/\|\nabla g(\theta_k)\|^2 \cdot \nabla g(\theta_k)$ correctly solves $\min \frac{1}{2}\|\Delta\theta\|^2$ s.t. $g(\theta_k) + \nabla g(\theta_k)^\top \Delta\theta = 0$. Tested across dimensions $d \in \{2,5,10,50\}$ with random gradients — all passed.

2. **Shortest signed distance** (Eq. 10): Verified $\|\Delta\theta^*\| = |g(\theta_k)|/\|\nabla g(\theta_k)\|$. Tested across $d \in \{2,5,10\}$ — all passed.

3. **Weight formula** (Eq. 12): Verified $w_k = 1/\|\nabla g(\theta_k)\|^2$ correctly scales the update. Tested across $d \in \{2,5,10,50,100\}$ — all passed.

4. **Code implementation**: Confirmed the CSPO implementation at https://github.com/serval-uni-lu/CSPO/tree/962e696 matches the paper:
   - `_compute_w()` computes $w = 1/(\|\nabla g\|^2 + \epsilon)$ (line 81 of cspo.py)
   - `_loss_pi_cost()` computes $\lambda_{\text{eff}} = \lambda + \alpha \cdot w \cdot [g(\theta)]_+$ (line 114-119)
   - EMA smoothing is applied with $\beta = 0.9$ (line 83)

5. **Geometric intuition**: Confirmed flat gradients ($\|\nabla g\| = 0.32$) produce $w = 10.0$ (strong correction), while steep gradients ($\|\nabla g\| = 31.62$) produce $w = 0.001$ (cautious correction).

**Result**: Claim 1 is **supported** — the derivation is mathematically sound and the implementation faithfully follows the paper.

**Code**: [verify_claim1.py](verify_claim1.py)
**Repo**: https://github.com/serval-uni-lu/CSPO/tree/962e696