| --- |
| license: mit |
| tags: |
| - reinforcement-learning |
| - trpo |
| - policy-optimization |
| - mujoco |
| - cartpole |
| library_name: entropy-trpo |
| --- |
| |
| # Entropy-TRPO Model Weights |
|
|
| PyTorch checkpoints for the comparative study in *A Review of Entropy-Based Extensions to Trust Region Policy Optimization*. |
|
|
| ## Repository layout |
|
|
| Each checkpoint directory contains: |
|
|
| | File | Description | |
| |------|-------------| |
| | `policy.pt` | Policy network state dict | |
| | `value.pt` | Value network state dict | |
| | `config.json` | Training hyperparameters | |
| | `metadata.json` | Paper source, variant flags, final metrics | |
|
|
| Hub path: `{env_id}/{variant}/latest/` (e.g. `CartPole-v1/entrpo/latest/`). |
|
|
| The repo **README** is updated automatically during training with a **Training progress** table |
| (epoch `n/N`, eval return, best return, KL) from `results/summary.md`, plus a JSON index of |
| available checkpoints. |
|
|
| ## Notation |
|
|
| - $\rho_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{\theta_{\text{old}}}(a_t|s_t)$, GAE advantages $\hat{A}_t$, trust-region radius $\delta$ |
| - **$\alpha$** — Roostaie advantage entropy; Xu ERO objective entropy (distinct roles, same symbol in each paper's row) |
| - **$\beta$** — Xu ERC constraint coefficient (Xu Eq. 49) |
| - **$c_{\mathrm{ent}}$** — PPO entropy bonus (Schulman et al., 2017; config field `entropy_coef`) |
|
|
| ## Variant definitions |
|
|
| | Key | Paper name | Surrogate / constraint | |
| | --- | --- | --- | |
| | `trpo` | TRPO | $\mathbb{E}[\rho_t \hat{A}_t]$; $\bar D_{\mathrm{KL}} \le \delta$ | |
| | `entrpo_entropy` | EnTRPO-Entropy | $\mathbb{E}[\rho_t \tilde{A}_t]$, $\tilde{A}_t=\hat{A}_t+\alpha\,\mathcal{H}(\pi_{\theta_{\text{old}}}(\cdot\|s_t))$ (fixed during step); $\bar D_{\mathrm{KL}} \le \delta$ | |
| | `ero_trpo` | ERO-TRPO | $\mathbb{E}[\rho_t \hat{A}_t]+\alpha\,\mathbb{E}[\mathcal{H}(\pi_\theta)]$; $\bar D_{\mathrm{KL}} \le \delta$ | |
| | `erc_trpo` | ERC-TRPO | $\mathbb{E}[\rho_t \hat{A}_t]$; $\bar D_{\mathrm{KL}} \le \delta+\beta\,\mathbb{E}[\mathcal{H}(\pi_\theta)]$ (Xu Eq. 49) | |
| | `entrpo_buffer` | EnTRPO-Buffer | $\mathbb{E}[\rho_t \hat{A}_t]$ with Roostaie on-policy replay | |
| | `entrpo` | EnTRPO | $\mathbb{E}[\rho_t \tilde{A}_t]$ + Roostaie buffer | |
| | `ppo` | PPO | $\mathbb{E}[\min(\rho_t \hat{A}_t,\mathrm{clip}(\rho_t)\hat{A}_t)]+c_{\mathrm{ent}}\mathbb{E}[\mathcal{H}(\pi_\theta)]$ | |
|
|
| $\mathcal{H}$ in EnTRPO rows is evaluated at the behavior policy $\pi_{\theta_{\text{old}}}$; in ERO/ERC/PPO rows at the candidate policy $\pi_\theta$. |
| |
| **ERO-TRPO implementation:** Xu Table 1 — $H\mathbf{d}=\mathbf{g}+\alpha\mathbf{h}$, step size $\eta\le\sqrt{2\delta/(\mathbf{g}+\alpha\mathbf{h})^\top H(\mathbf{g}+\alpha\mathbf{h})}$, Xu line search ($\eta_0=0.5$, strict objective improvement, hard $\bar{D}_{\mathrm{KL}}\le\delta$). |
| |
| **ERC-TRPO implementation:** Xu Table 1 — two CG solves ($\mathbf{u},\mathbf{v}$), $\eta\mathbf{u}+\beta\mathbf{v}$, Eq.~(49) acceptance $\bar{D}_{\mathrm{KL}}\le\delta+\beta\,\mathbb{E}[\mathcal{H}]$, same Xu line search. **Random ERC** shares the ERC step; only the acceptance bonus is randomized. |
|
|
| Older Hub folders (`trpo_entropy`, `trpo_buffer`, …) remain valid; training resumes from them automatically. |
|
|
| ## Environments |
|
|
| | Environment | Obs / action | Training budget | Hyperparameter source | |
| |-------------|--------------|-----------------|------------------------| |
| | CartPole-v1 | Gymnasium classic control | 1M steps | Roostaie + Xu Tables 4--5 | |
| | Humanoid-v5 | 348 / 17 | $10^6$ steps | PPO/baselines backbone; Xu ERO/ERC proxied from Walker2d | |
| | HumanoidStandup-v5 | 348 / 17 | $10^6$ steps | Same backbone; Xu ERO/ERC proxied from BipedalWalker | |
|
|
| See [HYPERPARAMETERS.md](HYPERPARAMETERS.md) for per-field provenance and [paper/results/annex_hyperparameters.tex](../paper/results/annex_hyperparameters.tex) for tables. |
|
|
| ## Variants and paper sources |
|
|
| | Variant | Paper | |
| |---------|-------| |
| | `trpo` | Schulman et al. (2015), *Trust Region Policy Optimization*, ICML | |
| | `entrpo_entropy` | Roostaie & Ebadzadeh (2021), *EnTRPO* — entropy-in-advantage ablation | |
| | `entrpo_buffer` | Roostaie & Ebadzadeh (2021), *EnTRPO* — replay-buffer ablation | |
| | `entrpo` | Roostaie & Ebadzadeh (2021), *EnTRPO* — full method | |
| | `ero_trpo` | Xu et al. (2024), *ERO-TRPO* | |
| | `erc_trpo` | Xu et al. (2024), *ERC-TRPO* | |
| | `ppo` | Schulman et al. (2017), *Proximal Policy Optimization* | |
|
|
| See `metadata.json` in each folder for full author names and URLs. |
|
|
| ## Usage |
|
|
| Training and evaluation code: [GitHub — entropy-trpo](https://github.com/pre63/entropy-trpo) (update URL when published). |
|
|
| ```bash |
| git clone https://github.com/pre63/entropy-trpo.git |
| cd entropy-trpo |
| make setup # install deps + create .env |
| # edit .env with HF_TOKEN and HF_REPO_ID |
| make download-weights |
| make eval-checkpoints |
| ``` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{entropytrporeview2026, |
| title = {A Review of Entropy-Based Extensions to Trust Region Policy Optimization}, |
| author = {Green, Simon}, |
| journal = {IEEE Transactions}, |
| year = {2026} |
| } |
| ``` |
|
|
| ```bibtex |
| @article{roostaie2021entrpo, |
| title = {EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization}, |
| author = {Roostaie, Sahar and Ebadzadeh, Mohammad Mehdi}, |
| journal = {arXiv:2110.13373}, |
| year = {2021} |
| } |
| ``` |
|
|
| ```bibtex |
| @article{xu2024trpo, |
| title = {Trust region policy optimization via entropy regularization for {Kullback--Leibler} divergence constraint}, |
| author = {Xu, Haotian and Xuan, Junyu and Zhang, Guangquan and Lu, Jie}, |
| journal = {Neurocomputing}, |
| volume = {589}, |
| pages = {127716}, |
| year = {2024} |
| } |
| ``` |
|
|
|
|
| ## Training progress |
|
|
| _Last updated: 2026-07-05 19:43:57 UTC_ |
|
|
| - **Xu window** — mean eval return over the last **W** epochs (Xu Table 6 protocol). |
| - **Full run** — mean eval return over all epochs. |
|
|
| - **Device:** `cpu` |
| - **Config:** `configs/e4/e4_entrpo_substitute_cartpole_v5.yaml` |
| - **Jobs complete:** 72/72 |
| - **Running:** 0 |
|
|
| ## CartPole-v1 (1M benchmark) |
|
|
| | Variant | Status | Epoch | Timesteps | Eval (final) | Best | Xu window | Full run | W | KL | |
| | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | |
| | TRPO (s0) | done | **1953/1953** | 999,936 | 325.5 ± 62.6 | 500.0 | 284.9 | 382.5 | 326 | 0.0032 | |
| | TRPO (s1) | done | **1953/1953** | 999,936 | 56.1 ± 62.4 | 500.0 | 438.6 | 399.8 | 326 | 0.0032 | |
| | TRPO (s2) | done | **1953/1953** | 999,936 | 290.1 ± 184.0 | 500.0 | 459.8 | 378.0 | 326 | 0.0021 | |
| | EnTRPO-Entropy (s0) | done | **200/200** | 1,000,000 | 497.9 ± 6.3 | 500.0 | 365.4 | 372.3 | 33 | 0.0035 | |
| | EnTRPO-Entropy (s1) | done | **200/200** | 1,000,000 | 420.3 ± 37.0 | 500.0 | 448.7 | 434.1 | 33 | 0.0059 | |
| | EnTRPO-Entropy (s2) | done | **200/200** | 1,000,000 | 494.9 ± 12.1 | 500.0 | 425.4 | 409.5 | 33 | 0.0050 | |
| | ERO-TRPO (s0) | done | **2000/2000** | 1,000,000 | 182.8 ± 66.6 | 232.8 | 168.1 | 86.8 | 333 | 0.0000 | |
| | ERO-TRPO (s1) | done | **2000/2000** | 1,000,000 | 153.6 ± 68.2 | 280.2 | 198.5 | 103.7 | 333 | 0.0000 | |
| | ERO-TRPO (s2) | done | **2000/2000** | 1,000,000 | 175.8 ± 110.2 | 254.6 | 157.7 | 84.4 | 333 | -0.0000 | |
| | ERC-TRPO (s0) | done | **2000/2000** | 1,000,000 | 229.2 ± 81.0 | 258.3 | 186.2 | 97.4 | 333 | -0.0000 | |
| | ERC-TRPO (s1) | done | **2000/2000** | 1,000,000 | 22.0 ± 9.3 | 43.7 | 25.6 | 25.7 | 333 | -0.0000 | |
| | ERC-TRPO (s2) | done | **2000/2000** | 1,000,000 | 20.3 ± 5.2 | 41.7 | 21.7 | 21.8 | 333 | 0.0000 | |
| | EnTRPO-Buffer (s0) | done | **200/200** | 1,000,000 | 170.3 ± 41.3 | 199.7 | 153.4 | 157.5 | 33 | 0.0091 | |
| | EnTRPO-Buffer (s1) | done | **200/200** | 1,000,000 | 309.2 ± 62.2 | 500.0 | 330.6 | 320.8 | 33 | 0.0064 | |
| | EnTRPO-Buffer (s2) | done | **200/200** | 1,000,000 | 205.0 ± 8.4 | 485.7 | 224.4 | 251.9 | 33 | 0.0070 | |
| | EnTRPO (s0) | done | **200/200** | 1,000,000 | 101.7 ± 6.1 | 489.2 | 106.1 | 129.7 | 33 | 0.0089 | |
| | EnTRPO (s1) | done | **200/200** | 1,000,000 | 99.0 ± 22.0 | 500.0 | 123.0 | 213.5 | 33 | 0.0076 | |
| | EnTRPO (s2) | done | **200/200** | 1,000,000 | 168.9 ± 9.4 | 500.0 | 399.5 | 287.1 | 33 | 0.0067 | |
| | PPO (s0) | done | **31250/31250** | 1,000,000 | 218.9 ± 84.4 | 340.9 | 207.5 | 150.0 | 1000 | 4.6836 | |
| | PPO (s1) | done | **31250/31250** | 1,000,000 | 500.0 ± 0.0 | 500.0 | 499.5 | 265.0 | 1000 | 0.0824 | |
| | PPO (s2) | done | **31250/31250** | 1,000,000 | 134.3 ± 51.4 | 500.0 | 299.0 | 275.5 | 1000 | -0.0170 | |
|
|
| ## Humanoid-v5 (1M benchmark) |
|
|
| | Variant | Status | Epoch | Timesteps | Eval (final) | Best | Xu window | Full run | W | KL | |
| | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | |
| | TRPO (s0) | done | **976/976** | 1,000,448 | 310.7 ± 62.0 | 385.1 | 303.1 | 253.8 | 163 | 0.0060 | |
| | TRPO (s1) | done | **976/976** | 999,424 | 321.3 ± 88.9 | 379.0 | 309.6 | 257.8 | 163 | 0.0043 | |
| | TRPO (s2) | done | **976/976** | 999,424 | 309.2 ± 105.4 | 365.1 | 304.2 | 251.8 | 163 | -0.0009 | |
| | EnTRPO-Entropy (s0) | done | **488/488** | 999,424 | 239.5 ± 64.2 | 332.1 | 271.8 | 194.6 | 81 | -0.0000 | |
| | EnTRPO-Entropy (s1) | done | **488/488** | 999,424 | 276.8 ± 63.1 | 333.5 | 268.0 | 183.7 | 81 | 0.0042 | |
| | EnTRPO-Entropy (s2) | done | **488/488** | 999,424 | 249.7 ± 82.2 | 328.3 | 268.7 | 173.1 | 81 | 0.0000 | |
| | ERO-TRPO (s0) | done | **976/976** | 999,424 | 280.8 ± 101.4 | 387.9 | 310.7 | 262.6 | 163 | -0.0012 | |
| | ERO-TRPO (s1) | done | **976/976** | 999,424 | 330.0 ± 83.0 | 369.2 | 304.4 | 253.4 | 163 | 0.0046 | |
| | ERO-TRPO (s2) | done | **976/976** | 999,424 | 316.5 ± 60.8 | 420.3 | 309.0 | 245.6 | 163 | 0.0092 | |
| | ERC-TRPO (s0) | done | **976/976** | 999,424 | 277.9 ± 89.4 | 383.3 | 302.5 | 257.0 | 163 | 0.8317 | |
| | ERC-TRPO (s1) | done | **976/976** | 999,424 | 281.9 ± 85.2 | 399.6 | 312.0 | 258.6 | 163 | 0.0051 | |
| | ERC-TRPO (s2) | done | **976/976** | 999,424 | 254.2 ± 46.3 | 392.6 | 311.4 | 257.9 | 163 | -0.0025 | |
| | EnTRPO-Buffer (s0) | done | **488/488** | 999,424 | 233.2 ± 46.0 | 314.2 | 262.7 | 181.5 | 81 | 0.0000 | |
| | EnTRPO-Buffer (s1) | done | **488/488** | 999,424 | 293.6 ± 75.7 | 308.4 | 257.5 | 178.3 | 81 | 0.0000 | |
| | EnTRPO-Buffer (s2) | done | **488/488** | 999,424 | 263.0 ± 78.6 | 310.7 | 250.2 | 161.5 | 81 | 0.0082 | |
| | EnTRPO (s0) | done | **488/488** | 999,424 | 276.8 ± 52.6 | 352.6 | 256.4 | 186.2 | 81 | 0.0000 | |
| | EnTRPO (s1) | done | **488/488** | 999,424 | 227.8 ± 31.5 | 334.7 | 256.3 | 174.8 | 81 | 0.0089 | |
| | EnTRPO (s2) | done | **488/488** | 999,424 | 286.4 ± 68.8 | 323.3 | 255.7 | 163.5 | 81 | 0.0093 | |
| | PPO (s0) | done | **1953/1953** | 999,936 | 286.4 ± 70.3 | 411.1 | 305.6 | 291.3 | 326 | 0.0171 | |
| | PPO (s1) | done | **1953/1953** | 999,936 | 324.4 ± 72.3 | 403.0 | 309.6 | 288.9 | 326 | -0.0023 | |
| | PPO (s2) | done | **1953/1953** | 999,936 | 291.7 ± 65.1 | 409.1 | 307.0 | 290.9 | 326 | -0.0067 | |
|
|
| ## HumanoidStandup-v5 (1M benchmark) |
|
|
| | Variant | Status | Epoch | Timesteps | Eval (final) | Best | Xu window | Full run | W | KL | |
| | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | |
| | TRPO (s0) | done | **976/976** | 999,424 | 65564.3 ± 13149.1 | 75225.9 | 65080.2 | 55528.2 | 163 | 0.0086 | |
| | TRPO (s1) | done | **976/976** | 999,424 | 85331.8 ± 15316.0 | 93973.1 | 83992.1 | 69408.5 | 163 | 0.0013 | |
| | TRPO (s2) | done | **976/976** | 999,424 | 71929.0 ± 11498.5 | 81636.4 | 73421.4 | 61150.4 | 163 | 0.0018 | |
| | EnTRPO-Entropy (s0) | done | **488/488** | 999,424 | 64478.0 ± 14627.4 | 73943.7 | 61896.3 | 50554.0 | 81 | 0.0022 | |
| | EnTRPO-Entropy (s1) | done | **488/488** | 999,424 | 79008.0 ± 7394.7 | 83421.9 | 75278.5 | 59057.8 | 81 | 0.0088 | |
| | EnTRPO-Entropy (s2) | done | **488/488** | 999,424 | 74557.1 ± 10527.1 | 75892.7 | 66491.7 | 53834.4 | 81 | 0.0032 | |
| | ERO-TRPO (s0) | done | **976/976** | 999,424 | 45868.3 ± 2401.2 | 48375.1 | 45105.9 | 40929.5 | 163 | 0.0005 | |
| | ERO-TRPO (s1) | done | **976/976** | 999,424 | 52085.0 ± 6858.4 | 54183.2 | 50235.2 | 45674.6 | 163 | -0.0008 | |
| | ERO-TRPO (s2) | done | **976/976** | 999,424 | 54828.2 ± 5440.4 | 57270.1 | 53365.4 | 45660.3 | 163 | 0.0006 | |
| | ERC-TRPO (s0) | done | **976/976** | 999,424 | 44648.7 ± 1975.6 | 47619.2 | 43774.6 | 40086.5 | 163 | -0.0007 | |
| | ERC-TRPO (s1) | done | **976/976** | 999,424 | 53205.4 ± 4787.5 | 56941.8 | 52720.3 | 46702.6 | 163 | -0.0042 | |
| | ERC-TRPO (s2) | done | **976/976** | 999,424 | 51636.4 ± 5581.1 | 56086.9 | 51601.2 | 44689.4 | 163 | -0.0001 | |
| | EnTRPO-Buffer (s0) | done | **488/488** | 999,424 | 36194.0 ± 1992.8 | 40890.8 | 36737.9 | 37064.1 | 81 | 0.0096 | |
| | EnTRPO-Buffer (s1) | done | **488/488** | 999,424 | 42289.7 ± 2855.5 | 46657.7 | 40983.5 | 41893.8 | 81 | 0.0078 | |
| | EnTRPO-Buffer (s2) | done | **488/488** | 999,424 | 37238.8 ± 1760.8 | 40826.2 | 37388.7 | 37541.4 | 81 | 0.0058 | |
| | EnTRPO (s0) | done | **488/488** | 999,424 | 37475.7 ± 2124.3 | 40184.2 | 38075.0 | 37742.2 | 81 | 0.0000 | |
| | EnTRPO (s1) | done | **488/488** | 999,424 | 42126.8 ± 1923.1 | 46326.0 | 41585.1 | 41616.1 | 81 | 0.0000 | |
| | EnTRPO (s2) | done | **488/488** | 999,424 | 36979.4 ± 2180.2 | 41487.5 | 38496.5 | 38440.7 | 81 | 0.0000 | |
| | PPO (s0) | done | **1953/1953** | 999,936 | 59353.4 ± 8311.7 | 86651.2 | 64306.1 | 67688.9 | 326 | 0.1466 | |
| | PPO (s1) | done | **1953/1953** | 999,936 | 71828.6 ± 12149.2 | 86146.6 | 70013.6 | 69247.9 | 326 | 0.2344 | |
| | PPO (s2) | done | **1953/1953** | 999,936 | 73034.0 ± 5080.3 | 83874.9 | 72769.7 | 69345.9 | 326 | -0.0429 | |
|
|
| ## CartPole-v5 (E4_CARTPOLE_V5 benchmark) |
|
|
| | Variant | Status | Epoch | Timesteps | Eval (final) | Best | Xu window | Full run | W | KL | |
| | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | |
| | EnTRPO-Entropy (s0) | done | **10/10** | 50,000 | 255.7 ± 71.7 | 263.5 | 254.9 | 152.9 | 2 | 0.0030 | |
| | EnTRPO-Entropy (s1) | done | **10/10** | 50,000 | 326.2 ± 98.8 | 337.1 | 305.4 | 188.7 | 2 | 0.0071 | |
| | EnTRPO-Entropy (s2) | done | **10/10** | 50,000 | 363.0 ± 105.1 | 393.6 | 378.3 | 181.3 | 2 | 0.0062 | |
| | Binomial EnTRPO (s0) | done | **10/10** | 50,000 | 323.2 ± 101.5 | 323.2 | 288.6 | 159.7 | 2 | 0.0093 | |
| | Binomial EnTRPO (s1) | done | **10/10** | 50,000 | 326.2 ± 98.8 | 337.1 | 305.4 | 188.7 | 2 | 0.0071 | |
| | Binomial EnTRPO (s2) | done | **10/10** | 50,000 | 392.6 ± 101.7 | 393.6 | 393.1 | 184.2 | 2 | 0.0061 | |
| | Constant EnTRPO (s0) | done | 9/10 | 50,000 | 382.3 ± 74.8 | 385.0 | 383.6 | 209.4 | 2 | 0.0054 | |
| | Constant EnTRPO (s1) | done | **10/10** | 50,000 | 268.7 ± 103.6 | 309.0 | 266.8 | 175.5 | 2 | 0.0054 | |
| | Constant EnTRPO (s2) | done | **10/10** | 50,000 | 392.6 ± 101.7 | 393.6 | 393.1 | 184.2 | 2 | 0.0061 | |
|
|
|
|
| ## Available checkpoints |
|
|
| ```json |
| { |
| "CartPole-v1": [ |
| "entrpo", |
| "entrpo_buffer", |
| "entrpo_entropy", |
| "ppo", |
| "trpo", |
| "trpo_buffer", |
| "trpo_entropy" |
| ] |
| } |
| ``` |
|
|
