ModelVerifiedExternal

Entropy Trpo Weights

PyTorch checkpoints for the comparative study in A Review of Entropy-Based Extensions to Trust Region Policy Optimization . Repository layout

Verified source

Sourcehuggingface.co/pre63/entropy-trpo-weights

Kindreinforcement-learningVersionvea67f998606a19e47733abc1cf25a48889da37b9LicensemitPublisher@pre63Cgrade

Model source

Kind: reinforcement-learning
Version: vea67f998606a19e47733abc1cf25a48889da37b9
License: mit
Publisher: @pre63
Source: Hugging Face

Source excerpt

--- license: mit tags: - reinforcement-learning - trpo - policy-optimization - mujoco - cartpole library_name: entropy-trpo --- # Entropy-TRPO Model Weights PyTorch checkpoints for the comparative study in *A Review of Entropy-Based Extensions to Trust Region Policy Optimization*. ## Repository layout Each checkpoint directory contains: | File | Description | |------|-------------| | `policy.pt` | Policy network state dict | | `value.pt` | Value network state dict | | `config.json` | Training hyperparameters | | `metadata.json` | Paper source, variant flags, final metrics | Hub path: `{env_id}/{variant}/latest/` (e.g. `CartPole-v1/entrpo/latest/`). The repo **README** is updated automatically during training with a **Training progress** table (epoch `n/N`, eval return, best return, KL) from `results/summary.md`, plus a JSON index of available checkpoints. ## Notation - $\rho_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{\theta_{\text{old}}}(a_t|s_t)$, GAE advantages $\hat{A}_t$, trust-region radius $\delta$ - **$\alpha$** — Roostaie advantage entropy; Xu ERO objective entropy (distinct roles, same symbol in each paper's row) - **$\beta$** — Xu ERC constraint coefficient (Xu Eq. 49) - **$c_{\mathrm{ent}}$** — PPO entropy bonus (Schulman et al., 2017; config field `entropy_coef`) ## Variant definitions | Key | Paper name | Surrogate / constraint | | --- | --- | --- | | `trpo` | TRPO | $\mathbb{E}[\rho_t \hat{A}_t]$; $\bar D_{\mathrm{KL}} \le \delta$ | | `entrpo_entropy` | EnTRPO-Entropy | $\mathbb{E}[\rho_t \tilde{A}_t]$, $\tilde{A}_t=\hat{A}_t+\alpha\,\mathcal{H}(\pi_{\theta_{\text{old}}}(\cdot\|s_t))$ (fixed during step); $\bar D_{\mathrm{KL}} \le \delta$ | | `ero_trpo` | ERO-TRPO | $\mathbb{E}[\rho_t \hat{A}_t]+\alpha\,\mathbb{E}[\mathcal{H}(\pi_\theta)]$; $\bar D_{\mathrm{KL}} \le \delta$ | | `erc_trpo` | ERC-TRPO | $\mathbb{E}[\rho_t \hat{A}_t]$; $\bar D_{\mathrm{KL}} \le \delta+\beta\,\mathbb{E}[\mathcal{H}(\pi_\theta)]$ (Xu Eq. 49) | | `entrpo_buffer` | EnTRPO-Buffer | $\mathbb{E}[\rho_t \hat{A}_t]$ with Roostaie on-policy replay | | `entrpo` | EnTRPO | $\mathbb{E}[\rho_t \tilde{A}_t]$ + Roostaie buffer | | `ppo` | PPO | $\mathbb{E}[\min(\rho_t \hat{A}_t,\mathrm{clip}(\rho_t)\hat{A}_t)]+c_{\mathrm{ent}}\mathbb{E}[\mathcal{H}(\pi_\theta)]$ | $\mathcal{H}$ in EnTRPO rows is evaluated at the behavior policy $\pi_{\theta_{\text{old}}}$; in ERO/ERC/PPO rows at the candidate policy $\pi_\theta$. **ERO-TRPO implementation:** Xu Table 1 — $H\mathbf{d}=\mathbf{g}+\alpha\mathbf{h}$, step size $\eta\le\sqrt{2\delta/(\mathbf{g}+\alpha\mathbf{h})^\top H(\mathbf{g}+\alpha\mathbf{h})}$, Xu line search ($\eta_0=0.5$, strict objective improvement, hard $\bar{D}_{\mathrm{KL}}\le\delta$). **ERC-TRPO implementation:** Xu Table 1 — two CG solves ($\mathbf{u},\mathbf{v}$), $\eta\mathbf{u}+\beta\mathbf{v}$, Eq.~(49) acceptance $\bar{D}_{\mathrm{KL}}\le\delta+\beta\,\mathbb{E}[\mathcal{H}]$, same Xu line search. **Random ERC** shares the ERC step; only the acceptance bonus is randomized. Older Hub folders (`trpo_entropy`, `trpo_buffer`, ...) remain valid; training resumes from them automatically. ## Environments | Environment | Obs / action | Training budget | Hyperparameter source | |-------------|--------------|-----------------|------------------------| | CartPole-v1 | Gymnasium classic control | 1M steps | Roostaie + Xu Tables 4--5 | | Humanoid-v5 | 348 / 17 | $10^6$ steps | PPO/baselines backbone; Xu ERO/ERC proxied from Walker2d | | HumanoidStandup-v5 | 348 / 17 | $10^6$ steps | Same backbone; Xu ERO/ERC proxied from BipedalWalker | See [HYPERPARAMETERS.md](HYPERPARAMETERS.md) for per-field provenance and [paper/results/annex_hyperparameters.tex](../paper/results/annex_hyperparameters.tex) for tables. ## Variants and paper sources | Variant | Paper | |---------|-------| | `trpo` | Schulman et al. (2015), *Trust Region Policy Optimization*, ICML | | `entrpo_entropy` | Roostaie & Ebadzadeh (2021), *EnTRPO* — entropy-in-advantage ablation | | `entrpo_buffer` | Roostaie & Ebadzadeh (2021), *EnTRPO* — replay-buffer ablation | | `entrpo` | Roostaie & Ebadzadeh (2021), *EnTRPO* — full method | | `ero_trpo` | Xu et al. (2024), *ERO-TRPO* | | `erc_trpo` | Xu et al. (2024), *ERC-TRPO* | | `ppo` | Schulman et al. (2017), *Proximal Policy Optimization* | See `metadata.json` in each folder for full author names and URLs. ## Usage Training and evaluation code: [GitHub — entropy-trpo](https://github.com/pre63/entropy-trpo) (update URL when published). ```bash git clone https://github.com/pre63/entropy-trpo.git cd entropy-trpo make setup # install deps + create .env # edit .env with HF_TOKEN and HF_REPO_ID make download-weights make eval-checkpoints ``` ## Citation ```bibtex @article{entropytrporeview2026, title = {A Review of Entropy-Based Extensions to Trust Region Policy Optimization}, author = {Green, Simon}, journal = {IEEE Transactions}, year = {2026} } ``` ```bibtex @article{roostaie2021entrpo, title = {EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization}, author = {Roostaie, Sahar and Ebadzadeh, Mohammad Mehdi}, journal = {arXiv:2110.13373}, year = {2021} } ``` ```bibtex @article{xu2024trpo, title = {Trust region policy optimization via entropy regularization for {Kullback--Leibler} divergence constraint}, author = {Xu, Haotian and Xuan, Junyu and Zhang, Guangquan and Lu, Jie}, journal = {Neurocomputing}, volume = {589}, pages = {127716}, year = {2024} } ``` ## Training progress _Last updated: 2026-07-05 19:43:57 UTC_ - **Xu window** — mean eval return over the last **W** epochs (Xu Table 6 protocol). - **Full run** — mean eval return over all epochs. - **Device:** `cpu` - **Config:** `configs/e4/e4_entrpo_substitute_cartpole_v5.yaml` - **Jobs complete:** 72/72 - **Running:** 0 ## CartPole-v1 (1M benchmark) | Variant | Status | Epoch | Timesteps | Eval (final) | Best | Xu window | Full run | W | KL | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | TRPO (s0) | done | **1953/1953** | 999,936 | 325.5 ± 62.6 | 500.0 | 284.9 | 382.5 | 326 | 0.0032 | | TRPO (s1) | done | **1953/1953** | 999,936 | 56.1 ± 62.4 | 500.0 | 438.6 | 399.8 | 326 | 0.0032 | | TRPO (s2) | done | **1953/1953** | 999,936 | 290.1 ± 184.0 | 500.0 | 459.8 | 378.0 | 326 | 0.0021 | | EnTRPO-Entropy (s0) | done | **200/200** | 1,000,000 | 497.9 ± 6.3 | 500.0 | 365.4 | 372.3 | 33 | 0.0035 | | EnTRPO-Entropy (s1) | done | **200/200** | 1,000,000 | 420.3 ± 37.0 | 500.0 | 448.7 | 434.1 | 33 | 0.0059 | | EnTRPO-Entropy (s2) | done | **200/200** | 1,000,000 | 494.9 ± 12.1 | 500.0 | 425.4 | 409.5 | 33 | 0.0050 | | ERO-TRPO (s0) | done | **2000/2000** | 1,000,000 | 182.8 ± 66.6 | 232.8 | 168.1 | 86.8 | 333 | 0.0000 | | ERO-TRPO (s1) | done | **2000/2000** | 1,000,000 | 153.6 ± 68.2 | 280.2 | 198.5 | 103.7 | 333 | 0.0000 | | ERO-TRPO (s2) | done | **2000/2000** | 1,000,000 | 175.8 ± 110.2 | 254.6 | 157.7 | 84.4 | 333 | -0.0000 | | ERC-TRPO (s0) | done | **2000/2000** | 1,000,000 | 229.2 ± 81.0 | 258.3 | 186.2 | 97.4 | 333 | -0.0000 | | ERC-TRPO (s1) | done | **2000/2000** | 1,000,000 | 22.0 ± 9.3 | 43.7 | 25.6 | 25.7 | 333 | -0.0000 | | ERC-TRPO (s2) | done | **2000/2000** | 1,000,000 | 20.3 ± 5.2 | 41.7 | 21.7 | 21.8 | 333 | 0.0000 | | EnTRPO-Buffer (s0) | done | **200/200** | 1,000,000 | 170.3 ± 41.3 | 199.7 | 153.4 | 157.5 | 33 | 0.0091 | | EnTRPO-Buffer (s1) | done | **200/200** | 1,000,000 | 309.2 ± 62.2 | 500.0 | 330.6 | 320.8 | 33 | 0.0064 | | EnTRPO-Buffer (s2) | done | **200/200** | 1,000,000 | 205.0 ± 8.4 | 485.7 | 224.4 | 251.9 | 33 | 0.0070 | | EnTRPO (s0) | done | **200/200** | 1,000,000 | 101.7 ± 6.1 | 489.2 | 106.1 | 129.7 | 33 | 0.0089 | | EnTRPO (s1) | done | **200/200** | 1,000,000 | 99.0 ± 22.0 | 500.0 | 123.0 | 213.5 | 33 | 0.0076 | | EnTRPO (s2) | done | **200/200** | 1,000,000 | 168.9 ± 9.4 | 500.0 | 399.5 | 287.1 | 33 | 0.0067 | | PPO (s0) | done | **31250/31250** | 1,000,000 | 218.9 ± 84.4 | 340.9 | 207.5 | 150.0 | 1000 | 4.6836 | | PPO (s1) | done | **31250/31250** | 1,000,000 | 500.0 ± 0.0 | 500.0 | 499.5 | 265.0 | 1000 | 0.0824 | | PPO (s2) | done | **31250/31250** | 1,000,000 | 134.3 ± 51.4 | 500.0 | 299.0 | 275.5 | 1000 | -0.0170 | ## Humanoid-v5 (1M benchmark) | Variant | Status | Epoch | Timesteps | Eval (final) | Best | Xu window | Full run | W | KL | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | TRPO (s0) | done | **976/976** | 1,000,448 | 310.7 ± 62.0 | 385.1 | 303.1 | 253.8 | 163 | 0.0060 | | TRPO (s1) | done | **976/976** | 999,424 | 321.3 ± 88.9 | 379.0 | 309.6 | 257.8 | 163 | 0.0043 | | TRPO (s2) | done | **976/976** | 999,424 | 309.2 ± 105.4 | 365.1 | 304.2 | 251.8 | 163 | -0.0009 | | EnTRPO-Entropy (s0) | done | **488/488** | 999,424 | 239.5 ± 64.2 | 332.1 | 271.8 | 194.6 | 81 | -0.0000 | | EnTRPO-Entropy (s1) | done | **488/488** | 999,424 | 276.8 ± 63.1 | 333.5 | 268.0 | 183.7 | 81 | 0.0042 | | EnTRPO-Entropy (s2) | done | **488/488** | 999,424 | 249.7 ± 82.2 | 328.3 | 268.7 | 173.1 | 81 | 0.0000 | | ERO-TRPO (s0) | done | **976/976** | 999,424 | 280.8 ± 101.4 | 387.9 | 310.7 | 262.6 | 163 | -0.0012 | | ERO-TRPO (s1) | done | **976/976** | 999,424 | 330.0 ± 83.0 | 369.2 | 304.4 | 253.4 | 163 | 0.0046 | | ERO-TRPO (s2) | done | **976/976** | 999,424 | 316.5 ± 60.8 | 420.3 | 309.0 | 245.6 | 163 | 0.0092 | | ERC-TRPO (s0) | done | **976/976** | 999,424 | 277.9 ± 89.4 | 383.3 | 302.5 | 257.0 | 163 | 0.8317 | | ERC-TRPO (s1) | done | **976/976** | 999,424 | 281.9 ± 85.2 | 399.6 | 312.0 | 258.6 | 163 | 0.0051 | | ERC-TRPO (s2) | done | **976/976** | 999,424 | 254.2 ± 46.3 | 392.6 | 311.4 | 257.9 | 163 | -0.0025 | | EnTRPO-Buffer (s0) | done | **488/488** | 999,424 | 233.2 ± 46.0 | 314.2 | 262.7 | 181.5 | 81 | 0.0000 | | EnTRPO-Buffer (s1) | done | **488/488** | 999,424 | 293.6 ± 75.7 | 308.4 | 257.5 | 178.3 | 81 | 0.0000 | | EnTRPO-Buffer (s2) | done | **488/488** | 999,424 | 263.0 ± 78.6 | 310.7 | 250.2 | 161.5 | 81 | 0.0082 | | EnTRPO (s0) | done | **488/488** | 999,424 | 276.8 ± 52.6 | 352.6 | 256.4 | 186.2 | 81 | 0.0000 | | EnTRPO (s1) | done | **488/488** | 999,424 | 227.8 ± 31.5 | 334.7 | 256.3 | 174.8 | 81 | 0.0089 | | EnTRPO (s2) | done | **488/488** | 999,424 | 286.4 ± 68.8 | 323.3 | 255.7 | 163.5 | 81 | 0.0093 | | PPO (s0) | done | **1953/1953** | 999,936 | 286.4 ± 70.3 | 411.1 | 305.6 | 291.3 | 326 | 0.0171 | | PPO (s1) | done | **1953/1953** | 999,936 | 324.4 ± 72.3 | 403.0 | 309.6 | 288.9 | 326 | -0.0023 | | PPO (s2) | done | **1953/1953** | 999,936 | 291.7 ± 65.1 | 409.1 | 307.0 | 290.9 | 326 | -0.0067 | ## HumanoidStandup-v5 (1M benchmark) | Variant | Status | Epoch | Timesteps | Eval (final) | Best | Xu window | Full run | W | KL | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | TRPO (s0) | done | **976/976** | 999,424 | 65564.3 ± 13149.1 | 75225.9 | 65080.2 | 55528.2 | 163 | 0.0086 | | TRPO (s1) | done | **976/976** | 999,424 | 85331.8 ± 15316.0 | 93973.1 | 83992.1 | 69408.5 | 163 | 0.0013 | | TRPO (s2) | done | **976/976** | 999,424 | 71929.0 ± 11498.5 | 81636.4 | 73421.4 | 61150.4 | 163 | 0.0018 | | EnTRPO-Entropy (s0) | done | **488/488** | 999,424 | 64478.0 ± 14627.4 | 73943.7 | 61896.3 | 50554.0 | 81 | 0.0022 | | EnTRPO-Entropy (s1) | done | **488/488** | 999,424 | 79008.0 ± 7394.7 | 83421.9 | 75278.5 | 59057.8 | 81 | 0.0088 | | EnTRPO-Entropy (s2) | done | **488/488** | 999,424 | 74557.1 ± 10527.1 | 75892.7 | 66491.7 | 53834.4 | 81 | 0.0032 | | ERO-TRPO (s0) | done | **976/976** | 999,424 | 45868.3 ± 2401.2 | 48375.1 | 45105.9 | 40929.5 | 163 | 0.0005 | | ERO-TRPO (s1) | done | **976/976** | 999,424 | 52085.0 ± 6858.4 | 54183.2 | 50235.2 | 45674.6 | 163 | -0.0008 | | ERO-TRPO (s2) | done | **976/976** | 999,424 | 54828.2 ± 5440.4 | 57270.1 | 53365.4 | 45660.3 | 163 | 0.0006 | | ERC-TRPO (s0) | done | **976/976** | 999,424 | 44648.7 ± 1975.6 | 47619.2 | 43774.6 | 40086.5 | 163 | -0.0007 | | ERC-TRPO (s1) | done | **976/976** | 999,424 | 53205.4 ± 4787.5 | 56941.8 | 52720.3 | 46702.6...

Sources & provenance

1 active source

Hugging FaceRepresentative

huggingface.co/pre63/entropy-trpo-weights

highest_eligible_score

Verified Jul 11

Source evidence

3 excerpts

Readme excerptJul 11huggingface.co/pre63/entropy-trpo-weights

license: mit tags: reinforcement-learning trpo policy-optimization mujoco cartpole libraryname: entropy-trpo

SummaryJul 11huggingface.co/pre63/entropy-trpo-weights