Du hast einen ComfyUI-Workflow? Erstelle direkt im Browser eine verständliche Workflow-Dokumentation.Workflow verständlich machen.Jetzt ausprobieren ↗
PyTorch checkpoints for the comparative study in A Review of Entropy-Based Extensions to Trust Region Policy Optimization.
Modellquelle
Quellenbeschreibung
PyTorch checkpoints for the comparative study in A Review of Entropy-Based Extensions to Trust Region Policy Optimization.
Each checkpoint directory contains:
Quellen
1 QuelleVerifiziert 4. Aug.
Modellartefakte
64 Artefaktebenchmark_e3_1_3m/CartPole-v1/trpo/seed_0/latest/policy.pt
pt · 20,8 KB · SHA-256 9a8babf83d6d…fe1d · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_0/latest/value.pt
pt · 20,5 KB · SHA-256 6388175bd42e…efb8 · Hugging Face
HerunterladenQuellenauszüge
3 Auszüge| File | Description |
|---|
policy.pt | Policy network state dict |
value.pt | Value network state dict |
config.json | Training hyperparameters |
metadata.json | Paper source, variant flags, final metrics |
Hub path: {env_id}/{variant}/latest/ (e.g. CartPole-v1/entrpo/latest/).
The repo README is updated automatically during training with a Training progress table
(epoch n/N, eval return, best return, KL) from results/summary.md, plus a JSON index of
available checkpoints.
entropy_coef)| Key | Paper name | Surrogate / constraint |
|---|---|---|
trpo | TRPO | $\mathbb{E}[\rho_t \hat{A}t]$; $\bar D{\mathrm{KL}} \le \delta$ |
entrpo_entropy | EnTRPO-Entropy | $\mathbb{E}[\rho_t \tilde{A}t]$, $\tilde{A}t=\hat{A}t+\alpha,\mathcal{H}(\pi{\theta{\text{old}}}(\cdot|s_t))$ (fixed during step); $\bar D{\mathrm{KL}} \le \delta$ |
ero_trpo | ERO-TRPO | $\mathbb{E}[\rho_t \hat{A}t]+\alpha,\mathbb{E}[\mathcal{H}(\pi\theta)]$; $\bar D_{\mathrm{KL}} \le \delta$ |
erc_trpo | ERC-TRPO | $\mathbb{E}[\rho_t \hat{A}t]$; $\bar D{\mathrm{KL}} \le \delta+\beta,\mathbb{E}[\mathcal{H}(\pi_\theta)]$ (Xu Eq. 49) |
entrpo_buffer | EnTRPO-Buffer | $\mathbb{E}[\rho_t \hat{A}_t]$ with Roostaie on-policy replay |
entrpo | EnTRPO | $\mathbb{E}[\rho_t \tilde{A}_t]$ + Roostaie buffer |
$\mathcal{H}$ in EnTRPO rows is evaluated at the behavior policy $\pi_{\theta_{\text{old}}}$; in ERO/ERC/PPO rows at the candidate policy $\pi_\theta$.
ERO-TRPO implementation: Xu Table 1 — $H\mathbf{d}=\mathbf{g}+\alpha\mathbf{h}$, step size $\eta\le\sqrt{2\delta/(\mathbf{g}+\alpha\mathbf{h})^\top H(\mathbf{g}+\alpha\mathbf{h})}$, Xu line search ($\eta_0=0.5$, strict objective improvement, hard $\bar{D}_{\mathrm{KL}}\le\delta$).
ERC-TRPO implementation: Xu Table 1 — two CG solves ($\mathbf{u},\mathbf{v}$), $\eta\mathbf{u}+\beta\mathbf{v}$, Eq.~(49) acceptance $\bar{D}_{\mathrm{KL}}\le\delta+\beta,\mathbb{E}[\mathcal{H}]$, same Xu line search. Random ERC shares the ERC step; only the acceptance bonus is randomized.
Older Hub folders (trpo_entropy, trpo_buffer, ...) remain valid; training resumes from them automatically.
| Environment | Obs / action | Training budget | Hyperparameter source |
|---|---|---|---|
| CartPole-v1 | Gymnasium classic control | 1M steps | Roostaie + Xu Tables 4--5 |
| Humanoid-v5 | 348 / 17 | $10^6$ steps | PPO/baselines backbone; Xu ERO/ERC proxied from Walker2d |
| HumanoidStandup-v5 | 348 / 17 | $10^6$ steps | Same backbone; Xu ERO/ERC proxied from BipedalWalker |
See HYPERPARAMETERS.md for per-field provenance and paper/results/annex_hyperparameters.tex for tables.
| Variant | Paper |
|---|---|
trpo | Schulman et al. (2015), Trust Region Policy Optimization, ICML |
entrpo_entropy | Roostaie & Ebadzadeh (2021), EnTRPO — entropy-in-advantage ablation |
entrpo_buffer | Roostaie & Ebadzadeh (2021), EnTRPO — replay-buffer ablation |
entrpo | Roostaie & Ebadzadeh (2021), EnTRPO — full method |
ero_trpo | Xu et al. (2024), ERO-TRPO |
erc_trpo | Xu et al. (2024), ERC-TRPO |
ppo | Schulman et al. (2017), Proximal Policy Optimization |
See metadata.json in each folder for full author names and URLs.
Training and evaluation code: GitHub — entropy-trpo (update URL when published).
git clone https://github.com/pre63/entropy-trpo.git
cd entropy-trpo
make setup # install deps + create .env
# edit .env with HF_TOKEN and HF_REPO_ID
make download-weights
make eval-checkpoints
@article{entropytrporeview2026,
title = {A Review of Entropy-Based Extensions to Trust Region Policy Optimization},
author = {Green, Simon},
journal = {IEEE Transactions},
year = {2026}
}
@article{roostaie2021entrpo,
title = {EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization},
author = {Roostaie, Sahar and Ebadzadeh, Mohammad Mehdi},
journal = {arXiv:2110.13373},
year = {2021}
}
@article{xu2024trpo,
title = {Trust region policy optimization via entropy regularization for {Kullback--Leibler} divergence constraint},
author = {Xu, Haotian and Xuan, Junyu and Zhang, Guangquan and Lu, Jie},
journal = {Neurocomputing},
volume = {589},
pages = {127716},
year = {2024}
}
Last updated: 2026-07-05 19:43:57 UTC
Xu window — mean eval return over the last W epochs (Xu Table 6 protocol).
Full run — mean eval return over all epochs.
Device: cpu
Config: configs/e4/e4_entrpo_substitute_cartpole_v5.yaml
Jobs complete: 72/72
Running: 0
| Variant | Status | Epoch | Timesteps | Eval (final) | Best | Xu window | Full run | W | KL |
|---|---|---|---|---|---|---|---|---|---|
| TRPO (s0) | done | 1953/1953 | 999,936 | 325.5 ± 62.6 | 500.0 | 284.9 | 382.5 | 326 | 0.0032 |
| TRPO (s1) | done | 1953/1953 | 999,936 | 56.1 ± 62.4 | 500.0 | 438.6 | 399.8 | 326 | 0.0032 |
| TRPO (s2) | done | 1953/1953 | 999,936 | 290.1 ± 184.0 | 500.0 | 459.8 | 378.0 | 326 |
| Variant | Status | Epoch | Timesteps | Eval (final) | Best | Xu window | Full run | W | KL |
|---|---|---|---|---|---|---|---|---|---|
| TRPO (s0) | done | 976/976 | 1,000,448 | 310.7 ± 62.0 | 385.1 | 303.1 | 253.8 | 163 | 0.0060 |
| TRPO (s1) | done | 976/976 | 999,424 | 321.3 ± 88.9 | 379.0 | 309.6 | 257.8 | 163 | 0.0043 |
| TRPO (s2) | done | 976/976 | 999,424 | 309.2 ± 105.4 | 365.1 | 304.2 | 251.8 | 163 |
| Variant | Status | Epoch | Timesteps | Eval (final) | Best | Xu window | Full run | W | KL |
|---|---|---|---|---|---|---|---|---|---|
| TRPO (s0) | done | 976/976 | 999,424 | 65564.3 ± 13149.1 | 75225.9 | 65080.2 | 55528.2 | 163 | 0.0086 |
| TRPO (s1) | done | 976/976 | 999,424 | 85331.8 ± 15316.0 | 93973.1 | 83992.1 | 69408.5 | 163 | 0.0013 |
| TRPO (s2) | done | 976/976 | 999,424 | 71929.0 ± 11498.5 | 81636.4 | 73421.4 | 61150.4 | 163 |
benchmark_e3_1_3m/CartPole-v1/trpo/seed_1/latest/policy.pt
pt · 20,8 KB · SHA-256 3a0accdca771…2a66 · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_1/latest/value.pt
pt · 20,5 KB · SHA-256 c18bcd062eb5…c79b · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_2/latest/policy.pt
pt · 20,8 KB · SHA-256 91bfbb10eadd…1e0f · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_2/latest/value.pt
pt · 20,5 KB · SHA-256 0bcff30fd7c7…1ea5 · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_3/latest/policy.pt
pt · 20,8 KB · SHA-256 4438c2770003…f500 · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_3/latest/value.pt
pt · 20,5 KB · SHA-256 5f657055d472…9fdb · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_4/latest/policy.pt
pt · 20,8 KB · SHA-256 77f6ce42a144…0525 · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_4/latest/value.pt
pt · 20,5 KB · SHA-256 1d87e7b3e5d9…0821 · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_5/latest/policy.pt
pt · 20,8 KB · SHA-256 dc9145f0b926…c743 · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_5/latest/value.pt
pt · 20,5 KB · SHA-256 08b25101b043…6134 · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_6/latest/policy.pt
pt · 20,8 KB · SHA-256 b3d694c2f962…5c3a · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_6/latest/value.pt
pt · 20,5 KB · SHA-256 62cd5d0a4981…7529 · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_7/latest/policy.pt
pt · 20,8 KB · SHA-256 2dccc8ba3f86…e3ce · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_7/latest/value.pt
pt · 20,5 KB · SHA-256 24bcb779a153…c093 · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_8/latest/policy.pt
pt · 20,8 KB · SHA-256 90b79643ada5…65e8 · Hugging Face
Herunterladenbenchmark_e3_1_3m/CartPole-v1/trpo/seed_8/latest/value.pt
pt · 20,5 KB · SHA-256 308a6eda7e31…58ef · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/constant_erc_trpo/seed_0/latest/policy.pt
pt · 5,83 KB · SHA-256 7da38f5c8dcb…f442 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/constant_erc_trpo/seed_0/latest/value.pt
pt · 5,32 KB · SHA-256 b62e5bada2fc…4ed9 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/constant_erc_trpo/seed_1/latest/policy.pt
pt · 5,83 KB · SHA-256 e46c0c35f6e2…9da8 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/constant_erc_trpo/seed_1/latest/value.pt
pt · 5,32 KB · SHA-256 860d7dc4acc9…96b7 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/constant_erc_trpo/seed_2/latest/policy.pt
pt · 5,83 KB · SHA-256 10fa273948fa…17ee · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/constant_erc_trpo/seed_2/latest/value.pt
pt · 5,32 KB · SHA-256 02a999019714…d0fe · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/erc_trpo/seed_0/latest/policy.pt
pt · 5,45 KB · SHA-256 cd434dc2502c…e2c6 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/erc_trpo/seed_0/latest/value.pt
pt · 4,94 KB · SHA-256 5f899d720659…3272 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/erc_trpo/seed_1/latest/policy.pt
pt · 5,45 KB · SHA-256 0ad8889e4ddb…b5c3 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/erc_trpo/seed_1/latest/value.pt
pt · 4,94 KB · SHA-256 3b60cfd54664…4ea8 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/erc_trpo/seed_2/latest/policy.pt
pt · 5,45 KB · SHA-256 9da4f86aec00…19c0 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/erc_trpo/seed_2/latest/value.pt
pt · 4,94 KB · SHA-256 f8cdd2a8bc7a…942b · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/random_erc_trpo/seed_0/latest/policy.pt
pt · 5,45 KB · SHA-256 cd434dc2502c…e2c6 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/random_erc_trpo/seed_0/latest/value.pt
pt · 4,94 KB · SHA-256 5f899d720659…3272 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/random_erc_trpo/seed_1/latest/policy.pt
pt · 5,45 KB · SHA-256 0ad8889e4ddb…b5c3 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/random_erc_trpo/seed_1/latest/value.pt
pt · 4,94 KB · SHA-256 3b60cfd54664…4ea8 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/random_erc_trpo/seed_2/latest/policy.pt
pt · 5,45 KB · SHA-256 9da4f86aec00…19c0 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/random_erc_trpo/seed_2/latest/value.pt
pt · 4,94 KB · SHA-256 f8cdd2a8bc7a…942b · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/trpo/seed_0/latest/policy.pt
pt · 20,4 KB · SHA-256 7360bf3741ac…ecaa · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/trpo/seed_0/latest/value.pt
pt · 20,1 KB · SHA-256 7a40d60c228b…7a3a · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/trpo/seed_1/latest/policy.pt
pt · 20,4 KB · SHA-256 1519ada9194a…4c4a · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/trpo/seed_1/latest/value.pt
pt · 20,1 KB · SHA-256 d2555aa53f32…9411 · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/trpo/seed_2/latest/policy.pt
pt · 20,4 KB · SHA-256 79b0b4f4cd69…da4e · Hugging Face
Herunterladenbenchmark_e3_1m/CartPole-v1/trpo/seed_2/latest/value.pt
pt · 20,1 KB · SHA-256 7d35800d946c…fa5d · Hugging Face
Herunterladenbenchmark_e3_3_xu_repro/CartPole-v1/erc_trpo/latest/policy.pt
pt · 5,83 KB · SHA-256 55dd9e74fb7e…9998 · Hugging Face
Herunterladenbenchmark_e3_3_xu_repro/CartPole-v1/erc_trpo/latest/value.pt
pt · 5,32 KB · SHA-256 83afa7ac381e…036e · Hugging Face
Herunterladenbenchmark_e3_3m/CartPole-v1/erc_trpo/seed_0/latest/policy.pt
pt · 5,83 KB · SHA-256 d97f37e45df2…d058 · Hugging Face
Herunterladenbenchmark_e3_3m/CartPole-v1/erc_trpo/seed_0/latest/value.pt
pt · 5,32 KB · SHA-256 d084c76e496e…37c9 · Hugging Face
Herunterladenbenchmark_e3_4_env/CartPole-v1-OpenAIGym/erc_trpo/latest/policy.pt
pt · 5,83 KB · SHA-256 b961a216129b…2909 · Hugging Face
Herunterladenbenchmark_e3_4_env/CartPole-v1-OpenAIGym/erc_trpo/latest/value.pt
pt · 5,32 KB · SHA-256 d7b4077ba4d9…747c · Hugging Face
Herunterladenbenchmark_e3_4_env/CartPole-v1/erc_trpo/latest/policy.pt
pt · 5,83 KB · SHA-256 55dd9e74fb7e…9998 · Hugging Face
Herunterladenbenchmark_e3_4_env/CartPole-v1/erc_trpo/latest/value.pt
pt · 5,32 KB · SHA-256 83afa7ac381e…036e · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/entrpo_entropy/seed_0/latest/policy.pt
pt · 20,8 KB · SHA-256 9392aa6e1dc6…f49e · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/entrpo_entropy/seed_0/latest/value.pt
pt · 46,2 KB · SHA-256 d8b2c8398558…2ba4 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/entrpo_entropy/seed_1/latest/policy.pt
pt · 20,8 KB · SHA-256 d17380bb79b7…8d29 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/entrpo_entropy/seed_1/latest/value.pt
pt · 46,2 KB · SHA-256 194bf3b742ba…e279 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/entrpo_entropy/seed_2/latest/policy.pt
pt · 20,8 KB · SHA-256 c971d32420bf…ce83 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/entrpo_entropy/seed_2/latest/value.pt
pt · 46,2 KB · SHA-256 53d685810cdf…6d11 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/ero_trpo/seed_0/latest/policy.pt
pt · 5,83 KB · SHA-256 59ee61a7956b…2fb9 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/ero_trpo/seed_0/latest/value.pt
pt · 5,32 KB · SHA-256 2a577ab6c848…de46 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/ero_trpo/seed_1/latest/policy.pt
pt · 5,83 KB · SHA-256 68b9f9b8aa93…db46 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/ero_trpo/seed_1/latest/value.pt
pt · 5,32 KB · SHA-256 2b48f648b5d8…e9c2 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/ero_trpo/seed_2/latest/policy.pt
pt · 5,83 KB · SHA-256 e2eaf514f751…e5e6 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/ero_trpo/seed_2/latest/value.pt
pt · 5,32 KB · SHA-256 7fdf9702e87b…ae86 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/random_entrpo_entropy/seed_0/latest/policy.pt
pt · 20,8 KB · SHA-256 3d1179a3376c…a7f6 · Hugging Face
Herunterladenbenchmark_e4_cartpole/CartPole-v1/random_entrpo_entropy/seed_0/latest/value.pt
pt · 46,2 KB · SHA-256 dd2c83e8fe5d…8959 · Hugging Face
Herunterladen--- license: mit tags: - reinforcement-learning - trpo - policy-optimization - mujoco - cartpole library_name: entropy-trpo --- # Entropy-TRPO Model Weights PyTorch checkpoints for the comparative study in *A Review of Entropy-Based Extensions to Trust Region Policy Optimization*. ## Repository layout Each checkpoint directory contains: | File | Description | |------|-------------| | `policy.pt` | Policy network state dict | | `value.pt` | Value network state dict | | `config.json` | Training hyperparameters | | `metadata.json` | Paper source, variant flags, final metrics | Hub path: `{env_id}/{variant}/latest/` (e.g. `CartPole-v1/entrpo/latest/`). The repo **README** is updated automatically during training with a **Training progress** table (epoch `n/N`, eval return, best return, KL) from `results/summary.md`, plus a JSON index of available checkpoints. ## Notation - $\rho_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{\theta_{\text{old}}}(a_t|s_t)$, GAE advantages $\hat{A}_t$, trust-region radius $\delta$ - **$\alpha$** — Roostaie advantage entropy; Xu ERO objective entropy (distinct roles, same symbol in each paper's row) - **$\beta$** — Xu ERC constraint coefficient (Xu Eq. 49) - **$c_{\mathrm{ent}}$** — PPO entropy bonus (Schulman et al., 2017; config field `entropy_coef`) ## Variant definitions | Key | Paper name | Surrogate / constraint | | --- | --- | --- | | `trpo` | TRPO | $\mathbb{E}[\rho_t \hat{A}_t]$; $\bar D_{\mathrm{KL}} \le \delta$ | | `entrpo_entropy` | EnTRPO-Entropy | $\mathbb{E}[\rho_t \tilde{A}_t]$, $\tilde{A}_t=\hat{A}_t+\alpha\,\mathcal{H}(\pi_{\theta_{\text{old}}}(\cdot\|s_t))$ (fixed during step); $\bar D_{\mathrm{KL}} \le \delta$ | | `ero_trpo` | ERO-TRPO | $\mathbb{E}[\rho_t \hat{A}_t]+\alpha\,\mathbb{E}[\mathcal{H}(\pi_\theta)]$; $\bar D_{\mathrm{KL}} \le \delta$ | | `erc_trpo` | ERC-TRPO | $\mathbb{E}[\rho_t \hat{A}_t]$; $\bar D_{\mathrm{KL}} \le \delta+\beta\,\mathbb{E}[\mathcal{H}(\pi_\theta)]$ (Xu Eq. 49) | | `entrpo_buffer` | EnTRPO-Buffer | $\mathbb{E}[\rho_t \hat{A}_t]$ with Roostaie on-policy replay | | `entrpo` | EnTRPO | $\mathbb{E}[\rho_t \tilde{A}_t]$ + Roostaie buffer | | `ppo` | PPO | $\mathbb{E}[\min(\rho_t \hat{A}_t,\mathrm{clip}(\rho_t)\hat{A}_t)]+c_{\mathrm{ent}}\mathbb{E}[\mathcal{H}(\pi_\theta)]$ | $\mathcal{H}$ in EnTRPO rows is evaluated at the behavior policy $\pi_{\theta_{\text{old}}}$; in ERO/ERC/...
Source context: 0 downloads · 1 likes · Pipeline reinforcement-learning · Library entropy-trpo · Repo pre63/entropy-trpo-weights
Source context: 0 downloads · 1 likes · Pipeline reinforcement-learning · Library entropy-trpo · Repo pre63/entropy-trpo-weights
ppo| PPO |
| $\mathbb{E}[\min(\rho_t \hat{A}t,\mathrm{clip}(\rho_t)\hat{A}t)]+c{\mathrm{ent}}\mathbb{E}[\mathcal{H}(\pi\theta)]$ |
| 0.0021 |
| EnTRPO-Entropy (s0) | done | 200/200 | 1,000,000 | 497.9 ± 6.3 | 500.0 | 365.4 | 372.3 | 33 | 0.0035 |
| EnTRPO-Entropy (s1) | done | 200/200 | 1,000,000 | 420.3 ± 37.0 | 500.0 | 448.7 | 434.1 | 33 | 0.0059 |
| EnTRPO-Entropy (s2) | done | 200/200 | 1,000,000 | 494.9 ± 12.1 | 500.0 | 425.4 | 409.5 | 33 | 0.0050 |
| ERO-TRPO (s0) | done | 2000/2000 | 1,000,000 | 182.8 ± 66.6 | 232.8 | 168.1 | 86.8 | 333 | 0.0000 |
| ERO-TRPO (s1) | done | 2000/2000 | 1,000,000 | 153.6 ± 68.2 | 280.2 | 198.5 | 103.7 | 333 | 0.0000 |
| ERO-TRPO (s2) | done | 2000/2000 | 1,000,000 | 175.8 ± 110.2 | 254.6 | 157.7 | 84.4 | 333 | -0.0000 |
| ERC-TRPO (s0) | done | 2000/2000 | 1,000,000 | 229.2 ± 81.0 | 258.3 | 186.2 | 97.4 | 333 | -0.0000 |
| ERC-TRPO (s1) | done | 2000/2000 | 1,000,000 | 22.0 ± 9.3 | 43.7 | 25.6 | 25.7 | 333 | -0.0000 |
| ERC-TRPO (s2) | done | 2000/2000 | 1,000,000 | 20.3 ± 5.2 | 41.7 | 21.7 | 21.8 | 333 | 0.0000 |
| EnTRPO-Buffer (s0) | done | 200/200 | 1,000,000 | 170.3 ± 41.3 | 199.7 | 153.4 | 157.5 | 33 | 0.0091 |
| EnTRPO-Buffer (s1) | done | 200/200 | 1,000,000 | 309.2 ± 62.2 | 500.0 | 330.6 | 320.8 | 33 | 0.0064 |
| EnTRPO-Buffer (s2) | done | 200/200 | 1,000,000 | 205.0 ± 8.4 | 485.7 | 224.4 | 251.9 | 33 | 0.0070 |
| EnTRPO (s0) | done | 200/200 | 1,000,000 | 101.7 ± 6.1 | 489.2 | 106.1 | 129.7 | 33 | 0.0089 |
| EnTRPO (s1) | done | 200/200 | 1,000,000 | 99.0 ± 22.0 | 500.0 | 123.0 | 213.5 | 33 | 0.0076 |
| EnTRPO (s2) | done | 200/200 | 1,000,000 | 168.9 ± 9.4 | 500.0 | 399.5 | 287.1 | 33 | 0.0067 |
| PPO (s0) | done | 31250/31250 | 1,000,000 | 218.9 ± 84.4 | 340.9 | 207.5 | 150.0 | 1000 | 4.6836 |
| PPO (s1) | done | 31250/31250 | 1,000,000 | 500.0 ± 0.0 | 500.0 | 499.5 | 265.0 | 1000 | 0.0824 |
| PPO (s2) | done | 31250/31250 | 1,000,000 | 134.3 ± 51.4 | 500.0 | 299.0 | 275.5 | 1000 | -0.0170 |
| -0.0009 |
| EnTRPO-Entropy (s0) | done | 488/488 | 999,424 | 239.5 ± 64.2 | 332.1 | 271.8 | 194.6 | 81 | -0.0000 |
| EnTRPO-Entropy (s1) | done | 488/488 | 999,424 | 276.8 ± 63.1 | 333.5 | 268.0 | 183.7 | 81 | 0.0042 |
| EnTRPO-Entropy (s2) | done | 488/488 | 999,424 | 249.7 ± 82.2 | 328.3 | 268.7 | 173.1 | 81 | 0.0000 |
| ERO-TRPO (s0) | done | 976/976 | 999,424 | 280.8 ± 101.4 | 387.9 | 310.7 | 262.6 | 163 | -0.0012 |
| ERO-TRPO (s1) | done | 976/976 | 999,424 | 330.0 ± 83.0 | 369.2 | 304.4 | 253.4 | 163 | 0.0046 |
| ERO-TRPO (s2) | done | 976/976 | 999,424 | 316.5 ± 60.8 | 420.3 | 309.0 | 245.6 | 163 | 0.0092 |
| ERC-TRPO (s0) | done | 976/976 | 999,424 | 277.9 ± 89.4 | 383.3 | 302.5 | 257.0 | 163 | 0.8317 |
| ERC-TRPO (s1) | done | 976/976 | 999,424 | 281.9 ± 85.2 | 399.6 | 312.0 | 258.6 | 163 | 0.0051 |
| ERC-TRPO (s2) | done | 976/976 | 999,424 | 254.2 ± 46.3 | 392.6 | 311.4 | 257.9 | 163 | -0.0025 |
| EnTRPO-Buffer (s0) | done | 488/488 | 999,424 | 233.2 ± 46.0 | 314.2 | 262.7 | 181.5 | 81 | 0.0000 |
| EnTRPO-Buffer (s1) | done | 488/488 | 999,424 | 293.6 ± 75.7 | 308.4 | 257.5 | 178.3 | 81 | 0.0000 |
| EnTRPO-Buffer (s2) | done | 488/488 | 999,424 | 263.0 ± 78.6 | 310.7 | 250.2 | 161.5 | 81 | 0.0082 |
| EnTRPO (s0) | done | 488/488 | 999,424 | 276.8 ± 52.6 | 352.6 | 256.4 | 186.2 | 81 | 0.0000 |
| EnTRPO (s1) | done | 488/488 | 999,424 | 227.8 ± 31.5 | 334.7 | 256.3 | 174.8 | 81 | 0.0089 |
| EnTRPO (s2) | done | 488/488 | 999,424 | 286.4 ± 68.8 | 323.3 | 255.7 | 163.5 | 81 | 0.0093 |
| PPO (s0) | done | 1953/1953 | 999,936 | 286.4 ± 70.3 | 411.1 | 305.6 | 291.3 | 326 | 0.0171 |
| PPO (s1) | done | 1953/1953 | 999,936 | 324.4 ± 72.3 | 403.0 | 309.6 | 288.9 | 326 | -0.0023 |
| PPO (s2) | done | 1953/1953 | 999,936 | 291.7 ± 65.1 | 409.1 | 307.0 | 290.9 | 326 | -0.0067 |
| 0.0018 |
| EnTRPO-Entropy (s0) | done | 488/488 | 999,424 | 64478.0 ± 14627.4 | 73943.7 | 61896.3 | 50554.0 | 81 | 0.0022 |
| EnTRPO-Entropy (s1) | done | 488/488 | 999,424 | 79008.0 ± 7394.7 | 83421.9 | 75278.5 | 59057.8 | 81 | 0.0088 |
| EnTRPO-Entropy (s2) | done | 488/488 | 999,424 | 74557.1 ± 10527.1 | 75892.7 | 66491.7 | 53834.4 | 81 | 0.0032 |
| ERO-TRPO (s0) | done | 976/976 | 999,424 | 45868.3 ± 2401.2 | 48375.1 | 45105.9 | 40929.5 | 163 | 0.0005 |
| ERO-TRPO (s1) | done | 976/976 | 999,424 | 52085.0 ± 6858.4 | 54183.2 | 50235.2 | 45674.6 | 163 | -0.0008 |
| ERO-TRPO (s2) | done | 976/976 | 999,424 | 54828.2 ± 5440.4 | 57270.1 | 53365.4 | 45660.3 | 163 | 0.0006 |
| ERC-TRPO (s0) | done | 976/976 | 999,424 | 44648.7 ± 1975.6 | 47619.2 | 43774.6 | 40086.5 | 163 | -0.0007 |
| ERC-TRPO (s1) | done... |