The key to fast iterations of research experiments are well-written baseline algorithms. Unfortunately, most big research companies write their code in TensorFlow, (openai/baselines, openai/spinningup, deepmind/trfl, google/dopamine) so the PyTorch implementations are less well known. To help PyTorch deep RL researchers, we compare and recommend open source implementations of policy gradient algorithms in PyTorch.

Note that due to the big difference between PyTorch 0.3 and 0.4, we only include repositories with PyTorch versions 0.4 or above.

Policy Gradient Methods

A3C

Asynchronous Advantage Actor Critic

[arXiv Paper]

A2C

Advantage Actor Critic

ACER

Actor Critic with Experience Replay

[arXiv Paper]

ACKTR

Actor Critic using Kronecker-Factored Trust Region

[arXiv Paper]

TRPO

Trust Region Policy Optimization

[arXiv Paper]

PPO

Proximal Policy Optimization

[arXiv Paper]

SAC

Soft Actor-Critic

[arXiv Paper]

Twin SAC

Combination of SAC and TD3

Recommendation

Although vitchyr/rlkit has SAC and Twin SAC, which are state-of-the-art methods in robotic control, it unfortunately does not include PPO, the standard baseline policy gradient algorithm. We found ikostrikov/pytorch-a2c-ppo-acktr and ShangtongZhang/DeepRL to be the best implementation of PPO, allowing us to run code almost immediately after cloning the repository. We gave bonus points to this repository because it also included some pretrained models.

Verdict: ikostrikov/pytorch-a2c-ppo-acktr

Deterministic Policy Gradient Methods

DDPG

Deep Deterministic Policy Gradient

[arXiv Paper]

TD3

Twin-Delayed Deep Deterministic Policy Gradient

[arXiv Paper]

HER+TD3

Hindsight Experience Replay + Twin-Delayed Deep Deterministic Policy Gradient

[arXiv Paper]

Recommendation

For Deterministic Policy Gradient methods, vitchyr/rlkit and higgsfield/RL-Adventure-2 were the only repositories with both DDPG and TD3 implemented. We found higgsfield/RL-Adventure-2 to be more suitable for understanding the algorithm than running it, so we recommend using vitchyr/rlkit as your baseline.

Verdict: vitchyr/rlkit

Checked Repositories

• vitchyr/rlkit
• ikostrikov/pytorch-a3c
• ikostrikov/pytorch-trpo
• ikostrikov/pytorch-a2c-ppo-acktr
• ikostrikov/pytorch-ddpg-naf
• Kaixhin/ACER
• higgsfield/RL-Adventure-2
• MillionIntegrals/vel
• ShangtongZhang/DeepRL (Added 2018/12/29)

If you believe we missed a great PyTorch RL repository, please tell us in the comment section!