• https://discord.gg/hnYRq7DsQh

• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning: https://arxiv.org/abs/2108.10470
• DeXtreme: Transfer of Agile In-Hand Manipulation from Simulation to Reality: https://dextreme.org/ https://arxiv.org/abs/2210.13702
• Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger: https://s2r2-ig.github.io/ https://arxiv.org/abs/2108.09779
• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? https://arxiv.org/abs/2011.09533
• Superfast Adversarial Motion Priors (AMP) implementation: https://twitter.com/xbpeng4/status/1506317490766303235 https://github.com/NVIDIA-Omniverse/IsaacGymEnvs
• OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation: https://cremebrule.github.io/oscar-web/ https://arxiv.org/abs/2110.00704
• EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine: https://arxiv.org/abs/2206.10558 and https://github.com/sail-sg/envpool
• TimeChamber: A Massively Parallel Large Scale Self-Play Framework: https://github.com/inspirai/TimeChamber

• NVIDIA Isaac Gym

• Dextreme

• DexPBT

• Starcraft 2 Multi Agents
• BRAX
• Mujoco Envpool
• DeepMind Envpool
• Atari Envpool
• Random Envs

Implemented in Pytorch:

• PPO with the support of asymmetric actor-critic variant
• Support of end-to-end GPU accelerated training pipeline with Isaac Gym and Brax
• Masked actions support
• Multi-agent training, decentralized and centralized critic variants
• Self-play

Implemented in Tensorflow 1.x (was removed in this version):

• Rainbow DQN
• A2C
• PPO

Explore RL Games quick and easily in colab notebooks:

• Mujoco training Mujoco envpool training example.
• Brax training Brax training example, with keeping all the observations and actions on GPU.
• Onnx discrete space export example with Cartpole envpool training example.
• Onnx continuous space export example with Pendulum envpool training example.
• Onnx continuous space with LSTM export example with Pendulum envpool training example.

For maximum training performance a preliminary installation of Pytorch 1.9+ with CUDA 11.1+ is highly recommended:

conda install pytorch torchvision cudatoolkit=11.3 -c pytorch -c nvidia or: pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

Then:

pip install rl-games

To run CPU-based environments either Ray or envpool are required pip install envpool or pip install ray To run Mujoco, Atari games or Box2d based environments training they need to be additionally installed with pip install gym[mujoco], pip install gym[atari] or pip install gym[box2d] respectively.

To run Atari also pip install opencv-python is required. In addition installation of envpool for maximum simulation and training perfromance of Mujoco and Atari environments is highly recommended: pip install envpool

If you use rl-games in your research please use the following citation:

@misc{rl-games2021,
title = {rl-games: A High-performance Framework for Reinforcement Learning},
author = {Makoviichuk, Denys and Makoviychuk, Viktor},
month = {May},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/Denys88/rl_games}},
}

poetry install
# install cuda related dependencies
poetry run pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

NVIDIA Isaac Gym

Download and follow the installation instructions of Isaac Gym: https://developer.nvidia.com/isaac-gym
And IsaacGymEnvs: https://github.com/NVIDIA-Omniverse/IsaacGymEnvs

Ant

python train.py task=Ant headless=True python train.py task=Ant test=True checkpoint=nn/Ant.pth num_envs=100

Humanoid

python train.py task=Humanoid headless=True python train.py task=Humanoid test=True checkpoint=nn/Humanoid.pth num_envs=100

Shadow Hand block orientation task

python train.py task=ShadowHand headless=True python train.py task=ShadowHand test=True checkpoint=nn/ShadowHand.pth num_envs=100

Other

Atari Pong

poetry install -E atari
poetry run python runner.py --train --file rl_games/configs/atari/ppo_pong.yaml
poetry run python runner.py --play --file rl_games/configs/atari/ppo_pong.yaml --checkpoint nn/PongNoFrameskip.pth

Brax Ant

poetry install -E brax
poetry run pip install --upgrade "jax[cuda]==0.3.13" -f https://storage.googleapis.com/jax-releases/jax_releases.html
poetry run python runner.py --train --file rl_games/configs/brax/ppo_ant.yaml
poetry run python runner.py --play --file rl_games/configs/brax/ppo_ant.yaml --checkpoint runs/Ant_brax/nn/Ant_brax.pth

rl_games support experiment tracking with Weights and Biases.

poetry install -E atari
poetry run python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --track
WANDB_API_KEY=xxxx poetry run python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --track
poetry run python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --wandb-project-name rl-games-special-test --track
poetry run python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --wandb-project-name rl-games-special-test -wandb-entity openrlbenchmark --track

We use torchrun to orchestrate any multi-gpu runs.

torchrun --standalone --nnodes=1 --nproc_per_node=2 runner.py --train --file rl_games/configs/ppo_cartpole.yaml

simple test network
This network takes dictionary observation. To register it you can add code in your init.py

from rl_games.envs.test_network import TestNetBuilder 
from rl_games.algos_torch import model_builder
model_builder.register_network('testnet', TestNetBuilder)

simple test environment example environment

Additional environment supported properties and functions

1.6.1 (Unreleased)

• Fixed Central Value RNN bug which occurs if you train ma multi agent environment.
• Added Deepmind Control PPO benchmark.
• Added a few more experimental ways to train value prediction (OneHot, TwoHot encoding and crossentropy loss instead of L2).
• New methods didn't. It is impossible to turn it on from the yaml files. Once we find an env which trains better it will be added to the config.
• Added shaped reward graph to the tensorboard.
• Fixed bug with SAC not saving weights with save_frequency.
• Added multi-node training support for GPU-accelerated training environments like Isaac Gym. No changes in training scripts are required. Thanks to @ankurhanda and @ArthurAllshire for assistance in implementation.
• Added evaluation feature for inferencing during training. Checkpoints from training process can be automatically picked up and updated in the inferencing process when enabled.
• Added get/set API for runtime update of rl training parameters. Thanks to @ArthurAllshire for the initial version of fast PBT code.
• Fixed SAC not loading weights properly.

1.6.0

• Added ONNX export colab example for discrete and continious action spaces. For continuous case LSTM policy example is provided as well.
• Improved RNNs training in continuous space, added option zero_rnn_on_done.
• Added NVIDIA CuLE support: https://github.com/NVlabs/cule
• Added player config everride. Vecenv is used for inference.
• Fixed multi-gpu training with central value.
• Fixed max_frames termination condition, and it's interaction with the linear learning rate: #212
• Fixed "deterministic" misspelling issue.
• Fixed Mujoco and Brax SAC configs.
• Fixed multiagent envs statistics reporting. Fixed Starcraft2 SMAC environments.

1.5.2

• Added observation normalization to the SAC.
• Returned back adaptive KL legacy mode.

1.5.1

• Fixed build package issue.

1.5.0

• Added wandb support.
• Added poetry support.
• Fixed various bugs.
• Fixed cnn input was not divided by 255 in case of the dictionary obs.
• Added more envpool mujoco and atari training examples. Some of the results: 15 min Mujoco humanoid training, 2 min atari pong.
• Added Brax and Mujoco colab training examples.
• Added 'seed' command line parameter. Will override seed in config in case it's > 0.
• Deprecated horovod in favor of torch.distributed (#171).

1.4.0

• Added discord channel https://discord.gg/hnYRq7DsQh :)
• Added envpool support with a few atari examples. Works 3-4x time faster than ray.
• Added mujoco results. Much better than openai spinning up ppo results.
• Added tcnn(https://github.com/NVlabs/tiny-cuda-nn) support. Reduces 5-10% of training time in the IsaacGym envs.
• Various fixes and improvements.

1.3.2

• Added 'sigma' command line parameter. Will override sigma for continuous space in case if fixed_sigma is True.

1.3.1

• Fixed SAC not working

1.3.0

• Simplified rnn implementation. Works a little bit slower but much more stable.
• Now central value can be non-rnn if policy is rnn.
• Removed load_checkpoint from the yaml file. now --checkpoint works for both train and play.

1.2.0

• Added Swish (SILU) and GELU activations, it can improve Isaac Gym results for some of the envs.
• Removed tensorflow and made initial cleanup of the old/unused code.
• Simplified runner.
• Now networks are created in the algos with load_network method.

1.1.4

• Fixed crash in a play (test) mode in player, when simulation and rl_devices are not the same.
• Fixed variuos multi gpu errors.

1.1.3

• Fixed crash when running single Isaac Gym environment in a play (test) mode.
• Added config parameter clip_actions for switching off internal action clipping and rescaling

1.1.0

• Added to pypi: pip install rl-games
• Added reporting env (sim) step fps, without policy inference. Improved naming.
• Renames in yaml config for better readability: steps_num to horizon_length amd lr_threshold to kl_threshold

• Some of the supported envs are not installed with setup.py, you need to manually install them
• Starting from rl-games 1.1.0 old yaml configs won't be compatible with the new version:
  • steps_num should be changed to horizon_length amd lr_threshold to kl_threshold

• Running a single environment with Isaac Gym can cause crash, if it happens switch to at least 2 environments simulated in parallel