NeatRL provides readable implementations of popular RL algorithms with a focus on simplicity and ease of use. Built with PyTorch and designed for research and experimentation.
-
DQN (Deep Q-Network) - Classic value-based RL algorithm
- Support for discrete action spaces
- Experience replay and target networks
- Atari preprocessing and frame stacking
-
Dueling DQN - Enhanced DQN with separate value and advantage streams
- Improved learning stability
- Better performance on complex environments
-
REINFORCE - Policy gradient method for discrete and continuous action spaces
- Atari game support with automatic CNN architecture
- Parallel environment training (
n_envs support)
- Continuous action space support
- Episode-based Monte Carlo returns
- Variance reduction through baseline subtraction
-
DDPG (Deep Deterministic Policy Gradient) - Actor-critic method for continuous action spaces
- Deterministic policy gradient for continuous control
- Experience replay and target networks
- Ornstein-Uhlenbeck noise for exploration
- Support for exact continuous action spaces
-
A2C (Advantage Actor-Critic) - Synchronous actor-critic algorithm
- Advantage function for reduced variance
- Support for both discrete and continuous action spaces
- Parallel environment training with vectorized environments
- Monte Carlo returns for value estimation
-
TD3 (Twin Delayed DDPG) - Actor-critic method for continuous control
- Twin Q-networks to reduce overestimation bias
- Delayed policy updates for improved stability
- Target policy smoothing with noise
- Experience replay and target networks
- CNN support for image-based environments
-
SAC (Soft Actor-Critic) - Maximum entropy reinforcement learning
- Stochastic Gaussian policies with entropy regularization
- Twin Q-networks for stable learning
- Automatic entropy tuning (alpha parameter)
- Balances exploration and exploitation
- CNN support for complex environments
-
PPO (Proximal Policy Optimization) - Policy gradient method with GAE
- Full PPO implementation with Generalized Advantage Estimation (GAE)
- Support for both discrete and continuous action spaces
- Atari game support with automatic CNN architecture
- Clipped surrogate objective for stable policy updates
- Value function clipping and entropy regularization
- Vectorized environments for parallel training
-
PPO-RND (Proximal Policy Optimization with Random Network Distillation) - Curiosity-driven exploration
- Intrinsic motivation through novelty detection
- Combined extrinsic and intrinsic rewards for better exploration
- Support for both discrete and continuous action spaces
- PPO with clipped surrogate objective
- Vectorized environments for parallel training
- Intrinsic reward normalization and advantage calculation
-
More algorithms coming soon...
1python -m venv neatrl-env
2source neatrl-env/bin/activate
3
4pip install neatrl
5
6# Install extras based on environments you want to use
7pip install neatrl[atari] # For Atari games
8pip install neatrl[box2d] # For BipedalWalker
9pip install neatrl[classic] # For Pendulum
10pip install neatrl[mujoco] # For HalfCheetah
1from neatrl import train_dqn
2
3model = train_dqn(
4 env_id="CartPole-v1",
5 total_timesteps=10000,
6 seed=42
7)
1from neatrl import train_ppo
2
3model = train_ppo(
4 env_id="CartPole-v1",
5 total_timesteps=50000,
6 n_envs=4, # Parallel environments
7 GAE=0.95, # Generalized Advantage Estimation lambda
8 clip_value=0.2, # PPO clipping parameter
9 use_wandb=True, # Track with WandB
10 seed=42
11)
1from neatrl import train_sac
2
3model = train_sac(
4 env_id="Pendulum-v1",
5 total_timesteps=50000,
6 alpha=0.2, # Entropy regularization coefficient
7 autotune_alpha=True, # Automatically tune alpha
8 use_wandb=True, # Track with WandB
9 seed=42
10)
1from neatrl import train_sac_cnn
2
3model = train_sac_cnn(
4 env_id="BreakoutNoFrameskip-v4",
5 total_timesteps=100000,
6 alpha=0.2,
7 autotune_alpha=True,
8 atari_wrapper=True, # Automatic Atari preprocessing
9 use_wandb=True,
10 seed=42
11)
Contributions are welcome. Please feel free to submit a Pull Request.
1git clone https://github.com/YuvrajSingh-mist/NeatRL.git
2cd NeatRL
3pip install -e .[dev]
For the complete changelog, see
CHANGELOG.md.
This project is licensed under the MIT License - see the
LICENSE file for details.
