Lunar Lander Deep Q-Learning Model

NOTE: I used only 10 episodes for the purpose of this video. I recommend using at least 500 episodes for better accuracy.

Table of Contents

• General Information
• Features
• Tools and Technologies
• Setup
• Usage
• How It Works
• Adjusting Hyperparameters
• Using the Trained Model
• Credits

General Information

• A fully implemented DQN algorithm.
• Real-time visualization of the training process using Pygame.
• Dynamic plotting of training progress using Matplotlib.

Features

• Deep Reinforcement Learning:
  • Neural networks approximate the Q-function.
  • Implements experience replay and a target network for stability.
• Interactive Visualization:
  • Displays the Lunar Lander environment in real-time using Pygame.
  • Dynamically plots training progress alongside the environment.
• Customizable Training:
  • Adjustable hyperparameters, including learning rate, batch size, discount factor, and more.
• Environment Solving:
  • Trains the agent to achieve an average score of 200 over 100 consecutive episodes.

Tools and Technologies

Libraries and Frameworks

• Reinforcement Learning:
  • PyTorch (for building and training the neural network)
  • NumPy (for efficient numerical computations)
• Visualization:
  • Pygame (for real-time visualization of the environment)
  • Matplotlib (for plotting training progress)
• Environment:
  • OpenAI Gym (Lunar Lander environment)

Hardware Support

• CUDA support for GPU acceleration.

Setup

Prerequisites

• Python 3.8 or higher
• A compatible GPU (optional but recommended for faster training)

Installation

1. Clone the Repository:
   bash

   1git clone https://github.com/yourusername/lunar-lander-dqn.git 2cd lunar-lander-dqn 3
2. Install Required Packages:
   bash

   1pip install -r requirements.txt 2
3. Verify Installation: Ensure that PyTorch is installed with CUDA support if you plan to use a GPU.

Usage

Training the Agent

1. Install Required Packages:
   bash

   1python main.py 2
2. Monitor Training
   • View real-time rendering of the Lunar Lander environment.
   • Observe the dynamic plot of training scores as the agent learns.
3. Save Model
   • The trained model is automatically saved as checkpoint.pth when the environment is solved (average score ≥ 200 over 100 episodes).

How It Works

1. Q-Network Architecture
   • A feedforward neural network with 2 hidden layers of 64 neurons each
   • Input: Current state of the environment
   • Output: Q-values for all possible actions
2. Target Network
   • Maintains a separate network to compute target Q-values.
   • Samples mini-batches for training to break correlations and stabilize learning.
3. Experience Replay
   • Stores past experiences in a replay buffer.
   • Updated periodically to stabilize training.
4. Epsilon-Greedy Policy
   • Balances exploration and exploitation.
   • Decays epsilon over time to focus on exploitation as training progresses.

Adjusting Hyperaramenters

Using the Trained Model

1. Load the Trained Model: Update your script to load the model:
   bash

   1agent.qnetwork_local.load_state_dict(torch.load('checkpoint.pth')) 2
2. Run Inference: Use the agent.act() function to make decisions for the Lunar Lander environment:
   bash

   1state = env.reset() 2done = False 3while not done: 4 action = agent.act(state) 5 next_state, reward, done, truncated, _ = env.step(action) 6 state = next_state

Credits