You can deploy this model and start earning money today!

epsill0n/
SB3-PPO-PhoenixNoFrameskip-v4-50M

Name: SB3-PPO-PhoenixNoFrameskip-v4-50M
Availability: InStock

tensorboard

Views

No views yet

PPO Agent for Atari PhoenixNoFrameskip-v4 (50M timesteps)

License: MIT

This repository hosts a PPO agent trained on PhoenixNoFrameskip-v4 for 50 million timesteps using Stable-Baselines3.

Repository Contents

.
├── train_phoenix.py        # Training script
├── evaluate_phoenix.py     # Evaluation script
├── requirements.txt        # Dependencies
├── LICENSE                 # MIT License
├── README.md
├── model_card.md
├── checkpoints/            # Trained model files (all steps)
├── config/
│ └── hyperparams.json
├── assets/
│ ├── charts/               # Training charts: reward, policy/loss, value, entropy
│ └── eval_video.mp4        # Demo video of trained agent
└── tensorboard/

Model Checkpoints

Saved every 5M timesteps:

Checkpoint	Timesteps	File
5M	5,000,000	`checkpoints/ppo_phoenix_5000000_steps.zip`
10M	10,000,000	`checkpoints/ppo_phoenix_10000000_steps.zip`
15M	15,000,000	`checkpoints/ppo_phoenix_15000000_steps.zip`
20M	20,000,000	`checkpoints/ppo_phoenix_20000000_steps.zip`
25M	25,000,000	`checkpoints/ppo_phoenix_25000000_steps.zip`
30M	30,000,000	`checkpoints/ppo_phoenix_30000000_steps.zip`
35M	35,000,000	`checkpoints/ppo_phoenix_35000000_steps.zip`
40M	40,000,000	`checkpoints/ppo_phoenix_40000000_steps.zip`
45M	45,000,000	`checkpoints/ppo_phoenix_45000000_steps.zip`
50M	50,000,000	`checkpoints/ppo_phoenix_50000000_steps.zip`

Hyperparameters

See config/hyperparams.json. The training script uses these values:

json

{
  "learning_rate": "2.5e-4 * linear_schedule",
  "n_steps": 2048,
  "batch_size": 256,
  "n_epochs": 4,
  "gamma": 0.99,
  "gae_lambda": 0.95,
  "clip_range": 0.2,
  "ent_coef": 0.005,
  "vf_coef": 0.5,
  "max_grad_norm": 0.5,
  "frame_stack": 4,
  "use_sde": false
}

Training

You can train the PPO agent from scratch or continue training from an existing checkpoint using train_phoenix.py.

Run:

bash

pip install -r requirements.txt
python train_phoenix.py

Key points about training:

Uses Stable-Baselines3 PPO with CnnPolicy.
Environment: PhoenixNoFrameskip-v4 (vectorized with VecFrameStack for 4 frames).
Default device: GPU if available (torch.cuda), otherwise CPU.
Total target timesteps: 50 million.
Training checkpoints are automatically saved every 500,000 steps in SAVE_PATH (ppo_phoenix_model by default).
If training is interrupted (e.g., keyboard interrupt), the model and training metadata are saved automatically every 100,000 steps via SaveOnInterruptCallback.
Hyperparameters are defined in the script and partially mirrored in config/hyperparams.json.
TensorBoard logs are written to ./tensorboard/ for monitoring metrics like reward, policy loss, value loss, and entropy.

Training Hardware

The model was trained on:

CPU: Intel Core i5-13600KF
GPU: NVIDIA GeForce RTX 3060 12GB
RAM: 32 GB DDR5 6000MHz CL30
Storage: NVMe SSD
OS: openSUSE Tumbleweed

Training Charts

Located in assets/charts/:

ppo_ep_rew_mean.png - Mean episode reward
ppo_policy_loss.png - Policy loss
ppo_value_loss.png - Value loss
ppo_entropy_loss.png - Entropy loss
ppo_total_loss.png - Total loss

Evaluation / Inference

To evaluate the trained agent, use evaluate_phoenix.py. This script allows you to load any checkpoint and run it for multiple episodes.

Example usage:

bash

python evaluate_phoenix.py \
    --model_path checkpoints/ppo_phoenix_50000000_steps.zip \
    --num_episodes 20 \
    --render

Options:

--model_path: Path to a trained model checkpoint (default: checkpoints/ppo_phoenix_50000000_steps.zip).
--num_episodes: Number of episodes to evaluate (default: 20).
--render: Enable rendering of the environment in human mode.
--use_monitor: Wrap the environment with Gym Monitor to record episode metrics.

Features:

Handles both Gymnasium and Gym outputs.
Returns:
- Mean and standard deviation of rewards over all episodes.
- Mean and standard deviation of episode lengths.
Can be used with all checkpoints to compare performance over training.

Minimal Python usage example:

python

from stable_baselines3 import PPO
import gymnasium as gym

from evaluate_phoenix import make_eval_env, evaluate_model

env = make_eval_env(render=False, use_monitor=True)
model = PPO.load("checkpoints/ppo_phoenix_25000000_steps.zip", env)

mean_reward, std_reward, mean_length, std_length = evaluate_model(model, env, num_episodes=20)

Demo Video

Short agent demo: assets/eval_video.mp4

License

MIT License (see LICENSE). Atari Phoenix game ROMs are not included. Install via gymnasium[atari] or ale-py.

Contact

Maintainer: Aleksandar Nikoloski

Hugging Face: @epsill0n

epsill0n/SB3-PPO-PhoenixNoFrameskip-v4-50M