Views
No views yet
# Vanilla DQN (overestimates):
next_q = q_target(s').max()
# Double DQN (this model):
best_a = q_online(s').argmax() # online picks action
next_q = q_target(s')[best_a] # target evaluates it
target = r + γ * (1 - done) * next_q| Step | Detail |
|---|---|
| Grayscale | cv2.cvtColor RGB → gray |
| Resize | 210×160 → 84×84 (INTER_AREA) |
| Frame stacking | 4 consecutive frames → state (4, 84, 84) |
| Normalization | uint8 stored in buffer, ÷255 at sample time |
| Reward clipping | Clipped to [−1, +1] during training |
| Frameskip | 4 (ALE built-in) |
| Fire-reset | FIRE pressed after reset and after every life loss |
Input: (4, 84, 84)
Conv2d(4→32, kernel=8, stride=4) → ReLU
Conv2d(32→64, kernel=4, stride=2) → ReLU
Conv2d(64→64, kernel=3, stride=1) → ReLU
Flatten → 3136
Linear(3136 → 512) → ReLU
Linear(512 → 4)
Output: Q(s,a) for [NOOP, FIRE, RIGHT, LEFT]| Parameter | Value |
|---|---|
| Environment | ALE/Breakout-v5 |
| Total steps | 750,000 |
| Parallel envs | 4 (AsyncVectorEnv) |
| Replay buffer | 100,000 (uint8, 4× memory saving) |
| Batch size | 64 |
| Learning rate | 1e-4 (Adam) |
| Discount γ | 0.99 |
| Target sync | Hard update every 1,000 steps |
| Gradient clip | Max norm 10 |
| Epsilon | 1.0 → 0.01 over first 5% of steps |
| Training start | After 5,000 steps |
1import torch
2import torch.nn as nn
3
4class DQN(nn.Module):
5 def __init__(self, n_actions=4):
6 super().__init__()
7 self.model = nn.Sequential(
8 nn.Conv2d(4, 32, kernel_size=8, stride=4), nn.ReLU(),
9 nn.Conv2d(32, 64, kernel_size=4, stride=2), nn.ReLU(),
10 nn.Conv2d(64, 64, kernel_size=3, stride=1), nn.ReLU(),
11 nn.Flatten(),
12 nn.Linear(3136, 512), nn.ReLU(),
13 nn.Linear(512, n_actions),
14 )
15 def forward(self, x):
16 return self.model(x)
17
18checkpoint = torch.load('best_breakout.pt')
19model = DQN()
20model.load_state_dict(checkpoint['model'])
21model.eval()