A policy-gradient agent with a value baseline + entropy bonus that solves the Gymnasium CartPole-v1 benchmark (greedy return near the 500 maximum).
Trained from scratch in Ropedia Academy — an interactive, bilingual course on embodied & spatial AI. Educational model: small and quick to train; the value is the method and a reproducible pipeline, not a leaderboard score. Try it live in the Ropedia demos Space.
At a glance
Base model
Trained from scratch (random initialization) — no pretrained base model.
Task
reinforcement learning
Training objective
REINFORCE policy gradient with a value baseline + entropy bonus (actor-critic).
Type: standard RL benchmark environment (no fixed dataset)
Size / stats: 4-D continuous state, 2 discrete actions; reward +1/step, episode cap 500; agent learns from its own rollouts
Split: online RL; greedy eval over 20 episodes
Source: Gymnasium (Farama Foundation) CartPole-v1
Training config
Actor-critic; Adam (lr 3e-3), 600 episodes, γ=0.99, entropy 0.01, value baseline. EPISODES env-overridable.
Evaluation results
metric
value
meaning
return (final)
452.7
greedy_eval
449.45
mean return of the greedy policy over 20 CartPole episodes (max 500)
figure
Robustness (mean ± std over 5 seeds)
Single-run numbers above are one seed; this is the distribution over independent re-trains (honest variance — no cherry-picking).
metric
mean ± std
greedy_eval
461.8 ± 76
seeds
Inference example
python
1import torch, torch.nn as nn
2policy = nn.Sequential(nn.Linear(4,64), nn.Tanh(), nn.Linear(64,2))3policy.load_state_dict(torch.load("policy.pt", map_location="cpu")); policy.eval()4# CartPole state s = [x, x_dot, theta, theta_dot]5# action = int(policy(torch.tensor(s, dtype=torch.float32)).argmax()) # 0 = push left, 1 = push right
Limitations
Educational scale. Trained quickly on CPU on small or synthetic data, so absolute numbers are not competitive with production systems — the value is the method and a reproducible pipeline. No large-scale data, no hyperparameter sweep, and no multi-seed variance is reported. Not for production use.
Solves CartPole only — not a general control policy; on-policy and high-variance.
Failure cases
High-variance gradients; a length-1 episode gives a NaN advantage (std 0) — fixed with unbiased=False + grad clipping; stalls without the entropy bonus.
Reproduce / train your own
One click: open the notebook in Colab → Runtime → GPU → Run all, then run its Publish to the Hugging Face Hub cell.
