PPO Agent Playing LunarLander-v3

Some tips about implementing ppo for LunarLander from scratch

• PPO on LunarLander is non-monotonic. A run can reach a great policy (reward 200+) mid-training and then destabilize and get worse by the end. Saving/evaluating only the final checkpoint is a real risk, not a hypothetical one — it's exactly what caused the bad video here. The fix: evaluate periodically during training and keep the best checkpoint seen, not just the last one.
• A single video is one sample, not the mean. If actions are sampled stochastically (probs.sample()) rather than greedily (argmax), even a genuinely decent policy can crash in any individual episode. A good mean eval reward does not guarantee a good-looking replay video.
• Variance can dominate over hyperparameters. I tried ~10 combinations (different seeds, larger batch sizes via more parallel envs, KL early stopping, truncation bootstrapping) and none reliably reached the "solved" ~200 mean reward — some "tuned" configs did consistently worse than the plain defaults. Don't assume a bigger/tuned run is automatically better; verify with real eval numbers over enough episodes (10 is noisy; ~30+ gives a more honest picture of the variance).
• Correctness fixes (e.g. bootstrapping value estimates on time-limit truncation) matter for principle but aren't a silver bullet for training instability — don't expect one flag to fix a high-variance policy.

Hyperparameters