You can deploy this model and start earning money today!

kabiriitb/
ppo_advantage_estimator

Name: ppo_advantage_estimator
Availability: InStock

Views

No views yet

PPO Advantage Bias Under UAV Turbulence

This repository studies whether PPO performance degradation under UAV turbulence follows the same quadratic scaling law previously observed for Generalized Advantage Estimation (GAE) bias.

The original NumPy/SciPy experiment is still available in run_experiment.py. It uses a linear UAV model, an LQR controller, the exact Riccati value function, and Monte Carlo simulations to show that advantage-estimation error scales as C sigma^2. The repository now also contains a complete Stable-Baselines3 PPO training and analysis pipeline for testing the same hypothesis with learned policies.

Research Question

Does PPO performance degradation under turbulence follow the same quadratic law as GAE bias?

The framework trains independent PPO policies for:

sigma = [0.00, 0.05, 0.10, 0.20, 0.30, 0.50, 0.80, 1.00]

For each turbulence level it measures:

Average episode return
Success rate
Policy loss
Value loss
Approximate KL divergence
Manual GAE advantage bias

It then fits:

Bias = C sigma^2
Performance Drop = K sigma^p

and reports C, R^2, K, and the performance exponent p.

Environment

The Gymnasium environment is envs/uav_env.py::UAVTurbulenceEnv.

State:

[x, vx, y, vy]

Action:

[ax, ay]

Dynamics:

text

s_{t+1} = A s_t + B a_t + E w_t
w_t ~ N(0, sigma^2)

The matrices match the original double-integrator experiment:

text

A = [[1, dt, 0,  0 ],
     [0, 1,  0,  0 ],
     [0, 0,  1,  dt],
     [0, 0,  0,  1 ]]

B = [[0,  0 ],
     [dt, 0 ],
     [0,  0 ],
     [0,  dt]]

E = [0, 0, 0, 1]

Reward:

text

r = -(s^T Q s + a^T R a)
Q = I
R = 0.1 I

Episode length is 200 steps.

Repository Structure

text

.
|-- envs/
|   |-- __init__.py
|   `-- uav_env.py
|-- training/
|   |-- __init__.py
|   `-- train_ppo.py
|-- analysis/
|   |-- __init__.py
|   |-- compute_gae_bias.py
|   |-- evaluate_models.py
|   `-- fit_models.py
|-- figures/
|-- models/
|-- results/
|-- config.py
|-- run_training.py
|-- run_analysis.py
|-- run_experiment.py
|-- make_plots.py
|-- README.md
`-- requirements.txt

Installation

Use Python 3.10 or newer.

pip install -r requirements.txt

The implementation is CPU-compatible. Stable-Baselines3 will use device="cpu" by default.

Train PPO Models

Run the full research training sweep:

python run_training.py

Default PPO settings are in config.py:

Timesteps per sigma: 1,000,000
Policy network: 64 -> 64
Value network: 64 -> 64
Activation: Tanh
gamma = 0.99
gae_lambda = 0.95
Seed: 0

Models are saved automatically under models/.

For a quick smoke test:

python run_training.py --sigmas 0.0 0.1 0.2 --total-timesteps 5000 --n-steps 512 --n-epochs 2

The smaller rollout length and epoch count make the smoke test much faster than the full research configuration. The training runner also stops at the requested timestep budget instead of letting Stable-Baselines3 round up to the next full rollout.

Run Analysis

After training, run:

python run_analysis.py

This writes:

results/ppo_evaluation.csv
results/gae_bias.csv
results/fit_statistics.csv
results/model_residuals.csv

For faster analysis while debugging:

python run_analysis.py --sigmas 0.0 0.1 0.2 --eval-episodes 3 --gae-episodes 5

Use at least two nonzero sigma values if you want the log-log bias plot to be generated during a smoke test.

Generate Figures

run_analysis.py generates all plots automatically. You can regenerate figures from existing result CSVs with:

python make_plots.py

Figures are saved in figures/ as both PNG and PDF:

fig_advantage_bias_vs_sigma
fig_ppo_return_vs_sigma
fig_advantage_bias_vs_return
fig_loglog_bias
fig_residual_analysis
fig_training_curves

Original Analytical Reproduction

The original paper-style numerical experiment is preserved:

python run_experiment.py

The analytical script still writes its original text and CSV outputs:

results_table.csv
regression_stats.txt
critical_threshold.txt

The pre-rendered analytical figures from the earlier repository remain in figures/. The updated make_plots.py is now dedicated to the PPO analysis CSVs produced by run_analysis.py.

Citation

If you use this code, please cite:

bibtex

@article{patil2026quadratic,
  title   = {A Quadratic Bound on PPO Advantage Bias Under UAV Turbulence},
  author  = {Sahil, Nasiruddin},
  journal = {International Journal of Engineering Research \& Technology (IJERT)},
  volume  = {15},
  number  = {05},
  year    = {2026}
}

kabiriitb/ppo_advantage_estimator

PPO Advantage Bias Under UAV Turbulence

Research Question

Environment

Repository Structure

Installation

Train PPO Models

Run Analysis

Generate Figures

Original Analytical Reproduction

Citation

kabiriitb/
ppo_advantage_estimator