# Agent Workspace Log ## Project Overview This project studies PPO/GAE degradation under aerodynamic turbulence for a linear 2D UAV control problem. The original repository provided a NumPy/SciPy analytical Monte Carlo experiment using double-integrator UAV dynamics, an LQR controller, and an exact Riccati value function to show that GAE bias scales approximately as `C sigma^2`. The workspace has now been extended into a Stable-Baselines3 PPO research framework that can train learned policies at multiple turbulence intensities and test whether PPO performance degradation matches the same quadratic law. ## Chat History & Context Summary - User requested a complete PPO-based research framework, not just the existing analytical GAE-bias experiment. - User specified the environment state `[x, vx, y, vy]`, action `[ax, ay]`, dynamics `s_{t+1} = A s_t + B a_t + E w_t`, episode length 200, reward `-(s^T Q s + a^T R a)`, `Q = I`, and `R = 0.1 I`. - User requested PPO with Stable-Baselines3, Tanh MLP networks `64 -> 64` for policy and value functions, one million training timesteps, seeded runs, and the turbulence ladder `[0.00, 0.05, 0.10, 0.20, 0.30, 0.50, 0.80, 1.00]`. - The existing `run_experiment.py`, `make_plots.py`, `README.md`, and `requirements.txt` were inspected. The dynamics were confirmed to match the paper: double-integrator matrices with scalar turbulence injected into `vy` through `E = [0, 0, 0, 1]`. - The provided `IJERT_May26_Sahil.pdf` was briefly extracted with `pypdf` to confirm the paper's framing and quadratic-bias hypothesis. The PDF was left unmodified and remains untracked. - User later reported a `git push` non-fast-forward rejection followed by `git pull origin` and a successful second `git push`. - User attached a smoke-test run log showing that PPO training completed but `--total-timesteps 5000` ran to 6,144 timesteps because Stable-Baselines3 rounded up to full 2,048-step rollouts. The analysis completed, but the log-log plot was skipped because the smoke test used only one nonzero sigma. ## Work Accomplished - Created `config.py` with dataclass-based experiment, PPO, dynamics, and analysis configuration. - Created `envs/__init__.py` and `envs/uav_env.py` implementing `UAVTurbulenceEnv` as a Gymnasium environment. - Created `training/__init__.py` and `training/train_ppo.py` implementing seeded Stable-Baselines3 PPO training, model saving, checkpointing, logging, progress bars, and training-metric CSV output. - Created `run_training.py` as the top-level training CLI. - Created `analysis/__init__.py`, `analysis/compute_gae_bias.py`, `analysis/evaluate_models.py`, and `analysis/fit_models.py`. - Implemented manual trajectory collection, critic-value evaluation, TD residual computation, GAE computation, nominal-vs-disturbed advantage-error comparison, PPO evaluation, loss/KL extraction, quadratic bias fitting, and performance-drop power-law fitting. - Created `run_analysis.py` to run evaluation, GAE-bias computation, model fitting, and figure generation. - Replaced `make_plots.py` with dynamic plotting from PPO analysis CSVs. It generates advantage-bias, return, bias-return, log-log, residual, and training-curve figures as PNG and PDF. - Updated `requirements.txt` to include PyTorch, Gymnasium, Stable-Baselines3, Pandas, and tqdm. - Updated `README.md` with installation, training, analysis, figure generation, repository structure, and original analytical experiment notes. - Updated `.gitignore` to ignore generated PPO models, result CSVs, training logs, and new generated PPO figures. - Ran an in-memory Python syntax check across the new Python files; it passed. - Verified imports for modules that do not require the newly added RL dependencies. - Ran a synthetic CSV smoke test through `analysis.fit_models` and `make_plots.generate_all_plots`; the fit/plot path succeeded. Training-curve plotting was skipped in that synthetic test because no PPO logs existed. - Cleaned up temporary smoke-test outputs and generated `__pycache__` files. - Checked Git state after the reported push issue. Local `main` and `origin/main` both point to commit `ddf304f`, and the working tree is clean. - Added an exact timestep stop callback to PPO training so short runs stop at the requested timestep budget. - Added training CLI overrides for `--n-steps`, `--batch-size`, and `--n-epochs` to make smoke tests much faster without changing the full research defaults. - Updated README smoke-test commands to use sigma values `0.0 0.1 0.2` and lightweight PPO settings so the log-log plot has at least two positive sigma points. ## Key Decisions - Preserved the original analytical experiment in `run_experiment.py`. - Matched the original UAV dynamics exactly, including disturbance injection into the `vy` channel. - Used `Stable-Baselines3 PPO` with separate models per turbulence level rather than one domain-randomized policy, matching the requested experimental design. - Used a frozen dataclass configuration in `config.py` so defaults are reproducible and CLI overrides are still available for quick smoke tests. - Used a nominal policy at `sigma=0.0` by default for GAE-bias comparison, with `--policy-sigma` available for alternative analysis. - Defined success as final state norm within `success_radius = 0.5`, recorded through the environment `info` dictionary. - Stored final PPO loss/KL summaries in `results/ppo_evaluation.csv` and per-sigma training metrics in `results/training_metrics_sigma_*.csv`. ## Next Steps - Install dependencies with `pip install -r requirements.txt`. - Run a short PPO smoke test, for example: `python run_training.py --sigmas 0.0 0.1 --total-timesteps 5000`. - Run a short analysis smoke test after those models exist: `python run_analysis.py --sigmas 0.0 0.1 --eval-episodes 3 --gae-episodes 5`. - Run the full research sweep with `python run_training.py`, then `python run_analysis.py`. - Inspect `results/fit_statistics.csv` to determine whether learned PPO advantage bias is quadratic and whether PPO performance drop has exponent near 2 or differs from the analytical GAE law. - No Git repair is currently needed for the reported push issue; future non-fast-forward rejections should be handled by pulling/rebasing remote changes before pushing. - Re-run the smoke test with: `python run_training.py --sigmas 0.0 0.1 0.2 --total-timesteps 5000 --n-steps 512 --n-epochs 2` followed by: `python run_analysis.py --sigmas 0.0 0.1 0.2 --eval-episodes 3 --gae-episodes 5`.