| # Multi-agent RL navigation — trustworthy autonomous systems (simulation only) |
|
|
| **Repo: `dummy033485/multi-agent-rl-navigation`** |
| **Status: model repo live. No Hugging Face Space has been created.** |
|
|
| A shared decentralized PPO policy for 2D multi-agent navigation, compared |
| honestly against a textbook classical controller on the *same* seeded |
| layouts. Built as a portfolio project for an application to Prof. Jiachen |
| Li's Trustworthy Autonomous Systems Lab (Georgia Tech): robotics, |
| trustworthy AI/ML, reinforcement learning, control, optimization and |
| multi-agent systems. |
|
|
| ## What this is — and is not |
|
|
| - **Is:** a self-contained 2D physics simulator (written for this project, |
| no external environment, no hardware, no proprietary data) + a learned |
| shared policy + a classical baseline + a fixed, seeded evaluation. |
| - **Is not:** a robot result. No hardware was used. The RL policy has **no |
| formal safety guarantee** — every safety number below is empirical, |
| measured on the simulator's fixed test layouts. The wall-trap layout is |
| deliberately constructed to defeat a pure potential-field controller. |
|
|
| ## Simulator (`sim.py`) |
|
|
| - Arena 12x12 with walls. Agents: double-integrator kinematics, |
| `dt=0.1`, max speed 1.2, max acceleration 3.0, agent radius 0.25, |
| episode cap 200 steps. |
| - Each layout: 3 circle obstacles (radius 0.5–0.8) **and** 1 wall/barrier |
| segment. Two layout families, generated with fixed seeds: |
| - *general*: a short (3 m) barrier segment at a random mid-arena angle. |
| - *trap*: a vertical segment at x=6, y in [2,10], with gaps only at the |
| ends; starts on the left, goals near-directly opposite on the right, |
| so a direct potential field is attracted right and repelled left and |
| stalls in the middle. |
| - Decentralized 20-dim observation per agent: own velocity, relative goal |
| vector + distance, nearest circle obstacle (direction, surface distance, |
| radius), nearest wall segment (direction, distance, both endpoint |
| vectors), nearest other agent (direction, distance), nearest arena-wall |
| distance. (Endpoint vectors were added after a failed training run: |
| with only the nearest segment point, the policy could not observe where |
| the wall ends — see Limitations and `BUILD_REPORT.md`.) |
| - Collision = distance to an obstacle/wall segment/arena wall below the |
| agent radius, or agent–agent distance below 0.5. A colliding arrival is |
| never counted as a success. Safety margin for violation counting: 0.15. |
| |
| ## Policies |
|
|
| - **Classical** (`policies.py`, exact gains documented in code): PD |
| controller toward the goal (KD=2.0) + radial repulsive potential field |
| (ETA=6.0, influence distance D0=2.0) from the nearest obstacle, segment |
| and neighbour. No learning, no tangential escape trick — the honest |
| textbook baseline. It stalls (timeout, 0 collisions) on every trap layout. |
| - **RL**: shared decentralized PPO, MLP 20→128→128 (the observation grew |
| from 15 to 20 dims when segment-endpoint features were added), |
| Gaussian actions, tanh-bounded to the acceleration limit. One set of |
| weights drives every agent from its own observation. |
| Training recipe (all in `train.py`, PPO from scratch with torch only): |
| the policy was warm-started by behaviour cloning the classical controller |
| on general layouts only (400 episodes, 4 agents, final action MSE 0.079), |
| then PPO fine-tuned in four phases — 1-agent mixed (trap ramp), 1-agent |
| trap-only with an exploration reset, 1-agent mixed consolidation, and |
| 4-agent mixed — 300 iterations × 128-step rollouts × 16 envs per phase |
| (614,400 environment steps per phase; 2,457,600 in the kept chain plus |
| the BC dataset). Earlier from-scratch PPO runs failed outright (success |
| 5–15%) and are documented, with their fixes, in `BUILD_REPORT.md` — |
| they are not hidden. The final phase's curve is `training_curve.csv` |
| (earlier phases have their own `training_curve_*.csv` files). |
|
|
| ## Evaluation protocol (exact) |
|
|
| Fixed layouts, identical for every policy and condition: |
| - Single-agent: seeds 1000–1019 (10 general + 10 trap, alternating). |
| - 4-agent: seeds 2000–2011 (6 general + 6 trap, alternating). |
| - RL evaluated with the deterministic (mean) action; classical is |
| deterministic. Noise conditions reuse the same noise RNG seed per |
| layout/condition for both policies. |
| - Conditions: clean; observation noise σ=0.05; observation noise σ=0.10; |
| dynamics disturbance (added acceleration noise σ=1.0); obstacle shift |
| +0.5 (circles and segments shifted +0.5 in x). |
| - Metrics are per agent-episode (20 single-agent and 48 4-agent episodes |
| per policy per condition; 728 rows in `eval.csv`). Steps-to-goal and |
| clearance are reported on successes only; safety-violation rate is the |
| fraction of active steps with clearance < 0.15. |
|
|
| ## Results (copied from `results.json` — no other numbers are claimed) |
|
|
| ### Clean |
|
|
| | Policy | Agents | Success | Collision | Obstacle coll. | Agent–agent coll. | Timeout | Steps (succ.) | Path len. | Min clearance (succ.) | Safety-viol. rate | General | Trap | |
| |---|---|---|---|---|---|---|---|---|---|---|---|---| |
| | Classical | 1 | 0.450 | 0.000 | 0.000 | 0.000 | 0.550 | 73.22 | 5.35 | 0.588 | 0.000 | 0.90 | 0.00 | |
| | RL | 1 | 0.700 | 0.200 | 0.200 | 0.000 | 0.100 | 86.00 | 9.59 | 0.419 | 0.033 | 0.60 | 0.80 | |
| | Classical | 4 | 0.396 | 0.000 | 0.000 | 0.000 | 0.604 | 160.42 | 5.87 | 0.690 | 0.000 | 0.79 | 0.00 | |
| | RL | 4 | 0.792 | 0.146 | 0.146 | 0.000 | 0.062 | 141.18 | 10.63 | 0.505 | 0.019 | 0.75 | 0.83 | |
|
|
| Reading, honestly: the RL policy wins on success everywhere, and is the |
| only policy that solves trap layouts or coordinates 4 agents (zero |
| agent–agent collisions clean). It loses on safety in the single-agent |
| case: 20% collisions and lower clearance than the classical controller's |
| 0% collisions. The classical controller never collides clean — it fails |
| by timing out at the wall instead. RL paths are longer (detours). |
|
|
| ### Robustness (success / collision) |
|
|
| | Condition | RL 1-agent | Classical 1-agent | RL 4-agent | Classical 4-agent | |
| |---|---|---|---|---| |
| | Clean | 0.700 / 0.200 | 0.450 / 0.000 | 0.792 / 0.146 | 0.396 / 0.000 | |
| | Obs. noise σ=0.05 | 0.750 / 0.150 | 0.450 / 0.100 | 0.646 / 0.292 | 0.333 / 0.125 | |
| | Obs. noise σ=0.10 | 0.600 / 0.300 | 0.200 / 0.250 | 0.542 / 0.396 | 0.104 / 0.396 | |
| | Dynamics disturbance σ=1.0 | 0.650 / 0.200 | 0.350 / 0.000 | 0.771 / 0.167 | 0.375 / 0.000 | |
| | Obstacle shift +0.5 | 0.550 / 0.300 | 0.350 / 0.000 | 0.708 / 0.250 | 0.375 / 0.000 | |
|
|
| The RL policy degrades gracefully on success under every perturbation and |
| stays well above classical, but its collision rate rises under noise |
| (up to 0.396 at σ=0.10, 4-agent) — the trustworthy claim is *measured |
| robustness of success*, not a safety proof. |
|
|
| ### Ablation — neighbour observation, 4-agent clean |
|
|
| | Policy | Success | Collision | Agent–agent collision | |
| |---|---|---|---| |
| | RL with neighbour observation | 0.792 | 0.146 | 0.000 | |
| | RL with neighbour observation zeroed | 0.146 | 0.708 | 0.333 | |
|
|
| Removing the neighbour observation collapses coordination: success falls |
| from 0.792 to 0.146 and agent–agent collisions jump from 0 to 0.333. |
| The shared policy genuinely uses the neighbour channel; it is not |
| four independent single-agent policies. |
|
|
| ## Files in this repo |
|
|
| - `ppo_policy.pt` — final shared PPO weights (ActorCritic, obs dim 20). |
| - `sim.py`, `policies.py`, `train.py`, `bc.py`, `evaluate.py`, `plots.py` |
| - `results.json`, `eval.csv` (728 rows), `training_curve.csv` (final |
| 4-agent phase) + `training_curve_single.csv`, |
| `training_curve_single2.csv`, `training_curve_ft1.csv`, |
| `training_curve_ft2.csv`, `training_curve_ft3.csv` (earlier phases) |
| - `plots/trap_trajectories.png`, `plots/four_agent.png`, |
| `plots/clean_comparison.png`, `plots/training_curve.png` |
| - `space/` — self-contained Gradio demo (copies of `sim.py`, |
| `policies.py`, `ppo_policy.pt` + `app.py`, `requirements.txt`). |
| See `SMOKE_TEST.md`. |
|
|
| ## Limitations |
|
|
| 1. 2D simulation only. No hardware, no real robot, no vision (the |
| observation is ground-truth state features + optional noise, not pixels). |
| 2. The wall trap is constructed. It demonstrates a known potential-field |
| failure mode; it does not prove the policy generalises to arbitrary |
| unseen maps (the +0.5 obstacle-shift condition is the only unseen-layout |
| test, and success there is 0.55–0.71, not 1.0). |
| 3. No formal safety guarantee. Collisions still occur (5–40% depending on |
| condition). A safety shield / control-barrier filter would be the |
| natural next step and is explicitly **not** claimed here. |
| 4. Small evaluation: 20 single-agent and 12 four-agent layouts. Differences |
| of a few percent are within layout noise; the large effects (trap |
| 0.00 → 0.80–0.83, 4-agent 0.396 → 0.792, ablation 0.792 → 0.146) are |
| the ones to rely on. |
| 5. Training required a behaviour-cloning warm-start and a phased |
| curriculum (including a trap-only phase that temporarily caused |
| catastrophic forgetting of general layouts, fixed by a mixed |
| consolidation phase). From-scratch PPO did not learn this task in |
| ~1.9M environment steps of attempts. See `BUILD_REPORT.md`. |
|
|
| ## Reproduce |
|
|
| ```bash |
| # shared venv (torch, numpy, matplotlib, gradio); proxy vars must be unset for gradio imports |
| export VENV=~/workspace/rul-project/venv/bin/python |
| env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY -u all_proxy -u ALL_PROXY -u no_proxy -u NO_PROXY $VENV bc.py --episodes 400 --n-agents 4 --epochs 30 --out bc_policy.pt |
| env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 1 --init bc_policy.pt --out ppo_ft1.pt --curve training_curve_ft1.csv |
| # (trap-only exploration-reset weights were made from ppo_ft1.pt by setting log_std to -0.5; see BUILD_REPORT.md) |
| env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 1 --init ppo_ft1_explore.pt --trap-prob 1.0 --trap-ramp-iters 1 --ent-coef 0.02 --out ppo_ft2.pt --curve training_curve_ft2.csv |
| env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 1 --init ppo_ft2.pt --trap-prob 0.5 --trap-ramp-iters 1 --out ppo_ft3.pt --curve training_curve_ft3.csv |
| env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 4 --init ppo_ft3.pt --trap-prob 0.5 --trap-ramp-iters 1 --out ppo_policy.pt --curve training_curve.csv |
| env -u ... $VENV evaluate.py --weights ppo_policy.pt --out results.json --csv eval.csv |
| env -u ... $VENV plots.py --weights ppo_policy.pt --results results.json --curve training_curve.csv --outdir plots |
| ``` |
|
|
| All commands are run from `~/workspace/li-project/`. Fixed seeds throughout |
| (training seed 0, BC seed 123, evaluation layout seeds 1000–1019 / |
| 2000–2011, noise seeds 90000+layout index). |
|
|
