# Multi-agent RL navigation — trustworthy autonomous systems (simulation only)

**Repo: `dummy033485/multi-agent-rl-navigation`**
**Status: model repo live. No Hugging Face Space has been created.**

A shared decentralized PPO policy for 2D multi-agent navigation, compared
honestly against a textbook classical controller on the *same* seeded
layouts. Built as a portfolio project for an application to Prof. Jiachen
Li's Trustworthy Autonomous Systems Lab (Georgia Tech): robotics,
trustworthy AI/ML, reinforcement learning, control, optimization and
multi-agent systems.

## What this is — and is not

- **Is:** a self-contained 2D physics simulator (written for this project,
  no external environment, no hardware, no proprietary data) + a learned
  shared policy + a classical baseline + a fixed, seeded evaluation.
- **Is not:** a robot result. No hardware was used. The RL policy has **no
  formal safety guarantee** — every safety number below is empirical,
  measured on the simulator's fixed test layouts. The wall-trap layout is
  deliberately constructed to defeat a pure potential-field controller.

## Simulator (`sim.py`)

- Arena 12x12 with walls. Agents: double-integrator kinematics,
  `dt=0.1`, max speed 1.2, max acceleration 3.0, agent radius 0.25,
  episode cap 200 steps.
- Each layout: 3 circle obstacles (radius 0.5–0.8) **and** 1 wall/barrier
  segment. Two layout families, generated with fixed seeds:
  - *general*: a short (3 m) barrier segment at a random mid-arena angle.
  - *trap*: a vertical segment at x=6, y in [2,10], with gaps only at the
    ends; starts on the left, goals near-directly opposite on the right,
    so a direct potential field is attracted right and repelled left and
    stalls in the middle.
- Decentralized 20-dim observation per agent: own velocity, relative goal
  vector + distance, nearest circle obstacle (direction, surface distance,
  radius), nearest wall segment (direction, distance, both endpoint
  vectors), nearest other agent (direction, distance), nearest arena-wall
  distance. (Endpoint vectors were added after a failed training run:
  with only the nearest segment point, the policy could not observe where
  the wall ends — see Limitations and `BUILD_REPORT.md`.)
- Collision = distance to an obstacle/wall segment/arena wall below the
  agent radius, or agent–agent distance below 0.5. A colliding arrival is
  never counted as a success. Safety margin for violation counting: 0.15.

## Policies

- **Classical** (`policies.py`, exact gains documented in code): PD
  controller toward the goal (KD=2.0) + radial repulsive potential field
  (ETA=6.0, influence distance D0=2.0) from the nearest obstacle, segment
  and neighbour. No learning, no tangential escape trick — the honest
  textbook baseline. It stalls (timeout, 0 collisions) on every trap layout.
- **RL**: shared decentralized PPO, MLP 20→128→128 (the observation grew
  from 15 to 20 dims when segment-endpoint features were added),
  Gaussian actions, tanh-bounded to the acceleration limit. One set of
  weights drives every agent from its own observation.
  Training recipe (all in `train.py`, PPO from scratch with torch only):
  the policy was warm-started by behaviour cloning the classical controller
  on general layouts only (400 episodes, 4 agents, final action MSE 0.079),
  then PPO fine-tuned in four phases — 1-agent mixed (trap ramp), 1-agent
  trap-only with an exploration reset, 1-agent mixed consolidation, and
  4-agent mixed — 300 iterations × 128-step rollouts × 16 envs per phase
  (614,400 environment steps per phase; 2,457,600 in the kept chain plus
  the BC dataset). Earlier from-scratch PPO runs failed outright (success
  5–15%) and are documented, with their fixes, in `BUILD_REPORT.md` —
  they are not hidden. The final phase's curve is `training_curve.csv`
  (earlier phases have their own `training_curve_*.csv` files).

## Evaluation protocol (exact)

Fixed layouts, identical for every policy and condition:
- Single-agent: seeds 1000–1019 (10 general + 10 trap, alternating).
- 4-agent: seeds 2000–2011 (6 general + 6 trap, alternating).
- RL evaluated with the deterministic (mean) action; classical is
  deterministic. Noise conditions reuse the same noise RNG seed per
  layout/condition for both policies.
- Conditions: clean; observation noise σ=0.05; observation noise σ=0.10;
  dynamics disturbance (added acceleration noise σ=1.0); obstacle shift
  +0.5 (circles and segments shifted +0.5 in x).
- Metrics are per agent-episode (20 single-agent and 48 4-agent episodes
  per policy per condition; 728 rows in `eval.csv`). Steps-to-goal and
  clearance are reported on successes only; safety-violation rate is the
  fraction of active steps with clearance < 0.15.

## Results (copied from `results.json` — no other numbers are claimed)

### Clean

| Policy | Agents | Success | Collision | Obstacle coll. | Agent–agent coll. | Timeout | Steps (succ.) | Path len. | Min clearance (succ.) | Safety-viol. rate | General | Trap |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Classical | 1 | 0.450 | 0.000 | 0.000 | 0.000 | 0.550 | 73.22 | 5.35 | 0.588 | 0.000 | 0.90 | 0.00 |
| RL | 1 | 0.700 | 0.200 | 0.200 | 0.000 | 0.100 | 86.00 | 9.59 | 0.419 | 0.033 | 0.60 | 0.80 |
| Classical | 4 | 0.396 | 0.000 | 0.000 | 0.000 | 0.604 | 160.42 | 5.87 | 0.690 | 0.000 | 0.79 | 0.00 |
| RL | 4 | 0.792 | 0.146 | 0.146 | 0.000 | 0.062 | 141.18 | 10.63 | 0.505 | 0.019 | 0.75 | 0.83 |

Reading, honestly: the RL policy wins on success everywhere, and is the
only policy that solves trap layouts or coordinates 4 agents (zero
agent–agent collisions clean). It loses on safety in the single-agent
case: 20% collisions and lower clearance than the classical controller's
0% collisions. The classical controller never collides clean — it fails
by timing out at the wall instead. RL paths are longer (detours).

### Robustness (success / collision)

| Condition | RL 1-agent | Classical 1-agent | RL 4-agent | Classical 4-agent |
|---|---|---|---|---|
| Clean | 0.700 / 0.200 | 0.450 / 0.000 | 0.792 / 0.146 | 0.396 / 0.000 |
| Obs. noise σ=0.05 | 0.750 / 0.150 | 0.450 / 0.100 | 0.646 / 0.292 | 0.333 / 0.125 |
| Obs. noise σ=0.10 | 0.600 / 0.300 | 0.200 / 0.250 | 0.542 / 0.396 | 0.104 / 0.396 |
| Dynamics disturbance σ=1.0 | 0.650 / 0.200 | 0.350 / 0.000 | 0.771 / 0.167 | 0.375 / 0.000 |
| Obstacle shift +0.5 | 0.550 / 0.300 | 0.350 / 0.000 | 0.708 / 0.250 | 0.375 / 0.000 |

The RL policy degrades gracefully on success under every perturbation and
stays well above classical, but its collision rate rises under noise
(up to 0.396 at σ=0.10, 4-agent) — the trustworthy claim is *measured
robustness of success*, not a safety proof.

### Ablation — neighbour observation, 4-agent clean

| Policy | Success | Collision | Agent–agent collision |
|---|---|---|---|
| RL with neighbour observation | 0.792 | 0.146 | 0.000 |
| RL with neighbour observation zeroed | 0.146 | 0.708 | 0.333 |

Removing the neighbour observation collapses coordination: success falls
from 0.792 to 0.146 and agent–agent collisions jump from 0 to 0.333.
The shared policy genuinely uses the neighbour channel; it is not
four independent single-agent policies.

## Files in this repo

- `ppo_policy.pt` — final shared PPO weights (ActorCritic, obs dim 20).
- `sim.py`, `policies.py`, `train.py`, `bc.py`, `evaluate.py`, `plots.py`
- `results.json`, `eval.csv` (728 rows), `training_curve.csv` (final
  4-agent phase) + `training_curve_single.csv`,
  `training_curve_single2.csv`, `training_curve_ft1.csv`,
  `training_curve_ft2.csv`, `training_curve_ft3.csv` (earlier phases)
- `plots/trap_trajectories.png`, `plots/four_agent.png`,
  `plots/clean_comparison.png`, `plots/training_curve.png`
- `space/` — self-contained Gradio demo (copies of `sim.py`,
  `policies.py`, `ppo_policy.pt` + `app.py`, `requirements.txt`).
  See `SMOKE_TEST.md`.

## Limitations

1. 2D simulation only. No hardware, no real robot, no vision (the
   observation is ground-truth state features + optional noise, not pixels).
2. The wall trap is constructed. It demonstrates a known potential-field
   failure mode; it does not prove the policy generalises to arbitrary
   unseen maps (the +0.5 obstacle-shift condition is the only unseen-layout
   test, and success there is 0.55–0.71, not 1.0).
3. No formal safety guarantee. Collisions still occur (5–40% depending on
   condition). A safety shield / control-barrier filter would be the
   natural next step and is explicitly **not** claimed here.
4. Small evaluation: 20 single-agent and 12 four-agent layouts. Differences
   of a few percent are within layout noise; the large effects (trap
   0.00 → 0.80–0.83, 4-agent 0.396 → 0.792, ablation 0.792 → 0.146) are
   the ones to rely on.
5. Training required a behaviour-cloning warm-start and a phased
   curriculum (including a trap-only phase that temporarily caused
   catastrophic forgetting of general layouts, fixed by a mixed
   consolidation phase). From-scratch PPO did not learn this task in
   ~1.9M environment steps of attempts. See `BUILD_REPORT.md`.

## Reproduce

```bash
# shared venv (torch, numpy, matplotlib, gradio); proxy vars must be unset for gradio imports
export VENV=~/workspace/rul-project/venv/bin/python
env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY -u all_proxy -u ALL_PROXY -u no_proxy -u NO_PROXY $VENV bc.py --episodes 400 --n-agents 4 --epochs 30 --out bc_policy.pt
env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 1 --init bc_policy.pt --out ppo_ft1.pt --curve training_curve_ft1.csv
# (trap-only exploration-reset weights were made from ppo_ft1.pt by setting log_std to -0.5; see BUILD_REPORT.md)
env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 1 --init ppo_ft1_explore.pt --trap-prob 1.0 --trap-ramp-iters 1 --ent-coef 0.02 --out ppo_ft2.pt --curve training_curve_ft2.csv
env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 1 --init ppo_ft2.pt --trap-prob 0.5 --trap-ramp-iters 1 --out ppo_ft3.pt --curve training_curve_ft3.csv
env -u ... $VENV train.py --iterations 300 --rollout 128 --n-agents 4 --init ppo_ft3.pt --trap-prob 0.5 --trap-ramp-iters 1 --out ppo_policy.pt --curve training_curve.csv
env -u ... $VENV evaluate.py --weights ppo_policy.pt --out results.json --csv eval.csv
env -u ... $VENV plots.py --weights ppo_policy.pt --results results.json --curve training_curve.csv --outdir plots
```

All commands are run from `~/workspace/li-project/`. Fixed seeds throughout
(training seed 0, BC seed 123, evaluation layout seeds 1000–1019 /
2000–2011, noise seeds 90000+layout index).