README.md

---
license: mit
library_name: pytorch
pipeline_tag: robotics
tags:
  - robotics
  - social-navigation
  - human-robot-interaction
  - pedestrian-trajectory-prediction
  - model-predictive-control
  - convex-optimization
---

# crowd-nav — human-response models for the Foresight controller

Trained weights for **Foresight**, a convex receding-horizon controller that predicts how each
nearby pedestrian will respond to the robot's *candidate plan*, and then chooses that plan by
exact convex optimization. These are the response models the controller calls; the planner is
plain code (no learned policy, no RL).

| file | model | role |
|---|---|---|
| `linres_head.pt` | **Linear-response operator** (headline) | best closed-loop performance; makes planning an exact convex QP |
| `residual_predictor_k1.pt` | Residual predictor | most accurate on real data; the safest variant in closed loop |
| `residual_predictor.pt` | *legacy* | **deprecated** — trained with a robot-force prior of 0 but deployed at 1; superseded by `residual_predictor_k1.pt` |

## Which one should I use?

**`linres_head.pt`** unless you have a reason not to. It is a few centimetres less accurate as a
pure forecaster, but its prediction is *exactly affine* in the robot's planned displacement, so the
planner gets an exact constant Jacobian and the safety constraints stay linear — the whole plan
selection becomes a convex QP that solves in ~17 ms with a checkable feasibility certificate.
That trade wins where it counts: it leads our benchmark, and it is the only variant that crossed a
90-pedestrian oncoming corridor with zero contacts.

## Measured results

Isaac Sim, 16 scenarios × 10 paired seeds (160 episodes per method), identical hardware:

| model | goal reach | hard collisions / ep | plan latency (mean) |
|---|---|---|---|
| **Foresight-LinRes** (`linres_head.pt`) | **94.4 %** | 0.013 | **16.6 ms** |
| Foresight-Residual (`residual_predictor_k1.pt`) | 90.0 % | **0.006** | 19.7 ms |
| Foresight-SFM (physics only, ablation) | 93.8 % | 0.013 | 17.2 ms |
| Foresight-CV (straight-line, ablation) | 90.0 % | 0.019 | 16.4 ms |
| SICNav [T-RO '24] (nonconvex bilevel MPC) | 89.2 % | 0.042 | 1843 ms |

Stress test — a 4.5 m × 105 m corridor with 90 pedestrians *all walking against* the robot and a
continuous inflow so it never empties: LinRes reached the goal in 98.5 s with **zero contacts**;
against pedestrian models it was never tuned on (SRFM, social-force) it reached in 85.0 s and
91.7 s, also contact-free.

Prediction accuracy on the held-out real split (factual ADE, metres — lower is better):

| model | no robot | stationary robot | moving robot |
|---|---|---|---|
| Constant velocity | 0.240 | 0.253 | 0.354 |
| Social force | 0.343 | 0.358 | 0.416 |
| Residual (k1) | **0.192** | **0.217** | 0.285 |
| Linear response | 0.204 | 0.248 | 0.319 |

## Architecture

Both models are a **physics prior plus a learned correction** — small by design, because the real
moving-robot data is scarce.

```
prior (no parameters, per pedestrian per 0.25 s step)
  f_goal  = (1.3·(g−p)/‖g−p‖ − v)/0.5
  f_ped   = Σ 2.0·exp(−d/0.4)·d̂      (d < 2.0 m)
  f_robot = k·exp(−d/0.5)·d̂          (d < 3.0 m)
  v ← clip(v + f·Δt, 1.8);  p ← p + v·Δt

residual_predictor_k1.pt     ŷ = prior(k=1) + f_θ(x)
  f_θ : MLP 22 → 128 → 128 → 16   (21,520 params)

linres_head.pt               ŷ = prior(k=0) + f_θ(x) + G_φ(x)·ΔR
  f_θ : MLP 22 → 128 → 128 → 16   (21,520 params)
  G_φ : MLP 22 → 64 → 96          (7,712 params) ⇒ rank-3 16×16 operator
        G = Σ_{r=1..3} a_r b_rᵀ,  ΔR = (R − 1⊗r₀)/5.0
        ⇒ ∂ŷ/∂R = G/5.0  — exact, constant, no differentiation needed
```

Input `x` (22-d, pedestrian local frame): 1 s of past positions (4 @ 0.25 s), velocity, goal
direction, robot relative position, a robot-nearby flag, the two nearest neighbours, and a 3-way
condition one-hot. Output: 8 × 2 future displacements — 2.0 s at 0.25 s.

Training: supervised (Adam, lr 1e-3, batch 128, ~25 epochs), synthetic pre-train then fine-tune on
the real **PeRoI** robot–pedestrian recordings. No reinforcement learning anywhere in this stack.
Both run in well under 1 ms per call on CPU.

## Load

```bash
pip install huggingface_hub torch
hf download elmoghany/crowd-nav linres_head.pt residual_predictor_k1.pt --local-dir .
```

```python
import torch
ck = torch.load("linres_head.pt", map_location="cpu")
ck["config"]
# {'in_dim': 22, 'horizon_dim': 16, 'rank': 3, 'hidden': 64, 'dr_scale': 5.0,
#  'sfm_k_robot': 0.0, 'convention': 'y = sfm(k=0) + f_resid(x_ctx) + head(x_ctx, dR)'}
ck["f_resid"], ck["head"]   # two state_dicts
```

The `config` block is authoritative: `sfm_k_robot` **must** match the robot-force strength used in
your prior at deployment. Mismatching them is a real bug we shipped once — the residual then
corrects an error that is not there. `residual_predictor.pt` is the artifact of that mistake and is
kept only for reproducibility.

Model classes (`ResidualPredictor`, `LinearResponseHead`) and the full controller live in the
**[github.com/elmoghany/crowd-nav](https://github.com/elmoghany/crowd-nav)** -- a one-file,
simulator-free controller with a `quickstart.py` that fetches these weights and verifies your
install in one command. The full research stack (convex-QP planner, Isaac benchmarks, paper) is at
[github.com/elmoghany/crowd-nav-legacy](https://github.com/elmoghany/crowd-nav-legacy).

## Honest limitations

- **Real-data forecasting is a tie.** On four real robot–pedestrian datasets (including JRDB),
  every learned model here ties the analytic baselines on factual accuracy. Our measured
  explanation: the counterfactual proxy those benchmarks rely on (constant-velocity extrapolation)
  correlates only r ≈ 0.30 with exact simulated counterfactuals — the target is mostly noise. The
  gains above are **closed-loop** gains, not forecasting gains.
- **Compliance bias.** The training recordings contain cooperative pedestrians. On deliberately
  stubborn crowds the k1 residual is the weakest learned variant; compliance-randomized training
  data is the known fix, not yet applied.
- **Cross-dataset transfer.** Moving `residual_predictor_k1.pt` to JRDB over-predicts robot-induced
  deviation by ~6× (its k=1 prior is calibrated to our robot). It makes a controller conservative,
  not unsafe — it still threaded real JRDB crowds contact-free — but retune `sfm_k_robot` for a
  different platform.