Model card: held-out AlpaSim results (344 scenes), test-drive section, Quickstart fix

README.md

@@ -66,7 +66,25 @@ each model's native camera rig (Pilot and the base: 3 × 1080p f-theta; Alpamayo
 **The route is given to every model that accepts one: Pilot and the base do; Alpamayo 1.5 and SimForge-D1.5 have no
 route input.** Pilot's plans take effect 0.2 s after they are issued. Paired bootstrap, 10,000 resamples.
 
-### Internal AlpaSim: 96 scenes, a tuning panel
+### Internal AlpaSim: 344 held-out scenes
+
+| | Pilot-4B v1.5 | Qwen-Drive-1.0-4B (base) |
+|---|---|---|
+| Average scene score ↑ [95% CI] | **0.672** [0.624, 0.720] | 0.465 [0.417, 0.513] |
+| Scenes ended by an at-fault collision (of 344) ↓ | **31** | 54 |
+| Scenes scored 0 (any failure) ↓ | **105** | 161 |
+| Pilot − base [95% CI] | +0.207 [+0.150, +0.264] (+45 %); better in 154 scenes, worse in 40 | — |
+
+The held-out set is AlpaSim's public `nurec_curated_val` split (441 scenes) without one scene and without the 96
+tuning scenes below. None of its clips was used to train Pilot or its initialisation, or to tune it (checked by clip
+ID). It is not untouched: it belongs to the validation panel on which v1.5 was chosen among training candidates, so
+some selection bias remains. 21 of the scenes were recorded within 50 m of a training clip; without them the result is
+the same (Pilot 0.672, base 0.467). Alpamayo 1.5 and SimForge-D1.5 have not been run on this set.
+
+By scene type, Pilot's lead over its base is largest in urban cruising (0.853 vs 0.538, 119 scenes) and smallest in
+fast cruising, where it is no better than its base (0.306 vs 0.274, 59 scenes; +0.03 [−0.13, +0.19]).
+
+### Internal AlpaSim: 96 tuning scenes, all four models
 
 | | Pilot-4B v1.5 | Alpamayo 1.5 (10B) | SimForge-D1.5 | Qwen-Drive-1.0-4B (base) |
 |---|---|---|---|---|
@@ -74,15 +92,16 @@ route input.** Pilot's plans take effect 0.2 s after they are issued. Paired boo
 | Pilot − model [95% CI] | — | +0.046 [−0.068, +0.161], **not significant** | +0.131 [+0.006, +0.255] | +0.179 [+0.067, +0.295] |
 
 **These 96 scenes were used to select and tune Pilot.** They share no clips with its training data, but the
-selection favours Pilot, so read this table as a within-family check. The clear result is that Pilot improves on its
-base by +0.179 (+42 %). Its lead over Alpamayo 1.5 is not statistically significant. On this panel Pilot does not
-have fewer collisions than Alpamayo 1.5; its edge is fewer exits from the lane corridor, and it does better in
+selection favours Pilot, so read this table as a within-family check. Pilot's gain over its base (+0.179, +42 %)
+holds on the held-out set above. Its lead over Alpamayo 1.5 is not statistically significant. On this panel Pilot does
+not have fewer collisions than Alpamayo 1.5; its edge is fewer exits from the lane corridor, and it does better in
 cruising scenes while Alpamayo 1.5 does better in slow and stationary ones.
 
-Protocol: our own run of AlpaSim 0.89.0 (not the public leaderboard); 200 steps × 0.1 s, the first 1.7 s replayed
-from the log; one rollout per scene; AlpaSim's default controller gains for every model; post-fix scorer `e121e59`.
-Scene score = 0 on an at-fault collision, off-road or a lateral corridor exit, otherwise
+Protocol for both AlpaSim tables: our own run of AlpaSim 0.89.0 (not the public leaderboard); 200 steps × 0.1 s, the
+first 1.7 s replayed from the log; one rollout per scene; AlpaSim's default controller gains for every model; post-fix
+scorer `e121e59`. Scene score = 0 on an at-fault collision, off-road or a lateral corridor exit, otherwise
 min(progress / 0.8, 1). The route is sent to all four models, but Alpamayo 1.5 and D1.5 ignore it in this version.
+Paired bootstrap, 10,000 resamples.
 
 ### Open loop: where Pilot is weaker
 
@@ -115,9 +134,51 @@ cameras' f-theta calibration for the veto.
 exact benchmarked profile:
 
 ```bash
-docker run --rm --gpus all -p 6789:6789 -v $PWD:/repo:ro pilot-4b /repo/run_alpasim_driver.sh
+docker run --rm --gpus all -p 6789:6789 -v $PWD:/repo:ro pilot-4b sh /repo/run_alpasim_driver.sh
 ```
 
+### Test drive in AlpaSim
+
+`test_drive_alpasim.sh` runs one complete closed-loop drive: it sets up the public
+[AlpaSim](https://github.com/NVlabs/alpasim) simulator, starts Pilot as its driver, drives a recorded scene for 20 s
+and prints the score, with a video of the drive.
+
+```bash
+hf download SIMFORGE-AI/Pilot-4B-v1.5 --local-dir Pilot-4B-v1.5
+cd Pilot-4B-v1.5
+export HF_TOKEN=hf_...          # a read token for an account with access to the scene dataset (below)
+bash test_drive_alpasim.sh      # or: bash test_drive_alpasim.sh <scene id>
+```
+
+```text
+scene      28621273-20ed-4570-98d1-cd0dcf451fc1
+score      1.000   (0 on an at-fault collision, off-road or lane-corridor exit; otherwise min(progress / 0.8, 1))
+outcome    no failure; progress 1.00 of the logged route (323 m)
+video      test-drives/28621273-…/alpasim/aggregate/videos/all/clipgt-28621273-…_camera_front_wide_120fov_default.mp4
+rollout    test-drives/28621273-…/alpasim/rollouts/clipgt-28621273-…/<rollout id>/rollout.asl
+summary    test-drives/28621273-…/alpasim/aggregate/results-summary.json
+wall time  8 min 36 s for the drive
+```
+
+You need Linux, an NVIDIA GPU with driver 570 or newer, Docker with the compose plugin and the NVIDIA Container
+Toolkit (usable without `sudo`), `git`, [`uv`](https://docs.astral.sh/uv/) 0.9.17 or newer and a Rust toolchain
+(AlpaSim's setup compiles an extension). The scenes come from the gated dataset
+[nvidia/PhysicalAI-Autonomous-Vehicles-NuRec](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec):
+request access on its page and accept its terms first. One GPU with 40 GB or more holds Pilot and the simulator; on
+24 GB cards, set `DRIVER_GPU` and `SIM_GPU` to two different GPUs. The first run clones AlpaSim into `./alpasim`,
+installs it, builds its images and Pilot's, pulls the NuRec renderer image and downloads the scene (about 1.7 GB),
+so it takes much longer than later runs.
+
+The default scene, `28621273-20ed-4570-98d1-cd0dcf451fc1`, is a left turn from AlpaSim's `nurec_curated_val` split.
+It was not used to train or tune Pilot. Any scene id in AlpaSim's `data/scenes/sim_scenes*.csv` works.
+
+The script pins AlpaSim to commit `e121e59` (version 0.89.0, branch `e2e_challenge`), whose scorer produced the
+AlpaSim results above, and drives with the same settings: the public challenge preset (`+e2e_challenge=dev`), the
+nonlinear MPC controller with default gains, the renderer without the NRE harmonizer, and no driver calls during the
+first 1.7 s, which are replayed from the log. Two things differ from our evaluation runs: those rendered only the three
+cameras Pilot reads, which gives the same pixels faster, and they seeded Pilot's sampling from a deterministic session
+id. A single drive is one sample, so its score can differ from ours on the same scene.
+
 ## Inputs and outputs
 
 | | |
@@ -140,7 +201,7 @@ weights. Two stages, both by SimForge:
    of them, with recovery targets mined from earlier versions' failures. 4 × A100, learning rate 5e-6, seed 43, about
    2 hours.
 
-None of the 96 AlpaSim scenes or the 60 SF scenarios above appears in the training clips (checked by clip ID; the SF
+None of the 440 AlpaSim scenes (344 held-out, 96 tuning) or the 60 SF scenarios above appears in the training clips (checked by clip ID; the SF
 scenarios are authored and have no recorded drive). The training manifests are not part of this release.
 
 ## Limitations
@@ -157,6 +218,8 @@ scenarios are authored and have no recorded drive). The training manifests are n
 - **The planning head was selected.** v1.5 was the best of four training seeds on a validation panel, so some of
   its gain over its predecessor may not carry over. During training, the route for recorded drives was built from
   the path the human driver went on to take.
+- **Fast cruising is weak.** On the held-out set Pilot scores 0.31 in fast-cruising scenes, no better than its base;
+  most of those failures are lane-corridor exits and leaving the road.
 - **Open loop is weaker** than the base model (see above).
 
 ## Files
@@ -170,6 +233,7 @@ scenarios are authored and have no recorded drive). The training manifests are n
 | `driver_profile.txt` | the benchmarked driver flags |
 | `inference.py`, `examples/` | single-decision example |
 | `run_alpasim_driver.sh`, `Dockerfile`, `requirements.txt` | closed-loop service and environment |
+| `test_drive_alpasim.sh` | one closed-loop test drive in AlpaSim (sets up AlpaSim, runs Pilot, prints the score) |
 | `SHA256SUMS` | checksums; the weights and `pilot/` match the serving image used for every result above |
 
 Weights SHA-256: VLM `b9de4bf4…caccc`, perception `e964ca94…450ae`, planner `a8e5d329…b4d14` (full hashes in `SHA256SUMS`).