---
license: apache-2.0
base_model:
- Qwen/Qwen-Drive-1.0-4B
base_model_relation: finetune
pipeline_tag: robotics
tags:
- autonomous-driving
- end-to-end-driving
- motion-planning
- closed-loop
- simulation
- qwen-drive
---

# SimForge Pilot-4B v1.5

An end-to-end driving model: three cameras and a route in, a 5-second trajectory out. It is
[Qwen-Drive-1.0-4B](https://huggingface.co/Qwen/Qwen-Drive-1.0-4B) with a **SimForge-trained, route-aware
planning head**, served with a **plan selector** and a **brake-only obstacle veto**. We tested it closed loop in
simulation against Alpamayo 1.5, SimForge-D1.5 and the Qwen-Drive base it is built on.

[simforge.ai](https://simforge.ai) · [Helios renderer](https://simforge.ai/helios) · related dataset: [SIMLOG-25k](https://huggingface.co/datasets/SIMFORGE-AI/simlog-25k)

> Research model for simulation. It has not been validated for, and must not be used to control, a real vehicle.

## What is in it

| Part | Params | Source |
|---|---|---|
| Vision-language model (`model.safetensors`) | 4.54 B, BF16 | Qwen-Drive-1.0-4B, **byte-identical** to `Qwen/Qwen-Drive-1.0-4B@28484089` |
| Perception head (`perception/`): 3D detection, map, occupancy | 0.13 B, F32 | Qwen-Drive-1.0-4B, **byte-identical**; used only by the veto and the selector |
| Planning head (`planner-navs2-002000/`) | 1.04 B, BF16 | **SimForge-trained.** 32 layers, flow matching, 50 poses at 10 Hz; inputs add a 16-point route |
| Driver (`pilot/`, `driver_profile.txt`) | — | **SimForge.** Plan selector + brake-only veto; the exact code and flags that produced the results below |

**Plan selector.** When the network's plan strays from the route (≥ 0.5 m), the driver draws up to 8 more candidates,
including ones under the alternative navigation commands, and keeps the one that makes the most route progress
without a predicted conflict. Otherwise the network's own plan is served.

**Brake-only veto.** The perception head runs on each frame set; if the chosen plan is predicted to close on an
obstacle (time-to-collision 1.5 s), the veto slows the plan (decelerations capped at 2 / 3.5 / 5 / 7 m/s²). It only
brakes, never steers, and only below 8 m/s.

"Pilot" in every result below means weights + this driver. The network alone (`--planner-only`) was not benchmarked
closed loop.

## Results

All numbers come from our November 2026 reruns on A100-40GB. They were run once per scene with one seed, so the
confidence intervals cover scene sampling only. Per-scene outcomes vary a lot between seeds and engine versions;
treat any single clip as an example, not a guarantee.

### SimForge closed loop: 60 authored San Francisco scenarios, Helios renderer

| | Pilot-4B v1.5 | Alpamayo 1.5 (10B) | SimForge-D1.5 | Qwen-Drive-1.0-4B (base) |
|---|---|---|---|---|
| Mean driving score ↑ | **0.537** | 0.405 | 0.405 | 0.301 |
| Runs ended by a collision (of 60) ↓ | **16** | 30 | 37 | 33 |
| Pilot − model, driving score [95% CI] | — | +0.132 [+0.046, +0.222] | +0.132 [+0.040, +0.222] | +0.236 [+0.160, +0.314] |
| Collision-ended rate, model − Pilot [95% CI] | — | +0.233 [+0.083, +0.383] | +0.350 [+0.167, +0.517] | +0.283 [+0.133, +0.433] |

Protocol: scenarios authored on SimForge's San Francisco map with no recorded drive behind them; SimForge engine 0.37,
scorer `simforge.offroad/v4` + `simforge.collision/v2` (driving score = route completion × infraction penalties);
20 s episodes, 200 steps at 10 Hz, a decision every 0.5 s, seed 42; the same pure-pursuit controller for every model;
each model's native camera rig (Pilot and the base: 3 × 1080p f-theta; Alpamayo 1.5 and D1.5: 4 pinhole).
**The route is given to every model that accepts one: Pilot and the base do; Alpamayo 1.5 and SimForge-D1.5 have no
route input.** Pilot's plans take effect 0.2 s after they are issued. Paired bootstrap, 10,000 resamples.

### Internal AlpaSim: 344 held-out scenes

| | Pilot-4B v1.5 | Qwen-Drive-1.0-4B (base) |
|---|---|---|
| Average scene score ↑ [95% CI] | **0.672** [0.624, 0.720] | 0.465 [0.417, 0.513] |
| Scenes ended by an at-fault collision (of 344) ↓ | **31** | 54 |
| Scenes scored 0 (any failure) ↓ | **105** | 161 |
| Pilot − base [95% CI] | +0.207 [+0.150, +0.264] (+45 %); better in 154 scenes, worse in 40 | — |

The held-out set is AlpaSim's public `nurec_curated_val` split (441 scenes) without one scene and without the 96
tuning scenes below. None of its clips was used to train Pilot or its initialisation, or to tune it (checked by clip
ID). It is not untouched: it belongs to the validation panel on which v1.5 was chosen among training candidates, so
some selection bias remains. 21 of the scenes were recorded within 50 m of a training clip; without them the result is
the same (Pilot 0.672, base 0.467). Alpamayo 1.5 and SimForge-D1.5 have not been run on this set.

By scene type, Pilot's lead over its base is largest in urban cruising (0.853 vs 0.538, 119 scenes) and smallest in
fast cruising, where it is no better than its base (0.306 vs 0.274, 59 scenes; +0.03 [−0.13, +0.19]).

### Internal AlpaSim: 96 tuning scenes, all four models

| | Pilot-4B v1.5 | Alpamayo 1.5 (10B) | SimForge-D1.5 | Qwen-Drive-1.0-4B (base) |
|---|---|---|---|---|
| Average scene score ↑ | **0.601** | 0.556 | 0.470 | 0.422 |
| Pilot − model [95% CI] | — | +0.046 [−0.068, +0.161], **not significant** | +0.131 [+0.006, +0.255] | +0.179 [+0.067, +0.295] |

**These 96 scenes were used to select and tune Pilot.** They share no clips with its training data, but the
selection favours Pilot, so read this table as a within-family check. Pilot's gain over its base (+0.179, +42 %)
holds on the held-out set above. Its lead over Alpamayo 1.5 is not statistically significant. On this panel Pilot does
not have fewer collisions than Alpamayo 1.5; its edge is fewer exits from the lane corridor, and it does better in
cruising scenes while Alpamayo 1.5 does better in slow and stationary ones.

Protocol for both AlpaSim tables: our own run of AlpaSim 0.89.0 (not the public leaderboard); 200 steps × 0.1 s, the
first 1.7 s replayed from the log; one rollout per scene; AlpaSim's default controller gains for every model; post-fix
scorer `e121e59`. Scene score = 0 on an at-fault collision, off-road or a lateral corridor exit, otherwise
min(progress / 0.8, 1). The route is sent to all four models, but Alpamayo 1.5 and D1.5 ignore it in this version.
Paired bootstrap, 10,000 resamples.

### Open loop: where Pilot is weaker

On the same 60 SF scenarios, scored against the authored path (minADE at 3 s, one sample, lower is better):
Qwen-Drive base **0.910 m**, Alpamayo 1.5 1.199 m, **Pilot 1.343 m**, SimForge-D1.5 1.461 m. Pilot is tuned to
recover and make progress in closed loop, and it does not imitate a reference path better than its base.

## Quickstart

Needs Docker and an NVIDIA GPU; we recommend 24 GB or more (the benchmarked service peaked at 15.1 GiB on an A100).

```bash
hf download SIMFORGE-AI/Pilot-4B-v1.5 --local-dir Pilot-4B-v1.5
cd Pilot-4B-v1.5
docker build -t pilot-4b .                       # Python 3.12, CUDA 12.6, the benchmarked package versions
docker run --rm --gpus all -v $PWD:/repo:ro pilot-4b python /repo/inference.py
```

`inference.py` plans one decision from `examples/sf-slowing/`, a moment from one of the SF closed-loop runs above: the
car is at 4.7 m/s approaching an all-way stop while another car crosses in front, and plans to slow almost to a stop before moving off. It prints the 5 s plan and how far it lies from the plan the benchmarked
system served at that moment. On our A100 the full driver's plan was within 0.41 m of it (mean 0.18 m), with the same
selection. Single-decision mode perceives the current frames, whereas closed loop uses the previous frame set's
detections, so small differences are expected. Add `--planner-only` for the network without selector and veto.

Your own inputs follow `examples/sf-slowing/input.json`: three cameras × four frames at 2 Hz (oldest first), 16 ego
poses at 10 Hz and the route, all in the vehicle frame at the current time (x forward, y left, z up), plus the
cameras' f-theta calibration for the veto.

**Closed loop in AlpaSim.** `run_alpasim_driver.sh` starts Pilot as an AlpaSim egodriver (gRPC, port 6789) with the
exact benchmarked profile:

```bash
docker run --rm --gpus all -p 6789:6789 -v $PWD:/repo:ro pilot-4b sh /repo/run_alpasim_driver.sh
```

### Test drive in AlpaSim

`test_drive_alpasim.sh` runs one complete closed-loop drive: it sets up the public
[AlpaSim](https://github.com/NVlabs/alpasim) simulator, starts Pilot as its driver, drives a recorded scene for 20 s
and prints the score, with a video of the drive.

```bash
hf download SIMFORGE-AI/Pilot-4B-v1.5 --local-dir Pilot-4B-v1.5
cd Pilot-4B-v1.5
export HF_TOKEN=hf_...          # a read token for an account with access to the scene dataset (below)
bash test_drive_alpasim.sh      # or: bash test_drive_alpasim.sh <scene id>
```

```text
scene      28621273-20ed-4570-98d1-cd0dcf451fc1
score      1.000   (0 on an at-fault collision, off-road or lane-corridor exit; otherwise min(progress / 0.8, 1))
outcome    no failure; progress 1.00 of the logged route (323 m)
video      test-drives/28621273-…/alpasim/aggregate/videos/all/clipgt-28621273-…_camera_front_wide_120fov_default.mp4
rollout    test-drives/28621273-…/alpasim/rollouts/clipgt-28621273-…/<rollout id>/rollout.asl
summary    test-drives/28621273-…/alpasim/aggregate/results-summary.json
wall time  8 min 36 s for the drive
```

You need Linux, an NVIDIA GPU with driver 570 or newer, Docker with the compose plugin and the NVIDIA Container
Toolkit (usable without `sudo`), `git`, [`uv`](https://docs.astral.sh/uv/) 0.9.17 or newer and a Rust toolchain
(AlpaSim's setup compiles an extension). The scenes come from the gated dataset
[nvidia/PhysicalAI-Autonomous-Vehicles-NuRec](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec):
request access on its page and accept its terms first. One GPU with 40 GB or more holds Pilot and the simulator; on
24 GB cards, set `DRIVER_GPU` and `SIM_GPU` to two different GPUs. The first run clones AlpaSim into `./alpasim`,
installs it, builds its images and Pilot's, pulls the NuRec renderer image and downloads the scene (about 1.7 GB),
so it takes much longer than later runs.

The default scene, `28621273-20ed-4570-98d1-cd0dcf451fc1`, is a left turn from AlpaSim's `nurec_curated_val` split.
It was not used to train or tune Pilot. Any scene id in AlpaSim's `data/scenes/sim_scenes*.csv` works.

The script pins AlpaSim to commit `e121e59` (version 0.89.0, branch `e2e_challenge`), whose scorer produced the
AlpaSim results above, and drives with the same settings: the public challenge preset (`+e2e_challenge=dev`), the
nonlinear MPC controller with default gains, the renderer without the NRE harmonizer, and no driver calls during the
first 1.7 s, which are replayed from the log. Two things differ from our evaluation runs: those rendered only the three
cameras Pilot reads, which gives the same pixels faster, and they seeded Pilot's sampling from a deterministic session
id. A single drive is one sample, so its score can differ from ours on the same scene.

## Inputs and outputs

| | |
|---|---|
| Cameras | front wide, cross left, cross right (120° f-theta, 1920 × 1080); 4 frames each at t − 1.5, − 1.0, − 0.5, 0 s |
| Ego state | 16 poses at 10 Hz ending at t; speed and acceleration are finite differences of them |
| Route | polyline ahead of the car; sets the navigation command (straight / left / right) and feeds the planner's route input |
| Output | 50 poses (x, y, heading) at 10 Hz, 5 s, in the vehicle frame at t |
| Latency (A100) | median of per-scene p95 Drive latency 1.62 s in AlpaSim; the example's single decision takes ~2 s with the full driver, 0.6 s planner-only |

## Training

Only the planning head was trained; the vision-language model and perception head were frozen at the upstream
weights. Two stages, both by SimForge:

1. **Preference stage** (the `r3_sn` initialisation): offline preference optimisation over branches certified in
   simulation, from 737 clips.
2. **v1.5** (`navs2`): 2,000 steps × batch 32 (0.3 epoch) of flow-matching imitation and recovery training on
   68,286 rows from 1,112 clips (about 6.2 hours of driving): recorded real-world driving logs and simulator re-renders
   of them, with recovery targets mined from earlier versions' failures. 4 × A100, learning rate 5e-6, seed 43, about
   2 hours.

None of the 440 AlpaSim scenes (344 held-out, 96 tuning) or the 60 SF scenarios above appears in the training clips (checked by clip ID; the SF
scenarios are authored and have no recorded drive). The training manifests are not part of this release.

## Limitations

- **Simulation only.** All results are closed loop in simulators (Helios and AlpaSim's neural reconstruction), on
  one seed. Expect different numbers on other engine versions; our own SF reruns moved Alpamayo 1.5's score by
  −0.04 between engine versions.
- **The benchmark compares a system with stock policies.** Pilot runs with its selector and veto; the other models
  run as released. Alpamayo 1.5 and SimForge-D1.5 receive no route in the SF scenarios, which favours Pilot in route
  completion (not shown for that reason).
- **The veto is narrow.** Measured on Pilot's predecessor planner, about 41 % of its brakes had no real conflict
  and about 40 % of real conflicts were missed; it is off at 8 m/s and above and cannot prevent side contacts. It
  was not ablated on this planner.
- **The planning head was selected.** v1.5 was the best of four training seeds on a validation panel, so some of
  its gain over its predecessor may not carry over. During training, the route for recorded drives was built from
  the path the human driver went on to take.
- **Fast cruising is weak.** On the held-out set Pilot scores 0.31 in fast-cruising scenes, no better than its base;
  most of those failures are lane-corridor exits and leaving the road.
- **Open loop is weaker** than the base model (see above).

## Files

| Path | What |
|---|---|
| `model.safetensors`, `config.json`, tokenizer and template files | Qwen-Drive-1.0-4B VLM (upstream bytes) |
| `perception/` | perception head (upstream bytes) |
| `planner-navs2-002000/` | the v1.5 planning head |
| `pilot/` | driver code: `driver/` (driver, selector, veto), `lib/` (Qwen-Drive modelling code), `qd/` (fast path), `train/` (planner definition) |
| `driver_profile.txt` | the benchmarked driver flags |
| `inference.py`, `examples/` | single-decision example |
| `run_alpasim_driver.sh`, `Dockerfile`, `requirements.txt` | closed-loop service and environment |
| `test_drive_alpasim.sh` | one closed-loop test drive in AlpaSim (sets up AlpaSim, runs Pilot, prints the score) |
| `SHA256SUMS` | checksums; the weights and `pilot/` match the serving image used for every result above |

Weights SHA-256: VLM `b9de4bf4…caccc`, perception `e964ca94…450ae`, planner `a8e5d329…b4d14` (full hashes in `SHA256SUMS`).

## License

Apache-2.0. Pilot is a derivative of Qwen-Drive-1.0-4B (Apache-2.0, © Alibaba Group); see `NOTICE`.

## Citation

```bibtex
@misc{simforge2026pilot,
  title  = {SimForge Pilot-4B v1.5},
  author = {{SimForge}},
  year   = {2026},
  url    = {https://huggingface.co/SIMFORGE-AI/Pilot-4B-v1.5}
}
```