drive-v5 and the checkpoints it came from, with a model card

README.md

@@ -0,0 +1,88 @@
+---
+library_name: pytorch
+pipeline_tag: image-segmentation
+tags:
+  - autonomous-driving
+  - go-kart
+  - racing-line
+  - semantic-segmentation
+  - imitation-learning
+  - robotics
+---
+
+# McQueen driving model (drive-v5)
+
+A small camera-only driving model for an autonomous go-kart on the Purdue Grand Prix track. One RGB frame goes in; three things come out: a **pavement mask**, a **racing line** (12 waypoints, 2 to 24 m ahead, in the kart's frame) and a **target speed** at each waypoint. It does not output steering or throttle. Controllers outside the model turn the line into commands from measured state, and a safety layer keeps the line inside the pavement the camera can see.
+
+> **Status: a simulator result, not a road-tested model.** It has never seen the kart's own camera and has never driven a real kart. See [What this does not show](#what-this-does-not-show).
+
+Code, training scripts, the simulator and the full version history are in the project repository: https://github.com/ExplodingCB/mcqueen-vip (branch `training-perception`, see `docs/13-driving-model-versions.md`).
+
+## Files
+
+| File | What it is |
+| --- | --- |
+| `drive-v5.pt` | **The recommended model.** Real-footage mask training and a clean simulated lap |
+| `checkpoints/drive-v3.pt` | Best simulated lap time (63.8 s); weaker on real footage |
+| `checkpoints/drive-v4.pt` | drive-v3 plus real helmet-cam footage for the mask |
+| `checkpoints/drive-v6.pt` | Trained on a faster teacher line. Best open-loop numbers, but **does not complete laps**; included as a research result |
+| `model.py`, `driver.py`, `example.py` | Everything needed to load a checkpoint and run it on an image |
+| `laptime-v2-best.json` | The searched expert line (racing line offsets and speed limits) the faster teacher uses |
+
+## Checksums (SHA-256)
+
+| File | SHA-256 | Size |
+| --- | --- | --- |
+| `checkpoints/drive-v3.pt` | `ce23623f702185460a999e423af3ac36670f49dfd3e08d41c65285bb1132ac5c` | 15.7 MB |
+| `checkpoints/drive-v4.pt` | `3d995f317eb3467ac0ca801500d060651d1ea6b3f68cea60bd9050f400431677` | 15.7 MB |
+| `checkpoints/drive-v6.pt` | `b1a7813b21b4e79b55e7c2f1c3f9a244fb7641a0550674d96e9f4e3058e057b5` | 15.7 MB |
+| `drive-v5.pt` | `0d6ba0f1a4e45fd28c4f609fbfb0cb7256a9feaf2071bab200498a0fcc12ddcc` | 15.7 MB |
+
+## Use
+
+```python
+# pip install torch numpy pillow
+import numpy as np
+from PIL import Image
+from driver import Driver  # driver.py and model.py sit beside it
+
+driver = Driver("drive-v5.pt")  # CPU by default
+frame = np.asarray(Image.open("frame.jpg").convert("RGB").resize((480, 270)))  # uint8, 270 x 480 x 3
+mask, line, speed = driver.predict_drive(frame)
+# mask : (270, 480) pavement probability
+# line : (12, 2) metres in the kart frame, x forward and y left, from 2 m to 24 m ahead
+# speed: (12,)   metres per second the model wants at each waypoint
+```
+
+The model assumes a camera about 0.7 m above the ground, pitched about 8 degrees down, with a 100 degree horizontal field of view. A different mount shifts the line, because it is in metres. `python example.py frame.jpg` prints the same thing.
+
+## Model
+
+`DriveNet`: a 3.9 M parameter encoder-decoder (a dilated bottleneck, all standard convolutions) with a mask head, plus a driving head that reads the bottleneck and the quarter-resolution decoder features (with pixel-coordinate channels) and regresses the line and speeds. Input is the 240 x 135 half-resolution frame. About 7 ms per frame on one CPU thread.
+
+## Training data
+
+* **Simulator frames:** the Purdue track with the 2023 Indiana orthophoto draped over a flat ground plane (Indiana's imagery service, marked CC0), camera mount and image appearance randomized, labeled with an expert's minimum-curvature racing line and the speed along it.
+* **DAgger frames:** frames from states the model itself reaches in closed loop (with steering bumps), labeled by the expert from where the kart actually is.
+* **Real kart footage, for the mask only:** about 2,300 frames from about 30 YouTube kart onboard videos (helmet cams and nose or bumper cams), pavement labeled automatically by SegFormer (ADE20K classes road, runway, sidewalk, path). **These are pseudo-labels with no human review.** The videos are third-party, so check their terms before redistributing or using this commercially.
+
+## Results (all in the simulator unless stated)
+
+| Measure | drive-v5 |
+| --- | --- |
+| Closed-loop lap, camera only, default limits, no speed cap, 3 seeds | **64.6 s**, 0.39 m closest to an edge |
+| Racing-line error on unseen simulator frames, 2 to 10 m / 12 to 24 m ahead | 0.27 m / 0.54 m |
+| Target-speed error | 0.34 m/s |
+| Pavement mask IoU on unseen simulator frames | 0.979 |
+| **Pavement mask IoU on 575 held-out real kart frames, against auto-labels** | **0.84** |
+
+For reference: a simulator-only model scores 0.39 on the real frames, below a fixed "the lower part of the picture is road" guess (0.47). Models trained on the faster teacher (drive-v6) fail to complete laps.
+
+## What this does not show
+
+* No version has seen the kart's own camera or driven a real kart. The held-out real frames are other people's cameras on other tracks, scored against automatic labels.
+* The racing line and speed were trained only on simulator frames. The mask is the only output that has been trained on real images, so how the line and speed behave on real frames is untested.
+* The track file is hand-traced from aerial imagery, and the simulated tyres, brakes and steering servo are placeholders. A 64.6 s simulated lap is not a prediction of the real kart.
+* The driving path (safety filter, corridor tracker) exists only in the Python simulator and is not yet ported to the kart's software or timed on the Jetson.
+
+The intended use today is research, the simulator, and **shadow mode** (a person drives while the model only watches).

checkpoints/drive-v3.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce23623f702185460a999e423af3ac36670f49dfd3e08d41c65285bb1132ac5c
+size 15691971

checkpoints/drive-v4.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d995f317eb3467ac0ca801500d060651d1ea6b3f68cea60bd9050f400431677
+size 15691907

checkpoints/drive-v6.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1a7813b21b4e79b55e7c2f1c3f9a244fb7641a0550674d96e9f4e3058e057b5
+size 15691907

drive-v5.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d6ba0f1a4e45fd28c4f609fbfb0cb7256a9feaf2071bab200498a0fcc12ddcc
+size 15691907

driver.py

@@ -0,0 +1,55 @@
+"""The driving model behind the simulator's camera-model contract.
+
+    python -m mcq_sim evaluate --policy stack --perception driver:create
+
+with ``PYTHONPATH=training/perception:src/mcq_sim``. Weights come from
+``MCQ_DRIVER_WEIGHTS``, else ``runs/drive-latest/model.pt`` beside this file.
+
+``predict(rgb)`` is the pavement probability, as for a segmenter, so the stack
+can build its corridor. ``predict_drive(rgb)`` adds the racing line the model
+intends to drive, as waypoints in base_link (x forward, y left), and the speed
+it wants at each. The stack drives that line when it stays inside the corridor
+and falls back to its own planner when it does not.
+"""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+
+from model import DriveNet
+
+HERE = Path(__file__).resolve().parent
+
+
+class Driver:
+    def __init__(self, weights: str | Path, device="cpu"):
+        torch.set_num_threads(1)  # many drivers run side by side in rollouts; one thread each
+        checkpoint = torch.load(weights, map_location=device, weights_only=True)
+        self.net = DriveNet(fine=bool(checkpoint.get("fine", False))).to(device).eval()
+        self.net.load_state_dict(checkpoint["state"])
+        self.device = device
+        self.weights = str(weights)
+
+    @torch.no_grad()
+    def predict_drive(self, rgb: np.ndarray):
+        """uint8 HWC RGB (270 x 480) in; (pavement probability 270 x 480,
+        waypoints K x 2 in metres, speeds K in m/s) out."""
+        x = torch.from_numpy(np.ascontiguousarray(rgb)).to(self.device).permute(2, 0, 1)[None].float()
+        h, w = x.shape[-2:]
+        logits, points, speeds = self.net.forward_all(F.avg_pool2d(x, 2))
+        probability = torch.sigmoid(F.interpolate(logits[None], size=(h, w), mode="bilinear", align_corners=False))
+        out = probability[0, 0].cpu().numpy(), points[0].cpu().numpy(), speeds[0].cpu().numpy()
+        return out
+
+    def predict(self, rgb: np.ndarray) -> np.ndarray:
+        return self.predict_drive(rgb)[0]
+
+
+def create():
+    path = os.environ.get("MCQ_DRIVER_WEIGHTS", HERE / "runs" / "drive-latest" / "model.pt")
+    return Driver(path)

example.py

@@ -0,0 +1,20 @@
+"""Run a checkpoint on one image:  python example.py frame.jpg [drive-v5.pt]"""
+import sys
+
+import numpy as np
+from PIL import Image
+
+from driver import Driver
+
+image = sys.argv[1] if len(sys.argv) > 1 else None
+weights = sys.argv[2] if len(sys.argv) > 2 else "drive-v5.pt"
+driver = Driver(weights)
+frame = (
+    np.asarray(Image.open(image).convert("RGB").resize((480, 270)))
+    if image
+    else np.full((270, 480, 3), 110, dtype=np.uint8)
+)
+mask, line, speed = driver.predict_drive(frame)
+print(f"{weights}: pavement share {float((mask > 0.5).mean()):.2f}")
+for k, ((x, y), v) in enumerate(zip(line, speed, strict=True)):
+    print(f"  waypoint {k:2d}: {x:5.1f} m ahead, {y:+5.2f} m left, wants {v:4.1f} m/s")

laptime-v2-best.json

@@ -0,0 +1,78 @@
+{
+  "theta": [
+    -0.020713883569653507,
+    0.7986371171163591,
+    -0.06007090210526833,
+    -0.13692955258757095,
+    0.3164787844956688,
+    0.22910077232445988,
+    0.4085683741777101,
+    0.1732443452654597,
+    0.31628061199629176,
+    0.2936319802034716,
+    -0.007033094360031467,
+    0.0013200730496496782,
+    0.40243794379289366,
+    0.765853695695947,
+    0.5178593376332585,
+    0.33366131041959013,
+    1.0801067821758314,
+    0.7981998198285803,
+    -0.3385555048128157,
+    -0.3040821312994424,
+    1.6182991758783387,
+    0.3389952005170975,
+    -0.22789996980313015,
+    0.4039164824230546,
+    0.4619468070190555,
+    0.21125474388800541,
+    -0.5006066348041253,
+    -0.8610472716707411,
+    -0.20847536428974067,
+    -0.2394684425191484,
+    -0.9391657225767008,
+    -0.9217363080423312,
+    0.06200212942970203,
+    0.6460498410297608,
+    0.9070343531284966,
+    1.2993982884320447,
+    0.6067228170245667,
+    0.8575524174357774,
+    0.9950192746873036,
+    -0.16623329240627815,
+    4.464017710110649,
+    1.7967631217917692,
+    2.0052742170641062
+  ],
+  "score": 55.65999999998595,
+  "results": [
+    {
+      "passed": true,
+      "lap_s": 54.64999999998654,
+      "progress_m": 435.10000000004015,
+      "clearance_m": 0.2806985629754275,
+      "max_speed": 12.532168425742148,
+      "corrections": 14,
+      "stop": null
+    },
+    {
+      "passed": true,
+      "lap_s": 55.29999999998595,
+      "progress_m": 435.1000000000418,
+      "clearance_m": 0.2562416972740049,
+      "max_speed": 12.49933797767215,
+      "corrections": 12,
+      "stop": null
+    },
+    {
+      "passed": true,
+      "lap_s": 54.64999999998654,
+      "progress_m": 435.25000000004,
+      "clearance_m": 0.4983136894426503,
+      "max_speed": 12.505934752719071,
+      "corrections": 0,
+      "stop": null
+    }
+  ],
+  "generation": 103
+}

model.py

@@ -0,0 +1,127 @@
+"""A small encoder-decoder for the pavement mask.
+
+About 1.6 M parameters, all standard convolutions, so it exports to TensorRT
+without surprises and runs far inside the 20 fps budget on the Orin. The input is
+the 240 x 135 half-resolution frame; the output is one logit per pixel at the
+same size. A dilated bottleneck gives the receptive field to see the whole road
+shape at once, which matters more than texture: pavement and grass can be the
+same color, and the shape of the corridor still says which is which.
+"""
+
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+def block(cin, cout, stride=1, dilation=1):
+    return nn.Sequential(
+        nn.Conv2d(cin, cout, 3, stride, padding=dilation, dilation=dilation, bias=False),
+        nn.BatchNorm2d(cout),
+        nn.ReLU(inplace=True),
+        nn.Conv2d(cout, cout, 3, 1, padding=dilation, dilation=dilation, bias=False),
+        nn.BatchNorm2d(cout),
+        nn.ReLU(inplace=True),
+    )
+
+
+class PavementNet(nn.Module):
+    def __init__(self, width=24):
+        super().__init__()
+        w = width
+        self.e1 = block(3, w, stride=2)  # 1/2
+        self.e2 = block(w, 2 * w, stride=2)  # 1/4
+        self.e3 = block(2 * w, 4 * w, stride=2)  # 1/8
+        self.e4 = block(4 * w, 6 * w, stride=2)  # 1/16
+        self.context = nn.Sequential(block(6 * w, 6 * w, dilation=2), block(6 * w, 6 * w, dilation=4))
+        self.d3 = block(6 * w + 4 * w, 4 * w)
+        self.d2 = block(4 * w + 2 * w, 2 * w)
+        self.d1 = block(2 * w + w, w)
+        self.head = nn.Conv2d(w, 1, 1)
+
+    @staticmethod
+    def _up(x, like):
+        return F.interpolate(x, size=like.shape[-2:], mode="bilinear", align_corners=False)
+
+    def forward(self, x):
+        size = x.shape[-2:]
+        x = x / 255.0 - 0.5
+        e1 = self.e1(x)
+        e2 = self.e2(e1)
+        e3 = self.e3(e2)
+        e4 = self.context(self.e4(e3))
+        d3 = self.d3(torch.cat([self._up(e4, e3), e3], 1))
+        d2 = self.d2(torch.cat([self._up(d3, e2), e2], 1))
+        d1 = self.d1(torch.cat([self._up(d2, e1), e1], 1))
+        return F.interpolate(self.head(d1), size=size, mode="bilinear", align_corners=False)[:, 0]
+
+
+class DriveNet(PavementNet):
+    """PavementNet with a driving head: from the same encoder, the racing line
+    as ``n_waypoints`` points in the kart's frame (x forward, y left) and the
+    target speed at each. The mask head stays, so the stack can still check the
+    driving output against the pavement it sees.
+
+    The head reads the whole bottleneck, not a pooled vector, because where the
+    road goes is a matter of where in the image it is. Outputs are scaled to
+    about unit size: x / 25 m, y / 10 m, speed / 15 m/s.
+    """
+
+    SCALE = (25.0, 10.0, 15.0)
+
+    def __init__(self, width=24, n_waypoints=12, grid=(9, 15), fine=False):
+        super().__init__(width)
+        self.n_waypoints = n_waypoints
+        self.squeeze = nn.Sequential(nn.Conv2d(6 * width, 32, 1), nn.BatchNorm2d(32), nn.ReLU(inplace=True))
+        self.grid = grid
+        # ``fine``: the head also reads the decoder's quarter-resolution features
+        # with two pixel-coordinate channels, then strides them down to the grid.
+        # Where the kart sits relative to the line is read from where the edges
+        # are 5 to 10 m ahead, a matter of a few pixels the 1/16 bottleneck blurs.
+        self.fine = fine
+        if fine:
+            self.fine_path = nn.Sequential(
+                nn.Conv2d(2 * width + 2, 64, 3, 2, 1, bias=False),
+                nn.BatchNorm2d(64),
+                nn.ReLU(inplace=True),
+                nn.Conv2d(64, 64, 3, 2, 1, bias=False),
+                nn.BatchNorm2d(64),
+                nn.ReLU(inplace=True),
+                nn.Conv2d(64, 32, 1),
+                nn.BatchNorm2d(32),
+                nn.ReLU(inplace=True),
+            )
+        head_in = 64 if fine else 32
+        self.drive = nn.Sequential(
+            nn.Flatten(),
+            nn.Linear(head_in * grid[0] * grid[1], 256),
+            nn.ReLU(inplace=True),
+            nn.Dropout(0.1),
+            nn.Linear(256, 3 * n_waypoints),
+        )
+
+    def forward_all(self, x):
+        """(mask logits N x H x W, waypoints N x K x 2 in metres, speeds N x K in m/s)."""
+        size = x.shape[-2:]
+        x = x / 255.0 - 0.5
+        e1 = self.e1(x)
+        e2 = self.e2(e1)
+        e3 = self.e3(e2)
+        e4 = self.context(self.e4(e3))
+        d3 = self.d3(torch.cat([self._up(e4, e3), e3], 1))
+        d2 = self.d2(torch.cat([self._up(d3, e2), e2], 1))
+        d1 = self.d1(torch.cat([self._up(d2, e1), e1], 1))
+        mask = F.interpolate(self.head(d1), size=size, mode="bilinear", align_corners=False)[:, 0]
+        code = F.adaptive_avg_pool2d(self.squeeze(e4), self.grid)
+        if self.fine:
+            n, _, h, w = d2.shape
+            ys = torch.linspace(-1, 1, h, device=d2.device).view(1, 1, h, 1).expand(n, 1, h, w)
+            xs = torch.linspace(-1, 1, w, device=d2.device).view(1, 1, 1, w).expand(n, 1, h, w)
+            detail = self.fine_path(torch.cat([d2, ys, xs], 1))
+            code = torch.cat([code, F.adaptive_avg_pool2d(detail, self.grid)], 1)
+        out = self.drive(code).view(-1, self.n_waypoints, 3)
+        sx, sy, sv = self.SCALE
+        points = torch.stack([out[..., 0] * sx, out[..., 1] * sy], -1)
+        speeds = out[..., 2] * sv
+        return mask, points, speeds