Add pictures to the model card: training frames and labels, a full lap, and the mask on held-out real footage

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+images/01-drape-before-after.png filter=lfs diff=lfs merge=lfs -text
+images/02-expert-labels.png filter=lfs diff=lfs merge=lfs -text
+images/07-drive-v5-lap-top-down.png filter=lfs diff=lfs merge=lfs -text
+images/08-real-footage-heldout-typical.png filter=lfs diff=lfs merge=lfs -text
+images/09-real-footage-heldout-worst.png filter=lfs diff=lfs merge=lfs -text
+images/progress.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -16,6 +16,10 @@ A small camera-only driving model for an autonomous go-kart on the Purdue Grand
 
 > **Status: a simulator result, not a road-tested model.** It has never seen the kart's own camera and has never driven a real kart. See [What this does not show](#what-this-does-not-show).
 
+![drive-v5 driving a full lap of the simulated Purdue track from the camera alone](images/07-drive-v5-lap-top-down.png)
+
+*drive-v5 driving a full lap (64.6 s) of the simulated track from the camera alone. The path is coloured by speed; the thin orange lines are the lines the model chose along the way.*
+
 Code, training scripts, the simulator and the full version history are in the project repository: https://github.com/ExplodingCB/mcqueen-vip (branch `training-perception`, see `docs/13-driving-model-versions.md`).
 
 ## Files
@@ -27,6 +31,7 @@ Code, training scripts, the simulator and the full version history are in the pr
 | `checkpoints/drive-v4.pt` | drive-v3 plus real helmet-cam footage for the mask |
 | `checkpoints/drive-v6.pt` | Trained on a faster teacher line. Best open-loop numbers, but **does not complete laps**; included as a research result |
 | `model.py`, `driver.py`, `example.py` | Everything needed to load a checkpoint and run it on an image |
+| `images/` | The pictures on this page |
 | `laptime-v2-best.json` | The searched expert line (racing line offsets and speed limits) the faster teacher uses |
 
 ## Checksums (SHA-256)
@@ -56,6 +61,26 @@ mask, line, speed = driver.predict_drive(frame)
 
 The model assumes a camera about 0.7 m above the ground, pitched about 8 degrees down, with a 100 degree horizontal field of view. A different mount shifts the line, because it is in metres. `python example.py frame.jpg` prints the same thing.
 
+## Pictures
+
+**What the model learned from.** The simulator camera drapes the real 2023 aerial photo of the track over the ground (right), instead of the flat grey and green it started with (left). It is still a flat ground with no buildings and one day's lighting, so it is a harder test, not a photograph.
+
+![Simulator camera: flat render on the left, the aerial photo draped over the ground on the right](images/01-drape-before-after.png)
+
+**The labels.** Each training frame carries the expert's racing line (magenta dots) and the speed it wants at each point (yellow, m/s). The camera mount and the image's appearance are randomized so the model does not depend on one exact mount.
+
+![Training frames with the expert's racing line and speeds drawn on them](images/02-expert-labels.png)
+
+**Real footage, for the pavement mask.** drive-v5's pavement mask on frames from three real nose-cam videos it was never trained on, against automatic SegFormer labels. Magenta: the model and the label agree. Red: the model says pavement and the label does not. Blue: the label says pavement and the model missed it. Each tile shows its own score. The first sheet is frames near the median; the second is the **lowest-scoring** frames, so you can see what it gets wrong. Across the 404 frames of these three clips the mean IoU is 0.91 (median 0.95, worst tenth 0.78), against 0.67 for a fixed "the lower part of the picture is road" guess. That guess is already strong on easy clips (0.90 on one of them), so the clip-by-clip numbers matter: 0.97, 0.93 and 0.82, against 0.90, 0.76 and 0.32.
+
+![drive-v5 pavement mask on held-out real frames, near the median](images/08-real-footage-heldout-typical.png)
+
+![drive-v5 pavement mask on the lowest-scoring held-out real frames](images/09-real-footage-heldout-worst.png)
+
+**Progress across versions.** Open-loop errors on unseen simulator frames, the real-footage mask score, and the simulated lap-time search. drive-v6 has the best open-loop numbers but does not complete laps in the closed loop, which is why those numbers alone are not the measure that matters.
+
+![Progress across model versions](images/progress.png)
+
 ## Model
 
 `DriveNet`: a 3.9 M parameter encoder-decoder (a dilated bottleneck, all standard convolutions) with a mask head, plus a driving head that reads the bottleneck and the quarter-resolution decoder features (with pixel-coordinate channels) and regresses the line and speeds. Input is the 240 x 135 half-resolution frame. About 7 ms per frame on one CPU thread.
@@ -75,6 +100,7 @@ The model assumes a camera about 0.7 m above the ground, pitched about 8 degrees
 | Target-speed error | 0.34 m/s |
 | Pavement mask IoU on unseen simulator frames | 0.979 |
 | **Pavement mask IoU on 575 held-out real kart frames, against auto-labels** | **0.84** |
+| Pavement mask IoU on 404 frames from three held-out real nose-cam videos | **0.91** (a fixed guess: 0.67) |
 
 For reference: a simulator-only model scores 0.39 on the real frames, below a fixed "the lower part of the picture is road" guess (0.47). Models trained on the faster teacher (drive-v6) fail to complete laps.