FPV gate racing: a camera-only pilot and its state-based teacher

A small CNN that flies a quadrotor through six-gate race tracks from the FPV camera image alone, outputting the four stick commands a human pilot would (thrust and roll / pitch / yaw rate) at 50 Hz. It was trained by imitating a state-based PPO policy. Everything is simulation: Crazyflow does the physics and Project AirSim (Unreal Engine 5) only draws the camera image.

Code and write-up: https://github.com/Abuudiii/rl-experiments/tree/main/fpv-racing

Camera student tuned on two tracks: figure-8 and climb-and-dive

The camera student below was trained directly on these two tracks in Unreal, the same way the single-circuit student was: it flies, the teacher labels what it should have done, and it is retrained (DAgger). It started from the random-track checkpoint of the next section and, like it, sees the last five frames with the next gate drawn green. It is tracks/student_two/ckpt.pkl (2.44 M parameters, 3.1 ms per call on an H100).

Track (trained on) ≥ 3 laps, 96 randomised episodes [95% interval] Crashes Lap time Nominal track: ≥ 3 laps (24 episodes) Teacher, same episodes
Figure-8 100% [96, 100] 0% 6.74 s 100% 100% (6.75 s)
Climb and dive 100% [96, 100] 0% 5.10 s 100% 100% (5.09 s)

One evaluation pass per track, 30 s episodes, with randomised drone physics, gate positions (±0.3 m, ±0.2 rad per gate) and spawns, on a seed not used for training data. The checkpoint was fixed beforehand on a separate selection seed (48 of 48 episodes with three laps on each track). Training used about 447,000 Unreal frames flown on these two tracks (four rounds), mixed with the earlier random-track frames at 30% of every batch. With zero crashes in 96 episodes the interval still allows a crash rate of up to about 4%.

Three-lap rate by track: fine-tuned vs never trained on

The grey bars are the untuned checkpoint of the next section.

Stick error against the teacher over the fine-tuning rounds

Stick error against the teacher after each fine-tuning round (8 flights per round, so a trend rather than a precise number). The published checkpoint is round 4, the one checked on the selection seed; later rounds were not evaluated.

Each video is the first flight from a seed fixed in advance: 27 gates in 30 s on the figure-8 and 35 on the climb-and-dive, no crash.

Figure-8 (trained on)

Climb and dive (trained on)

In each video, top left is what the pilot sees, with the sticks it commands as bars (white tick: what the teacher would have done in the same state); top right is a chase camera; bottom, a fixed overview camera and a map. The drone model is drawn at twice its true size so it is visible; the pilot's view never contains it.

Camera student on four other tracks (not tuned)

A second camera student was trained on randomly generated tracks only, then frozen. The table and videos below show how that frozen checkpoint (tracks/student_asis/ckpt.pkl) flies four named tracks in Unreal. It was not tuned on any of them, and none of the four was in its training data. It is published as it flies today, weak spots included.

Two things differ from the single-circuit student below: it sees the last five frames instead of three, and the next gate to fly through is drawn green (the others stay magenta), because on a track it has never seen the image alone does not say which gate comes next.

Track (not tuned) ≥ 3 laps, 48 randomised episodes [95% interval] Crashes Lap time Nominal track: ≥ 3 laps (24 episodes) Teacher, same episodes
Circuit 91.7% [80, 97] 10.4% 4.97 s 100% 100%
Slalom 79.2% [66, 88] 27.1% 6.62 s 87.5% 100%
Clockwise 72.9% [59, 83] 33.3% 5.10 s 100% 100%
Sprint into a hairpin 35.4% [23, 50] 72.9% 6.44 s 8.3% 100%

One evaluation pass per track, 30 s episodes, with randomised drone physics and gate positions; "nominal" is the track and drone without randomisation. On the hairpin track 31 of its 35 crashes are on the leg right after the hairpin: it drifts off the teacher's line there, loses sight of the next gate and sinks.

Each video is the first flight from a seed fixed in advance, kept however it went: 36 gates on the circuit, 37 clockwise and 27 on the slalom without a crash; on the hairpin track it hits the ground after 14.8 s and 14 gates.

Circuit (not tuned)

Slalom (not tuned)

Clockwise (not tuned)

Sprint into a hairpin (not tuned; crashes at 14.8 s)

The figure-8 and the climb-and-dive are not in this table: the student of the previous section was trained on those two. That tuned student was not evaluated on these four tracks.

Single-circuit camera student

The first student: three frames, no green gate, one six-gate circuit.

The student flying 30 s on the nominal circuit from its camera alone (39 gates, no crash).

Result

Closed loop in Unreal, student alone, 20 s per episode. The teacher is flown through the same harness on the same layouts, physics and spawns.

Pilot Input Randomised: ≥ 3 laps (192 held-out episodes) Crashes Lap time Nominal circuit: ≥ 3 laps (24 episodes)
Student (student/ckpt.pkl) 3 camera frames + its previous action 95.8% (184) 4.2% (8) 4.60 s 100%
Teacher (teacher/ckpt_latest.pkl) exact state (43 numbers) 100% 0% 4.59 s 100%

"Randomised" draws a new gate layout (±0.3 m, ±0.2 rad per gate), mass, thrust, drag and command delay per episode. All 8 student crashes are gate-frame hits and 7 of them happen in the first 4 seconds, before the third gate. student/eval.json covers 216 randomised episodes (96.3% with three laps). The first 24 of them repeat the seed, and so the layouts and spawns, of an earlier evaluation whose flights went into the training data; the table leaves them out (all 24 completed three laps). Per-episode results are in student/eval.episodes.json.

Top-down paths of student and teacher

The teacher on the other track shapes

The single-circuit student knows one circuit. A second teacher, trained with PPO on randomly generated six-gate tracks, flies the track shapes below. The videos below show that state-based teacher, not a camera pilot: the camera students' flights on these tracks are in the sections above.

Track ≥ 3 laps (2,048 randomised episodes) Lap time Crashes per flight-minute with stick noise (std 0.2)
Circuit (the original) 100% 4.73 s 0.009
Clockwise 100% 4.70 s 0.008
Figure-8 100% 6.71 s 0.059
Slalom 100% 6.21 s 0.025
Climb and dive 100% 5.07 s 0.024
Sprint into a hairpin 100% 5.41 s 0.036
Random tracks (what it trained on) 100% 5.92 s 0.039

Five of the six named tracks were held out of training; the sixth, the original circuit, is where the teacher it started from was trained. The figure-8 is close to one of the generator's own shape families, so it is the weakest test; the hairpin and the climb-and-dive are the furthest from the training mix. The slalom weaves through three of its six gates, since a closed six-gate loop cannot weave all the way round. Files: tracks/teacher/ (checkpoint, settings, evaluation, scenarios.png), tracks/track.json (the layouts).

Figure-8

Slalom

Climb and dive

Sprint into a hairpin

Clockwise

The two policies

Teacher Student
Trained with PPO, reward for progress through gates Supervised imitation of the teacher with DAgger (no reward)
Input Velocity, attitude, body rates, corners of the next two gates in the body frame, previous action Last three 320×180 RGB frames (camera pitched up 20°, 110° horizontal field of view) and its previous action
Network MLP, 3 × 256, separate actor and critic 6 conv layers + MLP, 2.43 M parameters
Output [thrust, roll rate, pitch rate, yaw rate] in [-1, 1] at 50 Hz same
Training 6 billion steps in two stages, about 13 minutes each on one H100 (8,192 drones in parallel) Pre-trained on a fast synthetic renderer, then fine-tuned on about 291,000 Unreal frames, the last three rounds flown by the student in Unreal

Starting the student on Unreal frames from the synthetic-renderer weights mattered: with the same first 171,000 Unreal frames, the pre-trained network completed three laps in 24 of 24 test episodes and a network trained from scratch in none.

Use

The checkpoints are Python pickles holding only numpy arrays and plain config values. Unpickling can run code, so load them only from a source you trust. Loading needs fpv/ from the GitHub repository plus jax, flax, optax.

import numpy as np
from huggingface_hub import hf_hub_download
from fpv.student import StudentRunner, load_student

cfg, params, _ = load_student(hf_hub_download("Abuudiii/fpv-gate-racing", "student/ckpt.pkl"))
runner = StudentRunner(cfg, params, n=1)                  # keeps the 3-frame history
frame = np.zeros((1, cfg.height, cfg.width, 3), np.uint8)  # the current FPV frame, RGB
action = runner.act(frame, prev_action=np.zeros((1, 4), np.float32), new=np.array([True]))
# action[0] = [thrust, roll rate, pitch rate, yaw rate] in [-1, 1]; fpv.env.scale_action maps it to N and rad/s

One call takes 5.8 ms on an H100, against the 20 ms control step. Flying it closed loop (and loading the teacher) needs Crazyflow and Project AirSim; see scripts/eval_student.py and fpv/closed_loop.py in the repository.

Limits

  • One Unreal scene (Blocks). The single-circuit student knows one circuit; the two-track student was trained on the figure-8 and the climb-and-dive only; the four other tracks are shown with an untuned checkpoint that is weak on the hairpin track. The five-frame students need the next gate drawn green. Another look is untested.
  • Cruise speed, about 4 m/s, on a simulated Crazyflie-class drone (cf21B_500); this is not aggressive racing.
  • Simulation only. It has not been run against real flight-controller firmware or a real camera.

Files

File What
student/ckpt.pkl Camera-only student (final round)
student/eval.json, student/eval.episodes.json Final closed-loop evaluation in Unreal, student and teacher
student/rounds.jsonl, student/train_config.json Unreal correction rounds and their settings
teacher/ckpt_latest.pkl State-based PPO teacher that labelled the student's data
teacher/config.json, teacher/eval.json Teacher environment / PPO settings and its evaluation in Crazyflow
third_person/flight_combined.mp4 The flight above; flight_fpv.mp4, flight_chase.mp4, flight_overview.mp4 are its single views
tracks/student_two/ckpt.pkl Camera student trained on the figure-8 and the climb-and-dive in Unreal (5 frames, next gate drawn green)
tracks/student_two/eval_<track>.json, .episodes.json, table.md Its final evaluation in Unreal on those two tracks, with the teacher on the same episodes
tracks/student_two/videos/<track>_combined.mp4, <track>_flight.json One flight per track and its record
tracks/student_asis/ckpt.pkl Camera student trained on random tracks only (5 frames, next gate drawn green), frozen and not tuned on any named track
tracks/student_asis/eval_<track>.json, .episodes.json, table.md Its one evaluation pass in Unreal on circuit, clockwise, slalom and sprint_hairpin, with the teacher on the same episodes
tracks/student_asis/videos/<track>_combined.mp4, <track>_flight.json One flight per track (pilot's view, chase, overview, map) and its record
fpv_flight.mp4, trajectory_student_vs_teacher.png An earlier flight of the same student (pilot's view only) and its top-down path
Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading