racingmania-pqn-seed1
PQN, seed 1: PQN baseline
Status: evaluated (evaluated; scores available); physics version v1; card generated 2026-11-15 from the result files.
RacingMania is an original 3D time-attack racing game for the browser (three.js + cannon-es RaycastVehicle) with reinforcement-learning drivers trained on a batched PyTorch port of the same car physics. Code, paper and full results: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game.
Videos
Official evaluation runs of this model replayed in the browser game ("Watch replay"): the car states the evaluator recorded while scoring the run, so each clip is exactly the run its caption describes. Clip numbers are those of the README, which has more clips, including the controls: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game#videos.
Best lap of the PQN baseline
- What you see: The baseline algorithm, PQN (parallelised Q-learning), trained with the same inputs, actions, reward and step budget as the PPO generalist on generated tracks. The track was never part of its training.
- Model:
pqn/seed1: PQN, trained by reinforcement learning for 1 billion agent steps on 48,949 generated training tracks (seed 1, 1.1 h on one GPU). - Map: Technical (
campaign-04), a hand-made campaign track, 666 m long with no jumps and up to 5° of banking; this model never saw it during training. - Evaluation: official evaluation in
results/pqn/seed1/eval_campaign.csv(standing start, most probable action, final checkpoint): finished in 26.04 s; the scripted driver needs 42.52 s, so this is 1.63× its speed. The clip replays this run. - Why this clip: chosen by a fixed rule as the highest-scoring finished track of the 5 campaign tracks it finished in its official evaluation.
Model
| Item | Value |
|---|---|
| Algorithm | PQN |
| Architecture | Q-network 2x256, LayerNorm, one 9-way head |
| Objective | centreline progress |
| Observation | car state + 20 look-ahead points |
| Training data | 48,949 generated tracks |
| Start states | standing + random centreline (p=0.5) |
| Agent steps trained | 1,000,341,504 |
| Wall-clock hours | 1.10 |
| Git commit | 70a35809b494c89e7fa074064a9c8dc961ef0360 |
Evaluation
Success-weighted speed (SWS): mean over tracks of finished x (scripted driver's time / agent's time); 1.0 means finishing every track as fast as the scripted reference driver. One deterministic run per track from a standing start with the most probable action; intervals are 95 % percentile bootstrap intervals over tracks (one seed).
| Track set | Tracks | SWS [95 % CI] | Finished |
|---|---|---|---|
| Campaign (hand-made) | 5 | 1.391 [1.270, 1.517] | 100.0 % |
| Out-of-range test | 187 | 0.725 [0.631, 0.820] | 55.1 % |
| Test (unseen generated) | 971 | 1.054 [1.020, 1.086] | 79.8 % |
| Unfiltered test | 200 | 1.105 [1.035, 1.173] | 83.0 % |
| Own training tracks | 1000 | 1.052 [1.019, 1.085] | 79.7 % |
| Training sample | 1000 | 1.052 [1.019, 1.085] | 79.7 % |
| Validation | 197 | 1.034 [0.956, 1.109] | 78.2 % |
How to use
Files: final.pt (PyTorch checkpoint: agent and norm state dicts, ppo and env_cfg settings, obs_dim), policy.onnx (observation function + frozen normaliser + actor in one graph, as run by the browser game), run.json, training and validation curves, and per-track evaluation CSVs under eval/.
PyTorch, inside a clone of the repository (uv sync):
PQN checkpoints hold a Q-network; load them with racingmania.rl.pqn (see the repository).
ONNX (inputs and outputs are specified in docs/formats.md, section "Policy model"):
import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("policy.onnx")
inputs = {i.name: np.zeros([d if isinstance(d, int) else 1 for d in i.shape], np.float32) for i in sess.get_inputs()}
steer_logits, pedal_logits = sess.run(None, inputs)
steer, pedal = int(steer_logits.argmax()), int(pedal_logits.argmax()) # 0/1/2 = left/none/right, gas/coast/brake
To watch it drive, open the game (web/index.html, or the GitHub Pages site), choose it in a track's AI drivers table and press "Watch AI" (live in the browser) or "Watch replay" (its official evaluation run).
Training configuration (from run.json)
| Setting | Value |
|---|---|
total_steps |
1000000000 |
num_envs |
16384 |
num_steps |
32 |
lr |
0.00025 |
adaptive_lr |
False |
desired_kl |
0.01 |
lr_min |
1e-05 |
lr_max |
0.01 |
gamma |
0.99 |
gae_lambda |
0.95 |
num_minibatches |
4 |
update_epochs |
4 |
clip_coef |
0.2 |
ent_coef |
0.01 |
vf_coef |
1.0 |
max_grad_norm |
10.0 |
weight_decay |
0.0 |
hidden |
[256, 256] |
action_mode |
joint |
eval_every_minutes |
10.0 |
checkpoint_every_minutes |
10.0 |
rotate_tracks_every |
20 |
loaded_tracks |
4096 |
seed |
1 |
compile |
True |
q_lambda |
0.65 |
start_e |
1.0 |
end_e |
0.05 |
exploration_fraction |
0.5 |
env.random_start_prob |
0.5 |
env.random_start_speed |
[0.0, 30.0] |
env.random_start_lateral |
0.5 |
env.max_steps |
1500 |
env.reward_scale |
0.1 |
env.window_k |
20 |
env.window_stride |
3 |
env.blind |
False |
env.time_cost |
0.0 |
env.fall_penalty |
0.0 |
env.hint_weight |
0.0 |
env.scripted_starts |
False |
Licence and citation
MIT. Author: Libor "Burny" Burian. Repository: https://github.com/BurnyCoder/reinforcement-learning-in-trackmania-like-racing-game.
Collection including BurnyCoder/racingmania-pqn-seed1
Evaluation results
- SWS on RacingMania test (unseen generated) tracksself-reported1.054
- Finish rate on RacingMania test (unseen generated) tracksself-reported0.798
- SWS on RacingMania validation tracksself-reported1.034
- Finish rate on RacingMania validation tracksself-reported0.782
- SWS on RacingMania training sample tracksself-reported1.052
- Finish rate on RacingMania training sample tracksself-reported0.797
- SWS on RacingMania own training tracksself-reported1.052
- Finish rate on RacingMania own training tracksself-reported0.797