jordykieto/neural-simulators

Neural Simulator

_{More research: The Legend of PPO: Learning to Fight in Hyrule · Learning in Dreams, Winning in Reality · Imitation Learning for Autonomous Driving in CARLA · all research ↗}

Two compact, action-conditioned world models that run as interactive, browser-playable simulators. Every frame is predicted from the model's own previous output; the original simulator is never executed.

	CARLA driving	TinySkies flight
Model	one-step CNN + MLP transition	VAE + 2-layer GRU latent dynamics
Predicts	next RGB 96×96, telemetry, route waypoints, LiDAR embedding	next frame via a 6×6×64 latent
Parameters	1.58 M (6.4 MB fp32)	3.79 M (15.2 MB fp32)
Step time, browser WASM (M1)	5.0 ms	18.3 ms

Files

Path	Contents
carla/world_transition.onnx / .keras	CARLA transition: (vision, lidar_latent, telemetry, waypoints, controls) → next of each
carla/lidar_encoder.onnx / .keras	LiDAR BEV 64×64 → 64-d embedding (applied once, to the seed)
tinyskies/encoder.onnx, decoder.onnx	VAE: RGB 96×96 ↔ 6×6×64 latent
tinyskies/latent_sim.onnx	latent dynamics: 4-step latent + control history, current control → next latent
tinyskies/vae.keras, latent_sim.keras	Keras sources of the TinySkies ONNX files (load with tinyskies/model.py from GitHub)
browser_seeds.json	the six real CARLA states the browser demo starts from
MANIFEST.json	md5, parameter count and input/output signature of every file

The ONNX files are byte-identical to the deployed demo. Provenance is documented in the GitHub repository (PROVENANCE.md).

Usage: drive the CARLA simulator

import json, numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download

get = lambda f: hf_hub_download("jordykieto/neural-simulators", f)
enc = ort.InferenceSession(get("carla/lidar_encoder.onnx"))
sim = ort.InferenceSession(get("carla/world_transition.onnx"))
seed = json.load(open(get("browser_seeds.json")))["samples"][0]
arr = lambda k, s: np.asarray(seed[k], np.float32).reshape((1,) + s)

state = {"vision": arr("vision", (96, 96, 3)), "telemetry": arr("telemetry", (5,)),
         "waypoints": arr("waypoints", (25, 3)),
         "lidar_latent": enc.run(None, {"lidar": arr("lidar", (64, 64, 1))})[0]}
names = [o.name.replace("next_", "") for o in sim.get_outputs()]
for _ in range(100):                                           # 5 s at 20 Hz
    controls = np.array([[0.3, 0.0, 0.0]], np.float32)         # throttle, brake, steer
    state = dict(zip(names, sim.run(None, {**state, "controls": controls})))
frame = state["vision"][0]                                     # (96, 96, 3) in [0, 1]
speed_mps = 20 * np.arctanh(state["telemetry"][0, 0])

TinySkies: encode one 96×96 frame with encoder.onnx, repeat the latent four times as the history (zero control history), then step latent_sim.onnx with (forward, brake, turn) controls and decode next_latent with decoder.onnx.

Limitations

These are compact approximations of slices of their source environments, not replacements. Fine visual detail is never reconstructed; the CARLA speed dynamics have too little damping under sustained heavy throttle; constant inputs drive both models to fixed points; and the CARLA LiDAR embedding is encoded but not propagated by the dynamics. All of this is measured in the paper.

TinySkies is a game by Danny Limanseta (https://github.com/dannylimanseta/tinyskies). The gameplay recordings are in jordykieto/tinyskies-gameplay, credited to the author; the game's imagery remains his.

Citation

@techreport{kieto2026neuralsimulator,
  title  = {Neural Simulator: Lightweight World Models for Continuous Interactive Environments},
  author = {Kieto, Jordy},
  year   = {2026},
  type   = {Technical Report},
  url    = {https://github.com/JordyKieto/neural-simulators}
}