colichar/space-invaders-dqn

DQN — ALE/SpaceInvaders-v5

A from-scratch PyTorch reimplementation of DeepMind's DQN Atari papers (Mnih et al., 2013, Mnih et al., 2015), trained on ALE/SpaceInvaders-v5.

• Code / training pipeline: https://github.com/colichar/deep-q-learning
• Architecture: DeepMind's original conv/FC Q-network (3 conv layers + 2 FC layers), operating on stacks of 4 grayscale 84x84 frames.
• Framework: PyTorch.

Training setup

Trained for 50M frames (~12.5 hours) on a single RTX 4070 Ti Super, with a single gymnasium environment (num_envs=1). Hyperparameters match the DeepMind 2015 paper, with a centered RMSprop optimizer:

Hyperparameter	Value
Optimizer	RMSprop (centered, alpha=0.95, eps=0.01)
Learning rate	0.00025
Discount (gamma)	0.99
Replay memory size	1,000,000 frames
Memory warmup	50,000 frames
Batch size	32
Target network sync	every 10,000 frames
Main network update	every 4 frames
Frame skip	4 (with max-pooling over the last 2 frames for flicker removal)
Reward clipping	{-1, 0, 1}

See the repo README for the full flag list.

Results

Over the last 1M frames of training (501 episodes): average episode reward ≈ 1968.6, max episode reward 4155.

Reward is the 200-episode rolling mean (shaded band: raw per-episode reward); loss is the Huber loss averaged every 400 frames, on a log scale.

Files

• model.pth — torch.save dict with model_state_dict and optimizer_state_dict for the main Q-network.
• episodes.csv — per-episode frame_num,episode_num,episode_reward,epsilon,wall_clock_elapsed_seconds, full training run.
• losses.csv — frame_num,avg_loss, Huber loss averaged every 400 frames.
• training-curves.png — the plot above, generated from the two CSVs.

The replay memory buffer (~14GB) used to resume training locally is intentionally not included here — it's only needed to resume training, not to run or evaluate the model.

Usage

The checkpoint's state dict matches src.models.cnn_model_py.CNNModelPY in the linked repo and is loaded through SpaceInvaderAgent, not standalone torch.load + a bare nn.Module — see scripts/evaluate.py:

git clone https://github.com/colichar/deep-q-learning
cd deep-q-learning
uv sync --extra cpu   # or --extra cuda
uv run AutoROM --accept-license -y

# download this checkpoint's model/ dir from the Hub into, e.g., checkpoint/model/
uv run python scripts/evaluate.py --checkpoint checkpoint --optimizer rmsprop --episodes 5

Limitations

• Trained and evaluated on a single game (ALE/SpaceInvaders-v5); the network is not general-purpose across Atari games without retraining.
• Intended for research/education (reproducing the DQN papers), not as a production game-playing agent.

License

MIT — see LICENSE in the linked repo.

Citation

@article{mnih2013playing,
  title={Playing atari with deep reinforcement learning},
  author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis and Wierstra, Daan and Riedmiller, Martin},
  journal={arXiv preprint arXiv:1312.5602},
  year={2013}
}

@article{mnih2015human,
  title={Human-level control through deep reinforcement learning},
  author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Rusu, Andrei A and Veness, Joel and Bellemare, Marc G and Graves, Alex and Riedmiller, Martin and Fidjeland, Andreas K and Ostrovski, Georg and others},
  journal={Nature},
  volume={518},
  number={7540},
  pages={529--533},
  year={2015},
  publisher={Nature Publishing Group}
}