README.md

---
license: mit
tags:
- reinforcement-learning
- dqn
- deep-q-learning
- atari
- ale
- space-invaders
- pytorch
library_name: pytorch
pipeline_tag: reinforcement-learning
---

# DQN — ALE/SpaceInvaders-v5

<p align="center">
  <img src="./result-50M-frames.gif" alt="Trained agent playing Space Invaders"/>
</p>

A from-scratch PyTorch reimplementation of DeepMind's DQN Atari papers ([Mnih et al.,
2013](https://arxiv.org/pdf/1312.5602.pdf), [Mnih et al.,
2015](https://www.nature.com/articles/nature14236/)), trained on `ALE/SpaceInvaders-v5`.

- **Code / training pipeline:** https://github.com/colichar/deep-q-learning
- **Architecture:** DeepMind's original conv/FC Q-network (3 conv layers + 2 FC layers), operating on stacks of 4
  grayscale 84x84 frames.
- **Framework:** PyTorch.

## Training setup

Trained for 50M frames (~12.5 hours) on a single RTX 4070 Ti Super, with a single `gymnasium` environment
(`num_envs=1`). Hyperparameters match the DeepMind 2015 paper, with a centered RMSprop optimizer:

| Hyperparameter | Value |
| --- | --- |
| Optimizer | RMSprop (centered, `alpha=0.95`, `eps=0.01`) |
| Learning rate | 0.00025 |
| Discount (gamma) | 0.99 |
| Replay memory size | 1,000,000 frames |
| Memory warmup | 50,000 frames |
| Batch size | 32 |
| Target network sync | every 10,000 frames |
| Main network update | every 4 frames |
| Frame skip | 4 (with max-pooling over the last 2 frames for flicker removal) |
| Reward clipping | {-1, 0, 1} |

See the [repo README](https://github.com/colichar/deep-q-learning#start-training) for the full flag list.

## Results

Over the last 1M frames of training (501 episodes): **average episode reward ≈ 1968.6**, max episode reward 4155.

<p align="center">
  <img src="./training-curves.png" alt="Episode reward and training loss over 50M frames"/>
</p>

Reward is the 200-episode rolling mean (shaded band: raw per-episode reward); loss is the Huber loss averaged
every 400 frames, on a log scale.

## Files

- `model.pth` — `torch.save` dict with `model_state_dict` and `optimizer_state_dict` for the main Q-network.
- `episodes.csv` — per-episode `frame_num,episode_num,episode_reward,epsilon,wall_clock_elapsed_seconds`, full
  training run.
- `losses.csv` — `frame_num,avg_loss`, Huber loss averaged every 400 frames.
- `training-curves.png` — the plot above, generated from the two CSVs.

The replay memory buffer (~14GB) used to resume training locally is intentionally **not** included here — it's
only needed to resume training, not to run or evaluate the model.

## Usage

The checkpoint's state dict matches `src.models.cnn_model_py.CNNModelPY` in the linked repo and is loaded through
`SpaceInvaderAgent`, not standalone `torch.load` + a bare `nn.Module` — see
[`scripts/evaluate.py`](https://github.com/colichar/deep-q-learning/blob/main/scripts/evaluate.py):

```bash
git clone https://github.com/colichar/deep-q-learning
cd deep-q-learning
uv sync --extra cpu   # or --extra cuda
uv run AutoROM --accept-license -y

# download this checkpoint's model/ dir from the Hub into, e.g., checkpoint/model/
uv run python scripts/evaluate.py --checkpoint checkpoint --optimizer rmsprop --episodes 5
```

## Limitations

- Trained and evaluated on a single game (`ALE/SpaceInvaders-v5`); the network is not general-purpose across
  Atari games without retraining.
- Intended for research/education (reproducing the DQN papers), not as a production game-playing agent.

## License

MIT — see [LICENSE](https://github.com/colichar/deep-q-learning/blob/main/LICENSE) in the linked repo.

## Citation

```bibtex
@article{mnih2013playing,
  title={Playing atari with deep reinforcement learning},
  author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis and Wierstra, Daan and Riedmiller, Martin},
  journal={arXiv preprint arXiv:1312.5602},
  year={2013}
}

@article{mnih2015human,
  title={Human-level control through deep reinforcement learning},
  author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Rusu, Andrei A and Veness, Joel and Bellemare, Marc G and Graves, Alex and Riedmiller, Martin and Fidjeland, Andreas K and Ostrovski, Georg and others},
  journal={Nature},
  volume={518},
  number={7540},
  pages={529--533},
  year={2015},
  publisher={Nature Publishing Group}
}
```