Zahhak-snakehandler

A Deep Reinforcement Learning Agent for Snake — From Scratch

Zahhak-snakehandler is a PyTorch-based reinforcement learning project that learns to play Snake through reward-driven interaction with its environment. It implements Double Deep Q-Network (Double DQN), experience replay, a target network, and configurable training on CPU, NVIDIA CUDA GPUs, and Apple Silicon MPS.

The project includes a live Pygame dashboard, headless training, resumable checkpoints, and a dedicated evaluation workflow comparing a greedy policy against a random-action baseline.

Source code: kheradafrouz/Zahhak-snakehandler

Model Details

Property	Description
Model name	Zahhak-snakehandler
Model type	Deep Reinforcement Learning Agent
Algorithm	Double Deep Q-Network (Double DQN)
Framework	PyTorch 2.x
Environment	Custom Snake environment
Observation space	14 binary-valued features
Action space	3 relative actions
Network architecture	Multilayer Perceptron (MLP)
Default hidden layers	128, 128
Default optimizer	Adam
Default learning rate	0.001
Default discount factor	0.95
License	MIT

Intended Use

This agent is intended for:

• Learning and experimenting with deep reinforcement learning.
• Demonstrating Double DQN and experience replay.
• Studying reward shaping and exploration strategies.
• Comparing greedy policy performance with a random baseline.
• Experimenting with training speed across CPU and GPU devices.
• Educational projects involving game-playing agents.

This project is designed for the Snake game environment. It is not a general-purpose language model or a model for real-world autonomous control.

Architecture

The agent uses a feed-forward neural network to estimate action values, Q(s, a), for the three available actions.

The implementation includes:

• Policy network: Selects actions and estimates Q-values.
• Target network: Provides more stable target estimates during learning.
• Experience replay buffer: Samples past transitions for training.
• Double DQN: Uses the policy network to select the next action and the target network to evaluate it.
• Huber loss: Reduces sensitivity to large temporal-difference errors.
• Gradient clipping: Helps control large gradients.
• Adam optimizer: Updates the policy network parameters.

The default network has two hidden layers of 128 units each, with ReLU activations. Hidden layer sizes are configurable.

Observation Space

The agent receives a 14-feature state representation:

Feature group	Number of features	Description
Danger indicators	6	Collision danger one and two cells ahead for straight, right, and left directions
Heading	4	One-hot encoding of the snake's current direction
Food direction	4	Relative food position: left, right, up, and down
Total	14	Binary-valued features

The observation is a compact representation of local danger, orientation, and food direction rather than a raw image of the board.

Action Space

The agent chooses among three relative actions:

1. Move straight.
2. Turn right.
3. Turn left.

A direct 180-degree reversal is excluded by the environment's action design.

Reward Function

The default reward configuration is:

Event	Reward
Eating food	+10.0
Death	-10.0
Starvation timeout	-10.0
Moving closer to food	+0.1
Moving farther from food	-0.15

The environment also terminates an episode after 100 times the current snake length in steps without eating food, preventing indefinite loops.

These rewards encourage food collection and survival while discouraging collisions, starvation, and movement away from food.

Training Configuration

Default training settings:

Parameter	Default
Grid size	15 × 15
Hidden layer sizes	128, 128
Learning rate	0.001
Discount factor (gamma)	0.95
Batch size	64
Replay buffer capacity	50,000
Learning frequency	Every 2 environment steps
Target network synchronization	Every 500 gradient steps
Initial epsilon	1.0
Minimum epsilon	0.02
Epsilon decay	0.997 per episode
Random seed	42
Default training duration	3,000 episodes

The project also provides a GPU preset with larger hidden layers, batches, and replay buffer capacity.

All settings can be adjusted through the project configuration and command-line interface.

Exploration Strategy

The agent uses epsilon-greedy exploration.

At the beginning of training, the agent explores extensively with epsilon set to 1.0. Epsilon decays by a factor of 0.997 per episode until it reaches the minimum value of 0.02.

This allows the agent to gradually shift from random exploration toward actions selected by its learned Q-values.

Evaluation

Zahhak-snakehandler includes a separate evaluation module that compares:

• A greedy policy with epsilon = 0.
• A random-action baseline.

Evaluation runs in a separate seeded environment and reports metrics such as average score, maximum score, and average survival length.

No fixed benchmark results are claimed in this model card because performance can vary with random seeds, training duration, hardware, and software versions.

For reproducible comparisons, evaluate multiple seeds and report the training configuration alongside the results.

Evaluation instructions and implementation: GitHub repository

Hardware and Performance

Supported compute backends:

• CPU: Standard PyTorch installation.
• NVIDIA CUDA: GPU training with a compatible PyTorch CUDA build and NVIDIA driver.
• Apple Silicon MPS: Supported where the installed PyTorch version and device provide the required functionality.

The GPU preset uses hidden layers of 256 units, a batch size of 512, and a replay buffer capacity of 200,000.

A GPU is not guaranteed to be faster for this workload. The network is relatively small, and action selection often processes one state at a time. Performance should be measured on the target machine.

Limitations

• The agent is designed for the project's custom Snake environment.
• Learning quality depends on the reward design, hyperparameters, random seed, and training duration.
• Training reward does not necessarily reflect the performance of the final greedy policy.
• Successful execution does not guarantee successful learning.
• The project does not claim a particular score or level of performance without measured evaluation results.
• Checkpoints must be loaded with compatible model architecture and appropriate software dependencies.

Intended Audience

This project is suitable for students, researchers, and developers exploring reinforcement learning, PyTorch, Q-learning, and game-playing agents.

It is intended for educational and experimental use rather than safety-critical applications.

Usage and Installation

The complete installation instructions, dependencies, training commands, and dashboard controls are documented in the source repository.

Repository: https://github.com/kheradafrouz/Zahhak-snakehandler

Clone and install the project:

git clone https://github.com/kheradafrouz/Zahhak-snakehandler.git
cd Zahhak-snakehandler

python -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

Train the agent without rendering:

python -m zahhak_snakehandler.train --episodes 3000

Launch the dashboard:

python -m zahhak_snakehandler.main

Evaluate a saved checkpoint:

python -m zahhak_snakehandler.evaluate \
  --model checkpoints/Zahhak-snakehandler_best.pt \
  --episodes 200

For CUDA-specific installation and commands, see the repository's CPU vs GPU section.

Checkpoints

The training system supports resumable checkpoints that can store:

• Policy and target network weights.
• Optimizer state.
• Exploration parameter (epsilon).
• Replay buffer.
• Training metric history.

Checkpoints are written atomically to reduce the risk of incomplete files during saving.

The commands and exact checkpoint formats are documented in the source repository.

Reproducibility

The project supports configurable random seeds and exposes its training hyperparameters through its configuration module.

For reproducible experiments:

1. Record the software versions and hardware.
2. Record the seed and training configuration.
3. Evaluate the greedy policy separately from training.
4. Compare against the random-action baseline.
5. Report results across multiple seeds.

Training Data

The agent learns through interactions with the Snake environment using rewards and state transitions. It does not require a conventional labeled training dataset.

The environment rules and reward implementation are available in snake_env.py.

Ethical Considerations

This is a game-playing research and educational project with no intended direct impact on people. Its learned behavior is specific to the environment and reward function used during training.

Results should not be generalized to unrelated environments without further training and evaluation.

Citation

If you use this project in research or educational work, cite the repository:

@software{zahhak_snakehandler,
  title = {Zahhak-snakehandler},
  author = {Kheradafrouz},
  url = {https://github.com/kheradafrouz/Zahhak-snakehandler},
  license = {MIT}
}

Links

• Source code: GitHub repository
• Configuration: config.py
• Environment: snake_env.py
• Agent implementation: agent.py
• Training: train.py
• Evaluation: evaluate.py
• License: MIT License