---
library_name: pytorch
tags:
- LunarLander-v2
- deep-reinforcement-learning
- reinforcement-learning
- ppo
- pytorch
- gymnasium
- deep-rl-course
model-index:
- name: PPO
  results:
  - task:
      type: reinforcement-learning
      name: reinforcement-learning
    dataset:
      name: LunarLander-v2
      type: LunarLander-v2
    metrics:
    - type: mean_reward
      value: -167.32 +/- 88.93
      name: mean_reward
      verified: false
---

# **PPO** Agent playing **LunarLander-v2**

This is a trained **Proximal Policy Optimization (PPO)** agent playing **LunarLander-v2**.

This project was completed as part of the **Hugging Face Deep Reinforcement Learning Course, Unit 8 - Part 1**.

The PPO agent was implemented from scratch using **PyTorch** and **Gymnasium**, following the PPO implementation and concepts covered in the course.

## Environment

The agent was trained on **LunarLander-v2**.

The goal is to learn a policy that controls a lunar lander and successfully lands it on the landing pad while maximizing the cumulative reward.

The environment provides an 8-dimensional observation describing:

- Horizontal position
- Vertical position
- Horizontal velocity
- Vertical velocity
- Angle
- Angular velocity
- Left leg contact
- Right leg contact

The action space contains four discrete actions:

| Action | Description |
| --- | --- |
| 0 | Do nothing |
| 1 | Fire left orientation engine |
| 2 | Fire main engine |
| 3 | Fire right orientation engine |

## Algorithm

The agent uses **Proximal Policy Optimization (PPO)**.

PPO is an on-policy policy-gradient reinforcement learning algorithm that improves the policy while limiting excessively large policy updates.

The implementation includes:

- Actor-Critic architecture
- Generalized Advantage Estimation (GAE)
- PPO clipped surrogate objective
- Clipped value loss
- Advantage normalization
- Entropy regularization
- Gradient clipping
- Learning-rate annealing

## PPO Clipped Objective

The probability ratio between the current and old policies is:

`r_t(theta) = pi_theta(a_t | s_t) / pi_theta_old(a_t | s_t)`

The PPO clipped objective is:

`L_CLIP = E[min(r_t A_t, clip(r_t, 1-epsilon, 1+epsilon) A_t)]`

The clipping coefficient used is:

`epsilon = 0.2`

Therefore, the clipping range is:

`[0.8, 1.2]`

Clipping prevents the policy from making excessively large updates.

When the ratio is within the clipping range, the policy can be updated normally.

When the ratio moves outside the range in a direction that would make the policy update excessively large, the clipped objective limits the update.

## Generalized Advantage Estimation

The implementation uses **Generalized Advantage Estimation (GAE)**.

The temporal-difference error is:

`delta_t = r_t + gamma * V(s_t+1) - V(s_t)`

The advantage is estimated recursively using:

`A_t = delta_t + gamma * lambda * A_t+1`

The implementation uses:

- Gamma = 0.99
- GAE Lambda = 0.95

GAE provides a balance between bias and variance when estimating advantages.

## Model Architecture

The PPO agent uses an **Actor-Critic architecture**.

### Actor

The Actor receives the environment observation and produces action logits.

Architecture:

`Input -> Linear(64) -> Tanh -> Linear(64) -> Tanh -> Linear(4)`

The output is used to create a categorical probability distribution over the four possible actions.

### Critic

The Critic estimates the value of the current state.

Architecture:

`Input -> Linear(64) -> Tanh -> Linear(64) -> Tanh -> Linear(1)`

## Training Configuration

| Parameter | Value |
| --- | ---: |
| Environment | LunarLander-v3 |
| Algorithm | PPO |
| Framework | PyTorch |
| Environment library | Gymnasium |
| Total timesteps | 100,000 |
| Learning rate | 0.00025 |
| Number of environments | 8 |
| Steps per rollout | 128 |
| Minibatches | 4 |
| Update epochs | 4 |
| Discount factor | 0.99 |
| GAE Lambda | 0.95 |
| PPO clip coefficient | 0.2 |
| Entropy coefficient | 0.01 |
| Value function coefficient | 0.5 |
| Maximum gradient norm | 0.5 |
| Advantage normalization | Enabled |
| GAE | Enabled |
| Learning-rate annealing | Enabled |
| Clipped value loss | Enabled |
| Random seed | 1 |

## Evaluation

The trained agent was evaluated for **10 episodes**.

The evaluation results are stored in `evaluation.txt`.

The file contains:

- Number of evaluation episodes
- Mean reward
- Standard deviation
- Individual episode rewards

The evaluation result shown at the top of this model card should be replaced with the actual mean reward from `evaluation.txt`.

## Usage

The trained model is stored as `model.pt`.

The model contains the PyTorch state dictionary of the trained Actor-Critic agent.

The same `Agent` architecture must be recreated before loading the weights.

Example:

    import torch

    agent.load_state_dict(
        torch.load(
            "model.pt",
            map_location="cpu"
        )
    )

    agent.eval()

The environment can be created using:

    import gymnasium as gym

    env = gym.make("LunarLander-v2")

    observation, info = env.reset()

## Repository Contents

- `model.pt` - trained PPO Actor-Critic model
- `hyperparameters.txt` - PPO training hyperparameters
- `evaluation.txt` - evaluation results
- `README.md` - model card

## Hugging Face Deep Reinforcement Learning Course

This project was completed as part of the:

**Hugging Face Deep Reinforcement Learning Course**

**Unit 8 - Part 1: Proximal Policy Optimization (PPO)**

The project demonstrates the implementation of PPO from scratch and its application to the LunarLander environment.

The main learning objectives include:

- Understanding PPO
- Implementing an Actor-Critic architecture
- Collecting experience from multiple environments
- Implementing Generalized Advantage Estimation
- Implementing the PPO clipped objective
- Training and evaluating the agent
- Sharing the trained model on the Hugging Face Hub

## References

- Hugging Face Deep Reinforcement Learning Course
- Proximal Policy Optimization Algorithms - Schulman et al.
- Gymnasium
- PyTorch

## Limitations

This model was developed as an educational project for the Hugging Face Deep Reinforcement Learning Course.

Performance may vary depending on:

- Random seed
- Training duration
- Hyperparameters
- Environment version
- Hardware
- Stochasticity of the environment