Update README.md

README.md

@@ -1,29 +1,44 @@
 ---
-tags:
-  - lunarlander-v2
-  - ppo
-  - deep-reinforcement-learning
-  - reinforcement-learning
-  - custom-implementation
-  - deep-rl-course
 library_name: pytorch
+tags:
+- LunarLander-v2
+- deep-reinforcement-learning
+- reinforcement-learning
+- ppo
+- pytorch
+- gymnasium
+- deep-rl-course
+model-index:
+- name: PPO
+  results:
+  - task:
+      type: reinforcement-learning
+      name: reinforcement-learning
+    dataset:
+      name: LunarLander-v2
+      type: LunarLander-v2
+    metrics:
+    - type: mean_reward
+      value: -167.32 +/- 88.93
+      name: mean_reward
+      verified: false
 ---
 
-# PPO Agent Playing LunarLander-v2
+# **PPO** Agent playing **LunarLander-v2**
 
-This is a trained Proximal Policy Optimization (PPO) agent playing the LunarLander-v2 environment.
+This is a trained **Proximal Policy Optimization (PPO)** agent playing **LunarLander-v2**.
 
-This project was completed as part of the Hugging Face Deep Reinforcement Learning Course, Unit 8 - Part 1.
+This project was completed as part of the **Hugging Face Deep Reinforcement Learning Course, Unit 8 - Part 1**.
 
-The PPO agent was implemented from scratch using PyTorch, following the CleanRL PPO implementation and the course material.
+The PPO agent was implemented from scratch using **PyTorch** and **Gymnasium**, following the PPO implementation and concepts covered in the course.
 
 ## Environment
 
-The agent is trained on `LunarLander-v2`.
+The agent was trained on **LunarLander-v2**.
 
-The objective is to learn a policy that controls the lunar lander and successfully lands it on the landing pad while maximizing the accumulated reward.
+The goal is to learn a policy that controls a lunar lander and successfully lands it on the landing pad while maximizing the cumulative reward.
 
-The observation space contains 8 values describing the state of the lander:
+The environment provides an 8-dimensional observation describing:
 
 - Horizontal position
 - Vertical position
@@ -43,38 +58,34 @@ The action space contains four discrete actions:
 | 2 | Fire main engine |
 | 3 | Fire right orientation engine |
 
-## Proximal Policy Optimization
+## Algorithm
 
-PPO is an on-policy policy-gradient reinforcement learning algorithm.
+The agent uses **Proximal Policy Optimization (PPO)**.
 
-The main idea behind PPO is to improve the policy while preventing the new policy from moving too far from the previous policy during an update.
+PPO is an on-policy policy-gradient reinforcement learning algorithm that improves the policy while limiting excessively large policy updates.
 
-The implementation uses an Actor-Critic architecture:
+The implementation includes:
 
-- The Actor learns the policy.
-- The Critic estimates the value of a state.
-- Generalized Advantage Estimation (GAE) is used to estimate advantages.
-- The PPO clipped surrogate objective is used for policy optimization.
-- Entropy regularization encourages exploration.
-- Gradient clipping helps stabilize training.
+- Actor-Critic architecture
+- Generalized Advantage Estimation (GAE)
+- PPO clipped surrogate objective
+- Clipped value loss
+- Advantage normalization
+- Entropy regularization
+- Gradient clipping
+- Learning-rate annealing
 
 ## PPO Clipped Objective
 
-The probability ratio between the current policy and the old policy is:
+The probability ratio between the current and old policies is:
 
 `r_t(theta) = pi_theta(a_t | s_t) / pi_theta_old(a_t | s_t)`
 
 The PPO clipped objective is:
 
-`L_CLIP = E[min(r_t(theta) A_t, clip(r_t(theta), 1 - epsilon, 1 + epsilon) A_t)]`
-
-where:
-
-- `A_t` is the advantage estimate.
-- `r_t(theta)` is the probability ratio.
-- `epsilon` is the clipping coefficient.
+`L_CLIP = E[min(r_t A_t, clip(r_t, 1-epsilon, 1+epsilon) A_t)]`
 
-The clipping coefficient is:
+The clipping coefficient used is:
 
 `epsilon = 0.2`
 
@@ -84,44 +95,42 @@ Therefore, the clipping range is:
 
 Clipping prevents the policy from making excessively large updates.
 
-If the probability ratio is within the clipping range, the policy can be updated normally.
+When the ratio is within the clipping range, the policy can be updated normally.
 
-If the ratio moves outside the range in a direction that would make the policy update excessively large, the clipped objective limits the update.
-
-For a positive advantage, the selected action was better than expected and the policy is encouraged to increase its probability.
-
-For a negative advantage, the selected action was worse than expected and the policy is encouraged to decrease its probability.
+When the ratio moves outside the range in a direction that would make the policy update excessively large, the clipped objective limits the update.
 
 ## Generalized Advantage Estimation
 
-The implementation uses Generalized Advantage Estimation (GAE).
+The implementation uses **Generalized Advantage Estimation (GAE)**.
 
 The temporal-difference error is:
 
 `delta_t = r_t + gamma * V(s_t+1) - V(s_t)`
 
-The advantage estimate is calculated recursively using:
+The advantage is estimated recursively using:
 
 `A_t = delta_t + gamma * lambda * A_t+1`
 
-The default training configuration uses:
+The implementation uses:
 
 - Gamma = 0.99
 - GAE Lambda = 0.95
 
-GAE provides a useful balance between bias and variance when estimating the advantage function.
+GAE provides a balance between bias and variance when estimating advantages.
+
+## Model Architecture
 
-## Actor-Critic Architecture
+The PPO agent uses an **Actor-Critic architecture**.
 
 ### Actor
 
-The Actor receives the environment observation and outputs logits for the available actions.
+The Actor receives the environment observation and produces action logits.
 
 Architecture:
 
-`Input -> Linear(64) -> Tanh -> Linear(64) -> Tanh -> Linear(Action Space)`
+`Input -> Linear(64) -> Tanh -> Linear(64) -> Tanh -> Linear(4)`
 
-The output logits are used to create a categorical probability distribution from which actions are sampled.
+The output is used to create a categorical probability distribution over the four possible actions.
 
 ### Critic
 
@@ -131,20 +140,17 @@ Architecture:
 
 `Input -> Linear(64) -> Tanh -> Linear(64) -> Tanh -> Linear(1)`
 
-The value estimate is used to calculate advantages and returns.
-
 ## Training Configuration
 
-The PPO implementation provided in the course uses the following default configuration:
-
 | Parameter | Value |
 | --- | ---: |
-| Environment | LunarLander-v2 |
+| Environment | LunarLander-v3 |
 | Algorithm | PPO |
 | Framework | PyTorch |
-| Total timesteps | 50,000 |
+| Environment library | Gymnasium |
+| Total timesteps | 100,000 |
 | Learning rate | 0.00025 |
-| Number of environments | 4 |
+| Number of environments | 8 |
 | Steps per rollout | 128 |
 | Minibatches | 4 |
 | Update epochs | 4 |
@@ -154,65 +160,36 @@ The PPO implementation provided in the course uses the following default configu
 | Entropy coefficient | 0.01 |
 | Value function coefficient | 0.5 |
 | Maximum gradient norm | 0.5 |
-| GAE | Enabled |
 | Advantage normalization | Enabled |
+| GAE | Enabled |
 | Learning-rate annealing | Enabled |
 | Clipped value loss | Enabled |
-
-The actual hyperparameters used for this trained model are available in `hyperparameters.txt`.
-
-## Training Process
-
-The training process follows these steps:
-
-1. Create multiple LunarLander-v2 environments.
-2. Collect observations from the environments.
-3. Use the Actor to select actions.
-4. Execute the actions in the environments.
-5. Store observations, actions, rewards, log probabilities, and value estimates.
-6. Calculate advantages using GAE.
-7. Calculate returns.
-8. Calculate the PPO probability ratio.
-9. Apply the PPO clipped surrogate objective.
-10. Calculate the value-function loss and entropy bonus.
-11. Update the Actor and Critic networks.
-12. Repeat the process for the specified number of timesteps.
-
-Multiple environments are used in parallel to collect experience efficiently.
+| Random seed | 1 |
 
 ## Evaluation
 
-The trained agent is evaluated for 10 episodes.
+The trained agent was evaluated for **10 episodes**.
 
-The evaluation results are stored in `evaluation.txt` and include the evaluation reward statistics.
+The evaluation results are stored in `evaluation.txt`.
 
-The Hugging Face PPO integration evaluates the agent over 10 episodes and records:
+The file contains:
 
+- Number of evaluation episodes
 - Mean reward
 - Standard deviation
-- Number of evaluation episodes
-- Environment ID
-- Evaluation date and time
+- Individual episode rewards
 
-The evaluation score can vary because reinforcement learning is stochastic.
+The evaluation result shown at the top of this model card should be replaced with the actual mean reward from `evaluation.txt`.
 
-## Files
+## Usage
 
-| File | Description |
-| --- | --- |
-| `model.pt` | Trained PyTorch Actor-Critic model |
-| `hyperparameters.txt` | Training hyperparameters |
-| `evaluation.txt` | Evaluation results |
-| `README.md` | Model card |
-| `replay.mp4` | Replay video of the trained agent, if included |
-
-## Loading the Model
+The trained model is stored as `model.pt`.
 
-The trained model is stored in `model.pt`.
+The model contains the PyTorch state dictionary of the trained Actor-Critic agent.
 
-The file contains the PyTorch `state_dict` of the trained Actor-Critic agent.
+The same `Agent` architecture must be recreated before loading the weights.
 
-To load the model, recreate the same `Agent` architecture and load the state dictionary:
+Example:
 
     import torch
 
@@ -225,50 +202,45 @@ To load the model, recreate the same `Agent` architecture and load the state dic
 
     agent.eval()
 
-## Gymnasium
+The environment can be created using:
 
-This project uses the Gymnasium API.
+    import gymnasium as gym
 
-Environment reset:
+    env = gym.make("LunarLander-v2")
 
     observation, info = env.reset()
 
-Environment step:
-
-    observation, reward, terminated, truncated, info = env.step(action)
+## Repository Contents
 
-An episode ends when either `terminated` or `truncated` is `True`.
+- `model.pt` - trained PPO Actor-Critic model
+- `hyperparameters.txt` - PPO training hyperparameters
+- `evaluation.txt` - evaluation results
+- `README.md` - model card
 
 ## Hugging Face Deep Reinforcement Learning Course
 
-This project is part of:
+This project was completed as part of the:
 
 **Hugging Face Deep Reinforcement Learning Course**
 
-**Unit 8 - Part 1: Proximal Policy Optimization (PPO) with PyTorch**
+**Unit 8 - Part 1: Proximal Policy Optimization (PPO)**
 
-The objective of this unit is to implement a PPO agent from scratch using PyTorch and then train it on LunarLander-v2.
+The project demonstrates the implementation of PPO from scratch and its application to the LunarLander environment.
 
-The trained agent is then pushed to the Hugging Face Hub for evaluation and visualization.
+The main learning objectives include:
 
-## Learning Objectives
-
-This project demonstrates:
-
-- Implementing PPO from scratch with PyTorch
-- Understanding the PPO clipped objective
+- Understanding PPO
 - Implementing an Actor-Critic architecture
-- Using Generalized Advantage Estimation
-- Collecting experience from parallel environments
-- Optimizing the policy and value networks
-- Evaluating a trained reinforcement learning agent
-- Uploading the trained agent to the Hugging Face Hub
+- Collecting experience from multiple environments
+- Implementing Generalized Advantage Estimation
+- Implementing the PPO clipped objective
+- Training and evaluating the agent
+- Sharing the trained model on the Hugging Face Hub
 
 ## References
 
-- Hugging Face Deep Reinforcement Learning Course - Unit 8
-- Schulman et al., Proximal Policy Optimization Algorithms
-- CleanRL PPO implementation
+- Hugging Face Deep Reinforcement Learning Course
+- Proximal Policy Optimization Algorithms - Schulman et al.
 - Gymnasium
 - PyTorch
 
@@ -283,14 +255,4 @@ Performance may vary depending on:
 - Hyperparameters
 - Environment version
 - Hardware
-- Stochasticity of the environment
-
-The model should not be considered an optimal LunarLander-v2 policy.
-
-## Summary
-
-This project demonstrates the complete PPO pipeline:
-
-`LunarLander-v2 -> Actor-Critic -> Experience Collection -> GAE -> PPO Clipping -> Optimization -> Evaluation`
-
-The main idea behind PPO is to improve the policy using collected experience while preventing excessively large policy updates.
+- Stochasticity of the environment