hwihwalab/lunarlander-v3-d3qn

🚀 LunarLander-v3 D3QN // Mission Control Deck

Gymnasium Box2D LunarLander-v3 Reinforcement Learning Agent & Real-time Aerospace Telemetry System
🌐 English Documentation | 🇰🇷 한국어 매뉴얼

This repository contains a pre-trained Dueling Double Deep Q-Network (D3QN) agent trained on the Gymnasium LunarLander-v3 environment.

---

🌟 Model Highlights

• Algorithm: Dueling Double DQN (D3QN) with Soft Target Updates ($\tau = 0.001$)
• Architecture: Separated Value ($V(s)$) and Advantage ($A(s, a)$) streams ($8 \to 128 \to 128 \to 64 \to 4$)
• Optimization: Smooth L1 (Huber) Loss, AdamW, Gradient Clipping
• Custom Reward Shaping: Stabilized pitch angle, soft touchdown velocity damping, and center-pad landing rewards.

---

📂 Repository Contents

• best_model.pth: Pre-trained PyTorch Dueling Double DQN neural network weights (+311.16 score).
• config.json: Model architecture, hyperparameters, and environment specifications.
• dqn_agent.py: Complete PyTorch source code for DQNAgent and DuelingQNetwork.
• README.md: Global English Model Card and evaluation guide.
• README_KR.md: Full Korean comprehensive manual and telemetry specifications.

---

📊 Environment & Action Space

• Observation Space (8 Dimensions):

  1. Coordinate $X$
  2. Coordinate $Y$
  3. Linear Velocity $V_x$
  4. Linear Velocity $V_y$
  5. Pitch Angle $\theta$
  6. Angular Velocity $\omega$
  7. Left Leg Ground Contact (0 or 1)
  8. Right Leg Ground Contact (0 or 1)

• Action Space (4 Discrete Actions):

  • 0: IDLE (Coast)
  • 1: Fire Left Thruster
  • 2: Fire Main Engine Thruster
  • 3: Fire Right Thruster

---

⚙️ Hyperparameters

Hyperparameter	Value	Description
Learning Rate	5e-4	AdamW optimizer learning rate
Discount Factor ($\gamma$)	0.99	Future reward discount factor
Replay Buffer Size	100,000	Experience replay memory capacity
Batch Size	64	Mini-batch sample size for training
Target Network Update ($\tau$)	0.001	Polyak soft update rate
Exploration ($\epsilon$)	1.0 \to 0.05	100% exploration decaying to 5%

---

💻 How to Load & Evaluate

You can easily load and run this trained model in Python using PyTorch and Gymnasium:

import torch
import torch.nn as nn
import gymnasium as gym

# 1. Define Dueling DQN Architecture
class DuelingDQN(nn.Module):
    def __init__(self, state_dim=8, action_dim=4):
        super().__init__()
        self.feature_network = nn.Sequential(
            nn.Linear(state_dim, 128),
            nn.LayerNorm(128),
            nn.ReLU(),
            nn.Linear(128, 128),
            nn.LayerNorm(128),
            nn.ReLU(),
        )
        self.value_stream = nn.Sequential(
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 1)
        )
        self.advantage_stream = nn.Sequential(
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, action_dim)
        )

    def forward(self, state):
        features = self.feature_network(state)
        values = self.value_stream(features)
        advantages = self.advantage_stream(features)
        return values + (advantages - advantages.mean(dim=-1, keepdim=True))

# 2. Download weights from Hugging Face Hub
from huggingface_hub import hf_hub_download

weights_path = hf_hub_download(repo_id="hwihwalab/lunarlander-v3-d3qn", filename="best_model.pth")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = DuelingDQN().to(device)
model.load_state_dict(torch.load(weights_path, map_location=device))
model.eval()

# 3. Test Agent Flight
env = gym.make("LunarLander-v3", render_mode="human")
state, _ = env.reset()
total_reward = 0

for _ in range(1000):
    state_t = torch.FloatTensor(state).unsqueeze(0).to(device)
    with torch.no_grad():
        action = model(state_t).argmax(dim=-1).item()
    
    state, reward, terminated, truncated, _ = env.step(action)
    total_reward += reward
    if terminated or truncated:
        break

print(f"Final Flight Reward: {total_reward:.2f}")
env.close()

---

🛠️ Training & Telemetry

Developed and trained with the Lunar Lander Mission Control Deck real-time telemetry system.

• Full Dashboard UI & Source Code: https://github.com/Hwihwa-Lab/lunarlander-v3-d3qn

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Trained and deployed with LunarLander Mission Control by hwihwalab.