File size: 1,955 Bytes
6231d0f beba66a 6231d0f 78daf4f beba66a 6231d0f 78daf4f beba66a 6231d0f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | ---
license: mit
tags:
- reinforcement-learning
- ppo
- openfront
- game-ai
---
# OpenFront RL Agent
PPO-trained agent for [OpenFront.io](https://openfront.io), a multiplayer territory control game.
## Model Version: v13b
Current best model trained with normalized elimination reward and winner bonus.
## Training Details
- **Algorithm:** PPO (Proximal Policy Optimization)
- **Architecture:** Actor-Critic with shared backbone (512→512→256)
- **Observation dim:** 80 (16 player stats + 16 neighbors × 4 features)
- **Action space:** MultiDiscrete [17 action types, 16 targets, 5 troop fractions]
- **Maps:** plains, big_plains, world, giantworldmap, ocean_and_land, half_land_half_ocean (random per episode)
- **Parallel envs:** 16
- **Learning rate:** 1.5e-4 (constant)
- **Rollout steps:** 1024
- **Batch size:** 16,384
- **Value function coefficient:** 0.5
- **Updates trained:** 1550 (ongoing)
## Reward Design (v13)
Normalized elimination reward — total reward sums to +1.0 on a full win regardless of opponent count:
- **Per-kill:** `+1/N` per opponent eliminated (N = starting opponents)
- **Winner bonus:** remaining alive opponents credited as `aliveCount/N` when `game.getWinner()` fires
- **Death penalty:** -1.0
## Curriculum
Win-rate-gated 12-stage curriculum advancing through Easy → Medium → Hard difficulty and 2 → 15 opponents. Stages advance only when rolling win rate exceeds per-stage threshold (75% down to 45%) over 200 episodes.
## Eval Results
- **Easy/2 opponents:** 100% win rate (20/20 games)
## Usage
```python
from train import ActorCritic
import torch
model = ActorCritic(obs_dim=80, max_neighbors=16, hidden_sizes=[512, 512, 256])
checkpoint = torch.load("best_model.pt", map_location="cpu", weights_only=False)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()
```
## Repository
Trained from [josh-freeman/openfront-rl](https://github.com/josh-freeman/openfront-rl).
|