docs/medium_draft.md

# Medium Article Draft: Teaching an LLM to Play Pokemon with Reinforcement Learning

## Title Options
- "I Trained an LLM to Play Competitive Pokemon — Here's What I Learned"
- "From Rock-Paper-Scissors to Pokemon: Training LLMs with GRPO"
- "Building an RL Environment for LLM Pokemon Battles"

---

## Outline

### 1. Hook (2-3 paragraphs)
- Start with the core insight: competitive Pokemon is a reasoning problem with hidden information, constrained actions, and long-term tradeoffs
- Rock-paper-scissors shows that even simple cyclic matchups create nontrivial reasoning — Pokemon scales that into a much richer domain
- The goal: build an OpenEnv-compatible environment that lets you train any LLM to play Pokemon battles using reinforcement learning

### 2. Why Pokemon? (2-3 paragraphs)
- Hidden information: you don't know your opponent's full team, movesets, or items
- Legal action constraints: only 4 moves + 5 switches per turn, must be valid
- Long-term resource management: HP, PP, team composition matter across the battle
- Active opponent: the other player adapts, creating non-stationary dynamics
- Compare to other RL benchmarks (Atari, board games) — Pokemon sits in an interesting middle ground

### 3. Environment Design (main technical section)

#### State Representation
- Markdown-formatted state with 3 sections (Part A/B/C)
- Why markdown: LLMs already understand structured text
- What information is included and why (active field, roster, opponent history)
- Show an example state snippet

#### Action Space
- JSON schema: `{"action": "move"|"switch", "choice": "name"}`
- Why constrained JSON instead of free text
- Action validation with case-insensitive, space-normalized matching
- What happens when the model hallucinates (fallback + penalty)

#### Reward Shaping
- The challenge: Pokemon battles are long (10-30 turns), sparse win/loss signal isn't enough
- Multi-component shaped reward:
  - Damage dealt/taken
  - Knockouts (+3.0/-3.0)
  - Healing (capped to prevent exploitation)
  - Setup moves (capped per Pokemon)
  - Type effectiveness bonus/penalty
  - Illegal action penalty (-10.0)
  - Anti-stall step penalty
- Design philosophy: dense signal without turning it into a toy proxy

### 4. Training Pipeline (medium-length section)

#### The Two-Stage Approach
- Stage 1: JSON warm-up SFT — teach the model to output valid action JSON
- Stage 2: GRPO — optimize the policy using real rollout data

#### Why GRPO?
- Brief explanation of Group Relative Policy Optimization
- How it differs from PPO / DPO for this use case
- The rollout collection loop: play battles, record (state, action, reward) tuples

#### Infrastructure
- Local Pokemon Showdown server via poke-env
- Colab GPU runtime for model inference
- LoRA adapters for parameter efficiency
- Multiple training runs with iterative improvement

### 5. Results & Observations (2-3 paragraphs)
- What the trained model learned to do well
- Where it still struggles
- Interesting emergent behaviors (if any)
- Comparison across checkpoints (run1 → run2 → run3)
- Honest about limitations: small training budget, random opponent, Gen 4 format

### 6. The OpenEnv Integration (1-2 paragraphs)
- What OpenEnv is and why it matters
- How the environment is packaged as a reusable server
- Link to the HF Space demo

### 7. Takeaways (2-3 paragraphs)
- What worked: structured state format, shaped rewards, GRPO on real rollouts
- What was harder than expected: battle lifecycle management, async poke-env integration, reward design
- What I'd do differently: more training budget, better opponent (self-play), broader format coverage
- The bigger picture: LLMs as RL agents in complex interactive environments

### 8. Links & Resources
- GitHub repo
- HF model weights
- HF Space demo
- OpenEnv project

---

## Key Diagrams to Include
1. Architecture diagram (Pokemon Showdown → poke-env → PokemonShowdownEnv → OpenEnv Server)
2. Training pipeline diagram (Base Model → SFT → Rollouts → GRPO → LoRA)
3. Example battle state screenshot from the HF Space
4. Reward component breakdown chart

## Estimated Length
- 1500-2000 words
- 4-5 code snippets
- 2-3 diagrams/screenshots

## Tone
- Technical but accessible
- First-person, honest about the hackathon context
- Focus on design decisions and lessons learned, not just "here's what I built"