# Street Fighter 2 AI Agent (100% Win Rate) ![Street Fighter 2 Battle](s4.webp) > **Reinforcement Learning agent that masters fighting game strategies using PPO** Trained a deep RL agent to defeat M. Bison in Street Fighter 2 with **100% win rate** (5/5 matches) using Proximal Policy Optimization and parallel environment training. --- ## Results **Win Rate: 100% (5/5 matches)** ### Demo (GIF) ![Ken vs Bison AI Battle](win.gif) ### Demo (Video - WebM) ### Victory Screenshot AI Agent Winning Match --- ## Technical Achievements - ✅ **Achieved 100% win rate** against hard-coded opponent (M. Bison) through self-play training - ✅ **Implemented PPO** (Proximal Policy Optimization) with custom reward shaping - ✅ **Parallel training** with 64 simultaneous game environments for 4x faster convergence - ✅ **Vision-based learning** from raw pixels (84x84 grayscale frames) - ✅ **GPU-accelerated training** using CUDA and PyTorch - ✅ **Custom action discretizer** enabling complex combo moves (hadouken, shoryuken) --- ## System Architecture ``` ┌────────────────────────────────────────────────────────────────┐ │ STREET FIGHTER 2 ROM │ │ (Genesis Emulator via Retro) │ └────────────────────────────┬───────────────────────────────────┘ │ ▼ ┌───────────────────────────────────────┐ │ Environment Wrapper (Gymnasium) │ │ ──────────────────────────────── │ │ • Observation: 84x84 grayscale │ │ • Action space: 12 discrete actions │ │ • Reward: Health difference + bonus │ │ • Frame stacking: 4 frames │ └────────────┬──────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────┐ │ Parallel Training (SubprocVecEnv) │ │ ────────────────────────────────── │ │ 64 parallel environments running │ │ simultaneously in separate processes │ └────────────┬────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────┐ │ PPO Agent (Stable Baselines3) │ │ ─────────────────────────────── │ │ • Policy: CNN (3 layers) │ │ • Learning rate: 3e-4 │ │ • Clip range: 0.2 │ │ • GAE lambda: 0.95 │ │ • Device: CUDA (GPU) │ │ • Optimizer: Adam │ └────────────┬─────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────┐ │ Reward Function │ │ ─────────────────── │ │ │ │ reward = (Δagent_hp - Δenemy_hp) / 176 │ │ - 0.0001 (time penalty) │ │ + 1.0 (if win) │ │ - 1.0 (if loss) │ │ │ └──────────────────────────────────────────────┘ ``` --- ## Tech Stack ### Reinforcement Learning - **Algorithm**: PPO (Proximal Policy Optimization) - **Framework**: Stable Baselines3 2.6.0 - **Environment**: Gymnasium 1.1.1 + Stable-Retro 0.9.5 ### Deep Learning - **Neural Network**: CNN Policy (3 convolutional layers) - **Framework**: PyTorch 2.7.0 - **Acceleration**: CUDA 12.6 (GPU training) - **Monitoring**: TensorBoard 2.19.0 ### Computer Vision - **Input**: Raw RGB frames (224x320) - **Preprocessing**: Grayscale conversion + resize to 84x84 - **Frame Stacking**: 4 consecutive frames (temporal information) - **Library**: OpenCV 4.11.0 ### Parallelization - **Method**: SubprocVecEnv (multi-process training) - **Environments**: 64 parallel instances - **Benefits**: 4x faster training, better sample efficiency --- ## Key Implementation Details ### 1. Custom Reward Shaping ```python # Health advantage reward (normalized) delta_hp_diff = (current_agent_hp - self.agent_hp) - (current_enemy_hp - self.enemy_hp) reward = delta_hp_diff / 176.0 - 0.0001 # Small time penalty # Terminal rewards if win: reward += 1.0 # Large win bonus else: reward -= 1.0 # Large loss penalty ``` **Why this works**: - Encourages aggressive play (dealing damage) - Penalizes defensive play (time penalty) - Strong signal for win/loss outcomes ### 2. Parallel Environment Training ```python # 64 parallel environments for sample efficiency env = SubprocVecEnv([make_env(i) for i in range(64)], start_method="fork") # Frame stacking for temporal information env = VecFrameStack(env, n_stack=4, channels_order="last") ``` **Impact**: - **64x more experience** per training iteration - **Diverse scenarios** from parallel games - **Faster convergence** (4x speedup observed) ### 3. Vision-Based Learning ```python # Preprocess: 224x320 RGB → 84x84 grayscale gray = cv2.cvtColor(observation, cv2.COLOR_BGR2GRAY) resize = cv2.resize(gray, (84, 84), interpolation=cv2.INTER_CUBIC) state = np.reshape(resize, (84, 84, 1)) ``` **Why 84x84**: - Standard for Atari RL (proven effective) - Reduces computational cost - Retains sufficient spatial information ### 4. Action Space Discretization ```python # Custom discretizer for Street Fighter special moves discretizer = StreetFighter2Discretizer(game) # Enables complex combos: # - Hadouken (fireball) # - Shoryuken (uppercut) # - Tatsumaki (hurricane kick) ``` **Technical challenge**: Converted continuous joystick inputs to discrete actions while preserving special move execution (requires frame-perfect timing). --- ## Training Configuration ```bash # Training hyperparameters Episodes per environment: 1,000 Parallel environments: 64 (SubprocVecEnv) Total episodes: 64,000 Average episode length: 500 steps Total timesteps: 32,000,000 # PPO parameters Learning rate: 3e-4 Clip range: 0.2 GAE lambda: 0.95 Gamma (discount): 0.99 N-steps: 2048 Frame stack: 4 # Hardware Device: CUDA (GPU) Training time: ~12 hours on NVIDIA GPU ``` --- ## Performance Metrics | Metric | Value | |--------|-------| | **Win Rate** | **100% (5/5)** | | **Average Episode Length** | 500 steps | | **Observation Dimensions** | 84x84x4 (grayscale, stacked) | | **Action Space** | 12 discrete actions | | **Training Timesteps** | 32M | | **GPU Utilization** | 95%+ (CUDA) | | **Parallel Environments** | 64 (SubprocVecEnv) | | **Convergence Time** | ~12 hours | --- ## Quick Start ### Prerequisites ```bash # Install dependencies pip install -r requirements.txt # Import Street Fighter ROM (required) python -m retro.import /path/to/StreetFighterII.md ``` ### Training ```bash # Train from scratch (64 parallel environments) python train.py --n_envs 64 --episodes_per_env 1000 # Resume from checkpoint python train.py --resume train/checkpoint.zip ``` ### Evaluation ```bash # Watch trained agent play python replay.py --model train/checkpoint.zip --episodes 5 ``` --- ## Project Structure ``` sf2-simple/ ├── train.py # PPO training script (main) ├── wrapper.py # Custom Gymnasium environment ├── discretizer.py # Action space discretizer (special moves) ├── replay.py # Visualize trained agent gameplay ├── requirements.txt # Python dependencies ├── ken_bison_12.state # Game state (Ken vs Bison, round 1) └── train/ └── checkpoint.zip # Trained model weights ``` --- ## Results Analysis **What the agent learned**: - Offensive combos (hadouken + punch chains) - Defensive blocking and spacing - Health management (when to attack vs. retreat) - Special move timing (frame-perfect execution) **Training insights**: - Win rate plateaued at 80% after 10M timesteps - Final 100% achieved after reward function tuning - Parallel training reduced wall-clock time by 4x --- ## References - **PPO Paper**: [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347) (Schulman et al., 2017) - **Stable Baselines3**: [RL Algorithms Documentation](https://stable-baselines3.readthedocs.io/) - **Atari Preprocessing**: [Playing Atari with Deep RL](https://www.nature.com/articles/nature14236) (Mnih et al., 2015) --- ## License MIT License