| --- |
| license: apache-2.0 |
| pipeline_tag: reinforcement-learning |
| tags: |
| - rl |
| - ping_pong |
| - self-play |
| - fromzero |
| - reinforcement-learning |
| - ppo |
| - custom-architecture |
| - mlp |
| - 2d |
| - game |
| --- |
| |
| # Mr. Pong |
|
|
| In the realm of Ping Pong, **Mr. Pong** is no ordinary paddle. Feared across the table as the **Blue Beast**, he commands every rally with ruthless precision, striking despair into the hearts of all who dare face him. |
| <video |
| src="https://huggingface.co/fromziro/MrsPaleta/resolve/main/assets/video.mp4" |
| controls |
| autoplay |
| loop |
| muted |
| playsinline |
| width="45%"> |
| </video> |
|
|
| In other words, he is a reinforcement learning (RL) agent trained to compete at a grandmaster level in Ping Pong. |
|
|
| ## Model Architecture |
|
|
| - Base Architecture: `MrPongMLPForRL` |
| - Hidden Size: `160` |
| - Observation Size: `16` |
| - Number Of Layers: `3` |
| - Action Space: `3` (stay, up, down) |
| - Activation: `tanh` |
|
|
|
|
| Mr. Pong uses a shared Actor-Critic MLP designed for 2D table tennis control. The 16-dimensional observation vector is passed through two hidden layers of 192 units each, with Tanh activations between them. The resulting 192-dimensional representation is shared by both the actor and critic heads. The actor outputs categorical logits across the 3 movement actions (stay, up, down), while the critic uses the same representation to estimate the scalar state value V(s). This shared setup keeps the network small and fast on CPU while letting the model combine ball coordinates, velocities, paddle momentum, raycasted intercept points, and opponent court openings into a single control decision. |
|
|
| ## Training Configuration |
|
|
| * Number of Environments: `12` |
| * Rollout Steps: `128 |
| * PPO Epochs: `4` |
| * Minibatch Size: `64` |
| * Gamma: `0.99` |
| * GAE Lambda: `0.95` |
| * Clip Coefficient: `0.20` |
| * Value Coefficient: `0.50` |
| * Entropy Coefficient: `0.02` |
| * Maximum Gradient Norm: `0.75` |
| * Learning Rate: `3.5e-4` |
| * Adam Epsilon: `1e-5` |
| * Anneal Learning Rate: `true` |
| * Total Timesteps: `10,000,000` |
| * Maximum Rally Steps: `1500` |
|
|
| ### Training Opponents |
|
|
| - Realistic Hard Logic (human-like perception horizon and bounce raycasting) |
| - Medium Logic (linear trajectory extrapolation) |
| - Minimax Lookahead (depth 1 and depth 2) |
| - Current Self-Play (mirror matches against the active policy) |
| - Historical Self-Play (past checkpoints sampled between 5 and 75 saves ago) |
| - Easy Logic (reaction delay with targeting noise) |
| - Impossible Hard Logic (0ms reaction time) |
| - Random Agent |
|
|
| ## Training Results |
|
|
| | Opponent Name | Win % | Draw % | Loss % | Record (W / D / L) | Avg Rally | |
| | ---------------- | -----: | -----: | -----: | -----------------: | --------: | |
| | Easy Logic | 99.7% | 0.0% | 0.3% | 997W / 0D / 3L | 1.7 hits | |
| | Medium Logic | 99.0% | 0.2% | 0.8% | 990W / 2D / 8L | 17.3 hits | |
| | Realistic Hard | 86.6% | 8.1% | 5.3% | 866W / 81D / 53L | 33.9 hits | |
| | Impossible Hard | 0.0% | 98.4% | 1.6% | 0W / 984D / 16L | 64.8 hits | |
| | Minimax Depth 1 | 43.0% | 55.8% | 1.2% | 430W / 558D / 12L | 32.2 hits | |
| | Minimax Depth 2 | 78.4% | 20.3% | 1.3% | 784W / 203D / 13L | 23.7 hits | |
| | Random Agent | 100.0% | 0.0% | 0.0% | 1000W / 0D / 0L | 0.9 hits | |
| | Self-Play Mirror | 1.6% | 96.1% | 2.3% | 16W / 961D / 23L | 50.5 hits | |
|
|
| Mr. Pong has demonstrated the he has mastered the game Ping Pong; losing only <=5% of games. |
|
|
| ## Inference |
|
|
| First install the required dependencies: |
|
|
| ```bash |
| pip install torch transformers |
| ``` |
|
|
| Next, download `inference.py` and run: |
|
|
| ```bash |
| python inference.py |
| ``` |
|
|
| - Use `--mode play` to play against the agent live in the terminal. |
| - Use `--mode simulate` to run AI match simulations (default). |
| - Add `--opponent` to choose a specific opponent (realistic_hard, medium, minimax_d1, minimax_d2, impossible_hard, easy, random). |
| - Add `--video` to export an MP4 recording of the match. |
|
|
| ## License |
|
|
| Apache 2.0. |
|
|
| ## Citation |
|
|
| ``` |
| @misc{mrbalance, |
| title = {Mr. Pong: Teaching RL agents to Play Ping Pong}, |
| organization = {FromZero}, |
| authors = {Paul Courneya}, |
| year = {2026}, |
| url = {https://huggingface.co/fromziro/MrBalance] |
| } |
| ``` |
