{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "\"PyTorch\"\n", "\n", "\n", "\n", "# OpenEnv: Production RL Made Simple\n", "\n", "### *From \"Hello World\" to RL Training in 5 Minutes* āœØ\n", "\n", "---\n", "\n", "**What if RL environments were as easy to use as REST APIs?**\n", "\n", "That's OpenEnv. Type-safe. Isolated. Production-ready. šŸŽÆ\n", "\n", "[![GitHub](https://img.shields.io/badge/GitHub-meta--pytorch%2FOpenEnv-blue?logo=github)](https://github.com/meta-pytorch/OpenEnv)\n", "[![License](https://img.shields.io/badge/License-BSD%203--Clause-green.svg)](https://opensource.org/licenses/BSD-3-Clause)\n", "[![PyTorch](https://img.shields.io/badge/PyTorch-EE4C2C?logo=pytorch&logoColor=white)](https://pytorch.org/)\n", "\n", "Author: [Sanyam Bhutani](http://twitter.com/bhutanisanyam1/)\n", "\n", "
\n", "\n", "\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Why OpenEnv?\n", "\n", "Let's take a trip down memory lane:\n", "\n", "It's 2016, RL is popular. You read some papers, it looks promising. \n", "\n", "But in real world: Cartpole is the best you can run on a gaming GPU. \n", "\n", "What do you do beyond Cartpole?\n", "\n", "Fast-forward to 2025, GRPO is awesome and this time it's not JUST in theory, it works well in practise and is really here!\n", "\n", "The problem still remains, how do you take these RL algorithms and take them beyond Cartpole?\n", "\n", "A huge part of RL is giving your algorithms environment access to learn. \n", "\n", "We are excited to introduce an Environment Spec for adding Open Environments for RL Training. This will allow you to focus on your experiments and allow everyone to bring their environments.\n", "\n", "Focus on experiments, use OpenEnvironments, and build agents that go beyond Cartpole on a single spec.\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## šŸ“‹ What You'll Learn\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "**šŸŽÆ Part 1-2: The Fundamentals**\n", "- āš” RL in 60 seconds\n", "- šŸ¤” Why existing solutions fall short\n", "- šŸ’” The OpenEnv solution\n", "\n", "\n", "\n", "**šŸ—ļø Part 3-5: The Architecture**\n", "- šŸ”§ How OpenEnv works\n", "- šŸ” Exploring real code\n", "- šŸŽ® OpenSpiel integration example\n", "\n", "
\n", "\n", "**šŸŽ® Part 6-8: Hands-On Demo**\n", "- šŸ”Œ Use existing OpenSpiel environment\n", "- šŸ¤– Test 4 different policies\n", "- šŸ‘€ Watch learning happen live\n", "\n", "\n", "\n", "**šŸ”§ Part 9-10: Going Further**\n", "- šŸŽ® Switch to other OpenSpiel games\n", "- āœØ Build your own integration\n", "- šŸŒ Deploy to production\n", "\n", "
\n", "\n", "> šŸ’” **Pro Tip**: This notebook is designed to run top-to-bottom in Google Colab with zero setup!\n", ">\n", "> ā±ļø **Time**: ~5 minutes | šŸ“Š **Difficulty**: Beginner-friendly | šŸŽÆ **Outcome**: Production-ready RL knowledge\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## šŸ“‘ Table of Contents\n", "\n", "
\n", "\n", "**Quick Navigation** - Click any section to jump right there! šŸŽÆ\n", "\n", "### Foundation\n", "- [Part 1: RL in 60 Seconds ā±ļø](#part-1)\n", "- [Part 2: The Problem with Traditional RL šŸ˜¤](#part-2)\n", "- [Part 3: Setup šŸ› ļø](#part-3)\n", "\n", "### Architecture\n", "- [Part 4: The OpenEnv Pattern šŸ—ļø](#part-4)\n", "- [Part 5: Example Integration - OpenSpiel šŸŽ®](#part-5)\n", "\n", "### Hands-On Demo\n", "- [Part 6: Interactive Demo šŸŽ®](#part-6)\n", "- [Part 7: Four Policies šŸ¤–](#part-7)\n", "- [Part 8: Policy Competition! šŸ†](#part-8)\n", "\n", "### Advanced\n", "- [Part 9: Using Real OpenSpiel šŸŽ®](#part-9)\n", "- [Part 10: Create Your Own Integration šŸ› ļø](#part-10)\n", "\n", "### Wrap Up\n", "- [Summary: Your Journey šŸŽ“](#summary)\n", "- [Resources šŸ“š](#resources)\n", "\n", "
\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "# Part 1: RL in 60 Seconds ā±ļø\n", "\n", "
\n", "\n", "**Reinforcement Learning is simpler than you think.**\n", "\n", "It's just a loop:\n", "\n", "```python\n", "while not done:\n", " observation = environment.observe()\n", " action = policy.choose(observation)\n", " reward = environment.step(action)\n", " policy.learn(reward)\n", "```\n", "\n", "That's it. That's RL.\n", "\n", "
\n", "\n", "Let's see it in action:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "šŸŽ² ========================================================== šŸŽ²\n", " Number Guessing Game - The Simplest RL Example\n", "šŸŽ² ========================================================== šŸŽ²\n", "\n", "šŸŽÆ I'm thinking of a number between 1 and 10...\n", "šŸ’­ You have 3 guesses. Let's see how random guessing works!\n", "\n", "šŸ’­ Guess #1: 4 ā†’ ā„ļø Cold! (far)\n", "šŸ’­ Guess #2: 4 ā†’ ā„ļø Cold! (far)\n", "šŸ’­ Guess #3: 3 ā†’ ā„ļø Cold! (far)\n", "\n", "šŸ’” Out of guesses. The number was 8.\n", "\n", "==============================================================\n", "šŸ’” This is RL: Observe ā†’ Act ā†’ Reward ā†’ Repeat\n", " But this policy is terrible! It doesn't learn from rewards.\n", "==============================================================\n", "\n" ] } ], "source": [ "import random\n", "\n", "print(\"šŸŽ² \" + \"=\"*58 + \" šŸŽ²\")\n", "print(\" Number Guessing Game - The Simplest RL Example\")\n", "print(\"šŸŽ² \" + \"=\"*58 + \" šŸŽ²\")\n", "\n", "# Environment setup\n", "target = random.randint(1, 10)\n", "guesses_left = 3\n", "\n", "print(f\"\\nšŸŽÆ I'm thinking of a number between 1 and 10...\")\n", "print(f\"šŸ’­ You have {guesses_left} guesses. Let's see how random guessing works!\\n\")\n", "\n", "# The RL Loop - Pure random policy (no learning!)\n", "while guesses_left > 0:\n", " # Policy: Random guessing (no learning yet!)\n", " guess = random.randint(1, 10)\n", " guesses_left -= 1\n", " \n", " print(f\"šŸ’­ Guess #{3-guesses_left}: {guess}\", end=\" ā†’ \")\n", " \n", " # Reward signal (but we're not using it!)\n", " if guess == target:\n", " print(\"šŸŽ‰ Correct! +10 points\")\n", " break\n", " elif abs(guess - target) <= 2:\n", " print(\"šŸ”„ Warm! (close)\")\n", " else:\n", " print(\"ā„ļø Cold! (far)\")\n", "else:\n", " print(f\"\\nšŸ’” Out of guesses. The number was {target}.\")\n", "\n", "print(\"\\n\" + \"=\"*62)\n", "print(\"šŸ’” This is RL: Observe ā†’ Act ā†’ Reward ā†’ Repeat\")\n", "print(\" But this policy is terrible! It doesn't learn from rewards.\")\n", "print(\"=\"*62 + \"\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "\n", "# Part 2: The Problem with Traditional RL šŸ˜¤\n", "\n", "
\n", "\n", "## šŸ¤” Why Can't We Just Use OpenAI Gym?\n", "\n", "Good question! Gym is great for research, but production needs more...\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
ChallengeTraditional ApproachOpenEnv Solution
Type SafetyāŒ obs[0][3] - what is this?āœ… obs.info_state - IDE knows!
IsolationāŒ Same process (can crash your training)āœ… Docker containers (fully isolated)
DeploymentāŒ \"Works on my machine\" šŸ¤·āœ… Same container everywhere šŸ³
ScalingāŒ Hard to distributeāœ… Deploy to Kubernetes ā˜øļø
LanguageāŒ Python onlyāœ… Any language (HTTP API) šŸŒ
DebuggingāŒ Cryptic numpy errorsāœ… Clear type errors šŸ›
\n", "\n", "
\n", "\n", "## šŸ’” The OpenEnv Philosophy\n", "\n", "**\"RL environments should be like microservices\"**\n", "\n", "Think of it like this: You don't run your database in the same process as your web server, right? Same principle!\n", "\n", "- šŸ”’ **Isolated**: Run in containers (security + stability)\n", "- šŸŒ **Standard**: HTTP API, works everywhere\n", "- šŸ“¦ **Versioned**: Docker images (reproducibility!)\n", "- šŸš€ **Scalable**: Deploy to cloud with one command\n", "- šŸ›”ļø **Type-safe**: Catch bugs before they happen\n", "- šŸ”„ **Portable**: Works on Mac, Linux, Windows, Cloud\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The Architecture\n", "\n", "```\n", "ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”\n", "ā”‚ YOUR TRAINING CODE ā”‚\n", "ā”‚ ā”‚\n", "ā”‚ env = OpenSpielEnv(...) ā† Import the client ā”‚\n", "ā”‚ result = env.reset() ā† Type-safe! ā”‚\n", "ā”‚ result = env.step(action) ā† Type-safe! ā”‚\n", "ā”‚ ā”‚\n", "ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜\n", " ā”‚\n", " ā”‚ HTTP/JSON (Language-Agnostic)\n", " ā”‚ POST /reset, POST /step, GET /state\n", " ā”‚\n", "ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā–¼ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”\n", "ā”‚ DOCKER CONTAINER ā”‚\n", "ā”‚ ā”‚\n", "ā”‚ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” ā”‚\n", "ā”‚ ā”‚ FastAPI Server ā”‚ ā”‚\n", "ā”‚ ā”‚ ā””ā”€ Environment (reset, step, state) ā”‚ ā”‚\n", "ā”‚ ā”‚ ā””ā”€ Your Game/Simulation Logic ā”‚ ā”‚\n", "ā”‚ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ā”‚\n", "ā”‚ ā”‚\n", "ā”‚ Isolated ā€¢ Reproducible ā€¢ Secure ā”‚\n", "ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜\n", "```\n", "\n", "
\n", "\n", "**šŸŽÆ Key Insight**: You never see HTTP details - just clean Python methods!\n", "\n", "```python\n", "env.reset() # Under the hood: HTTP POST to /reset\n", "env.step(...) # Under the hood: HTTP POST to /step\n", "env.state() # Under the hood: HTTP GET to /state\n", "```\n", "\n", "The magic? OpenEnv handles all the plumbing. You focus on RL! āœØ\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "# Part 3: Setup šŸ› ļø\n", "\n", "
\n", "\n", "**Running in Colab?** This cell will clone OpenEnv and install dependencies automatically.\n", "\n", "**Running locally?** Make sure you're in the OpenEnv directory.\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "šŸ’» Running locally - Nice!\n", "āœ… Using local OpenEnv installation\n", "\n", "šŸš€ Ready to explore OpenEnv and build amazing things!\n", "šŸ’” Tip: Run cells top-to-bottom for the best experience.\n", "\n" ] } ], "source": [ "# Detect environment\n", "try:\n", " import google.colab\n", " IN_COLAB = True\n", " print(\"šŸŒ Running in Google Colab - Perfect!\")\n", "except ImportError:\n", " IN_COLAB = False\n", " print(\"šŸ’» Running locally - Nice!\")\n", "\n", "if IN_COLAB:\n", " print(\"\\nšŸ“¦ Cloning OpenEnv repository...\")\n", " !git clone https://github.com/meta-pytorch/OpenEnv.git > /dev/null 2>&1\n", " %cd OpenEnv\n", " \n", " print(\"šŸ“š Installing dependencies (this takes ~10 seconds)...\")\n", " !pip install -q fastapi uvicorn requests\n", " \n", " import sys\n", " sys.path.insert(0, './src')\n", " print(\"\\nāœ… Setup complete! Everything is ready to go! šŸŽ‰\")\n", "else:\n", " import sys\n", " from pathlib import Path\n", " sys.path.insert(0, str(Path.cwd().parent))\n", " print(\"āœ… Using local OpenEnv installation\")\n", "\n", "print(\"\\nšŸš€ Ready to explore OpenEnv and build amazing things!\")\n", "print(\"šŸ’” Tip: Run cells top-to-bottom for the best experience.\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "\n", "# Part 4: The OpenEnv Pattern šŸ—ļø\n", "\n", "
\n", "\n", "## Every OpenEnv Environment Has 3 Components:\n", "\n", "```\n", "envs/your_env/\n", "ā”œā”€ā”€ šŸ“ models.py ā† Type-safe contracts\n", "ā”‚ (Action, Observation, State)\n", "ā”‚\n", "ā”œā”€ā”€ šŸ“± client.py ā† What YOU import\n", "ā”‚ (HTTPEnvClient implementation)\n", "ā”‚\n", "ā””ā”€ā”€ šŸ–„ļø server/\n", " ā”œā”€ā”€ environment.py ā† Game/simulation logic\n", " ā”œā”€ā”€ app.py ā† FastAPI server\n", " ā””ā”€ā”€ Dockerfile ā† Container definition\n", "```\n", "\n", "
\n", "\n", "Let's explore the actual OpenEnv code to see how this works:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "======================================================================\n", " šŸ§© OPENENV CORE ABSTRACTIONS\n", "======================================================================\n", "\n", "šŸ–„ļø SERVER SIDE (runs in Docker):\n", "\n", " class Environment(ABC):\n", " '''Base class for all environment implementations'''\n", "\n", " @abstractmethod\n", " def reset(self) -> Observation:\n", " '''Start new episode'''\n", "\n", " @abstractmethod\n", " def step(self, action: Action) -> Observation:\n", " '''Execute action, return observation'''\n", "\n", " @property\n", " def state(self) -> State:\n", " '''Get episode metadata'''\n", "\n", "šŸ“± CLIENT SIDE (your training code):\n", "\n", " class EnvClient(ABC):\n", " '''Base class for HTTP clients'''\n", "\n", " def reset(self) -> StepResult:\n", " # HTTP POST /reset\n", "\n", " def step(self, action) -> StepResult:\n", " # HTTP POST /step\n", "\n", " def state(self) -> State:\n", " # HTTP GET /state\n", "\n", "======================================================================\n", "\n", "āœØ Same interface on both sides - communication via HTTP!\n", "šŸŽÆ You focus on RL, OpenEnv handles the infrastructure.\n", "\n" ] } ], "source": [ "# Import OpenEnv's core abstractions\n", "from openenv.core.env_server import Environment, Action, Observation, State\n", "from openenv.core.env_client import EnvClient\n", "\n", "print(\"=\"*70)\n", "print(\" šŸ§© OPENENV CORE ABSTRACTIONS\")\n", "print(\"=\"*70)\n", "\n", "print(\"\"\"\n", "šŸ–„ļø SERVER SIDE (runs in Docker):\n", "\n", " class Environment(ABC):\n", " '''Base class for all environment implementations'''\n", " \n", " @abstractmethod\n", " def reset(self) -> Observation:\n", " '''Start new episode'''\n", " \n", " @abstractmethod\n", " def step(self, action: Action) -> Observation:\n", " '''Execute action, return observation'''\n", " \n", " @property\n", " def state(self) -> State:\n", " '''Get episode metadata'''\n", "\n", "šŸ“± CLIENT SIDE (your training code):\n", "\n", " class EnvClient(ABC):\n", " '''Base class for HTTP clients'''\n", " \n", " def reset(self) -> StepResult:\n", " # HTTP POST /reset\n", " \n", " def step(self, action) -> StepResult:\n", " # HTTP POST /step\n", " \n", " def state(self) -> State:\n", " # HTTP GET /state\n", "\"\"\")\n", "\n", "print(\"=\"*70)\n", "print(\"\\nāœØ Same interface on both sides - communication via HTTP!\")\n", "print(\"šŸŽÆ You focus on RL, OpenEnv handles the infrastructure.\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "# Part 5: Example Integration - OpenSpiel šŸŽ®\n", "\n", "
\n", "\n", "## What is OpenSpiel?\n", "\n", "**OpenSpiel** is a library from DeepMind with **70+ game environments** for RL research.\n", "\n", "## OpenEnv's Integration\n", "\n", "We've wrapped **6 OpenSpiel games** following the OpenEnv pattern:\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "**šŸŽÆ Single-Player**\n", "1. **Catch** - Catch falling ball\n", "2. **Cliff Walking** - Navigate grid\n", "3. **2048** - Tile puzzle\n", "4. **Blackjack** - Card game\n", "\n", "\n", "\n", "**šŸ‘„ Multi-Player**\n", "5. **Tic-Tac-Toe** - Classic 3Ɨ3\n", "6. **Kuhn Poker** - Imperfect info poker\n", "\n", "
\n", "\n", "This shows how OpenEnv can wrap **any** existing RL library!\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "======================================================================\n", " šŸ”Œ HOW OPENENV WRAPS OPENSPIEL\n", "======================================================================\n", "\n", "class OpenSpielEnv(HTTPEnvClient[OpenSpielAction, OpenSpielObservation]):\n", "\n", " def _step_payload(self, action: OpenSpielAction) -> dict:\n", " '''Convert typed action to JSON for HTTP'''\n", " return {\n", " \"action_id\": action.action_id,\n", " \"game_name\": action.game_name,\n", " }\n", "\n", " def _parse_result(self, payload: dict) -> StepResult:\n", " '''Parse HTTP JSON response into typed observation'''\n", " return StepResult(\n", " observation=OpenSpielObservation(...),\n", " reward=payload['reward'],\n", " done=payload['done']\n", " )\n", "\n", "\n", "ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", "\n", "āœØ Usage (works for ALL OpenEnv environments):\n", "\n", " env = OpenSpielEnv(base_url=\"http://localhost:8000\")\n", "\n", " result = env.reset()\n", " # Returns StepResult[OpenSpielObservation] - Type safe!\n", "\n", " result = env.step(OpenSpielAction(action_id=2, game_name=\"catch\"))\n", " # Type checker knows this is valid!\n", "\n", " state = env.state()\n", " # Returns OpenSpielState\n", "\n", "ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", "\n", "šŸŽÆ This pattern works for ANY environment you want to wrap!\n", "\n" ] } ], "source": [ "from envs.openspiel_env.client import OpenSpielEnv\n", "\n", "print(\"=\"*70)\n", "print(\" šŸ”Œ HOW OPENENV WRAPS OPENSPIEL\")\n", "print(\"=\"*70)\n", "\n", "print(\"\"\"\n", "class OpenSpielEnv(HTTPEnvClient[OpenSpielAction, OpenSpielObservation]):\n", " \n", " def _step_payload(self, action: OpenSpielAction) -> dict:\n", " '''Convert typed action to JSON for HTTP'''\n", " return {\n", " \"action_id\": action.action_id,\n", " \"game_name\": action.game_name,\n", " }\n", " \n", " def _parse_result(self, payload: dict) -> StepResult:\n", " '''Parse HTTP JSON response into typed observation'''\n", " return StepResult(\n", " observation=OpenSpielObservation(...),\n", " reward=payload['reward'],\n", " done=payload['done']\n", " )\n", "\n", "\"\"\")\n", "\n", "print(\"ā”€\" * 70)\n", "print(\"\\nāœØ Usage (works for ALL OpenEnv environments):\")\n", "print(\"\"\"\n", " env = OpenSpielEnv(base_url=\"http://localhost:8000\")\n", " \n", " result = env.reset()\n", " # Returns StepResult[OpenSpielObservation] - Type safe!\n", " \n", " result = env.step(OpenSpielAction(action_id=2, game_name=\"catch\"))\n", " # Type checker knows this is valid!\n", " \n", " state = env.state()\n", " # Returns OpenSpielState\n", "\"\"\")\n", "\n", "print(\"ā”€\" * 70)\n", "print(\"\\nšŸŽÆ This pattern works for ANY environment you want to wrap!\\n\")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "======================================================================\n", " šŸŽ® OPENSPIEL INTEGRATION - TYPE-SAFE MODELS\n", "======================================================================\n", "\n", "šŸ“¤ OpenSpielAction (what you send):\n", " ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", " ā€¢ metadata : typing.Dict[str, typing.Any]\n", " ā€¢ action_id : \n", " ā€¢ game_name : \n", " ā€¢ game_params : typing.Dict[str, typing.Any]\n", "\n", "šŸ“„ OpenSpielObservation (what you receive):\n", " ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", " ā€¢ done : \n", " ā€¢ reward : bool | int | float | None\n", " ā€¢ metadata : typing.Dict[str, typing.Any]\n", " ā€¢ info_state : typing.List[float]\n", " ā€¢ legal_actions : typing.List[int]\n", " ā€¢ game_phase : \n", " ā€¢ current_player_id : \n", " ā€¢ opponent_last_action : typing.Optional[int]\n", "\n", "šŸ“Š OpenSpielState (episode metadata):\n", " ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", " ā€¢ episode_id : typing.Optional[str]\n", " ā€¢ step_count : \n", " ā€¢ game_name : \n", " ā€¢ agent_player : \n", " ā€¢ opponent_policy : \n", " ā€¢ game_params : typing.Dict[str, typing.Any]\n", " ā€¢ num_players : \n", "\n", "======================================================================\n", "\n", "šŸ’” Type safety means:\n", " āœ… Your IDE autocompletes these fields\n", " āœ… Typos are caught before running\n", " āœ… Refactoring is safe\n", " āœ… Self-documenting code\n", "\n" ] } ], "source": [ "from envs.openspiel_env.models import (\n", " OpenSpielAction,\n", " OpenSpielObservation,\n", " OpenSpielState\n", ")\n", "\n", "print(\"=\" * 70)\n", "print(\" šŸŽ® OPENSPIEL INTEGRATION - TYPE-SAFE MODELS\")\n", "print(\"=\" * 70)\n", "\n", "print(\"\\nšŸ“¤ OpenSpielAction (what you send):\")\n", "print(\" \" + \"ā”€\" * 64)\n", "for name, field in OpenSpielAction.model_fields.items():\n", " print(f\" ā€¢ {name:20s} : {field.annotation}\")\n", "\n", "print(\"\\nšŸ“„ OpenSpielObservation (what you receive):\")\n", "print(\" \" + \"ā”€\" * 64)\n", "for name, field in OpenSpielObservation.model_fields.items():\n", " print(f\" ā€¢ {name:20s} : {field.annotation}\")\n", "\n", "print(\"\\nšŸ“Š OpenSpielState (episode metadata):\")\n", "print(\" \" + \"ā”€\" * 64)\n", "for name, field in OpenSpielState.model_fields.items():\n", " print(f\" ā€¢ {name:20s} : {field.annotation}\")\n", "\n", "print(\"\\n\" + \"=\" * 70)\n", "print(\"\\nšŸ’” Type safety means:\")\n", "print(\" āœ… Your IDE autocompletes these fields\")\n", "print(\" āœ… Typos are caught before running\")\n", "print(\" āœ… Refactoring is safe\")\n", "print(\" āœ… Self-documenting code\\n\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### How the Client Works\n", "\n", "
\n", "\n", "The client **inherits from HTTPEnvClient** and implements 3 methods:\n", "\n", "1. `_step_payload()` - Convert action ā†’ JSON\n", "2. `_parse_result()` - Parse JSON ā†’ typed observation \n", "3. `_parse_state()` - Parse JSON ā†’ state\n", "\n", "That's it! The base class handles all HTTP communication.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "
\n", "\n", "# šŸŽ® Part 6: Using Real OpenSpiel\n", "\n", "### Now let's USE a production environment!\n", "\n", "We'll play **Catch** using OpenEnv's **OpenSpiel integration** šŸŽÆ
\n", "This is a REAL environment running in production at companies!\n", "\n", "
\n", "\n", "**Get ready for:**\n", "- šŸ”Œ Using existing environments (not building)\n", "- šŸ¤– Testing policies against real games\n", "- šŸ“Š Live gameplay visualization\n", "- šŸŽÆ Production-ready patterns\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The Game: Catch šŸ”“šŸ“\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "```\n", "ā¬œ ā¬œ šŸ”“ ā¬œ ā¬œ\n", "ā¬œ ā¬œ ā¬œ ā¬œ ā¬œ\n", "ā¬œ ā¬œ ā¬œ ā¬œ ā¬œ Ball\n", "ā¬œ ā¬œ ā¬œ ā¬œ ā¬œ\n", "ā¬œ ā¬œ ā¬œ ā¬œ ā¬œ falls\n", "ā¬œ ā¬œ ā¬œ ā¬œ ā¬œ\n", "ā¬œ ā¬œ ā¬œ ā¬œ ā¬œ down\n", "ā¬œ ā¬œ ā¬œ ā¬œ ā¬œ\n", "ā¬œ ā¬œ ā¬œ ā¬œ ā¬œ\n", "ā¬œ ā¬œ šŸ“ ā¬œ ā¬œ\n", " Paddle\n", "```\n", "\n", "\n", "\n", "**Rules:**\n", "- 10Ɨ5 grid\n", "- Ball falls from random column\n", "- Move paddle left/right to catch it\n", "\n", "**Actions:**\n", "- `0` = Move LEFT ā¬…ļø\n", "- `1` = STAY šŸ›‘\n", "- `2` = Move RIGHT āž”ļø\n", "\n", "**Reward:**\n", "- `+1` if caught šŸŽ‰\n", "- `0` if missed šŸ˜¢\n", "\n", "
\n", "\n", "
\n", "\n", "**šŸŽÆ Why Catch?**\n", "- Simple rules (easy to understand)\n", "- Fast episodes (~5 steps)\n", "- Clear success/failure\n", "- Part of OpenSpiel's 70+ games!\n", "\n", "**šŸ’” The Big Idea:**\n", "Instead of building this from scratch, we'll USE OpenEnv's existing OpenSpiel integration. Same interface, but production-ready!\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "šŸŽ® ================================================================ šŸŽ®\n", " āœ… Importing Real OpenSpiel Environment!\n", "šŸŽ® ================================================================ šŸŽ®\n", "\n", "šŸ“¦ What we just imported:\n", " ā€¢ OpenSpielEnv - HTTP client for OpenSpiel games\n", " ā€¢ OpenSpielAction - Type-safe actions\n", " ā€¢ OpenSpielObservation - Type-safe observations\n", " ā€¢ OpenSpielState - Episode metadata\n", "\n", "šŸ“‹ OpenSpielObservation fields:\n", " ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€\n", " ā€¢ done : \n", " ā€¢ reward : bool | int | float | None\n", " ā€¢ metadata : typing.Dict[str, typing.Any]\n", " ā€¢ info_state : typing.List[float]\n", " ā€¢ legal_actions : typing.List[int]\n", " ā€¢ game_phase : \n", " ā€¢ current_player_id : \n", " ā€¢ opponent_last_action : typing.Optional[int]\n", "\n", "======================================================================\n", "\n", "šŸ’” This is REAL OpenEnv code - used in production!\n", " ā€¢ Wraps 6 OpenSpiel games (Catch, Tic-Tac-Toe, Poker, etc.)\n", " ā€¢ Type-safe actions and observations\n", " ā€¢ Works via HTTP (we'll see that next!)\n", "\n" ] } ], "source": [ "from envs.openspiel_env import OpenSpielEnv\n", "from envs.openspiel_env.models import (\n", " OpenSpielAction,\n", " OpenSpielObservation,\n", " OpenSpielState,\n", ")\n", "\n", "print(\"šŸŽ® \" + \"=\" * 64 + \" šŸŽ®\")\n", "print(\" āœ… Importing Real OpenSpiel Environment!\")\n", "print(\"šŸŽ® \" + \"=\" * 64 + \" šŸŽ®\\n\")\n", "\n", "print(\"šŸ“¦ What we just imported:\")\n", "print(\" ā€¢ OpenSpielEnv - HTTP client for OpenSpiel games\")\n", "print(\" ā€¢ OpenSpielAction - Type-safe actions\")\n", "print(\" ā€¢ OpenSpielObservation - Type-safe observations\")\n", "print(\" ā€¢ OpenSpielState - Episode metadata\\n\")\n", "\n", "print(\"šŸ“‹ OpenSpielObservation fields:\")\n", "print(\" \" + \"ā”€\" * 60)\n", "for name, field in OpenSpielObservation.model_fields.items():\n", " print(f\" ā€¢ {name:25s} : {field.annotation}\")\n", "\n", "print(\"\\n\" + \"=\" * 70)\n", "print(\"\\nšŸ’” This is REAL OpenEnv code - used in production!\")\n", "print(\" ā€¢ Wraps 6 OpenSpiel games (Catch, Tic-Tac-Toe, Poker, etc.)\")\n", "print(\" ā€¢ Type-safe actions and observations\")\n", "print(\" ā€¢ Works via HTTP (we'll see that next!)\\n\")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "šŸš€ ================================================================ šŸš€\n", " Starting OpenSpiel Server (Catch Game)\n", "šŸš€ ================================================================ šŸš€\n", "\n", "āœ… OpenSpiel is installed!\n", "\n", "āš” Starting FastAPI server for OpenSpiel Catch...\n", " (This uses REAL OpenEnv + OpenSpiel integration)\n", "\n", "ā³ Waiting for server to start...\n", "\n", "āœ… OpenSpiel server is running!\n", "šŸŒ Server URL: http://localhost:8000\n", "šŸ“ Endpoints available:\n", " ā€¢ POST /reset\n", " ā€¢ POST /step\n", " ā€¢ GET /state\n", "\n", "šŸŽÆ This is REAL OpenEnv + OpenSpiel in action!\n", " ā€¢ Running actual OpenSpiel Catch game\n", " ā€¢ Exposed via FastAPI HTTP server\n", " ā€¢ Using OpenEnv's standard interface\n", "\n" ] } ], "source": [ "import subprocess\n", "import time\n", "import sys\n", "import os\n", "\n", "print(\"šŸš€ \" + \"=\"*64 + \" šŸš€\")\n", "print(\" Starting OpenSpiel Server (Catch Game)\")\n", "print(\"šŸš€ \" + \"=\"*64 + \" šŸš€\\n\")\n", "\n", "# Check if open_spiel is installed\n", "try:\n", " import pyspiel\n", " print(\"āœ… OpenSpiel is installed!\\n\")\n", "except ImportError:\n", " print(\"āš ļø OpenSpiel not found. Installing...\")\n", " import subprocess\n", " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"open_spiel\"])\n", " print(\"āœ… OpenSpiel installed!\\n\")\n", "\n", "# Start the OpenSpiel server in background\n", "print(\"āš” Starting FastAPI server for OpenSpiel Catch...\")\n", "print(\" (This uses REAL OpenEnv + OpenSpiel integration)\\n\")\n", "\n", "# Determine the correct path\n", "if IN_COLAB:\n", " work_dir = \"/content/OpenEnv\"\n", "else:\n", " from pathlib import Path\n", " work_dir = str(Path.cwd().parent.absolute())\n", "\n", "server_process = subprocess.Popen(\n", " [sys.executable, \"-m\", \"uvicorn\",\n", " \"envs.openspiel_env.server.app:app\",\n", " \"--host\", \"0.0.0.0\",\n", " \"--port\", \"8000\"],\n", " env={**os.environ,\n", " \"PYTHONPATH\": f\"{work_dir}/src\",\n", " \"OPENSPIEL_GAME\": \"catch\",\n", " \"OPENSPIEL_AGENT_PLAYER\": \"0\",\n", " \"OPENSPIEL_OPPONENT_POLICY\": \"random\"},\n", " stdout=subprocess.DEVNULL,\n", " stderr=subprocess.DEVNULL,\n", " text=True,\n", " cwd=work_dir\n", ")\n", "\n", "# Wait for server to start\n", "print(\"ā³ Waiting for server to start...\")\n", "time.sleep(5)\n", "\n", "# Check if server is running\n", "import requests\n", "try:\n", " response = requests.get('http://localhost:8000/health', timeout=2)\n", " print(\"\\nāœ… OpenSpiel server is running!\")\n", " print(\"šŸŒ Server URL: http://localhost:8000\")\n", " print(\"šŸ“ Endpoints available:\")\n", " print(\" ā€¢ POST /reset\")\n", " print(\" ā€¢ POST /step\")\n", " print(\" ā€¢ GET /state\")\n", " print(\"\\nšŸŽÆ This is REAL OpenEnv + OpenSpiel in action!\")\n", " print(\" ā€¢ Running actual OpenSpiel Catch game\")\n", " print(\" ā€¢ Exposed via FastAPI HTTP server\")\n", " print(\" ā€¢ Using OpenEnv's standard interface\\n\")\n", "except Exception as e:\n", " print(f\"\\nāŒ Server failed to start: {e}\")\n", " print(\"\\nšŸ“‹ Checking error output...\")\n", " server_process.poll()\n", " if server_process.stderr:\n", " stderr = server_process.stderr.read()\n", " if stderr:\n", " print(stderr)\n", " print(\"\\nšŸ’” Make sure open_spiel is installed:\")\n", " print(\" pip install open_spiel\")\n", " raise" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "šŸ“± ================================================================ šŸ“±\n", " Connecting to OpenSpiel Server via HTTP\n", "šŸ“± ================================================================ šŸ“±\n", "\n", "āœ… Client created!\n", "\n", "šŸ’” What just happened:\n", " ā€¢ OpenSpielEnv is an HTTPEnvClient subclass\n", " ā€¢ It knows how to talk to OpenSpiel servers\n", " ā€¢ All communication is type-safe and over HTTP\n", " ā€¢ Same client works for ALL OpenSpiel games!\n", "\n" ] } ], "source": [ "print(\"šŸ“± \" + \"=\"*64 + \" šŸ“±\")\n", "print(\" Connecting to OpenSpiel Server via HTTP\")\n", "print(\"šŸ“± \" + \"=\"*64 + \" šŸ“±\\n\")\n", "\n", "# Create HTTP client for OpenSpiel\n", "client = OpenSpielEnv(base_url=\"http://localhost:8000\")\n", "\n", "print(\"āœ… Client created!\")\n", "print(\"\\nšŸ’” What just happened:\")\n", "print(\" ā€¢ OpenSpielEnv is an HTTPEnvClient subclass\")\n", "print(\" ā€¢ It knows how to talk to OpenSpiel servers\")\n", "print(\" ā€¢ All communication is type-safe and over HTTP\")\n", "print(\" ā€¢ Same client works for ALL OpenSpiel games!\\n\")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "šŸŽ® ================================================================ šŸŽ®\n", " Testing Connection - Playing One Step\n", "šŸŽ® ================================================================ šŸŽ®\n", "\n", "šŸ“¤ Calling client.reset()...\n", " Under the hood: HTTP POST to http://localhost:8000/reset\n", "\n", "šŸ“„ Received OpenSpielObservation:\n", " ā€¢ info_state: [0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]... (first 10 values)\n", " ā€¢ number of info_state: 50\n", " ā€¢ legal_actions: [0, 1, 2]\n", " ā€¢ game_phase: initial\n", " ā€¢ done: False\n", "\n", "šŸ“¤ Calling client.step(OpenSpielAction(action_id=1, game_name='catch'))...\n", " Under the hood: HTTP POST to http://localhost:8000/step\n", "\n", "šŸ“„ Received response:\n", " ā€¢ Reward: 0.0\n", " ā€¢ Done: False\n", " ā€¢ legal_actions: [0, 1, 2]\n", "\n", "šŸ“Š Episode state:\n", " ā€¢ episode_id: aec2e599-5633-45d6-a7a3-30f46e8796ab\n", " ā€¢ step_count: 1\n", " ā€¢ game_name: catch\n", "\n", "======================================================================\n", "\n", "šŸŽ‰ IT WORKS! We're using REAL OpenSpiel via HTTP!\n", " āœ… Type-safe communication\n", " āœ… Same interface as any OpenEnv environment\n", " āœ… Production-ready architecture\n", "\n" ] } ], "source": [ "print(\"šŸŽ® \" + \"=\"*64 + \" šŸŽ®\")\n", "print(\" Testing Connection - Playing One Step\")\n", "print(\"šŸŽ® \" + \"=\"*64 + \" šŸŽ®\\n\")\n", "\n", "# Reset the environment (HTTP POST /reset)\n", "print(\"šŸ“¤ Calling client.reset()...\")\n", "print(\" Under the hood: HTTP POST to http://localhost:8000/reset\\n\")\n", "\n", "result = client.reset()\n", "\n", "print(\"šŸ“„ Received OpenSpielObservation:\")\n", "print(f\" ā€¢ info_state: {result.observation.info_state[:10]}... (first 10 values)\")\n", "print(f\" ā€¢ number of info_state: {len(result.observation.info_state)}\")\n", "print(f\" ā€¢ legal_actions: {result.observation.legal_actions}\")\n", "print(f\" ā€¢ game_phase: {result.observation.game_phase}\")\n", "print(f\" ā€¢ done: {result.done}\")\n", "\n", "# Take an action (HTTP POST /step)\n", "print(\"\\nšŸ“¤ Calling client.step(OpenSpielAction(action_id=1, game_name=\\'catch\\'))...\")\n", "print(\" Under the hood: HTTP POST to http://localhost:8000/step\\n\")\n", "\n", "action = OpenSpielAction(action_id=1, game_name=\"catch\") # STAY\n", "result = client.step(action)\n", "\n", "print(\"šŸ“„ Received response:\")\n", "print(f\" ā€¢ Reward: {result.reward}\")\n", "print(f\" ā€¢ Done: {result.done}\")\n", "print(f\" ā€¢ legal_actions: {result.observation.legal_actions}\")\n", "\n", "# Get state (HTTP GET /state)\n", "state = client.state()\n", "print(f\"\\nšŸ“Š Episode state:\")\n", "print(f\" ā€¢ episode_id: {state.episode_id}\")\n", "print(f\" ā€¢ step_count: {state.step_count}\")\n", "print(f\" ā€¢ game_name: {state.game_name}\")\n", "\n", "print(\"\\n\" + \"=\"*70)\n", "print(\"\\nšŸŽ‰ IT WORKS! We\\'re using REAL OpenSpiel via HTTP!\")\n", "print(\" āœ… Type-safe communication\")\n", "print(\" āœ… Same interface as any OpenEnv environment\")\n", "print(\" āœ… Production-ready architecture\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "# Part 7: Four Policies šŸ¤–\n", "\n", "
\n", "\n", "## Let's test 4 different AI strategies:\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
PolicyStrategyExpected Performance
šŸŽ² RandomPick random action every step~20% (pure luck)
šŸ›‘ Always StayNever move, hope ball lands in center~20% (terrible!)
šŸ§  SmartMove paddle toward ball100% (optimal!)
šŸ“ˆ LearningStart random, learn smart strategy~85% (improves over time)
\n", "\n", "**šŸ’” These policies work with ANY OpenSpiel game!**\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import random\n", "\n", "# ============================================================================\n", "# POLICIES - Different AI strategies (adapted for OpenSpiel)\n", "# ============================================================================\n", "\n", "class RandomPolicy:\n", " \"\"\"Baseline: Pure random guessing.\"\"\"\n", " name = \"šŸŽ² Random Guesser\"\n", "\n", " def select_action(self, obs: OpenSpielObservation) -> int:\n", " return random.choice(obs.legal_actions)\n", "\n", "\n", "class AlwaysStayPolicy:\n", " \"\"\"Bad strategy: Never moves.\"\"\"\n", " name = \"šŸ›‘ Always Stay\"\n", "\n", " def select_action(self, obs: OpenSpielObservation) -> int:\n", " return 1 # STAY\n", "\n", "\n", "class SmartPolicy:\n", " \"\"\"Optimal: Move paddle toward ball.\"\"\"\n", " name = \"šŸ§  Smart Heuristic\"\n", "\n", " def select_action(self, obs: OpenSpielObservation) -> int:\n", " # Parse OpenSpiel observation\n", " # For Catch: info_state is a flattened 10x5 grid\n", " # Ball position and paddle position encoded in the vector\n", " info_state = obs.info_state\n", "\n", " # Find ball and paddle positions from info_state\n", " # Catch uses a 10x5 grid, so 50 values\n", " grid_size = 5\n", "\n", " # Find positions (ball = 1.0 in the flattened grid, paddle = 1.0 in the last row of the flattened grid)\n", " ball_col = None\n", " paddle_col = None\n", "\n", " for idx, val in enumerate(info_state):\n", " if abs(val - 1.0) < 0.01: # Ball\n", " ball_col = idx % grid_size\n", " break\n", "\n", " last_row = info_state[-grid_size:]\n", " paddle_col = last_row.index(1.0) # Paddle\n", "\n", " if ball_col is not None and paddle_col is not None:\n", " if paddle_col < ball_col:\n", " return 2 # Move RIGHT\n", " elif paddle_col > ball_col:\n", " return 0 # Move LEFT\n", "\n", " return 1 # STAY (fallback)\n", "\n", "\n", "class LearningPolicy:\n", " \"\"\"Simulated RL: Epsilon-greedy exploration.\"\"\"\n", " name = \"šŸ“ˆ Learning Agent\"\n", "\n", " def __init__(self):\n", " self.steps = 0\n", " self.smart_policy = SmartPolicy()\n", "\n", " def select_action(self, obs: OpenSpielObservation) -> int:\n", " self.steps += 1\n", "\n", " # Decay exploration rate over time\n", " epsilon = max(0.1, 1.0 - (self.steps / 100))\n", "\n", " if random.random() < epsilon:\n", " # Explore: random action\n", " return random.choice(obs.legal_actions)\n", " else:\n", " # Exploit: use smart strategy\n", " return self.smart_policy.select_action(obs)\n", "\n", "\n", "print(\"šŸ¤– \" + \"=\"*64 + \" šŸ¤–\")\n", "print(\" āœ… 4 Policies Created (Adapted for OpenSpiel)!\")\n", "print(\"šŸ¤– \" + \"=\"*64 + \" šŸ¤–\\n\")\n", "\n", "policies = [RandomPolicy(), AlwaysStayPolicy(), SmartPolicy(), LearningPolicy()]\n", "for i, policy in enumerate(policies, 1):\n", " print(f\" {i}. {policy.name}\")\n", "\n", "print(\"\\nšŸ’” These policies work with OpenSpielObservation!\")\n", "print(\" ā€¢ Read info_state (flattened grid)\")\n", "print(\" ā€¢ Use legal_actions\")\n", "print(\" ā€¢ Work with ANY OpenSpiel game that exposes these!\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Watch a Policy Play!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "def run_episode(env, policy, visualize=True, delay=0.3):\n", " \"\"\"Run one episode with a policy against OpenSpiel environment.\"\"\"\n", "\n", " # RESET\n", " result = env.reset()\n", " obs = result.observation\n", "\n", " if visualize:\n", " print(f\"\\n{'='*60}\")\n", " print(f\" šŸŽ® {policy.name}\")\n", " print(f\" šŸŽ² Playing against OpenSpiel Catch\")\n", " print('='*60 + '\\n')\n", " time.sleep(delay)\n", "\n", " total_reward = 0\n", " step = 0\n", " action_names = [\"ā¬…ļø LEFT\", \"šŸ›‘ STAY\", \"āž”ļø RIGHT\"]\n", "\n", " # THE RL LOOP\n", " while not obs.done:\n", " # 1. Policy chooses action\n", " action_id = policy.select_action(obs)\n", "\n", " # 2. Environment executes (via HTTP!)\n", " action = OpenSpielAction(action_id=action_id, game_name=\"catch\")\n", " result = env.step(action)\n", " obs = result.observation\n", "\n", " # 3. Collect reward\n", " if result.reward is not None:\n", " total_reward += result.reward\n", "\n", " if visualize:\n", " print(f\"šŸ“ Step {step + 1}: {action_names[action_id]} ā†’ Reward: {result.reward}\")\n", " time.sleep(delay)\n", "\n", " step += 1\n", "\n", " if visualize:\n", " result_text = \"šŸŽ‰ CAUGHT!\" if total_reward > 0 else \"šŸ˜¢ MISSED\"\n", " print(f\"\\n{'='*60}\")\n", " print(f\" {result_text} Total Reward: {total_reward}\")\n", " print('='*60)\n", "\n", " return total_reward > 0\n", "\n", "\n", "print(\"šŸ“ŗ \" + \"=\"*64 + \" šŸ“ŗ\")\n", "print(\" Watch Smart Policy Play Against OpenSpiel!\")\n", "print(\"šŸ“ŗ \" + \"=\"*64 + \" šŸ“ŗ\\n\")\n", "\n", "# Demo: Watch Smart Policy in action\n", "policy = SmartPolicy()\n", "run_episode(client, policy, visualize=True, delay=0.5)\n", "\n", "print(\"\\nšŸ’” You just watched REAL OpenSpiel Catch being played!\")\n", "print(\" ā€¢ Every action was an HTTP call\")\n", "print(\" ā€¢ Game logic runs in the server\")\n", "print(\" ā€¢ Client only sends actions and receives observations\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "# Part 8: Policy Competition! šŸ†\n", "\n", "
\n", "\n", "Let's run **50 episodes** for each policy against **REAL OpenSpiel** and see who wins!\n", "\n", "This is production code - every action is an HTTP call to the OpenSpiel server!\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def evaluate_policies(env, num_episodes=50):\n", " \"\"\"Compare all policies over many episodes using real OpenSpiel.\"\"\"\n", " policies = [\n", " RandomPolicy(),\n", " AlwaysStayPolicy(),\n", " SmartPolicy(),\n", " LearningPolicy(),\n", " ]\n", "\n", " print(\"\\nšŸ† \" + \"=\"*66 + \" šŸ†\")\n", " print(f\" POLICY SHOWDOWN - {num_episodes} Episodes Each\")\n", " print(f\" Playing against REAL OpenSpiel Catch!\")\n", " print(\"šŸ† \" + \"=\"*66 + \" šŸ†\\n\")\n", "\n", " results = []\n", " for policy in policies:\n", " print(f\"āš” Testing {policy.name}...\", end=\" \")\n", " successes = sum(run_episode(env, policy, visualize=False)\n", " for _ in range(num_episodes))\n", " success_rate = (successes / num_episodes) * 100\n", " results.append((policy.name, success_rate, successes))\n", " print(f\"āœ“ Done!\")\n", "\n", " print(\"\\n\" + \"=\"*70)\n", " print(\" šŸ“Š FINAL RESULTS\")\n", " print(\"=\"*70 + \"\\n\")\n", "\n", " # Sort by success rate (descending)\n", " results.sort(key=lambda x: x[1], reverse=True)\n", "\n", " # Award medals to top 3\n", " medals = [\"šŸ„‡\", \"šŸ„ˆ\", \"šŸ„‰\", \" \"]\n", "\n", " for i, (name, rate, successes) in enumerate(results):\n", " medal = medals[i]\n", " bar = \"ā–ˆ\" * int(rate / 2)\n", " print(f\"{medal} {name:25s} [{bar:<50}] {rate:5.1f}% ({successes}/{num_episodes})\")\n", "\n", " print(\"\\n\" + \"=\"*70)\n", " print(\"\\nāœØ Key Insights:\")\n", " print(\" ā€¢ Random (~20%): Baseline - pure luck šŸŽ²\")\n", " print(\" ā€¢ Always Stay (~20%): Bad strategy - stays center šŸ›‘\")\n", " print(\" ā€¢ Smart (100%): Optimal - perfect play! šŸ§ \")\n", " print(\" ā€¢ Learning (~85%): Improves over time šŸ“ˆ\")\n", " print(\"\\nšŸŽ“ This is Reinforcement Learning + OpenEnv in action:\")\n", " print(\" 1. We USED existing OpenSpiel environment (didn\\'t build it)\")\n", " print(\" 2. Type-safe communication over HTTP\")\n", " print(\" 3. Same code works for ANY OpenSpiel game\")\n", " print(\" 4. Production-ready architecture\\n\")\n", "\n", "# Run the epic competition!\n", "print(\"šŸŽ® Starting the showdown against REAL OpenSpiel...\\n\")\n", "evaluate_policies(client, num_episodes=50)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "\n", "# Part 9: Switching to Other Games šŸŽ®\n", "\n", "
\n", "\n", "## What We Just Used: Real OpenSpiel! šŸŽ‰\n", "\n", "In Parts 6-8, we **USED** the existing OpenSpiel Catch environment:\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
What We DidHow It Works
ImportedOpenSpielEnv client (pre-built)
StartedOpenSpiel server via uvicorn
ConnectedHTTP client to server
PlayedReal OpenSpiel Catch game
\n", "\n", "**šŸŽÆ This is production code!** Every action was an HTTP call to a real OpenSpiel environment.\n", "\n", "
\n", "\n", "## šŸŽ® 6 Games Available - Same Interface!\n", "\n", "The beauty of OpenEnv? **Same code, different games!**\n", "\n", "```python\n", "# We just used Catch\n", "env = OpenSpielEnv(base_url=\"http://localhost:8000\")\n", "# game_name=\"catch\" was set via environment variable\n", "\n", "# Want Tic-Tac-Toe instead? Just change the game!\n", "# Start server with: OPENSPIEL_GAME=tic_tac_toe uvicorn ...\n", "# Same client code works!\n", "```\n", "\n", "
\n", "\n", "**šŸŽ® All 6 Games:**\n", "\n", "1. āœ… **`catch`** - What we just used!\n", "2. **`tic_tac_toe`** - Classic 3Ɨ3\n", "3. **`kuhn_poker`** - Imperfect information poker\n", "4. **`cliff_walking`** - Grid navigation\n", "5. **`2048`** - Tile puzzle\n", "6. **`blackjack`** - Card game\n", "\n", "**All use the exact same OpenSpielEnv client!**\n", "\n", "
\n", "\n", "### Try Another Game (Optional):\n", "\n", "```python\n", "# Stop the current server (kill the server_process)\n", "# Then start a new game:\n", "\n", "server_process = subprocess.Popen(\n", " [sys.executable, \"-m\", \"uvicorn\",\n", " \"envs.openspiel_env.server.app:app\",\n", " \"--host\", \"0.0.0.0\",\n", " \"--port\", \"8000\"],\n", " env={**os.environ,\n", " \"PYTHONPATH\": f\"{work_dir}/src\",\n", " \"OPENSPIEL_GAME\": \"tic_tac_toe\", # Changed!\n", " \"OPENSPIEL_AGENT_PLAYER\": \"0\",\n", " \"OPENSPIEL_OPPONENT_POLICY\": \"random\"},\n", " # ... rest of config\n", ")\n", "\n", "# Same client works!\n", "client = OpenSpielEnv(base_url=\"http://localhost:8000\")\n", "result = client.reset() # Now playing Tic-Tac-Toe!\n", "```\n", "\n", "**šŸ’” Key Insight**: You don't rebuild anything - you just USE different games with the same client!\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "\n", "# Part 10: Create Your Own Integration šŸ› ļø\n", "\n", "
\n", "\n", "## The 5-Step Pattern\n", "\n", "Want to wrap your own environment in OpenEnv? Here's how:\n", "\n", "
\n", "\n", "### Step 1: Define Types (`models.py`)\n", "\n", "```python\n", "from dataclasses import dataclass\n", "from openenv.core.env_server import Action, Observation, State\n", "\n", "@dataclass\n", "class YourAction(Action):\n", " action_value: int\n", " # Add your action fields\n", "\n", "@dataclass\n", "class YourObservation(Observation):\n", " state_data: List[float]\n", " done: bool\n", " reward: float\n", " # Add your observation fields\n", "\n", "@dataclass\n", "class YourState(State):\n", " episode_id: str\n", " step_count: int\n", " # Add your state fields\n", "```\n", "\n", "### Step 2: Implement Environment (`server/environment.py`)\n", "\n", "```python\n", "from openenv.core.env_server import Environment\n", "\n", "class YourEnvironment(Environment):\n", " def reset(self) -> Observation:\n", " # Initialize your game/simulation\n", " return YourObservation(...)\n", " \n", " def step(self, action: Action) -> Observation:\n", " # Execute action, update state\n", " return YourObservation(...)\n", " \n", " @property\n", " def state(self) -> State:\n", " return self._state\n", "```\n", "\n", "### Step 3: Create Client (`client.py`)\n", "\n", "```python\n", "from openenv.core.http_env_client import HTTPEnvClient\n", "from openenv.core.types import StepResult\n", "\n", "class YourEnv(HTTPEnvClient[YourAction, YourObservation]):\n", " def _step_payload(self, action: YourAction) -> dict:\n", " \"\"\"Convert action to JSON\"\"\"\n", " return {\"action_value\": action.action_value}\n", " \n", " def _parse_result(self, payload: dict) -> StepResult:\n", " \"\"\"Parse JSON to observation\"\"\"\n", " return StepResult(\n", " observation=YourObservation(...),\n", " reward=payload['reward'],\n", " done=payload['done']\n", " )\n", " \n", " def _parse_state(self, payload: dict) -> YourState:\n", " return YourState(...)\n", "```\n", "\n", "### Step 4: Create Server (`server/app.py`)\n", "\n", "```python\n", "from openenv.core.env_server import create_fastapi_app\n", "from .your_environment import YourEnvironment\n", "\n", "env = YourEnvironment()\n", "app = create_fastapi_app(env)\n", "\n", "# That's it! OpenEnv creates all endpoints for you.\n", "```\n", "\n", "### Step 5: Dockerize (`server/Dockerfile`)\n", "\n", "```dockerfile\n", "FROM python:3.11-slim\n", "\n", "WORKDIR /app\n", "COPY requirements.txt .\n", "RUN pip install --no-cache-dir -r requirements.txt\n", "\n", "COPY . .\n", "CMD [\"uvicorn\", \"app:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\"]\n", "```\n", "\n", "
\n", "\n", "### šŸŽ“ Examples to Study\n", "\n", "OpenEnv includes 3 complete examples:\n", "\n", "1. **`envs/echo_env/`**\n", " - Simplest possible environment\n", " - Great for testing and learning\n", "\n", "2. **`envs/openspiel_env/`**\n", " - Wraps external library (OpenSpiel)\n", " - Shows integration pattern\n", " - 6 games in one integration\n", "\n", "3. **`envs/coding_env/`**\n", " - Python code execution environment\n", " - Shows complex use case\n", " - Security considerations\n", "\n", "**šŸ’” Study these to understand the patterns!**\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "
\n", "\n", "# šŸŽ“ Summary: Your Journey\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What You Learned\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "### šŸ“š Concepts\n", "\n", "āœ… **RL Fundamentals**\n", "- The observe-act-reward loop\n", "- What makes good policies\n", "- Exploration vs exploitation\n", "\n", "āœ… **OpenEnv Architecture**\n", "- Client-server separation\n", "- Type-safe contracts\n", "- HTTP communication layer\n", "\n", "āœ… **Production Patterns**\n", "- Docker isolation\n", "- API design\n", "- Reproducible deployments\n", "\n", "\n", "\n", "### šŸ› ļø Skills\n", "\n", "āœ… **Using Environments**\n", "- Import OpenEnv clients\n", "- Call reset/step/state\n", "- Work with typed observations\n", "\n", "āœ… **Building Environments**\n", "- Define type-safe models\n", "- Implement Environment class\n", "- Create HTTPEnvClient\n", "\n", "āœ… **Testing & Debugging**\n", "- Compare policies\n", "- Visualize episodes\n", "- Measure performance\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## OpenEnv vs Traditional RL\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
FeatureTraditional (Gym)OpenEnvWinner
Type SafetyāŒ Arrays, dictsāœ… DataclassesšŸ† OpenEnv
IsolationāŒ Same processāœ… DockeršŸ† OpenEnv
DeploymentāŒ Manual setupāœ… K8s-readyšŸ† OpenEnv
LanguageāŒ Python onlyāœ… Any (HTTP)šŸ† OpenEnv
ReproducibilityāŒ \"Works on my machine\"āœ… Same everywherešŸ† OpenEnv
Communityāœ… Large ecosystemšŸŸ” GrowingšŸ¤ Both!
\n", "\n", "
\n", "\n", "**šŸŽÆ The Bottom Line**\n", "\n", "OpenEnv brings **production engineering** to RL:\n", "- Same environments work locally and in production\n", "- Type safety catches bugs early\n", "- Docker isolation prevents conflicts\n", "- HTTP API works with any language\n", "\n", "**It's RL for 2024 and beyond.**\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## šŸ“š Resources\n", "\n", "
\n", "\n", "### šŸ”— Essential Links\n", "\n", "- **šŸ  OpenEnv GitHub**: https://github.com/meta-pytorch/OpenEnv\n", "- **šŸŽ® OpenSpiel**: https://github.com/google-deepmind/open_spiel\n", "- **āš” FastAPI Docs**: https://fastapi.tiangolo.com/\n", "- **šŸ³ Docker Guide**: https://docs.docker.com/get-started/\n", "- **šŸ”„ PyTorch**: https://pytorch.org/\n", "\n", "### šŸ“– Documentation Deep Dives\n", "\n", "- **Environment Creation Guide**: `envs/README.md`\n", "- **OpenSpiel Integration**: `envs/openspiel_env/README.md`\n", "- **Example Scripts**: `examples/`\n", "- **RFC 001**: [Baseline API Specs](https://github.com/meta-pytorch/OpenEnv/pull/26)\n", "\n", "### šŸŽ“ Community & Support\n", "\n", "**Supported by amazing organizations:**\n", "- šŸ”„ Meta PyTorch\n", "- šŸ¤— Hugging Face\n", "- āš” Unsloth AI\n", "- šŸŒŸ Reflection AI\n", "- šŸš€ And many more!\n", "\n", "**License**: BSD 3-Clause (very permissive!)\n", "\n", "**Contributions**: Always welcome! Check out the issues tab.\n", "\n", "
\n", "\n", "---\n", "\n", "### šŸŒˆ What's Next?\n", "\n", "1. ā­ **Star the repo** to show support and stay updated\n", "2. šŸ”„ **Try modifying** the Catch game (make it harder? bigger grid?)\n", "3. šŸŽ® **Explore** other OpenSpiel games\n", "4. šŸ› ļø **Build** your own environment integration\n", "5. šŸ’¬ **Share** what you build with the community!" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 4 }