docs/planning/openenv-course-notes.md

OpenEnv Course — Complete Study Notes

Source: https://github.com/raun/openenv-course/tree/main Scraped: 2026-03-24

Building RL Environments with OpenEnv — A hands-on course for ML engineers, researchers, and hobbyists who want to use and build RL environments for LLM training.

---

Course Overview

• 5 modules, ~45-60 min each
• Format: Markdown README + Jupyter Notebook (Google Colab) per module
• Prerequisites: Basic Python, Hugging Face ecosystem familiarity, no RL experience required

#	Module	What You'll Learn
1	Why OpenEnv?	The RL loop, why Gym falls short, OpenEnv architecture
2	Using Existing Environments	Environment Hub, type-safe models, policies, competition
3	Deploying Environments	Local dev, Docker, HF Spaces, openenv push
4	Building Your Own Environment	The 3-component pattern, scaffold → deploy
5	Training with OpenEnv + TRL	GRPO, reward functions, Wordle training

---

Quick Start

# Install OpenEnv core
pip install openenv-core

# Clone the OpenEnv repo to get typed environment clients
git clone https://github.com/meta-pytorch/OpenEnv.git

import sys, os
repo = os.path.abspath('OpenEnv')
sys.path.insert(0, repo)
sys.path.insert(0, os.path.join(repo, 'src'))

# Echo environment — uses MCP tool-calling interface
from envs.echo_env import EchoEnv
with EchoEnv(base_url="https://openenv-echo-env.hf.space").sync() as env:
    env.reset()
    response = env.call_tool("echo_message", message="Hello, OpenEnv!")
    print(response)  # Hello, OpenEnv!

# OpenSpiel environments — use standard reset/step interface
from envs.openspiel_env import OpenSpielEnv
from envs.openspiel_env.models import OpenSpielAction
with OpenSpielEnv(base_url="https://openenv-openspiel-catch.hf.space").sync() as env:
    result = env.reset()
    result = env.step(OpenSpielAction(action_id=1, game_name="catch"))
    print(result.observation.legal_actions)

Every standard OpenEnv environment uses the same 3-method interface: reset(), step(), state().

---

Module 1: Why OpenEnv?

The RL Loop

Observe → Act → Reward → Repeat — the basic cycle where agents interact with environments, receive feedback, and improve.

Why Not Gymnasium?

Problem	Gymnasium	OpenEnv
Type Safety	Cryptic array indexing (obs[0][3])	Typed Pydantic models with autocomplete
Isolation	Same-process, crashes together	Containerized microservices
Deployment	Not reproducible across machines	Versioned Docker images
Scaling	Single-process only	Cloud-scalable, multi-container
Language	Python-only	Language-agnostic HTTP/WebSocket API
Debugging	Hard to inspect state	Type-safe, inspectable

OpenEnv Core Principle

"RL environments should be microservices" — environments deserve containerized isolation, like separating databases from application servers.

Architecture

Training Code  ←——WebSocket/HTTP——→  Containerized Environment Server
(any language)                        (Docker, FastAPI, versioned)

The 3-Method Interface

Every environment implements:

• reset() — initialize/restart episode
• step(action) — send action, receive observation + reward
• state() — inspect current environment state

The 3-Component Structure (per environment)

my_env/
├── models.py      ← Type contracts (Action, Observation, State)
├── client.py      ← What training code imports
└── server/
    ├── environment.py  ← Game logic (FastAPI)
    ├── app.py          ← Server wiring
    └── Dockerfile      ← Container definition

• Server-side: Abstract base classes, FastAPI
• Client-side: Async methods with sync wrappers for notebooks
• MCP-based environments: Use tool-calling patterns

---

Module 2: Using Existing Environments

The Environment Hub

Environments are hosted on Hugging Face Spaces. Each Space provides three components:

Component	Function	Access
Server	Running environment endpoint	https://<username>-<space-name>.hf.space
Repository	Installable Python package	pip install git+https://huggingface.co/spaces/<space>
Registry	Docker container image	docker pull registry.hf.space/<space>:latest

You don't need to build environments to use them. Install the client, point it at a server, and go.

Type-Safe Models

Environments define typed Pydantic models for actions, observations, and state:

class OpenSpielAction(Action):
    action_id: int
    game_name: str = "catch"
    game_params: Dict[str, Any] = Field(default_factory=dict)

class OpenSpielObservation(Observation):
    info_state: List[float]
    legal_actions: List[int]
    game_phase: str = "playing"
    current_player_id: int = 0
    opponent_last_action: Optional[int] = None

No more guessing what obs[0][3] means.

Available OpenSpiel Games

Game	Type	Description
Catch	Single-player	Catch falling ball
Cliff Walking	Single-player	Navigate grid
2048	Single-player	Tile puzzle
Blackjack	Single-player	Card game
Tic-Tac-Toe	Multi-player	Classic 3x3
Kuhn Poker	Multi-player	Imperfect information

Writing Policies — Example (Catch)

Policy	Success Rate	Logic
Random	~20%	random.choice(obs.legal_actions)
Stay	~20%	Always returns 1 (STAY)
Smart Heuristic	100%	Finds ball & paddle position, moves toward ball
Epsilon-Greedy	~85%	Mixes random exploration with smart policy

Key insight: All four policies operate with the identical OpenSpielObservation type.

Switching Games — Same Interface

# Catch
with OpenSpielEnv(base_url="https://openenv-openspiel-catch.hf.space").sync() as env:
    result = env.reset()

# Tic-Tac-Toe — same client, different URL
with OpenSpielEnv(base_url="https://openenv-openspiel-tictactoe.hf.space").sync() as env:
    result = env.reset()

Policy code stays the same; only game strategy changes.

---

Module 3: Deploying Environments

Three Access Methods (from a single HF Space)

1. Web server: https://<username>-<space-name>.hf.space
2. Pip package: pip install git+https://huggingface.co/spaces/<space>
3. Docker image: docker pull registry.hf.space/<space>:latest

Local Development

# Clone and run
git clone https://huggingface.co/spaces/<space>
cd <space>
uv sync && uv run server

# Or with uvicorn (auto-reload)
uvicorn server.app:app --reload

Docker Deployment

# Pull prebuilt
docker pull registry.hf.space/<space>:latest

# Or build from source
docker build -t my-env ./server

# Run with config
docker run -p 8000:8000 \
  -e WORKERS=4 \
  -e MAX_CONCURRENT_ENVS=100 \
  my-env

Deploy to HF Spaces

openenv push --repo-id user/my-env

Automatically provides: API docs, web UI, health check endpoints.

Configuration via openenv.yaml

Variable	Default	Description
WORKERS	4	Uvicorn worker processes
PORT	8000	Server port
HOST	0.0.0.0	Bind address
MAX_CONCURRENT_ENVS	100	Max WebSocket sessions per worker

HF Spaces Hardware Tiers

Tier	Specs	Cost
CPU Basic (free)	2 vCPUs, 16GB RAM	Free (~128 concurrent sessions)
CPU Upgrade	8 vCPUs, 32GB RAM	$0.03/hour

Deployment Workflow

Initialize → Implement logic → Test locally → Deploy → Install client

---

Module 4: Building Your Own Environment

The 3-Component Pattern

my_env/
├── models.py              ← Types: Action, Observation, State
├── client.py              ← HTTP/WebSocket client (what users import)
├── server/
│   ├── environment.py     ← Game logic (reset, step, state)
│   ├── app.py             ← FastAPI server
│   └── Dockerfile         ← Container definition
├── openenv.yaml           ← Manifest
└── pyproject.toml         ← Package metadata

~100 lines of meaningful code for a complete custom environment.

Step 1: Define Types (models.py)

from openenv.core.env_server import Action, Observation, State

class WordGameAction(Action):
    guess: str  # Player's guessed letter

class WordGameObservation(Observation):
    # Inherits: done, reward
    masked_word: str
    guessed_letters: list
    attempts_remaining: int
    message: str

class WordGameState(State):
    # Inherits: episode_id, step_count
    target_word: str
    max_attempts: int

Step 2: Implement Logic (server/environment.py)

class WordGameEnvironment:
    SUPPORTS_CONCURRENT_SESSIONS = True
    MAX_ATTEMPTS = 10
    WORDS = ["python", "neural", "tensor", "matrix", "vector",
             "kernel", "lambda", "signal", "binary", "cipher"]

    def reset(self):
        # Select random word, clear state, return initial observation
        ...

    def step(self, action):
        # Process guess, update state, check win/loss
        # reward: 1.0 for win, 0.0 otherwise
        ...

    def _mask(self):
        # Reveal guessed letters, hide others with underscores
        ...

Step 3: Create Client (client.py)

class WordGameEnv(EnvClient):
    def _step_payload(self, action: WordGameAction) -> dict:
        return {"guess": action.guess}

    def _parse_result(self, payload) -> WordGameObservation:
        # Reconstruct observation from server response
        ...

    def _parse_state(self, payload) -> WordGameState:
        # Reconstruct state from server response
        ...

EnvClient base class handles all WebSocket communication automatically.

Step 4: Wire FastAPI (server/app.py)

app = create_fastapi_app(WordGameEnvironment)

This single line auto-generates endpoints: /ws, /reset, /step, /state, /health, /web, /docs.

Step 5: Dockerize (server/Dockerfile)

Standard Python 3.11-slim container running uvicorn on port 8000.

The Fast Path — Scaffolding

openenv init word_game      # Generates full directory structure
# Customize: models.py, server/environment.py, client.py
uv run server               # Test locally
openenv push --repo-id user/word-game  # Deploy

---

Module 5: Training with OpenEnv + TRL

What is GRPO?

Group Relative Policy Optimization — RL for LLM fine-tuning:

1. Generate multiple completions per prompt
2. Score them via reward functions
3. Use relative ranking within groups to optimize the policy
4. No separate value model needed (unlike PPO)

TRL + OpenEnv Integration

GRPOTrainer
  → calls rollout_func with prompts
    → generates completions via model
      → each completion becomes an environment action
        → environment returns observations + rewards
          → TRL applies rewards to optimize model

trainer = GRPOTrainer(
    model=model_name,
    reward_funcs=[reward_correct, reward_greens, reward_yellows],
    rollout_func=rollout_func,
    train_dataset=dataset,
    args=grpo_config,
)
trainer.train()

Wordle Training Example

Environment: TextArena Wordle on HF Spaces

• Input: [WORD] (5-letter words in brackets)
• Feedback: G (green/correct), Y (yellow/misplaced), X (gray/absent)
• 6 attempts per game
• Reward: 1.0 for correct, 0.0 otherwise

from envs.textarena_env import TextArenaEnv
env = TextArenaEnv(base_url="https://burtenshaw-textarena.hf.space")

Reward Functions

Reward	What it measures	Range
reward_correct	Game win (solved word)	0.0–1.0
reward_greens	Correct letter positions	0.0–1.0
reward_yellows	Misplaced letter detection	0.0–1.0
reward_repetition	Penalizes duplicate guesses	0.0–1.0

Greens and yellows provide learning gradient even without victories. Repetition penalty discourages repeating the same guess.

Rollout Function (Simplified)

def rollout_once(trainer, env, tokenizer, prompt, system_prompt, max_turns):
    result = env.reset()
    observation = result.observation

    for turn in range(max_turns):
        if result.done:
            break
        messages = [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": format_game_state(observation)},
        ]
        rollout = generate_rollout_completions(trainer, [messages])
        guess = extract_guess(rollout["text"])
        result = env.step(TextArenaAction(message=guess))
        observation = result.observation

    return {
        "prompt_ids": ..., "completion_ids": ..., "logprobs": ...,
        "correct_reward": ..., "green_reward": ...,
    }

GRPO Config

grpo_config = GRPOConfig(
    num_train_epochs=1,
    learning_rate=5e-6,
    gradient_accumulation_steps=64,
    per_device_train_batch_size=1,
    num_generations=2,
    max_completion_length=8,        # Wordle = short responses
    max_prompt_length=1400,
    use_vllm=True,
    vllm_mode="colocate",           # Generation + training on same GPU
    vllm_gpu_memory_utilization=0.1,
    gradient_checkpointing=True,
    report_to="trackio",
)

Hardware Requirements

• GPU: A100 40GB (Colab Pro or equivalent)
• Training time: ~90 minutes
• Peak memory: ~37GB

What the Model Learns

After training:

• Strong opening moves (CRANE, SLATE)
• Feedback-driven candidate narrowing
• Strategic letter position confirmation
• Still struggles with repetitive guesses (common RL challenge)

Improvement Ideas

• More training epochs
• Stronger repetition penalties
• Larger models (Qwen3-8B+)
• Swap Wordle for any other environment (coding, math, your Module 4 build)

"OpenEnv makes the environment a plug-in. The training pipeline stays the same."

---

Scaling OpenEnv (Bonus)

WebSocket vs HTTP

OpenEnv uses WebSocket (/ws) for persistent sessions:

• step() = lightweight frame (~0.1ms overhead) over existing connection
• HTTP would require TCP handshake (~10-50ms) per call
• One container handles many isolated sessions (each WS connection = own environment instance)

Single Container Scaling

Variable	Default	Description
WORKERS	4	Uvicorn worker processes
MAX_CONCURRENT_ENVS	100	Max WebSocket sessions per worker

With 8 workers → ~2,048 concurrent sessions for simple text environments.

Multi-Container Scaling (Envoy Load Balancer)

Setup	Containers	Sessions/container	Total
Single	1	100	100
4× containers	4	100	400
8× containers	8	100	800

Benchmarks

Infrastructure	Max Concurrent (WS)	Cores	Sessions/Core
HF Spaces (free)	128	2	64
Local Uvicorn	2,048	8	256
Local Docker	2,048	8	256
SLURM multi-node	16,384	96	171

Recommendations

• Dev / moderate (<2K): Single Uvicorn or Docker. Best efficiency (256 sessions/core).
• Demos / published: HF Spaces free tier, reliable up to 128 concurrent.
• Large-scale training (>2K): Multi-node with Envoy. See tutorial/03-scaling.md.

---

Dependencies (requirements.txt)

Package	Purpose
openenv-core>=0.2.2	Core framework
fastapi	Server-side API
uvicorn	ASGI server
fastmcp	MCP tool-calling
pydantic	Type-safe models
trl	GRPO trainer (Module 5)
transformers	Model loading
datasets	Training data
accelerate	Multi-GPU training
trackio	Experiment tracking
huggingface-hub	Model/Space management
vllm (optional)	Fast inference (CUDA/Linux)
bitsandbytes (optional)	Quantization

pip install -r requirements.txt

GPU (A100 40GB) only required for Module 5 (GRPO training).

---

Key Links

• OpenEnv GitHub
• Environment Hub Collection (HF Spaces)
• TRL Documentation
• Scaling Experiments