README.md

metadata

title: Code Output Assessment Environment
emoji: 🧪
colorFrom: purple
colorTo: pink
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv
  - code-assessment
  - rl-environment
  - code-grading

Code Output Assessment Environment

An OpenEnv RL environment that tests an agent's ability to solve coding problems across three difficulty levels with automated grading and shaped rewards.

Overview

This environment challenges AI agents to:

• Solve coding problems at varying difficulty levels (Easy, Medium, Hard)
• Produce correct outputs for given test cases
• Maximize rewards through accuracy and maintaining solving streaks

Difficulty Levels

🟢 Easy (1x multiplier)

• Basic arithmetic operations (addition, max)
• Simple string manipulation (reversal, vowel counting)
• Example: Add two numbers: 3,5 → 8

🟡 Medium (2x multiplier)

• String/list processing (palindrome check, duplicate removal)
• Aggregation operations (sum of lists, character counting)
• Example: Check palindrome: racecar → true

🔴 Hard (5x multiplier)

• Advanced algorithms (Fibonacci, prime numbers)
• Complex logic (balanced parentheses, longest word)
• Example: Find primes up to n: 10 → 2,3,5,7

Grading & Reward System

Normalized Grading (0.0-1.0)

All graders produce normalized scores regardless of difficulty:

Score Range	Meaning	Feedback
1.0	Perfect answer	"✓ Correct!"
0.8-0.9	Very close	"⚡ Very close! 80-90% correct"
0.5-0.7	Moderate partial credit	"⚡ Partial credit: 50-70% correct"
0.2-0.4	Low partial credit	"⚡ Some correct elements"
0.1	Format credit only	"⚡ Correct format, wrong values"
0.0	Completely wrong	"✗ Incorrect"

Reward Calculation

Formula: reward = grader_score × difficulty_multiplier + bonuses

Difficulty	Multiplier	Perfect (1.0)	High Partial (0.7)	Low Partial (0.3)	Wrong (0.0)
Easy	1x	+1.0	+0.35	+0.15	0.0
Medium	2x	+2.0	+1.4	+0.6	0.0
Hard	5x	+5.0	+3.5	+1.5	-0.3

Bonuses:

• Streak Bonus: +0.5 for maintaining 3+ consecutive correct answers
• Penalty: -0.3 on hard problems for completely wrong answers (discourages random guessing)

Maximum Episode Reward: ~28.0 (perfect accuracy with streaks)

Quick Start

The simplest way to use the Code Assessment environment is through the CodeAssessmentEnv class:

from code_assessment_env import CodeAssessmentAction, CodeAssessmentEnv

# Create environment from Docker image
env = CodeAssessmentEnv.from_docker_image("code_assessment_env:latest").sync()

# Reset to get first problem
result = env.reset()
print(f"Problem: {result.observation.problem_description}")
print(f"Difficulty: {result.observation.difficulty}")
print(f"Test Input: {result.observation.test_case_input}")

# Submit an answer
result = env.step(CodeAssessmentAction(answer="8"))
print(f"Correct: {result.observation.is_correct}")
print(f"Reward: {result.reward}")
print(f"Feedback: {result.observation.feedback}")

# Continue solving problems
for _ in range(10):
    obs = result.observation
    # Your agent logic here to solve obs.problem_description with obs.test_case_input
    answer = solve_problem(obs.problem_description, obs.test_case_input)
    result = env.step(CodeAssessmentAction(answer=answer))
    
    if result.done:
        break

env.close()

Key Features

✅ Normalized Grading System

Each answer is graded on a 0.0-1.0 scale:

• Exact match detection: Full credit (1.0)
• Partial credit: 0.1-0.9 based on correctness percentage
• Format validation: Credit for proper structure even if values are wrong
• String similarity: Grading for text-based answers using overlap metrics

✅ Difficulty-Scaled Rewards

• Normalized grader scores (0.0-1.0) are multiplied by difficulty
• Easy: 1x, Medium: 2x, Hard: 5x multipliers
• Higher difficulty = higher potential rewards for correct answers
• Partial credit proportionally scaled by difficulty

✅ Progressive Difficulty

• Starts with Easy problems
• Advances to Medium after solving 4 problems
• Advances to Hard after solving 8 problems

✅ Shaped Rewards

• Base rewards scale with difficulty
• Partial credit for near-correct answers
• Streak bonuses for consecutive successes
• Penalties for repeated failures on hard problems

✅ Rich Feedback

Observations include:

• problem_description: What to solve
• difficulty: Current difficulty level
• test_case_input: Input to process
• feedback: Grading feedback ("✓ Correct!", "✗ Incorrect", etc.)
• is_correct: Boolean correctness flag
• partial_credit: Score between 0.0-1.0
• problems_solved: Total solved count
• current_streak: Consecutive correct answers

Running with LLM Agent

Use the included inference script to test with an LLM:

# Set environment variables
export IMAGE_NAME=code_assessment_env:latest
export HF_TOKEN=your_huggingface_token

# Run inference
uv run python inferency.py

Expected output:

[START] task=code_output_assessment env=code_assessment_env model=Qwen/Qwen2.5-72B-Instruct
[STEP] step=1 action=answer='8' | correct=True | difficulty=easy reward=1.00 done=false error=null
[STEP] step=2 action=answer='olleh' | correct=True | difficulty=easy reward=1.00 done=false error=null
...
[END] success=true steps=15 score=0.720 rewards=1.00,1.00,2.00,2.00,5.00,...

Development

Building the Docker Image

cd code_assessment_env
docker build -t code_assessment_env:latest .

Running Locally

# Start the server
docker run -p 8000:8000 code_assessment_env:latest

# Test with API
curl http://localhost:8000/docs  # Swagger UI

API Endpoints

• POST /reset - Start new episode
• POST /step - Submit answer
• GET /state - Get episode state
• GET /schema - Get action/observation schemas
• GET /health - Health check
• GET /docs - Interactive API documentation

Problem Examples

Easy Problems

# Addition
Input: "3,5" → Output: "8"

# String Reversal  
Input: "hello" → Output: "olleh"

# Vowel Counting
Input: "hello" → Output: "2"

Medium Problems

# Palindrome Check
Input: "racecar" → Output: "true"

# Sum List
Input: "1,2,3" → Output: "6"

# Remove Duplicates
Input: "1,2,2,3" → Output: "1,2,3"

Hard Problems

# Fibonacci
Input: "10" → Output: "55"

# Balanced Parentheses
Input: "({[]})" → Output: "true"

# Prime Numbers
Input: "20" → Output: "2,3,5,7,11,13,17,19"

Training Tips

1. Start Simple: Master easy problems before advancing
2. Pay Attention to Format: Exact formatting matters (lowercase true/false, comma-separated lists)
3. Build Streaks: Maintain accuracy for streak bonuses
4. Learn from Feedback: Use partial credit signals to improve
5. Optimize for Speed: Solve quickly to maximize problems per episode

License

BSD-style license - see LICENSE file for details.

• Connecting to the environment
• Container cleanup when you call close()

Building the Docker Image

Before using the environment, you need to build the Docker image:

# From project root
docker build -t first_rl_proj-env:latest -f server/Dockerfile .

Deploying to Hugging Face Spaces

You can easily deploy your OpenEnv environment to Hugging Face Spaces using the openenv push command:

# From the environment directory (where openenv.yaml is located)
openenv push

# Or specify options
openenv push --namespace my-org --private

The openenv push command will:

1. Validate that the directory is an OpenEnv environment (checks for openenv.yaml)
2. Prepare a custom build for Hugging Face Docker space (enables web interface)
3. Upload to Hugging Face (ensuring you're logged in)

Prerequisites

• Authenticate with Hugging Face: The command will prompt for login if not already authenticated

Options

• --directory, -d: Directory containing the OpenEnv environment (defaults to current directory)
• --repo-id, -r: Repository ID in format 'username/repo-name' (defaults to 'username/env-name' from openenv.yaml)
• --base-image, -b: Base Docker image to use (overrides Dockerfile FROM)
• --private: Deploy the space as private (default: public)

Examples

# Push to your personal namespace (defaults to username/env-name from openenv.yaml)
openenv push

# Push to a specific repository
openenv push --repo-id my-org/my-env

# Push with a custom base image
openenv push --base-image ghcr.io/meta-pytorch/openenv-base:latest

# Push as a private space
openenv push --private

# Combine options
openenv push --repo-id my-org/my-env --base-image custom-base:latest --private

After deployment, your space will be available at: https://huggingface.co/spaces/<repo-id>

The deployed space includes:

• Web Interface at /web - Interactive UI for exploring the environment
• API Documentation at /docs - Full OpenAPI/Swagger interface
• Health Check at /health - Container health monitoring
• WebSocket at /ws - Persistent session endpoint for low-latency interactions

Environment Details

Action

FirstRlProjAction: Contains a single field

• message (str) - The message to echo back

Observation

FirstRlProjObservation: Contains the echo response and metadata

• echoed_message (str) - The message echoed back
• message_length (int) - Length of the message
• reward (float) - Reward based on message length (length × 0.1)
• done (bool) - Always False for echo environment
• metadata (dict) - Additional info like step count

Reward

The reward is calculated as: message_length × 0.1

• "Hi" → reward: 0.2
• "Hello, World!" → reward: 1.3
• Empty message → reward: 0.0

Advanced Usage

Connecting to an Existing Server

If you already have a First Rl Proj environment server running, you can connect directly:

from first_rl_proj import FirstRlProjEnv

# Connect to existing server
first_rl_projenv = FirstRlProjEnv(base_url="<ENV_HTTP_URL_HERE>")

# Use as normal
result = first_rl_projenv.reset()
result = first_rl_projenv.step(FirstRlProjAction(message="Hello!"))

Note: When connecting to an existing server, first_rl_projenv.close() will NOT stop the server.

Using the Context Manager

The client supports context manager usage for automatic connection management:

from first_rl_proj import FirstRlProjAction, FirstRlProjEnv

# Connect with context manager (auto-connects and closes)
with FirstRlProjEnv(base_url="http://localhost:8000") as env:
    result = env.reset()
    print(f"Reset: {result.observation.echoed_message}")
    # Multiple steps with low latency
    for msg in ["Hello", "World", "!"]:
        result = env.step(FirstRlProjAction(message=msg))
        print(f"Echoed: {result.observation.echoed_message}")

The client uses WebSocket connections for:

• Lower latency: No HTTP connection overhead per request
• Persistent session: Server maintains your environment state
• Efficient for episodes: Better for many sequential steps

Concurrent WebSocket Sessions

The server supports multiple concurrent WebSocket connections. To enable this, modify server/app.py to use factory mode:

# In server/app.py - use factory mode for concurrent sessions
app = create_app(
    FirstRlProjEnvironment,  # Pass class, not instance
    FirstRlProjAction,
    FirstRlProjObservation,
    max_concurrent_envs=4,  # Allow 4 concurrent sessions
)

Then multiple clients can connect simultaneously:

from first_rl_proj import FirstRlProjAction, FirstRlProjEnv
from concurrent.futures import ThreadPoolExecutor

def run_episode(client_id: int):
    with FirstRlProjEnv(base_url="http://localhost:8000") as env:
        result = env.reset()
        for i in range(10):
            result = env.step(FirstRlProjAction(message=f"Client {client_id}, step {i}"))
        return client_id, result.observation.message_length

# Run 4 episodes concurrently
with ThreadPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(run_episode, range(4)))

Development & Testing

Direct Environment Testing

Test the environment logic directly without starting the HTTP server:

# From the server directory
python3 server/first_rl_proj_environment.py

This verifies that:

• Environment resets correctly
• Step executes actions properly
• State tracking works
• Rewards are calculated correctly

Running Locally

Run the server locally for development:

uvicorn server.app:app --reload

Project Structure

first_rl_proj/
├── .dockerignore         # Docker build exclusions
├── __init__.py            # Module exports
├── README.md              # This file
├── openenv.yaml           # OpenEnv manifest
├── pyproject.toml         # Project metadata and dependencies
├── uv.lock                # Locked dependencies (generated)
├── client.py              # FirstRlProjEnv client
├── models.py              # Action and Observation models
└── server/
    ├── __init__.py        # Server module exports
    ├── first_rl_proj_environment.py  # Core environment logic
    ├── app.py             # FastAPI application (HTTP + WebSocket endpoints)
    └── Dockerfile         # Container image definition