required.md

# Round 1 Requirements And Project Compliance Plan

## Official Problem Statement

Round 1 requires building a complete, real-world OpenEnv environment that an AI agent can learn from through the standard `step()` / `reset()` / `state()` API.

### Key requirements at a glance

- must simulate a real-world task, not a game or toy
- must implement the full OpenEnv spec with typed models and `openenv.yaml`
- must include at least 3 tasks with agent graders spanning easy -> medium -> hard
- graders must return scores in `[0.0, 1.0]`
- reward must provide meaningful partial-progress signal
- must include a reproducible baseline `inference.py`
- must deploy to Hugging Face Spaces with a working Dockerfile
- README must include environment description, action / observation spaces, setup, usage, and baseline scores

## Official Functional Requirements

### Real-world task simulation

The environment must simulate a task humans actually do. The official examples include:

- email triage
- code review
- data cleaning
- scheduling
- customer support
- content moderation

### OpenEnv spec compliance

The environment must implement the OpenEnv interface with:

- typed Observation model
- typed Action model
- typed state model
- `step(action)`
- `reset()`
- `state()`
- `openenv.yaml`

This is expected to be checked through `openenv validate`.

### Minimum 3 tasks with agent graders

Each task must have:

- a concrete objective
- a programmatic grader
- score output in `[0.0, 1.0]`
- deterministic success / failure criteria
- clear difficulty progression from easy to hard

### Meaningful reward function

The reward should:

- provide signal across the full trajectory
- reward partial progress
- penalize clearly undesirable behavior

### Baseline inference script

The baseline must:

- use the OpenAI client for LLM calls
- live at the project root as `inference.py`
- produce reproducible scores
- complete successfully across all 3 tasks

## Official Non-Functional Requirements

### Hugging Face Spaces

- must deploy as a containerized HF Space
- should be tagged with `openenv`
- should respond successfully when pinged

### Containerized execution

- must include a working Dockerfile
- should start cleanly with `docker build` + `docker run`

### Documentation

README must include:

- environment description and motivation
- action space definition
- observation space definition
- task descriptions with difficulty expectations
- setup and usage instructions
- baseline scores

## Official Evaluation Criteria

### Weights

| Parameter | Weight | What judges look for |
|-----------|--------|----------------------|
| Real-world utility | 30% | Genuine practical task and value |
| Task & grader quality | 25% | Clear objectives, fair graders, real progression |
| Environment design | 20% | Clean state, sensible API, good reward shaping |
| Code quality & spec compliance | 15% | OpenEnv compliance, structure, typing, tests, Docker |
| Creativity & novelty | 10% | Original domain, mechanics, reward ideas |

### Phase 1: Automated validation

Pass / fail gate:

- HF Space deploys
- OpenEnv spec compliance
- Dockerfile builds
- baseline reproduces
- 3+ tasks with graders

### Phase 2: Agentic evaluation

Scored:

- baseline agent rerun
- standard Open LLM agent run against the environment
- score variance check

### Phase 3: Human review

Top submissions are reviewed by Meta and Hugging Face engineers for:

- real-world utility
- creativity
- exploit resistance

## Official Disqualification Criteria

- environment does not deploy or respond
- plagiarized or trivially modified existing environment
- graders always return the same score
- no baseline inference script

## Official Pre-Submission Checklist

All of these must pass:

- HF Space deploys and responds
- automated ping to the Space URL returns `200`
- reset path works on the deployed environment
- `openenv validate` passes
- Dockerfile builds
- baseline inference completes and produces scores
- 3+ tasks with graders are present and score in `[0.0, 1.0]`

## Mandatory Additional Instructions

### Required inference environment variables

  - `API_BASE_URL`
  - `MODEL_NAME`
  - `API_KEY`
  - `HF_TOKEN`

Use `API_KEY` as the primary evaluator-injected credential for the OpenAI client. `HF_TOKEN` can remain as a backward-compatible local fallback, but submission-time LLM traffic should flow through the injected proxy key.

### Inference script constraints

- script must be named `inference.py`
- it must live in the project root
- all LLM calls must use the OpenAI client
- stdout logs must strictly follow the `[START]`, `[STEP]`, and `[END]` format from the official sample

### Infra restrictions

- inference runtime should stay under 20 minutes
- env and inference should run on a machine with `vcpu=2` and `memory=8gb`

### Validator

- run the official pre-submission validation script before final submission if possible

---

## Project Compliance Plan

## Project Goal

Build a polished OpenEnv environment for IT helpdesk ticket routing that satisfies:

- real-world utility
- strong task and grader quality
- clean environment design
- OpenEnv spec compliance
- reproducible baseline inference
- Docker and Hugging Face deployment readiness

## Current Product Definition

The environment simulates a helpdesk queue. An agent receives one ticket at a time and predicts:

- `issue_type`
- `priority`
- `assignment_group`
- `resolution_action`

The project keeps three tasks:

1. Issue Type Classification
2. Issue Type And Priority
3. Full Ticket Routing

## What Must Be True At Submission

### Pass / fail requirements

- the environment responds correctly
- OpenEnv metadata is valid
- `reset()`, `step()`, and `state()` work
- there are at least 3 tasks
- graders return scores in `[0.0, 1.0]`
- `inference.py` runs and prints reproducible results
- `inference.py` uses the OpenAI client and required env vars
- structured stdout logging matches the official format
- `openenv validate` passes
- Docker builds and starts cleanly
- HF Space responds and reset works

### Scored requirements

- the task clearly feels like real helpdesk work
- the hard task requires meaningful reasoning
- partial credit is useful and deterministic
- docs are clear enough for judges to understand quickly
- reward is informative over the trajectory, not only at the end

## Core Files

### Runtime

- `models.py`
- `server/environment.py`
- `server/grader.py`
- `server/reward.py`
- `server/tasks.py`
- `server/app.py`
- `client.py`
- `inference.py`

### Data and metadata

- `data/dataset.json`
- `openenv.yaml`
- `server/Dockerfile`
- `pyproject.toml`
- `requirements.txt`

### Docs

- `README.md`
- `KNOWLEDGE.md`
- `required.md`

## Technical Priorities

### P0

1. keep environment behavior correct
2. verify task definitions and graders
3. make the baseline script reliable and compliant with official logging format
4. confirm dataset coverage and label consistency
5. validate the official submission gates, not just local behavior

### P1

1. validate Docker
2. validate deployment assumptions
3. record baseline scores
4. polish docs
5. verify the runtime envelope and structured inference logs

### P2

1. strengthen ticket wording for realism
2. expand hard-case examples if needed
3. remove low-signal artifacts from the repo

## Quality Checks To Perform

### Environment

- reset starts a clean episode
- each step advances the queue correctly
- the final step returns trajectory reward
- state reflects the real internal status
- episode boundaries are sensible

### Grader

- exact matches score `1.0`
- near misses get partial credit where intended
- unsupported task IDs fail clearly
- scores vary across examples
- graders do not collapse to constant scores

### Inference

- heuristic mode works without model credentials
- LLM mode reads `API_BASE_URL`, `MODEL_NAME`, and `API_KEY` (`HF_TOKEN` remains a local fallback)
- uses the OpenAI client
- stdout follows `[START]`, `[STEP]`, and `[END]`
- output is reproducible when the seed is fixed
- runtime stays below the official time budget

### Deployment and validation

- `openenv validate` passes
- Docker build succeeds
- Docker run succeeds
- HF ping / reset behavior works
- official validator script is run if practical

### Docs

- no outdated domain references remain
- team and project metadata are correct
- setup and run instructions are accurate
- README reflects the current inference and deployment path

## Risks

### Runtime risk

The first local execution pass, merged-state rerun, clean-copy rerun, and local validator pass have already succeeded. The remaining runtime risk is submission-day deployment execution, not first-pass local behavior.

### Benchmark risk

The current local benchmark is already recorded. Remaining benchmark risk is whether deployment / validation changes expose a mismatch late.

### Deployment risk

Docker smoke coverage, `openenv validate`, and structured inference logging are now verified in the repo state. The remaining deployment risk is the live Hugging Face Space ping and reset check after the final push if a fresh deployment is created.

## Definition Of Done

The project is ready when:

1. the environment runs locally end to end
2. unit, smoke, and integration tests cover the critical paths
3. the heuristic baseline runs successfully
4. the inference script is compliant with the official logging format
5. `openenv validate` passes
6. Docker build and run both succeed
7. HF deployment checks succeed or are as close to verified as possible before submission
8. the docs are clean, current, and submission-ready
9. the repo clearly presents Hackstreet Boys as the team

## Current Compliance Snapshot

As of April 8, 2026, the core submission requirements and the major benchmark upgrades are in place:

- real-world task definition is clear and stable
- typed models, `reset()`, `step()`, `state()`, and `openenv.yaml` are present in the repo
- 3-task easy -> medium -> hard ladder is present
- graders are deterministic and bounded to `[0.0, 1.0]`
- unit tests now prove scorer crispness, task invariants, and dataset coverage
- smoke tests now prove environment behavior, seeded determinism, score bounds, and full-episode completion
- integration tests now cover `/health`, `/tasks`, `/reset`, `/step`, `/state`, full seeded episodes, and heuristic regression
- baseline heuristic results are recorded in the docs
- the README now includes Hugging Face Spaces frontmatter and a judge-facing grounded-scoring explanation
- the label space and partial-credit policy were reviewed against public IT-support references during development
- `.openenvignore` is present
- Docker smoke coverage exists through the checked-in GitHub Actions workflow and recorded April 6 run
- `inference.py` structured `[START]`, `[STEP]`, and `[END]` logging is verified
- `uv.lock` is checked in and `openenv validate` now passes on the current repo state
- a clean-copy install-and-run pass has been completed

The remaining work is optional benchmark expansion rather than submission readiness work:

- make the simulator even more emergent instead of partially authored
- broaden the data distribution further
- replace the local policy search loop with a more training-oriented learning setup if needed later

The short TRL / GRPO README example remains optional and is still deferred because it is not required for this project to be understandable, runnable, or judgeable.