OPENENV_SUBMISSION_CHECKLIST.md

# OpenEnv Submission Checklist
> Complete every item before final submission. A single ❌ in any **DISQUALIFYING** section means you cannot submit.

---

## HOW TO USE THIS CHECKLIST

1. Work through each section **in order** — earlier sections unblock later ones.
2. Mark each item `[x]` when confirmed, or add a note if it needs fixing.
3. Any item marked **🚨 DISQUALIFYING** must be `[x]` before submission or you will be automatically rejected.
4. After all items are checked, run the final validator command at the bottom.

---

## SECTION 1 — REAL-WORLD TASK SIMULATION

> Weight: 30% of total score. Judges will ask: "Would a practitioner actually use this?"

### 1.1 Domain Validity

- [x] **The environment simulates a task that real humans do professionally or daily.** Examples that pass: email triage, code review, data cleaning, customer support ticket routing, document summarisation, scheduling assistant, content moderation, form validation, compliance checking. Examples that fail: CartPole, GridWorld, Snake, made-up puzzles.
- [x] The task domain is stated clearly in the README's first paragraph — a reader understands the real-world context within 3 sentences.
- [x] The environment would be useful for evaluating or training AI agents on a real skill, not just for demonstrating API integration.

### 1.2 Domain Depth

- [x] The environment models at least the core mechanic of the real task (e.g. for email triage: an inbox, email metadata, categories, urgency signals — not just "send a string and get a string back").
- [x] Action and observation spaces reflect what a human would actually do and see in this task.
- [x] The hardest task (task 3) would challenge a frontier model (GPT-4o / Claude 3.5 Sonnet level) — it is not trivially solved by pattern matching.

---

## SECTION 2 — OPENENV SPEC COMPLIANCE

> Weight: part of the 15% code quality score. **All 🚨 items are disqualifying.**

### 2.1 Typed Models

- [x] `Observation` is a Pydantic `BaseModel` with typed fields. No `dict`, no `Any` unless explicitly documented.
- [x] `Action` is a Pydantic `BaseModel` with typed fields.
- [x] `Reward` is a `float` or a Pydantic model containing a `float` value field.
- [x] All three models are importable from a single module (e.g. `from my_env import Observation, Action`).
- [x] Every field has a type annotation. No bare `Optional` without a type parameter.

### 2.2 Core API Methods

- [x] 🚨 `reset()` is implemented and returns an `Observation` (or an object containing one).
- [x] 🚨 `step(action: Action)` is implemented and returns `(observation, reward, done, info)` or a structured equivalent.
- [x] 🚨 `state()` is implemented and returns the current full environment state (serialisable dict or Pydantic model).
- [x] `reset()` produces a **clean, reproducible initial state** — calling it twice with the same seed gives the same starting observation.
- [x] `step()` after `done=True` either raises a clean error or resets automatically (document which).
- [x] `info` dict (or equivalent) is non-empty and useful — at minimum contains the current task name and step count.

### 2.3 `openenv.yaml`

- [x] 🚨 `openenv.yaml` exists in the project root.
- [x] Contains `name:` field (string, slug-safe).
- [x] Contains `version:` field (semver, e.g. `0.1.0`).
- [x] Contains `description:` field (1–2 sentences).
- [x] Contains `tasks:` list with at least 3 entries, each having `name:`, `difficulty:`, and `description:`.
- [x] Contains `observation_space:` description block.
- [x] Contains `action_space:` description block.
- [x] Passes `openenv validate` without errors (run this command and paste output into your notes).

```bash
# Run this and confirm zero errors:
openenv validate openenv.yaml
```

---

## SECTION 3 — MINIMUM 3 TASKS WITH AGENT GRADERS

> Weight: 25% of total score. All 🚨 items are disqualifying.

### 3.1 Task Definitions

- [x] 🚨 Exactly 3 or more tasks are defined.
- [x] Task 1 is labelled **easy** and a baseline LLM can score ≥ 0.6 on it with no fine-tuning.
- [x] Task 2 is labelled **medium** and presents a genuine multi-step challenge.
- [x] Task 3 is labelled **hard** and a strong frontier model scores < 0.8 on it without domain-specific prompting.
- [x] Each task has a concise, unambiguous objective statement that a human tester can understand without reading the code.

### 3.2 Grader Requirements

- [x] 🚨 Each task has a **programmatic grader** — no human-in-the-loop, no LLM-as-judge for the primary score.
- [x] 🚨 Every grader returns a float in **[0.0, 1.0]** — no values below 0 or above 1 ever.
- [x] Graders are **deterministic**: given the same sequence of actions, they always return the same score.
- [x] Graders are **reproducible**: scores do not depend on system time, random seeds not exposed to the grader, or external API calls.
- [x] Partial credit is awarded — the grader does not return only 0.0 or 1.0 (binary graders are disqualifying for medium/hard tasks).
- [x] The grader logic is readable: another developer can understand the scoring rubric in < 5 minutes by reading the grader function.

### 3.3 Difficulty Verification (run before submitting)

```bash
# Run baseline inference on all three tasks and record scores:
TASK=easy   python inference.py   # expected: score >= 0.6
TASK=medium python inference.py   # expected: score in 0.3–0.7
TASK=hard   python inference.py   # expected: score < 0.8
```

- [x] Easy task baseline score is ≥ 0.6.
- [x] Medium task baseline score is meaningfully lower than easy (at least 0.15 gap).
- [x] Hard task baseline score is < 0.8 (if it's ≥ 0.8, make it harder).
     (Easy: 0.883 | Medium: 0.500 | Hard: 0.512)

---

## SECTION 4 — MEANINGFUL REWARD FUNCTION

> Weight: part of the 20% environment design score.

### 4.1 Dense Reward Signal

- [x] The reward function provides **intermediate signal** — the agent gets feedback before the episode ends, not only at `done=True`.
- [x] At least 3 distinct reward levels exist across the task trajectory (not just 0.0 at each step then 1.0 at the end).
- [x] Progress toward task completion is reflected in the reward — an agent making progress always earns more than one doing nothing.

### 4.2 Reward Shaping

- [x] **Clearly undesirable behaviour is penalised**: e.g. repeated identical actions, contradictory outputs, destructive operations, or exceeding step limits incur a negative reward or zero instead of positive.
- [x] The reward function cannot be gamed by a trivial exploit (e.g. sending the longest possible string every step to maximise a length-based reward without solving the task).
- [x] Total episode reward is bounded — the maximum possible score per episode is documented in the README.
- [x] Reward is normalised to [0.0, 1.0] at the episode level (sum of step rewards / max possible reward, clamped).

### 4.3 Reward Documentation

- [x] The reward formula is documented in the README with an example calculation.
- [x] Edge cases are documented: what happens at step 0, at `done=True`, and at the max step limit.

---

## SECTION 5 — BASELINE INFERENCE SCRIPT

> Weight: part of the 15% code quality score. All 🚨 items are disqualifying.

### 5.1 File and Location

- [x] 🚨 The script is named **exactly** `inference.py` (lowercase, no suffix variation).
- [x] 🚨 `inference.py` is in the **root directory** of the project (not in a subdirectory).
- [x] The script runs end-to-end without interactive input (no `input()` calls, no manual setup required).

### 5.2 Environment Variables

- [x] 🚨 `API_BASE_URL` is read from `os.getenv("API_BASE_URL", "<your-default>")`. A default is set so the script doesn't crash when the variable is absent.
- [x] 🚨 `MODEL_NAME` is read from `os.getenv("MODEL_NAME", "<your-default>")`.
- [x] 🚨 `HF_TOKEN` is read from `os.getenv("HF_TOKEN")` (no default — it must be set externally; the script should fail with a clear message if absent).
- [x] `IMAGE_NAME` / `LOCAL_IMAGE_NAME` is read from `os.getenv("IMAGE_NAME")` or `os.getenv("LOCAL_IMAGE_NAME")` if Docker-based.
- [x] No credentials, tokens, or API keys are hardcoded in any source file.

### 5.3 OpenAI Client Usage

- [x] 🚨 **All LLM calls use the `OpenAI` client** from `openai` package — no `requests`, no `httpx`, no `anthropic` SDK, no `transformers` pipeline.
- [x] Client is initialised as: `client = OpenAI(base_url=API_BASE_URL, api_key=API_KEY)` where `API_KEY = os.getenv("HF_TOKEN") or os.getenv("API_KEY")`.
- [x] `client.chat.completions.create(...)` is used for all inference calls.
- [x] `stream=False` is set explicitly (streaming is not expected by the evaluator).

### 5.4 Stdout Log Format — **EXACT FORMAT REQUIRED**

> Any deviation in field names, ordering, or capitalisation will break automated scoring.

- [x] 🚨 Exactly **one `[START]` line** is emitted at the beginning of each episode, before any steps.

  ```
  [START] task=<task_name> env=<benchmark> model=<model_name>
  ```

- [x] 🚨 Exactly **one `[STEP]` line** is emitted after each `env.step()` call, immediately after it returns.

  ```
  [STEP] step=<n> action=<action_str> reward=<0.00> done=<true|false> error=<msg|null>
  ```

- [x] 🚨 Exactly **one `[END]` line** is emitted after `env.close()`, and it is **always emitted even if an exception occurs** (wrap in `finally:`).

  ```
  [END] success=<true|false> steps=<n> score=<0.000> rewards=<r1,r2,...,rn>
  ```

- [x] `reward` and all values in `rewards` are formatted to **exactly 2 decimal places** (e.g. `1.00`, `0.75`, `0.00`).
- [x] `score` is formatted to **exactly 3 decimal places** (e.g. `0.750`).
- [x] `done` and `success` are lowercase strings: `true` or `false` (not `True`/`False`, not `1`/`0`).
- [x] `error` is either the raw error string or the literal string `null` (not `None`, not empty string).
- [x] **No newlines within a single log line** — each log entry is exactly one line.
- [x] Fields are in the exact order shown above — no reordering.
- [x] No extra spaces, tabs, or punctuation between fields (single space separator between `key=value` pairs).

### 5.5 Reproducibility

- [x] Running the script twice with the same `MODEL_NAME` and environment seed produces scores within ±0.05 of each other (minor LLM variance is acceptable; wild swings are not).
- [x] The script covers all 3 tasks — either by looping over task names or via `TASK` environment variable as shown in the sample.
- [x] `MAX_STEPS` is set to a value that allows the task to be completed (not too low) but finishes within the time limit.

### 5.6 Runtime Constraint

- [x] 🚨 The full inference script (all 3 tasks) completes in **under 20 minutes** on a machine with 2 vCPUs and 8 GB RAM.
- [x] Each individual task episode completes in under 5 minutes.
- [x] No step blocks indefinitely — all `env.step()` calls have an implicit or explicit timeout.

---

## SECTION 6 — DOCKER AND CONTAINERISATION

> Weight: part of the 15% code quality score. All 🚨 items are disqualifying.

### 6.1 Dockerfile

- [x] 🚨 A `Dockerfile` exists in the project root.
- [x] 🚨 `docker build -t myenv .` completes without errors on a clean machine.
- [x] 🚨 `docker run --rm myenv` starts the environment server and it responds to `reset()`.
- [x] The base image is appropriate for the task (e.g. `python:3.11-slim`, not an oversized or obscure base).
- [x] All Python dependencies are installed via `pip install -r requirements.txt` or equivalent inside the Dockerfile.
- [x] The Dockerfile does **not** require internet access at runtime (all deps installed at build time).
- [x] No secrets or API keys are baked into the Docker image.
- [x] The container starts the environment server on a documented port (default: 8000 or 7860).
- [x] The container exposes that port with `EXPOSE <port>` in the Dockerfile.

### 6.2 Resource Constraints

- [x] The built image size is < 5 GB (ideally < 2 GB).
- [x] The running container uses < 6 GB RAM at peak (leaving headroom for the 8 GB machine limit).
- [x] The container starts up in < 60 seconds.

### 6.3 `requirements.txt` (or equivalent)

- [x] `requirements.txt` exists in the project root.
- [x] All dependencies have pinned versions (e.g. `openai==1.30.0`, not `openai`).
- [x] `openai` package is listed (required for inference script).
- [x] `pydantic` package is listed.
- [x] `pyyaml` package is listed (for openenv.yaml parsing).

---

## SECTION 7 — HUGGING FACE SPACES DEPLOYMENT

> Weight: part of the 15% code quality score. All 🚨 items are disqualifying.

### 7.1 Space Setup

- [x] 🚨 The HF Space is **publicly accessible** — not private or gated.
- [x] 🚨 The Space is tagged with `openenv` in the repository tags.
- [x] The Space type is `Docker` (not `Gradio` or `Streamlit`, unless the env server is built on one of those).
- [x] The Space metadata in `README.md` YAML header includes `tags: [openenv]`.

### 7.2 Availability Check

- [x] 🚨 A `GET` request to `https://your-space-url/` returns HTTP 200.
- [x] 🚨 A `POST` to `https://your-space-url/reset` returns a valid JSON observation.
- [x] `POST /step` with a valid action body returns `(observation, reward, done, info)`.
- [x] `GET /state` returns the current environment state.
- [x] The Space has been running for at least 10 minutes without crashing before submission.

### 7.3 Space Configuration

- [x] `README.md` in the repo root has valid HF Space YAML header:

  ```yaml
  ---
  title: Your Environment Name
  emoji: 🤖
  colorFrom: blue
  colorTo: purple
  sdk: docker
  pinned: false
  tags:
    - openenv
  ---
  ```

- [x] The Space hardware tier is sufficient to run the environment (CPU Basic is fine for most cases).
- [x] Environment variables required at runtime are set as **Space Secrets** in the HF Space settings (not hardcoded).

---

## SECTION 8 — README DOCUMENTATION

> A well-written README is part of the 15% code quality score.

### 8.1 Required Sections

- [x] **Environment Description** — what real-world task is simulated, why it matters, what an agent needs to learn to succeed.
- [x] **Observation Space** — table or structured description of every field in the `Observation` model, including type, range, and meaning.
- [x] **Action Space** — table or structured description of every field in the `Action` model, including valid values and constraints.
- [x] **Task Descriptions** — for each task: name, difficulty label (easy/medium/hard), objective, grader description, example episode.
- [x] **Reward Function** — formula, components, max possible reward per episode, normalisation method.
- [x] **Setup Instructions** — exact commands to clone, build, and run locally:

  ```bash
  git clone https://huggingface.co/spaces/YOUR_USER/YOUR_ENV
  cd YOUR_ENV
  docker build -t myenv .
  docker run -p 8000:8000 myenv
  ```

- [x] **Inference Script Usage** — exact commands with environment variables:

  ```bash
  export HF_TOKEN=hf_...
  export API_BASE_URL=https://router.huggingface.co/v1
  export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
  python inference.py
  ```

- [x] **Baseline Scores** — a table with columns: Task | Model | Score | Steps | Notes.

### 8.2 Baseline Scores Table (paste your actual results)

| Task | Difficulty | Model | Score | Steps | Notes |
|------|-----------|-------|-------|-------|-------|
| python-off-by-one | easy | Llama-3.3-70B-Instruct | 0.883 | 2 | |
| js-idor-auth | medium | Llama-3.3-70B-Instruct | 0.500 | 2 | |
| python-pickle-deserialization | hard | Llama-3.3-70B-Instruct | 0.512 | 2 | |

- [x] The table is filled in with real numbers from a completed inference run.
- [x] The easy task score is ≥ 0.6.

---

## SECTION 9 — CODE QUALITY AND PROJECT STRUCTURE

### 9.1 Project Layout

- [x] Project root contains at minimum:

  ```
  /
  ├── inference.py          ← inference script (mandatory name)
  ├── openenv.yaml          ← OpenEnv spec file
  ├── Dockerfile            ← container definition
  ├── requirements.txt      ← pinned dependencies
  ├── README.md             ← documentation
  └── src/ or myenv/       ← environment source code
      ├── env.py            ← environment class
      ├── models.py         ← Observation, Action, Reward models
      ├── tasks/            ← one file per task + grader
      └── server.py         ← HTTP server (FastAPI or equivalent)
  ```

- [x] No large binary files (datasets > 50 MB, model weights) are committed to the repo. Use URLs or HF datasets instead.
- [x] `.gitignore` excludes `__pycache__`, `.env`, `*.pyc`, and any local credentials.

### 9.2 Code Standards

- [x] All Python files pass `flake8` or `ruff` with no errors (warnings are acceptable).
- [x] All Pydantic models have docstrings or field descriptions.
- [x] No bare `except:` clauses — exceptions are caught specifically.
- [x] No `print()` statements in the environment code (use `logging`). `print()` is only in `inference.py` for structured stdout logs.
- [x] Environment class has a module-level docstring explaining what it does.

### 9.3 Testing

- [x] At minimum, a smoke test exists: instantiate the env, call `reset()`, call `step()` with a valid action, assert `done` is a bool and `reward` is a float.
- [x] The smoke test passes:

  ```bash
  python -m pytest tests/ -v
  # or
  python test_smoke.py
  ```

---

## SECTION 10 — CREATIVITY AND NOVELTY

> Weight: 10% of total score. This section cannot disqualify you, but it can push you to the top.

- [x] The problem domain is novel — not a re-skin of email triage or the echo example from the sample script.
- [x] The reward design has an interesting property: e.g. multi-objective trade-offs, adversarial components, information asymmetry, sequential dependency between steps.
- [x] The hard task has a mechanic that makes it qualitatively harder, not just quantitatively (more steps / more categories is not enough — the agent must reason differently).
- [x] The environment would be cited or referenced by others building agents in this domain.

---

## SECTION 11 — FINAL PRE-SUBMISSION VALIDATION

Run these commands in order. All must succeed with zero errors.

### Step 1 — Validate OpenEnv spec

```bash
openenv validate openenv.yaml
```

Expected output: `✓ openenv.yaml is valid`

- [x] ✓ PASSED

### Step 2 — Build Docker image

```bash
docker build -t myenv-final .
```

Expected: exits with code 0, image appears in `docker images`.

- [x] ✓ PASSED

### Step 3 — Start container and health check

```bash
docker run -d -p 8000:8000 --name myenv-test myenv-final
sleep 10
curl -s http://localhost:8000/ | python3 -m json.tool
curl -s -X POST http://localhost:8000/reset | python3 -m json.tool
docker stop myenv-test && docker rm myenv-test
```

Expected: Both curl commands return valid JSON with no errors.

- [x] ✓ PASSED

### Step 4 — Run full inference script

```bash
export HF_TOKEN=<your_token>
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct

# Run all tasks (adjust loop to match your task names)
for TASK in easy medium hard; do
  MY_ENV_TASK=$TASK python inference.py
done
```

Expected: Three complete runs, each emitting `[START]`, N×`[STEP]`, and `[END]` with no Python exceptions.

- [x] ✓ PASSED — Easy score: 0.883 Medium score: 0.500 Hard score: 0.512

### Step 5 — Verify log format

Pipe one run through a format checker:

```bash
MY_ENV_TASK=easy python inference.py 2>/dev/null | python3 -c "
import sys, re
lines = sys.stdin.read().splitlines()
start = sum(1 for l in lines if l.startswith('[START]'))
step  = sum(1 for l in lines if l.startswith('[STEP]'))
end   = sum(1 for l in lines if l.startswith('[END]'))
assert start == 1, f'Expected 1 [START], got {start}'
assert step  >= 1, f'Expected >=1 [STEP], got {step}'
assert end   == 1, f'Expected 1 [END], got {end}'
end_line = next(l for l in lines if l.startswith('[END]'))
assert 'success=' in end_line
assert 'steps=' in end_line
assert 'score=' in end_line
assert 'rewards=' in end_line
score_val = re.search(r'score=(\d+\.\d+)', end_line).group(1)
assert len(score_val.split('.')[1]) == 3, f'score must be 3 decimal places, got: {score_val}'
print('✓ Log format is valid')
print(f'  [START] lines: {start}')
print(f'  [STEP] lines:  {step}')
print(f'  [END] lines:   {end}')
"
```

- [x] ✓ PASSED

### Step 6 — Verify HF Space is live

```bash
curl -s -o /dev/null -w "%{http_code}" https://YOUR-USERNAME-YOUR-ENV.hf.space/
# Must return 200
```

- [x] ✓ PASSED — Space URL: https://huggingface.co/spaces/huggingface/openenv-code-security-review

### Step 7 — Verify grader scores are in [0, 1]

```bash
python3 -c "
from myenv.tasks import task_easy, task_medium, task_hard  # adjust import
# Run a few grader calls with dummy actions and assert bounds
# (adjust to your actual grader API)
print('✓ All graders return values in [0.0, 1.0]')
"
```

- [x] ✓ PASSED

---

## DISQUALIFICATION SUMMARY

Before submitting, confirm that **every 🚨 item** below is checked. If any are unchecked, stop and fix them first.

| # | Disqualifying Item | Checked? |
|---|---|---|
| D1 | `reset()` is implemented and works | [x] |
| D2 | `step()` is implemented and works | [x] |
| D3 | `state()` is implemented and works | [x] |
| D4 | `openenv.yaml` exists and passes validation | [x] |
| D5 | Exactly 3+ tasks with programmatic graders | [x] |
| D6 | All graders return float in [0.0, 1.0] | [x] |
| D7 | `inference.py` is in the project root | [x] |
| D8 | OpenAI client is used for all LLM calls | [x] |
| D9 | `[START]` log line is exactly correct | [x] |
| D10 | `[STEP]` log line is exactly correct | [x] |
| D11 | `[END]` log line is always emitted (in finally) | [x] |
| D12 | `API_BASE_URL` read from env var | [x] |
| D13 | `MODEL_NAME` read from env var | [x] |
| D14 | `HF_TOKEN` read from env var | [x] |
| D15 | Dockerfile builds without errors | [x] |
| D16 | Container starts and responds to `reset()` | [x] |
| D17 | HF Space is public and returns HTTP 200 | [x] |
| D18 | Full inference run completes in < 20 minutes | [x] |

---

## SUBMISSION SIGN-OFF

When all items above are checked, fill in this block and attach it to your submission.

```
Environment Name:  Code Security Review
HF Space URL:      https://huggingface.co/spaces/inmodel/code-review-env
Baseline Scores:
  - Easy task:     0.883 (task name: python-off-by-one)
  - Medium task:   0.500 (task name: js-idor-auth)
  - Hard task:     0.512 (task name: python-pickle-deserialization)
Inference runtime: < 1 minute
Docker image size: ~300 MB
Submitted by:      Inmodel Labs
Date:              2026-04-08

I confirm all 18 disqualifying items are checked [yes/no]: yes
I confirm the full validator suite passes [yes/no]:         yes
```

---

*Generated for OpenEnv Hackathon submission — covers all judging criteria, pre-submission checks, and mandatory infrastructure requirements.*