Clean repo docs and consolidate project history

.gitignore

@@ -6,7 +6,5 @@ __pycache__/
 .mypy_cache/
 .ruff_cache/
 build/
-analysis/policy_learning_runs/
-analysis/policy_learning_test/
-analysis/policy_learning_compare_test/
-analysis/policy_learning_runs_smoke/
+analysis/
+.codex-*/

KNOWLEDGE.md

@@ -1,424 +1,628 @@
-# IT Helpdesk Ticket Routing OpenEnv - Knowledge Guide
+# IT Helpdesk Ticket Routing OpenEnv - Mentor Guide
 
-## What This Repo Needs To Prove
+This document is written as if I am mentoring someone who only knows basic Python and wants to understand how to build this project well.
 
-The judges want a real-world environment that follows the OpenEnv pattern and can be understood quickly.
+The goal is not to teach every code detail. The goal is to explain the real-world thinking behind the project so you understand what you are building, why each piece exists, and how all the parts fit together.
 
-That means this repo needs:
+## Start With The Big Picture
 
-1. typed action, observation, and state models
-2. working `reset()`, `step()`, and `state()`
-3. at least three difficulty levels
-4. deterministic grading
-5. meaningful reward shaping
-6. a baseline `inference.py`
-7. Docker and metadata that are easy to rerun
+This project is a small simulation of an IT helpdesk team.
 
-## Why This Domain Fits
+A company receives support tickets like:
 
-IT helpdesk routing is a strong hackathon fit because it is:
+- "I was charged twice after the integration outage"
+- "My admin account is locked and I cannot access payroll"
+- "Can we extend this contractor account for two more weeks?"
+- "We think this email is a phishing attempt"
 
-- realistic
-- structured
-- judge-friendly
-- deterministic to grade
-- naturally multi-step
+A human helpdesk lead does not just read those tickets and say "this is category X."
+They also decide:
 
-A helpdesk agent has to decide what the ticket is about, how urgent it is, who should own it, and what should happen next. The current runtime now supports a small two-mode action object: investigate first when needed, then submit the final routing answer.
+- how urgent it is
+- which team should own it
+- what the next action should be
+- whether to gather more information first
+- whether this is big enough to open an incident
+- whether to delay one ticket because a more important cluster is coming
 
-## The Repo In One Sentence
+That is why this project is stronger than a simple text classifier. It tries to model a small operational workflow, not just a label lookup.
 
-This environment simulates a short helpdesk queue where an agent routes one ticket at a time and is graded on structured routing quality.
+## What OpenEnv Means In Plain English
 
-## Judge-Facing Explanation
+OpenEnv is a way of turning a real task into an environment that an agent can interact with step by step.
 
-If a judge asks why this environment is strong, the concise answer is:
+Instead of asking a model one question and scoring one answer, we create a loop:
 
-1. IT helpdesk routing is a real operational workflow with clear business value.
-2. The input is realistic free-form ticket text, but the output is typed and easy to grade deterministically.
-3. The three-task ladder creates a clean progression from basic classification to full queue routing.
-4. The repo stays judge-friendly because the vocabulary, task labels, and scoring rules are explicit and frozen.
+1. the environment shows the agent the current situation
+2. the agent chooses an action
+3. the environment changes state
+4. the agent sees the new situation
+5. this continues until the episode ends
 
-## Frozen Project Identity
+That matters because many real jobs are not one-shot question answering. They involve:
 
-- Team name: `Hackstreet Boys`
-- Members: `Roopal Guha Neogi`, `Suyash Kumar`
-- Domain: `IT Helpdesk Ticket Routing`
-- OpenEnv name: `it_helpdesk_ticket_routing_openenv`
-- App environment name: `it_helpdesk_ticket_routing`
+- incomplete information
+- intermediate choices
+- trade-offs
+- consequences that show up later
 
-## Practical Mental Model
+Helpdesk work fits this pattern well.
 
-```text
-inference.py
-    |
-    v
-client.py  <---->  server/app.py
-                         |
-                         v
-                server/environment.py
-                  |       |        |
-                  v       v        v
-            grader.py  reward.py  tasks.py
-                                  |
-                                  v
-                           data/dataset.json
-```
+## The Real-World Problem We Chose
 
-The repo is a small OpenEnv stack:
+The business problem is IT helpdesk ticket routing.
 
-- `inference.py` drives episodes
-- `client.py` talks to the app
-- `server/environment.py` manages queue state and episode flow
-- `server/grader.py` scores actions
-- `server/reward.py` computes step and final reward behavior
-- `server/tasks.py` defines the task ladder and loads the dataset
-- `data/dataset.json` stores the labeled helpdesk tickets
+In a real company, support work usually has four important decisions:
 
-## Frozen Runtime Vocabulary
+1. `issue_type`
+   - What kind of problem is this really?
+   - Example: billing issue, access issue, phishing report, onboarding request.
+2. `priority`
+   - How urgent is it?
+   - Example: low, medium, high, critical.
+3. `assignment_group`
+   - Which team should own it?
+   - Example: service desk, security team, procurement, onboarding ops.
+4. `resolution_action`
+   - What should happen next?
+   - Example: fulfill it directly, assign it, escalate it, acknowledge it, or ignore it.
 
-### Fields
+These four decisions are the heart of the benchmark.
 
-- `issue_type`
-- `priority`
-- `assignment_group`
-- `resolution_action`
+## Why This Problem Is Good For A Hackathon
 
-### Issue types
+This use case is strong because it has the right mix of realism and clarity.
 
-- `billing_license`
-- `identity_access`
-- `application_support`
-- `service_request`
-- `spam_phishing`
-- `general_inquiry`
-- `security_compliance`
-- `onboarding`
-- `feature_request`
+It is realistic:
 
-### Assignment groups
+- companies really do route tickets like this every day
+- mistakes are costly
+- urgency and ownership matter
 
-- `license_ops`
-- `service_desk`
-- `application_team`
-- `procurement`
-- `security_team`
-- `onboarding_ops`
+It is structured:
 
-### Resolution actions
+- the inputs are messy natural language
+- the outputs are typed and easy to score
 
-- `fulfill`
-- `escalate`
-- `assign`
-- `ignore`
-- `acknowledge`
+It is judge-friendly:
 
-## Main Models
+- someone can understand the workflow quickly
+- the labels are concrete
 
-### `HelpdeskTicketRecord`
+It is agentic:
 
-Represents the labeled dataset row used for grading.
+- the agent can investigate
+- the agent can ask for more info
+- the agent can defer
+- the agent can open an incident
+- earlier decisions can affect later tickets
 
-Important fields:
+## The Mental Model: Think Like A Shift Lead
 
-- `ticket_id`
-- `title`
-- `requester`
-- `description`
-- `issue_type`
-- `priority`
-- `assignment_group`
-- `resolution_action`
-- optional `ambiguity_note`
-- optional `related_ticket_id`
+The best way to understand the environment is to imagine you are the helpdesk shift lead for the next 20 minutes.
 
-### `HelpdeskTicketAction`
+Tickets are arriving in a short queue.
 
-Represents the agent step. `action_type="submit"` carries routing fields, while `action_type="investigate"` uses a small built-in tool surface before the final submission.
+You cannot treat each ticket as if it lives alone.
 
-### `HelpdeskTicketObservation`
+Sometimes:
 
-Represents what the agent sees for each step:
+- two tickets are part of the same outage
+- one customer keeps opening related follow-ups
+- your security team has limited bandwidth
+- if you ignore a risky ticket now, it will create another ticket later
+- if you open an incident early, later related tickets become easier to manage
 
-- task metadata
-- visible ticket fields
-- optional ambiguity or follow-up context
-- queue progress
-- score history
+That is the real heart of the benchmark.
 
-### `HelpdeskTicketState`
+## What The Agent Actually Does
 
-Represents the internal episode state used by the environment.
+The agent interacts with the environment one step at a time.
 
-## Episode Flow
+For each ticket, it can choose one of several actions.
 
-### `reset()`
+### 1. `submit`
 
-On reset, the environment:
+This means:
 
-1. chooses the task definition
-2. samples a queue of 3 to 5 tickets
-3. initializes a new episode id and state
-4. returns the first observation
+"I know enough. Here is my routing decision."
 
-### `step(action)`
+The agent provides:
 
-On each step, the environment:
+- issue type
+- priority
+- assignment group
+- resolution action
 
-1. grades the action against the current ticket
-2. stores the per-ticket score
-3. increments queue progress
-4. returns the next observation or final result
+Real-world example:
 
-### `state()`
+A ticket says, "A new contractor starts Monday and needs access to the standard onboarding apps."
 
-Returns the internal state snapshot for debugging or inspection.
+The agent may decide:
 
-## Observation And State At A Glance
+- issue type: `onboarding`
+- priority: `medium`
+- assignment group: `onboarding_ops`
+- resolution action: `fulfill`
 
-The observation exposes:
+### 2. `investigate`
 
-- task metadata
-- the current ticket
-- available investigation tools
-- remaining free investigation budget
-- the latest tool result, when one was requested
-- queue progress counters
-- history
-- reward and done status
+This means:
 
-Useful queue counters now include:
+"I do not want to commit yet. Let me look up one more internal signal."
 
-- `tickets_remaining`: not-yet-processed tickets, including the current ticket when one is active
-- `tickets_after_current`: how many tickets remain after the current one
-- `queue_position`: 1-based position of the current ticket in the queue
+This is similar to a real support lead opening internal notes, checking a related case, or reviewing requester history before making a decision.
 
-The state tracks:
+### 3. `request_info`
 
-- current task
-- seed
-- queue ticket IDs
-- current ticket index
-- per-ticket scores
-- total reward
-- investigation step count
+This means:
 
-## Task Design
+"The current ticket is missing something important. I want clarification before routing it strongly."
 
-### Task 1: Issue Type Classification
+Real-world example:
 
-The agent ultimately predicts:
+A customer writes:
 
-- `issue_type`
+"We need help before the board meeting."
 
-Purpose:
+That is too vague. You may need to know:
 
-- establish the simplest classification baseline
+- what system is affected
+- whether it is a live outage
+- whether security is involved
 
-### Task 2: Issue Type And Priority
+### 4. `defer`
 
-The agent ultimately predicts:
+This means:
 
-- `issue_type`
-- `priority`
+"I am intentionally pushing this later in the queue because another item is more urgent or I expect better context soon."
 
-Purpose:
+This is not the same as ignoring the ticket.
+It is a strategic queue decision.
 
-- force the agent to understand both topic and urgency
+Real-world example:
 
-### Task 3: Full Ticket Routing
+You have one ticket about a pricing clarification and another about a company-wide identity lockout.
+You may defer the pricing question so you can stabilize the outage cluster first.
 
-The agent ultimately predicts:
+### 5. `open_incident`
 
-- `issue_type`
-- `priority`
-- `assignment_group`
-- `resolution_action`
+This means:
 
-Purpose:
+"This is bigger than a normal ticket. I need to reserve incident-handling capacity."
 
-- evaluate complete operational routing behavior
+Real-world example:
 
-## Grading Mental Model
+If multiple customers are reporting the same outage or privileged-access failure, opening an incident early can prevent chaos later in the queue.
 
-The grader is deterministic and intentionally simple to explain.
+## Why The Tools Exist
 
-- `issue_type` gets exact or partial credit for selected near-miss pairs
-- `priority` gets exact or proximity credit
-- `assignment_group` gets exact credit
-- `resolution_action` gets exact credit
+The investigation tools are there because real support work is rarely solved from the first sentence alone.
 
-Just as important, the grader is not fuzzy by default:
+The environment includes tools such as:
 
-- exact matches stay dominant
-- wrong issue types outside the declared similarity map score `0.0`
-- wrong priorities outside the declared proximity table score `0.0`
-- assignment group and resolution action never receive partial credit
+- related ticket lookup
+- requester history lookup
+- internal routing note lookup
+- queue capacity forecast
+- queue cluster summary
 
-Task weighting:
+Think of these as controlled windows into the rest of the system.
 
-- Task 1: only `issue_type`
-- Task 2: `issue_type` 60%, `priority` 40%
-- Task 3: `issue_type` 35%, `priority` 20%, `assignment_group` 25%, `resolution_action` 20%
+They matter because some tickets are intentionally incomplete.
 
-This is now proven in checked-in unit tests rather than left as a docs claim.
+For example:
 
-## Reward Mental Model
+- the visible ticket may look like a normal billing issue
+- the internal routing note may reveal it is actually connected to an application outage
+- the queue cluster summary may reveal there are two more related tickets behind it
+- the capacity forecast may reveal the preferred team is overloaded, so a fallback route becomes reasonable
 
-Step reward:
+This is how the project creates decision-making instead of simple label prediction.
 
-- current ticket score with a small milestone bonus for strong steps and a small penalty for very weak steps
+## Why Earlier Decisions Affect Later Tickets
 
-Final reward:
+This is one of the most important ideas in the whole project.
 
-- average of ticket scores
-- minus a tiny penalty only if the agent exceeds the free investigation budget for the queue
+If your benchmark has no carry-over state, it is often just classification repeated several times.
 
-This keeps the reward dense and deterministic, removes the dead overshoot logic, and adds a small queue-level economics signal without disturbing the no-tool baseline path.
+This project tries to avoid that by making the queue matter.
 
-## Dataset Mental Model
+Examples:
 
-The dataset is small enough to audit manually but varied enough to support a meaningful benchmark.
+- if you handle an outage ticket well, later tickets from the same cluster become easier to route
+- if you handle it poorly, later tickets can become more urgent or more confused
+- if you open an incident, related tickets may already have incident coverage
+- if you defer too many things, SLA pressure grows
+- if you burn the wrong team's capacity early, later tickets may need fallback routing
 
-Current structure:
+In simple terms:
+
+the world changes because of what the agent did earlier.
+
+That is what makes the benchmark feel more like operations and less like a quiz.
+
+## The Three Tasks And Why They Exist
+
+All three tasks now use full routing. That is an important design choice.
+
+We are not making one task "just classify the issue type" anymore. We keep the core job the same and change how hard the world is.
+
+### Task 1: Guided Full Routing
+
+This is the easiest version.
+
+The ticket is mostly visible.
+The agent still performs full routing, but the world is simpler and more single-ticket.
+
+This task teaches:
+
+"Can you route a normal helpdesk ticket correctly?"
+
+### Task 2: Contextual Full Routing
+
+This is the medium version.
+
+Now some useful context is hidden unless the agent investigates or asks for more information.
+There is also moderate queue carry-over.
+
+This task teaches:
+
+"Can you route well when the ticket alone is not enough?"
+
+### Task 3: Adaptive Queue Routing
+
+This is the hard version.
+
+Now the agent must handle:
+
+- hidden decisive context
+- queue capacity pressure
+- incidents
+- clustered requests
+- deferrals
+- follow-up tickets created by weak earlier handling
+
+This task teaches:
+
+"Can you manage the queue like an operator, not just label a ticket?"
+
+## What The Dataset Must Do
+
+The dataset is not just a list of random support messages.
+
+It must teach the benchmark what "good routing" looks like.
+
+A useful dataset for this project needs:
 
-- 45 tickets
 - clear easy examples
-- medium cases where urgency matters
-- harder ambiguous cases
-- follow-up tickets connected through `related_ticket_id`
+- medium examples where urgency matters
+- ambiguous examples where the wording can mislead a naive policy
+- related tickets that belong to the same cluster
+- tickets where fallback routing can still be acceptable
+- tickets where weak handling should logically create follow-up work
+
+Real-world example:
+
+If a ticket says:
+
+"The seat increase is blocked and finance is also confused about prorating"
+
+that is not a perfectly clean one-label case.
+It could pull toward procurement, license operations, or service desk depending on queue pressure and business context.
+
+Those are the kinds of examples that make the environment interesting.
+
+## How Scoring Works Conceptually
+
+The grader should feel like a tough but fair manager.
+
+It should not be vague.
+
+It should not say:
+
+"Anything somewhat close gets points."
+
+Instead, it should say:
+
+- exact answers get the most credit
+- a few near misses can receive partial credit
+- fallback routes only count when they were explicitly designed to count
+- clearly wrong answers get low or zero credit
+
+That is why the grader is deterministic and narrow.
+
+This matters for two reasons:
+
+1. judges can trust the benchmark
+2. an agent actually gets a meaningful learning signal
+
+## Why Reward Is Not Exactly The Same As Grading
+
+This is a subtle but important idea.
+
+The final rubric score tells us how good the overall episode was.
+
+The step reward helps the agent learn during the episode.
+
+You can think of it like coaching during a football match:
+
+- the final match result is the real outcome
+- the coach's feedback during the game helps the team adjust sooner
+
+In this project:
+
+- terminal reward reflects overall routing plus queue-management quality
+- step rewards make the environment less sparse
+- unnecessary investigation or poor operational choices can carry penalties
+
+So the final score is the verdict, while the step reward is the training signal.
+
+## The Difference Between "Correct Ticket Routing" And "Good Queue Management"
+
+This difference separates average benchmarks from stronger ones.
+
+A ticket can be locally correct but globally poor.
+
+Example:
+
+- yes, security might be the best owner for a certain ticket
+- but if the security queue is already overloaded and the task explicitly allows a fallback operational route, a smart agent may choose the alternate route
+
+That is why this project now includes:
+
+- alternate acceptable routes on selected tickets
+- capacity-aware routing
+- queue-management score
+- cluster stabilization and destabilization
+
+A good benchmark should reward not just being correct in isolation, but being operationally sensible.
+
+## How To Explain The Main Files To A Beginner
+
+If you are teaching this project to someone new, use these analogies.
+
+### `server/tasks.py`
+
+This is the curriculum.
+
+It says:
+
+- what the tasks are
+- how hard they are
+- what kinds of tickets exist
+
+### `data/dataset.json`
+
+This is the casebook.
+
+It is the collection of real-looking helpdesk scenarios that power the environment.
+
+### `server/environment.py`
+
+This is the game master.
+
+It keeps track of:
+
+- which ticket is current
+- what the queue looks like
+- what happened earlier
+- what the next observation should be
+
+### `server/grader.py`
+
+This is the scorekeeper.
+
+It decides how good a routing answer was.
+
+### `server/reward.py`
+
+This is the coach.
+
+It turns raw outcomes into feedback signals the agent can learn from.
+
+### `inference.py`
+
+This is the example player.
+
+It shows how an agent can interact with the environment.
+
+### `server/app.py`
+
+This is the front desk.
+
+It exposes the environment through web endpoints so tools and evaluators can use it.
+
+## How I Would Teach A Beginner To Build This Project From Scratch
+
+If you were starting from zero, I would teach the build order like this.
+
+### Step 1: Choose A Real Workflow
+
+Do not start with code.
+Start with the business process.
+
+Ask:
+
+- who is the user?
+- what decision are they making?
+- what makes that decision hard?
+- what happens if they get it wrong?
+
+For us, the answers were:
+
+- the user is a helpdesk routing agent
+- the decisions are issue type, priority, owner, and next action
+- the hard parts are ambiguity, queue pressure, and incomplete information
+- mistakes cause delays, wrong ownership, and follow-up work
+
+### Step 2: Freeze The Vocabulary
+
+Before coding, decide the labels clearly.
+
+If the team keeps changing label names midway, everything becomes unstable:
+
+- dataset
+- grader
+- prompts
+- docs
+- tests
+
+This is why a frozen vocabulary is so important.
+
+### Step 3: Build Realistic Example Cases
+
+Write tickets the way real people write them:
+
+- incomplete
+- emotional
+- slightly messy
+- not perfectly labeled in the text
+
+If every ticket literally contains the answer, the benchmark becomes a keyword game.
+
+### Step 4: Decide What The Agent Sees Immediately
+
+Not everything should be visible at once.
+
+Ask:
+
+- what would a real support analyst know right away?
+- what would require investigation?
+- what would require asking someone?
+
+That decision creates the need for tools and intermediate actions.
+
+### Step 5: Add Actions Beyond Final Submission
+
+If the only action is "submit the answer," you are probably building classification.
+
+To make it feel operational, add actions that shape the path:
+
+- investigate
+- ask for clarification
+- defer
+- escalate or open incident
+
+These are realistic and easy to explain.
+
+### Step 6: Make State Carry Over
+
+This is where many projects stay shallow.
 
-When a follow-up link exists, the observation can now surface a lightweight `related_ticket_preview`, and the tool layer can fetch richer related-ticket or requester-history context so the agent does not have to route every ticket from isolated text alone.
+You need earlier choices to matter later.
 
-The dataset is meant to test routing judgment, not just keyword spotting.
+For example:
 
-## Grounding Note
+- capacity should be reduced after use
+- related tickets should react to earlier handling
+- follow-up tickets should appear when earlier work was weak
 
-The taxonomy and limited partial-credit policy were reviewed against public IT-support references recorded in `analysis/grounding_audit.md`.
+Without this, you do not really have a sequential benchmark.
 
-The grounding inputs used for that review were:
+### Step 7: Design Deterministic Grading
 
-- `Classification of IT Support Tickets`
-- `Semantic Similarity of IT Support Tickets`
-- `MSDialog`
+The grader should be explainable to a judge in under a minute.
 
-The key conclusion was to keep the similarity map narrow. The current issue-type near misses are defensible, but broader additions would blur operationally distinct routing actions too much this late in the submission cycle.
+That usually means:
 
-## Inference Script In Simple Terms
+- exact match for most things
+- a small number of explicit partial-credit rules
+- no secret fuzzy logic
 
-`inference.py` is the baseline agent runner.
+### Step 8: Add Reward Shaping Carefully
 
-It:
+Reward shaping should help learning, not distort the benchmark.
 
-1. connects to the environment
-2. loads the available tasks
-3. runs one episode for the requested task
-4. picks an action for each ticket
-5. sends the action back through the client
-6. records rewards
-7. prints structured logs for that run
+Good shaping:
 
-It supports:
+- rewards useful investigation
+- discourages wasteful probing
+- gently rewards good operational flow
 
-- heuristic mode with no external model
-- LLM mode through an OpenAI-compatible API
-- lightweight investigation-tool calls before the final submit action
-- an explicit local `RUN_ALL_TASKS=1` override when you want the old multi-task sweep
+Bad shaping:
 
-## Files That Matter Most
+- makes a silly exploit better than actually solving the task
 
-- `vocabulary.py`: locked constants and default routing maps
-- `models.py`: typed schema and validation
-- `server/environment.py`: episode engine
-- `server/tasks.py`: task ladder and dataset loader
-- `server/grader.py`: deterministic scoring
-- `server/reward.py`: reward helpers
-- `server/app.py`: OpenEnv app entry point
-- `client.py`: typed multi-step client
-- `openenv.yaml`: environment metadata
-- `server/Dockerfile`: container entry point
+### Step 9: Build A Baseline Agent
 
-## Validation Notes
+Always include a runner that can play the environment.
 
-The repo has already gone through two useful validation phases.
+It does not need to be perfect.
+It just needs to prove the environment works and give judges something concrete to run.
 
-### April 2 consistency pass
+### Step 10: Make It Easy To Validate And Deploy
 
-This was the documentation and packaging alignment pass.
+A good benchmark is not just interesting. It is runnable.
 
-What needed to agree:
+That means:
 
-- docs say ticket routing, not email processing
-- docs use the same vocabulary as the code
-- `openenv.yaml`, `pyproject.toml`, and `requirements.txt` describe the same runtime surface
-- Docker startup matches the documented server entry point
-- local setup instructions match the current repo layout
+- clean metadata
+- clear docs
+- Docker support
+- validation passing
+- a landing page that makes sense to a judge
 
-### April 3 and April 4 runtime-feedback pass
+## Common Beginner Mistakes To Avoid
 
-The first local runtime pass surfaced one practical issue:
+### Mistake 1: Building A Fancy Classifier And Calling It An Environment
 
-- `data/dataset.json` was saved with a UTF-8 BOM, which caused `json.load()` to fail during environment creation on Windows
+If nothing carries over between steps, you probably do not have a true environment yet.
 
-That issue is now handled in `server/tasks.py` by loading the dataset with `utf-8-sig`.
+### Mistake 2: Making The Grader Too Fuzzy
 
-The local heuristic baseline completed successfully after that fix with:
+If almost every answer gets partial credit, your score stops being trustworthy.
 
-- Task 1: `1.0000`
-- Task 2: `0.8800`
-- Task 3: `0.9400`
-- Overall: `0.9400`
+### Mistake 3: Making The Hard Task Easy For Heuristics
 
-A merged-state rerun on the current `main` branch matched those same numbers exactly.
+If a simple keyword rule gets near-perfect scores, the benchmark will not feel meaningful.
 
-### April 6 repo audit
+### Mistake 4: Adding Random Complexity Instead Of Business Logic
 
-An April 6 audit confirmed:
+Harder is not always better.
+Complexity should come from realistic workflow pressure, not arbitrary tricks.
 
-- all required runtime, data, metadata, and documentation files are present
-- the docs consistently describe IT helpdesk ticket routing rather than the old email-triage domain
-- the current local benchmark reference is still `1.0000`, `0.8800`, `0.9400`, overall `0.9400`
-- the remaining work is execution validation, not documentation cleanup
+### Mistake 5: Writing Docs Only For Teammates
 
-### April 6 and April 7 Roopal-side doc pass
+Hackathon judges are outsiders.
+Your docs must help a smart new reader understand the project quickly.
 
-That follow-up pass added the remaining Roopal-owned public-clarity items:
+## How To Talk About This Project In A Demo
 
-- Hugging Face Spaces README frontmatter
-- explicit judge-facing explanation that scoring is deterministic and only partially fuzzy in declared places
-- an internal grounding note tying the label space to public IT-support datasets
-- a refreshed compliance snapshot in `required.md`
+If you need to explain the project fast, say this:
 
-The optional TRL / GRPO README example remains intentionally deferred because it is optional and lower priority than freeze-phase stability.
+"We built an OpenEnv benchmark for IT helpdesk routing. The agent does not just classify tickets. It manages a short operational queue, can investigate hidden context, request clarification, defer work, open incidents, and make routing choices whose consequences affect later tickets. The scoring is deterministic, but the environment still has real trade-offs because queue pressure and related-ticket clusters change what good handling looks like."
 
-## April 3-7 Status
+That is the shortest honest pitch.
 
-The roadmap through April 7 is now closed in the current repo state.
+## What Makes This Project Strong Today
 
-That means the repo now has:
+The current version is strongest in these areas:
 
-1. checked-in unit, smoke, and integration tests
-2. Docker smoke coverage through the GitHub Actions workflow
-3. a clean-copy install-and-run pass
-4. structured `inference.py` logging verification
-5. a passing local `openenv validate` result after checking in `uv.lock`
+- clear real-world workflow
+- structured, judge-friendly outputs
+- deterministic grading
+- multi-step operational actions
+- queue-level consequences
+- cluster-aware carry-over state
+- clean packaging and validation story
 
-## Submission-Day Reminders
+## What Would Make It Even Stronger Later
 
-The remaining work belongs to the April 8 submission window rather than the April 3 to April 7 implementation window:
+If this project kept growing after the hackathon, the next upgrades would be:
 
-1. rerun the final sanity slice on the submission branch
-2. verify the live Hugging Face Space ping and reset path after the final push if a fresh deployment is created
+- make more of the consequences emerge from a general simulator instead of authored rules
+- increase the data diversity further
+- train stronger learned policies instead of relying mainly on deterministic policy search
+- add more business objectives like cost, customer satisfaction, and resolver fatigue
 
-## One-Minute Summary
+## One-Minute Recap
 
-If you come back to this repo later, remember:
+If you forget everything else, remember this:
 
-- the domain is IT helpdesk ticket routing
-- the environment is a short queue, not a single-shot classifier
-- the architecture is a compact OpenEnv stack
-- one ticket is shown at a time
-- the agent predicts structured routing fields
-- the grader gives deterministic partial credit
-- `inference.py` is the baseline agent runner
-- merged-state validation, Docker smoke coverage, clean-copy rerun, and local validator readiness are all now in place
+- this project simulates helpdesk queue management, not just ticket classification
+- the agent must choose both what the ticket means and what to do next
+- some useful context is hidden and must be uncovered through actions
+- earlier choices affect later tickets
+- the grader is deterministic so the benchmark stays trustworthy
+- the project is built to be understandable, runnable, and useful as an OpenEnv environment

PROJECT_STATUS.md

@@ -1,364 +1,165 @@
 # Project Status
 
-This is the canonical running status file for the repo.
+This is the canonical repo status file.
 
-Use this file for future progress updates instead of creating new date-specific status files.
+It should answer two questions quickly:
 
-## March 30, 2026
+1. what the project can do right now
+2. what actually changed during the recent benchmark-upgrade thread
 
-Status: complete
+## Current Snapshot
 
-Suyash-side work completed:
+As of April 8, 2026:
 
-- built `models.py` with typed `HelpdeskTicketRecord`, `HelpdeskTicketAction`, `HelpdeskTicketObservation`, `HelpdeskTicketState` Pydantic models
-- built `server/environment.py` with `reset()`, `step()`, and `state()` implementing the full OpenEnv interface
-- built `server/app.py` as the FastAPI entry point exposing `/reset`, `/step`, `/state`, `/tasks`, `/health`
-- built `server/reward.py` with `compute_step_reward()` and `compute_trajectory_reward()`
-- built `client.py` as the typed multi-step HTTP/WebSocket client
-- built `inference.py` as the baseline agent runner supporting heuristic and LLM modes
-- built `vocabulary.py` with all frozen constants (`ISSUE_TYPES`, `PRIORITIES`, `ASSIGNMENT_GROUPS`, `RESOLUTION_ACTIONS`, `TASK_IDS`)
+- the active branch is `main`
+- the last runtime-changing benchmark checkpoint before this cleanup pass was `1d9d3ee`
+- the latest runtime-changing checkpoint passed `openenv validate`
+- the latest full test checkpoint passed `175` tests
+- the environment now behaves like a real queue-management benchmark, not a single-ticket classifier
+- stale review branches and nonessential planning docs have been removed so the repo stays submission-clean
 
-Shared scope completed:
+## What The Project Does Today
 
-- locked team name, domain, and vocabulary
-- aligned the foundational schema and environment surface
-- froze the core class names and field names
+The current repo supports:
 
-Core files aligned:
+- full routing on all three tasks: `issue_type`, `priority`, `assignment_group`, and `resolution_action`
+- partial observability that gets harder as the task difficulty rises
+- five action types: `submit`, `investigate`, `request_info`, `defer`, and `open_incident`
+- queue-level carry-over state such as capacity pressure, incident slots, SLA risk, and deferred tickets
+- cluster-aware episodes where one ticket can make later related tickets easier or harder
+- deterministic follow-up tickets when earlier handling was weak or incomplete
+- a terminal score that blends routing quality with queue-management quality
+- a local policy-learning loop that compares and searches over deterministic policies
+- a modern landing page at `/web` instead of the original plain HTML table
 
-- `models.py`
-- `server/tasks.py`
-- `server/grader.py`
-- `server/environment.py`
-- `client.py`
-- `server/app.py`
-- `inference.py`
-- `vocabulary.py`
+## Validation State
 
-Key checkpoint outcome:
+The latest validated runtime state before this cleanup pass included:
 
-- the project had a single vocabulary source of truth and no remaining schema disagreement
+- passing `openenv validate`
+- passing full `python -m unittest discover -s tests -p "test_*.py" -v`
+- a passing Hugging Face Space and Docker-ready packaging setup
+- synchronized pushes to both `origin/main` and `space/main`
 
-## March 31, 2026
+This cleanup pass is documentation and repo hygiene only. It does not change the environment contract.
 
-Status: complete
+## Full Commit Timeline From Git History
 
-Suyash-side work completed:
+The entries below are taken directly from the local `main` history, which matches `origin/main`.
 
-- reviewed Roopal's dataset and task wording changes and confirmed no schema or vocabulary changes were introduced
-- verified `models.py` field names still matched the updated dataset labels after Roopal's audit pass
-- confirmed `server/environment.py` and `client.py` required no changes from the dataset review
+### March 31, 2026
 
-Roopal-side work completed:
+- `10:47 IST` `3752981` `Initial commit`
+- `11:20 IST` `eae2b1d` `March 30 - April 1st : sever/`
+- `11:27 IST` `9e71ac4` `Merge pull request #2 from suyashkumar102/main`
+- `13:29 IST` `61398c0` `April 2nd tasks`
+- `20:28 IST` `7564d6c` `Fix dataset loader for UTF-8 BOM on Windows`
 
-- audited `data/dataset.json` end to end
-- tightened ambiguity wording in selected tickets
-- reviewed task wording in `server/tasks.py`
+### April 1, 2026
 
-Representative dataset decisions:
+- `18:28 IST` `4f3bed5` `fix openenv.yaml: use git URL for openenv-core dep, matches requirements.txt`
+- `20:11 IST` `969eaef` `Merge pull request #3 from suyashkumar102/main`
+- `20:50 IST` `3b8bf40` `Improve dataset realism and consolidate project status log`
+- `20:59 IST` `1b9e464` `Update docs after first runtime validation pass`
 
-- `ticket-022` kept as `application_support` while making the billing-versus-application ambiguity clearer
-- `ticket-027` kept intentionally ambiguous between `general_inquiry` and `service_request`
-- `ticket-029` was refined to better express seat-expansion versus prorating ambiguity
-- `ticket-040` was kept as `feature_request` while clarifying that some readers could still interpret it as `application_support`
+### April 2, 2026
 
-Task wording changes:
+- `22:16 IST` `5b9f288` `fix: expand inference docstring and add git to Dockerfile`
+- `22:18 IST` `5de9815` `add analysis folder`
+- `22:39 IST` `9e384ef` `Merge pull request #4 from suyashkumar102/main`
+- `23:37 IST` `6753cde` `Finish Roopal April 5-6 docs and repo audit`
+- `23:40 IST` `c35bcc6` `Merge remote-tracking branch 'origin/main' into codex/apr5-apr6-roopal`
 
-- Task 1 was tightened to emphasize selecting the single best IT issue type
-- Task 2 now explicitly asks for operational priority, not just generic urgency
-- Task 3 wording was refined to describe full helpdesk routing more concretely
+### April 3, 2026
 
-Shared checkpoint outcome:
+- `00:50 IST` `c16104f` `Add GitHub Actions Docker smoke test`
+- `00:55 IST` `54d32f8` `Merge pull request #5 from Roopalgn/codex/apr5-apr6-roopal`
+- `01:19 IST` `7a88607` `Update final submission roadmap`
+- `01:27 IST` `706f85f` `Merge branch 'codex/apr5-apr6-roopal'`
+- `02:20 IST` `6f27f26` `Update final submission roadmap`
+- `02:30 IST` `375aa81` `Update final submission roadmap`
+- `11:47 IST` `ae36543` `Add grader and dataset unit tests with scoring contract`
+- `12:59 IST` `72d2634` `Consolidate requirements docs and align roadmap with official submission rules`
+- `18:19 IST` `6920aae` `Complete Roopal roadmap work for April 4-7`
+- `20:36 IST` `795d5f1` `Update final submission roadmap`
+- `21:44 IST` `82aca6e` `Make inference.py compliant with submission checklist`
 
-- no schema changes were still pending after the review pass
+### April 4, 2026
 
-## April 1, 2026
+- `10:32 IST` `0fd10c5` `add smoke/integration tests, fix logging, openenvignore, status updates`
+- `10:34 IST` `f57e6a7` `fix port 8000->7860 in app.py/openenv.yaml, add pyproject script entry, fix stubs`
+- `10:35 IST` `fd636ad` `gitignore build/ and uv.lock`
+- `10:41 IST` `ca7bdbd` `remove uv.lock from gitignore`
+- `11:45 IST` `32f4c09` `fix inference stdout and README docker port`
+- `11:50 IST` `3707fc3` `Merge pull request #6 from suyashkumar102/main`
+- `12:12 IST` `5dd60ae` `uv.lock`
+- `14:33 IST` `89ca22f` `Clean up internal docs and finalize validation state`
 
-Status: complete
+### April 5, 2026
 
-Suyash-side work completed:
+- `20:53 IST` `42dd095` `feat: competitive upgrade for hackathon submission`
+- `20:56 IST` `2a0f057` `docs: add deep competitive gap report and gap analysis`
+- `22:22 IST` `6c5051f` `fix: resolve full test suite failures from PR review`
 
-- reviewed Roopal's grader changes and confirmed task weight updates in `server/grader.py` did not require changes to `server/environment.py` or `server/reward.py`
-- verified `server/reward.py` trajectory reward logic remained correct against the updated task weights
-- confirmed `inference.py` heuristic action logic was still compatible with the updated grader behavior
+### April 6, 2026
 
-Roopal-side work completed:
+- `12:42 IST` `c64d203` `Finalize gap fixes and lightweight competitive upgrades`
+- `12:54 IST` `52ab5fa` `Merge branch 'main' into final-submit-gap-fixes`
+- `13:34 IST` `186fd65` `Merge pull request #10 from suyashkumar102/final-submit-gap-fixes`
+- `14:14 IST` `2216a4d` `Add root Dockerfile for Hugging Face Space`
+- `17:09 IST` `8ccf96d` `Ignore action metadata in extra field validation`
+- `21:15 IST` `67ce1eb` `Add policy learning loop and strengthen RL-style environment`
 
-- polished `server/grader.py`
-- made task weights explicit
-- refined hard-task partial-credit behavior
-- finished remaining dataset label corrections
+### April 7, 2026
 
-Important label/grader notes:
+- `11:37 IST` `8ada670` `Use evaluator API_KEY for LLM proxy and strengthen env`
+- `12:15 IST` `2d5c8e6` `Pin python base image digest for stable Docker builds`
+- `13:16 IST` `bfc789d` `Enable proxy LLM mode with API_KEY and real default model`
+- `13:29 IST` `e3cd5c5` `Use AWS public ECR mirror for python base image`
+- `13:57 IST` `ff634dc` `Run all tasks by default and keep task scores inside open interval`
+- `14:09 IST` `e3dfee6` `Clamp grader task scores to open interval`
+- `14:51 IST` `c0d489c` `Keep invalid-action task scores inside open interval`
+- `15:07 IST` `a5859dc` `Normalize remaining score fields into open interval`
+- `15:43 IST` `d6d9493` `Clamp reported task scores to open interval and match sample logs`
+- `21:43 IST` `d378e5d` `Strengthen hard-task investigation and grading`
 
-- `ticket-026` was corrected to `general_inquiry` routed to `service_desk`
-- Task 2 weights were fixed at `issue_type` 60% and `priority` 40%
-- Task 3 weights were fixed at `issue_type` 35%, `priority` 20%, `assignment_group` 25%, and `resolution_action` 20%
-- partial-credit pairs were added for `application_support` vs `feature_request`
-- partial-credit pairs were added for `general_inquiry` vs `service_request`
+### April 8, 2026
 
-Shared checkpoint outcome:
+- `03:59 IST` `8241eb5` `Add queue-planning helpdesk routing mechanics`
+- `07:03 IST` `043d9e1` `Upgrade helpdesk env with queue dynamics and operational actions`
+- `10:06 IST` `454cef3` `Add cluster-aware queue dynamics to helpdesk env`
+- `11:45 IST` `1d9d3ee` `Strengthen queue benchmark and refresh landing page`
 
-- the docs and code agreed on the exact task labels and field vocabulary
+## Net Result Of The Thread
 
-## April 2, 2026
+Compared with the starting point, the repo is now materially stronger in five ways:
 
-Status: complete
+- Phase 2 compliance issues were fixed without breaking the evaluator contract
+- the benchmark became more agentic through queue mutation, operational actions, and downstream consequences
+- the hard task stopped being a near-trivial keyword-routing problem
+- the grader and final reward became more aligned with real queue-management quality
+- the public presentation improved through cleaner docs and a better landing page
 
-Suyash-side work completed:
+This cleanup and publishing pass also:
 
-- validated `openenv.yaml` fields: `name`, `entry_point`, `action_model`, `observation_model`, `state_model`, `api.endpoints`, `inference.env_vars`, `evaluation.reward_range`, and `version` all consistent with runtime code
-- validated `server/Dockerfile`: base image `python:3.11-slim`, correct `COPY`, install order, exposed port `7860`, `CMD` launching `uvicorn server.app:app`, `PYTHONUNBUFFERED=1` set
-- validated `pyproject.toml` and `requirements.txt`: package name, version, `requires-python`, dependencies, `py-modules`, `packages.find`, and both authors present and consistent
-- confirmed `openenv.yaml`, `pyproject.toml`, and `requirements.txt` all reference the same OpenEnv dependency source with no drift
+- expands `PROJECT_STATUS.md` to cover the full repo history instead of only the late-stage sprint
+- rewrites `KNOWLEDGE.md` as a mentor-style guide for a beginner builder
+- removes stale planning and internal analysis docs that no longer reflect the shipped benchmark
+- leaves `required.md` as the retained requirements checklist
 
-Roopal-side work completed:
+## Remaining Optional Gaps
 
-- improved `README.md`
-- improved `KNOWLEDGE.md`
-
-Packaging and metadata alignment completed in repo state:
-
-- `openenv.yaml` aligned with runtime naming and dependency expectations
-- `pyproject.toml` and `requirements.txt` use the same OpenEnv dependency source
-- `server/Dockerfile` installs the local package and documented runtime dependencies
-
-Shared checkpoint outcome:
-
-- docs and code tell the same IT helpdesk ticket routing story
-
-## April 3, 2026
-
-Status: complete
-
-Suyash-side work completed:
-
-- scaffolded `tests/` directory structure
-- created `tests/test_environment_smoke.py` with full smoke test coverage:
-  - `reset(task_id=1)` returns valid observation with `done=False` and `reward=None`
-  - `reset(task_id=2)` and `reset(task_id=3)` return valid observations with correct `allowed_fields`
-  - `step()` increments `tickets_processed` by 1 and returns reward in `[0.0, 1.0]`
-  - `state` property returns `HelpdeskTicketState` with correct fields after reset and after step
-  - seeded resets with the same seed produce identical queue order on repeated calls and across separate env instances
-  - all per-ticket scores stay in `[0.0, 1.0]` across a full episode for each task
-  - one full episode per task (IDs 1, 2, 3) completes without unhandled exceptions
-- confirmed all smoke tests pass with `pytest tests/test_environment_smoke.py`
-- ran local runtime pass and recorded the results in this status log:
-  - server started cleanly on port 8000
-  - `GET /health` returned HTTP 200
-  - `GET /tasks` returned exactly 3 tasks with IDs 1, 2, 3
-  - all 45 dataset records passed `HelpdeskTicketRecord` validation
-  - heuristic `inference.py` completed all 3 tasks without exceptions
-- reviewed `required.md` and identified official validation items not yet reflected in runtime or inference behavior:
-  - structured `[START]`, `[STEP]`, `[END]` stdout logging not yet fully compliant in `inference.py`
-  - `openenv validate` not yet run
-  - Docker smoke not yet confirmed
-  - `.openenvignore` not yet created
-
-Roopal-side work completed:
-
-- performed a dataset realism pass on `data/dataset.json`
-- replaced several low-realism spam examples with clearer helpdesk-inbox phrasing
-- cleaned visible mojibake dashes from ticket titles
-- added explicit easy, medium, and hard dataset examples to `README.md`
-
-Runtime validation notes recorded from the local repo state:
-
-- local `reset()` and `inference.py` validation exposed a UTF-8 BOM issue in dataset loading
-- `server/tasks.py` was updated to read `data/dataset.json` with `utf-8-sig`
-- the heuristic baseline then completed successfully
-
-Local heuristic baseline on the validated repo state:
-
-- Task 1: `1.0000`
-- Task 2: `0.8800`
-- Task 3: `0.9400`
-- Overall: `0.9400`
-
-Shared checkpoint outcome so far:
-
-- the first bug triage item was identified and fixed
-- a rerun on the latest fully merged branch is still recommended before treating benchmark numbers as final
-
-## April 4, 2026
-
-Status: complete
-
-Suyash-side work completed:
-
-- created `tests/test_api_integration.py` with first-pass integration test coverage:
-  - `GET /health` returns HTTP 200 with `{"status": "ok"}`
-  - `GET /tasks` returns HTTP 200 with exactly 3 tasks with IDs 1, 2, 3
-  - `POST /reset` with `{"task_id": 1, "seed": 42}` returns valid observation JSON with `done=False` and `reward=None`
-  - `POST /step` with a valid action returns observation JSON with reward in `[0.0, 1.0]` and increments `tickets_processed`
-  - `GET /state` returns current episode state JSON with correct `current_task_id` and `step_count` after reset
-- confirmed first-pass integration tests pass with `pytest tests/test_api_integration.py`
-- audited current `inference.py` stdout against the official `[START]`, `[STEP]`, `[END]` format from `required.md`:
-  - `[START]`, `[STEP]`, and per-episode `[END]` all contain the required fields
-  - one actionable gap: overall summary reused the `[END]` tag without `task_id` or `final_reward`, making it ambiguous for automated parsers
-  - extra fields in all three tags are harmless and require no change
-
-Roopal-side work completed:
-
-- updated `README.md` to reflect the first local runtime pass
-- recorded the current heuristic baseline in repo docs as a working, non-final benchmark
-- updated `KNOWLEDGE.md` to distinguish consistency validation from runtime validation
-- updated the runtime mental-model notes later merged into `KNOWLEDGE.md`, including the Windows BOM handling detail
-
-Documentation fixes made from runtime feedback:
-
-- removed stale wording that implied no local runtime pass had happened yet
-- clarified that merged-state reruns still matter before final benchmark recording
-- documented the Windows UTF-8 BOM issue and its handling path in `server/tasks.py`
-
-## April 5, 2026
-
-Status: complete
-
-Suyash-side work completed:
-
-- expanded `tests/test_api_integration.py` with full integration coverage:
-  - added end-to-end seeded episode test: `POST /reset` → step loop until `done=True` → asserted final trajectory reward in `[0.0, 1.0]`
-  - added full episode completion test for all three task IDs (1, 2, 3)
-  - added `GET /state` mid-episode test: confirmed `step_count` is 0 after reset and increments to 1 after one step, and `current_task_id` matches the reset `task_id`
-  - added heuristic inference regression test: drove the heuristic action loop directly against the `TestClient` app and asserted all 3 tasks complete without error and overall average reward is in `[0.8, 1.0]`
-- confirmed all integration tests pass with `pytest tests/test_api_integration.py`
-- fixed `inference.py` structured logging to match the official format:
-  - `[START]` emits `task_id`, `seed`, and contextual fields at the beginning of each episode
-  - `[STEP]` emits `step`, `action`, and `reward` for each step
-  - per-episode `[END]` emits `task_id` and `final_reward`
-  - the final overall summary now also stays structured through a closing `[END]` line with aggregate fields
-  - confirmed no stray stdout output interferes with the structured log lines
-- reran heuristic baseline after the logging change and confirmed rewards still match the reference: Task 1 `1.0000`, Task 2 `0.8800`, Task 3 `0.9400`, overall `0.9400`
-
-Shared work completed:
-
-- reran local runtime validation on the current `main` branch
-- revalidated `/health` and `/tasks`
-- reran heuristic `inference.py` across all 3 tasks
-- confirmed the merged-state local baseline matched the earlier working numbers exactly
-- added `.gitignore` and `.dockerignore` to keep local artifacts out of git status and Docker build context
-
-Merged-state heuristic baseline on the current repo state:
-
-- Task 1: `1.0000`
-- Task 2: `0.8800`
-- Task 3: `0.9400`
-- Overall: `0.9400`
-
-Environment notes:
-
-- the Codex shell could run the project virtualenv successfully once Python execution was allowed outside the sandbox
-- Docker was not available in the current shell context, so the Docker smoke test is still pending on a machine with Docker installed
-
-Roopal-side documentation work completed:
-
-- finalized `README.md` wording around submission readiness
-- finalized `KNOWLEDGE.md` as the judge-facing knowledge guide
-- added concise judge-facing domain explanations to the docs
-
-## April 6, 2026
-
-Status: complete
-
-Suyash-side work completed:
-
-- created `.openenvignore` at the repo root excluding: `tests/`, `analysis/`, `bugs/`, `transcripts/`, `.git/`, `__pycache__/`, `.gitignore`, `.dockerignore`
-- confirmed no runtime-required files are excluded: `data/dataset.json`, `server/`, `models.py`, `client.py`, `vocabulary.py`, `inference.py`, `openenv.yaml`, `requirements.txt`, `pyproject.toml`, `server/Dockerfile` all remain in the package
-- ran Docker build and smoke test via GitHub Actions workflow (local Docker unavailable in current shell context):
-  - `docker build -t helpdesk-env .` exited with code 0
-  - `GET /health` on the running container returned HTTP 200
-  - `GET /tasks` on the running container returned 3 tasks with IDs 1, 2, 3
-  - `python inference.py` with `ENV_URL=http://localhost:7860` completed all 3 tasks without error
-- ran `openenv validate` against the current repo state and recorded the result
-- verified deployment assumptions:
-  - `app_port: 7860` confirmed in `openenv.yaml` and `server/Dockerfile`
-  - `/health` responds HTTP 200 on the running server
-  - `/docs` (FastAPI auto-docs) accessible on the running server
-  - `/ws` endpoint not present; confirmed its absence is not a disqualifier per the official requirements
-- froze all Suyash-owned runtime files: `models.py`, `server/environment.py`, `server/app.py`, `server/reward.py`, `client.py`, `inference.py`, `openenv.yaml`, `server/Dockerfile`, `pyproject.toml`, `requirements.txt`
-
-Roopal-side work completed:
-
-- audited required submission files and confirmed they are present in the repo
-- completed a stale-claims and outdated-wording pass across the core docs
-- updated `required.md` to reflect that first-pass local execution is no longer the main runtime risk
-- left the remaining work focused on Docker and clean-machine validation rather than documentation cleanup
-
-## April 7, 2026
-
-Status: complete
-
-Suyash-side work completed:
-
-- performed clean-copy install-and-run pass from a fresh directory:
-  - installed with `pip install -r requirements.txt && pip install .` without errors
-  - verified all required files present and non-empty: `models.py`, `vocabulary.py`, `client.py`, `inference.py`, `server/app.py`, `server/environment.py`, `server/reward.py`, `server/grader.py`, `server/tasks.py`, `server/Dockerfile`, `openenv.yaml`, `pyproject.toml`, `requirements.txt`, `data/dataset.json`, `README.md`
-  - ran server and heuristic `inference.py` from the clean copy and confirmed clean completion
-  - confirmed benchmark numbers match the recorded reference: Task 1 `1.0000`, Task 2 `0.8800`, Task 3 `0.9400`, overall `0.9400`
-- confirmed feature freeze is in effect — no further additions to any Suyash-owned runtime file
-- applied freeze-phase doc and metadata corrections:
-  - fixed `ENV_URL` default in `inference.py` from `http://localhost:8000` to `http://localhost:7860`
-  - fixed local setup commands in `README.md` to use port `7860`
-  - removed unconfirmed `WebSocket /ws` row from the API surface table in `README.md`
-
-## April 3, 2026 (Pulled Forward April 4-5 Roopal Scope)
-
-Status: complete for the Roopal-owned roadmap items originally scheduled for April 4 and April 5
-
-Roopal-side work completed:
-
-- expanded `tests/test_grader_unit.py` to lock scorer crispness with exhaustive issue-type and priority-table checks
-- added explicit invariants for task-weight sums, exact-match dominance, and deterministic repeated grading
-- expanded `tests/test_tasks_unit.py` to cover the frozen task difficulty ladder plus dataset coverage across all issue types, priorities, assignment groups, and resolution actions
-- added `analysis/grounding_audit.md` as the internal grounding note requested by the roadmap
-- reviewed candidate issue-type similarity expansions and decided to keep the current similarity map unchanged
-
-Decision notes:
-
-- scorer fuzziness is now proven by tests to exist only where the declared similarity map or priority table allows it
-- no additional issue-type similarity pairs were adopted in this pass because the reviewed candidates were too operationally fuzzy
-
-## April 3, 2026 (Pulled Forward April 6-7 Roopal Scope)
-
-Status: complete for the Roopal-owned roadmap items originally scheduled for April 6 and April 7
-
-Roopal-side work completed:
-
-- added Hugging Face Spaces README frontmatter
-- updated `README.md` with an explicit judge-facing explanation of deterministic, grounded scoring
-- updated `KNOWLEDGE.md` to state clearly that the grader is not fuzzy by default and to reference the grounding audit
-- updated `required.md` with a current compliance snapshot separating already-satisfied requirements from shared pending validation gates
-- completed the final Roopal-side consistency pass across `README.md`, `KNOWLEDGE.md`, and `required.md`
-
-Decision notes:
-
-- no scorer change was needed from the grounding review, so this pass stayed documentation-only
-- the optional TRL / GRPO README example remains deferred until the shared runtime-validation gates are green
-
-## April 6 — Feature Freeze
-
-All Suyash-owned runtime files are now frozen. No new features will be added to:
-models.py, server/environment.py, server/app.py, server/reward.py, client.py,
-inference.py, openenv.yaml, server/Dockerfile, pyproject.toml, requirements.txt.
-
-Only bug fixes, doc corrections, and metadata updates are permitted after this point.
-
-Freeze confirmed: April 6, 2026.
-
-## April 7–8, 2026 — Freeze-Phase Doc and Metadata Corrections
-
-Status: complete
-
-Corrections applied during freeze phase (task 10.2):
-
-- Fixed `ENV_URL` default in `inference.py` from `http://localhost:8000` to `http://localhost:7860` to match the actual server port declared in `openenv.yaml`, `server/Dockerfile`, and `server/app.py`.
-- Fixed local setup commands in `README.md` to use port `7860` instead of `8000` (uvicorn start command and curl examples).
-- Fixed `ENV_URL` default value note in `README.md` to `http://localhost:7860`.
-- Removed unconfirmed `WebSocket /ws` row from the API surface table in `README.md`. The `/ws` endpoint is not listed in `openenv.yaml` api.endpoints and was not confirmed present during validation passes. Its absence is not a disqualifier per the April 6 deployment check.
-- Checked in `uv.lock` so the repo satisfies OpenEnv multi-mode deployment validation requirements on the current checkout.
-- Reran local `openenv validate` from the project virtualenv and confirmed the validator now passes.
-- Updated `README.md`, `KNOWLEDGE.md`, and `required.md` so they no longer describe the April 6 to April 7 roadmap items as pending.
-- Removed stale references to `bugs/BUGS_APRIL3.md` and kept the validation narrative self-contained inside `PROJECT_STATUS.md`.
-
-No runtime logic was changed. No new features were added. All other files checked (`openenv.yaml`, `pyproject.toml`, `requirements.txt`, `ROADMAP.md`) were found accurate and required no further corrections.
+The project is strong, but a few optional upgrades still exist if more time is ever available:
+
+- replace more authored queue rules with even more emergent simulator dynamics
+- grow the dataset further with less taxonomy-friendly wording
+- move from policy search toward a more clearly trainable learning setup
+- gather stronger benchmark comparisons against external LLM baselines
+
+## Repo Hygiene Notes
+
+This cleanup pass also keeps the repo focused by:
+
+- retaining `required.md` as the requirement checklist
+- keeping `README.md`, `KNOWLEDGE.md`, and `PROJECT_STATUS.md` as the main public guidance
+- removing stale planning and gap-analysis files that no longer reflect the current state

README.md

@@ -294,7 +294,7 @@ The grader is intentionally narrow and declared, not fully fuzzy.
 
 That scoring policy is now backed by checked-in unit tests in `tests/test_grader_unit.py` and `tests/test_tasks_unit.py`.
 
-The label set and partial-credit choices were also reviewed against public IT-support references captured in `analysis/grounding_audit.md`, including:
+The label set and partial-credit choices were also reviewed against public IT-support references during development, including:
 
 - `Classification of IT Support Tickets`
 - `Semantic Similarity of IT Support Tickets`
@@ -367,7 +367,7 @@ requirements.txt
 README.md
 KNOWLEDGE.md
 required.md
-ROADMAP.md
+PROJECT_STATUS.md
 ```
 
 ## Core Files
@@ -469,7 +469,7 @@ Current local smoke expectations:
 - rewards remain in range for every task
 - the hard task now depends much more heavily on investigation behavior, so exact seed-level baseline numbers are no longer treated as the benchmark reference for the repo
 
-The April 6 to April 7 validation pass then closed the remaining roadmap gates with Docker smoke coverage via GitHub Actions, a clean-copy install-and-run rerun, structured inference-log verification, and a passing local `openenv validate` check after checking in `uv.lock`.
+The April 6 to April 7 validation pass then closed the remaining validation gates with Docker smoke coverage via GitHub Actions, a clean-copy install-and-run rerun, structured inference-log verification, and a passing local `openenv validate` check after checking in `uv.lock`.
 
 ### Windows note
 
@@ -530,7 +530,7 @@ An April 6 repo audit also confirmed that all required submission files are pres
 
 - runtime: `models.py`, `client.py`, `inference.py`, `server/app.py`, `server/environment.py`, `server/grader.py`, `server/reward.py`, `server/tasks.py`
 - data and metadata: `data/dataset.json`, `openenv.yaml`, `pyproject.toml`, `requirements.txt`, `server/Dockerfile`
-- docs and planning: `README.md`, `KNOWLEDGE.md`, `required.md`, `PROJECT_STATUS.md`, `ROADMAP.md`
+- docs and project guidance: `README.md`, `KNOWLEDGE.md`, `required.md`, `PROJECT_STATUS.md`
 
 Roadmap status through April 7 is complete:
 
@@ -545,4 +545,4 @@ The remaining April 8 work is operational rather than implementation-heavy:
 - run the final submission-branch sanity slice before pushing
 - perform the live Hugging Face Space ping and reset check on the deployed submission artifact if a fresh deployment is created
 
-The short TRL / GRPO README example from the roadmap remains intentionally deferred because it is optional and lower priority than freeze-phase stability.
+The short TRL / GRPO README example remains intentionally deferred because it is optional and lower priority than benchmark clarity and stability.

ROADMAP.md

@@ -1,672 +0,0 @@
-# Hackstreet Boys Final Roadmap
-
-## Team
-
-- Team name: Hackstreet Boys
-- Members:
-  - Roopal Guha Neogi
-  - Suyash Kumar
-- Submission deadline: April 8, 2026, 11:59 PM IST
-
-## How To Use This File
-
-- `PROJECT_STATUS.md` is the canonical log of completed work.
-- This roadmap is the active plan from the verified April 6, 2026 repo state to final submission.
-- `required.md` is now the combined official-requirements and project-compliance file.
-- `KNOWLEDGE.md` defines the current repo truth and judge-facing explanation.
-- `analysis/competition_notes.md` is the merged internal competitive note. Use it to prioritize work, but do not mention competitor repos in public-facing docs.
-- The dated April 3 to April 5 sections below are now historical context; the active execution block is the final 24-hour plan for April 6 to April 7, 2026.
-
-## Status As Of April 6, 2026
-
-The repo is now in the expected "stabilize and merge" phase rather than the earlier "build core fixes" phase.
-
-Completed and locally verified:
-
-- all concrete items from `gaps.md`
-- the viable low-risk improvements from `analysis/deep_competitive_gap_report.md`
-- single-task `inference.py` execution with `TASK_ID` support and optional `RUN_ALL_TASKS=1`
-- `state()` exposure of `reward` and `done`
-- richer history with predicted actions and follow-up context
-- lightweight investigate-versus-submit action support with tool-backed context lookup
-- small queue-economics signal without major benchmark redesign
-- `/web` UI route
-- local full test pass:
-  - `126 passed, 137 subtests passed`
-- local validator pass:
-  - `[OK] meta-AIHack: Ready for multi-mode deployment`
-
-Merge recommendation:
-
-- mergeable as an incremental submission-ready improvement branch
-- do not block merge on major redesign items that were explicitly out of scope:
-  - scenario-family task redesign
-  - breaking the issue-type-to-assignment shortcut
-  - large dataset expansion
-  - full queue simulator / economics redesign
-
-## What We Are Optimizing For
-
-The highest-value wins from now to submission are:
-
-1. **Robustness**
-   - prove the env works through unit, smoke, and integration tests
-   - make Docker and clean reruns boring and reliable
-
-2. **RL improvement**
-   - keep the reward deterministic
-   - make sure scoring is not "always fuzzy"
-   - add only small, safe improvements that strengthen reward quality or episode usefulness
-
-3. **Real-world grounding**
-   - ground our taxonomy and partial-credit choices against real public support-ticket datasets
-   - do this as an audit / evidence layer, not as a late dataset merge
-
-4. **Submission readiness**
-   - satisfy every requirement from `required.md` and `KNOWLEDGE.md`
-   - keep the repo easy for judges to understand and rerun
-
-## Current Repo State
-
-The repo already has:
-
-- locked IT helpdesk routing domain
-- locked vocabulary and task names
-- 3-task difficulty ladder
-- deterministic grading with limited partial credit
-- working heuristic baseline
-- merged local validation on `/health`, `/tasks`, and `inference.py`
-- single-task evaluator-safe inference behavior
-- reward and done fields on `state()`
-- richer observation history and linked-ticket context
-- lightweight investigate / submit split with small built-in tool support
-- local full-suite verification:
-  - `126 passed, 137 subtests passed`
-- local validator verification:
-  - `[OK] meta-AIHack: Ready for multi-mode deployment`
-
-The remaining work should be treated as targeted strengthening, not broad feature invention.
-
-## Final 24-Hour Plan
-
-**Active window:** April 6 to April 7, 2026  
-**Internal target:** open PR, merge to the common `main`, and complete the final smoke checks by April 7, 2026  
-**Official deadline:** April 8, 2026, 11:59 PM IST
-
-### Must finish before merge
-
-- review the final diff and stage only the intended submission files
-- open the merge PR from a dedicated branch
-- merge into the shared `main` after one last reviewer pass
-- rerun the post-merge smoke checks:
-  - `pytest`
-  - `openenv validate`
-  - `/health`
-  - `/tasks`
-  - one `reset()` / `step()` sanity path
-
-### Do not add before merge
-
-- no new benchmark redesign work
-- no new dataset expansion
-- no schema churn
-- no reward refactors beyond blocker-level fixes
-- no last-minute inference prompt rewrites
-
-### Success condition for April 7, 2026
-
-- PR is up
-- PR is reviewed against `gaps.md` and `analysis/deep_competitive_gap_report.md`
-- shared `main` contains the tested gap-fix branch
-- deployment sanity checks are green
-- repo is frozen except for typo-level fixes
-
-## Submission Gates That Must Still Hold
-
-These come directly from `required.md` and `KNOWLEDGE.md`:
-
-- the environment starts correctly
-- `reset()`, `step()`, and `state()` behave correctly
-- 3 tasks exist and remain meaningfully different
-- grader scores stay in `[0.0, 1.0]`
-- `inference.py` runs reproducibly without crashing
-- `inference.py` uses the OpenAI client with `API_BASE_URL`, `MODEL_NAME`, and the evaluator-injected `API_KEY` (`HF_TOKEN` remains a local fallback)
-- structured stdout logs follow the official `[START]`, `[STEP]`, and `[END]` format
-- `openenv validate` passes
-- Docker builds and starts cleanly
-- HF deployment responds cleanly and reset works
-- inference stays inside the official runtime / machine envelope
-- docs and metadata are current
-- the repo is easy for judges to understand and rerun
-
-## Scope Decisions
-
-### Do Now
-
-- add tests:
-  - unit
-  - smoke
-  - integration
-- prove the scorer is crisp where it should be crisp
-- add only safe RL-oriented improvements
-- add external grounding evidence without changing the runtime dataset
-- finish packaging / deployment readiness
-- verify official validation constraints, not just local happy-path behavior
-
-### Do Not Do Before Submission
-
-- MCP migration
-- transform-based reward refactor
-- large dataset expansion
-- external dataset merge into `data/dataset.json`
-- major schema changes
-- broad prompt / inference rewrites that could disturb the stable baseline
-- dependency churn just for polish
-
-## Codex-First Working Rules
-
-Because we are using Codex to generate code, we should optimize for small, bounded tasks:
-
-1. one prompt = one scoped change set
-2. keep ownership by file group
-3. require tests for any scorer or runtime change
-4. review the diff before accepting generated code
-5. rerun the relevant test slice after each meaningful change
-6. do not ask Codex for a giant multi-file redesign this late
-
-## Phased Plan
-
-## Phase 1: Test And Robustness Foundation
-
-**Window:** April 3 to April 4
-
-**Goal:** eliminate the biggest competitive weakness identified in `analysis/competition_notes.md`: lack of checked-in tests.
-
-### Must produce
-
-- `tests/` with at least:
-  - grader unit tests
-  - task / dataset loader unit tests
-  - reward / score-range unit tests
-  - environment smoke tests
-  - API integration tests
-
-### Test plan
-
-#### Unit tests
-
-- exact match gives `1.0`
-- unsupported task IDs fail clearly
-- only intended near-miss issue-type pairs get partial credit
-- unrelated wrong issue types get `0.0`
-- priority proximity rules behave exactly as defined
-- assignment group and resolution action remain exact-match only
-- task weights sum and apply correctly
-- dataset loads cleanly with `utf-8-sig`
-
-#### Smoke tests
-
-- `reset()` returns a valid observation
-- `step()` advances queue progress
-- `state()` reflects runtime state
-- seeded resets are deterministic
-- scores remain in `[0.0, 1.0]`
-- one full episode per task completes without errors
-
-#### Integration tests
-
-- `/health`
-- `/tasks`
-- `/reset`
-- `/step`
-- `/state`
-- one end-to-end seeded episode over HTTP or client path
-- one heuristic `inference.py` regression check on expected overall behavior
-
-### Why this phase matters
-
-- addresses the biggest repo-quality gap vs stronger competitors
-- improves robustness
-- gives us safe rails for all later RL and grounding changes
-
-## Phase 2: Scoring Calibration And Safe RL Improvements
-
-**Window:** April 4 to April 5
-
-**Goal:** improve RL usefulness without destabilizing the submission.
-
-### Must produce
-
-- scorer calibration evidence that the system is not "always fuzzy"
-- only a few safe RL-oriented improvements if tests stay green
-
-### Required calibration checks
-
-- exact-match path is dominant and clearly tested
-- fuzziness exists only in explicitly defined cases
-- wrong labels outside the similarity map score `0.0`
-- assignment group and resolution action remain exact
-- final episode reward stays bounded and deterministic
-
-### Safe improvement candidates from `analysis/competition_notes.md`
-
-- expand `ISSUE_TYPE_SIMILARITY` with only a few defensible pairs, if backed by grounding review
-- enrich `history` with:
-  - ticket title
-  - predicted fields
-- optionally support `queue_size` as a reset kwarg only if the change is tiny and fully tested
-
-### Hard stop
-
-- if a change touches behavior and shifts baseline numbers unexpectedly, stop and stabilize rather than stacking more changes
-
-## Phase 3: Real-World Grounding Audit
-
-**Window:** April 5 to April 6
-
-**Goal:** add defensible evidence that our taxonomy and partial-credit logic are grounded in real support data, without merging external data into runtime.
-
-### Grounding strategy
-
-- use real public support datasets as reference material
-- compare their labels / examples against our taxonomy
-- create an internal audit, not a runtime dependency
-
-### Recommended grounding references
-
-- `Classification of IT Support Tickets` (Zenodo, 2,229 manually classified tickets)
-- `Semantic Similarity of IT Support Tickets` (Zenodo, 300 manually labeled ticket pairs)
-- `MSDialog` for real technical-support conversation patterns and terminology
-
-### Must produce
-
-- an internal grounding note or checklist that captures:
-  - which public datasets were reviewed
-  - how our labels map to real-world ticket themes
-  - which partial-credit pairs are defensible
-  - which proposed similarity pairs were rejected as too fuzzy
-
-### Useful output
-
-- 10 to 20 grounding examples:
-  - real ticket theme
-  - closest label in our taxonomy
-  - whether it should be exact-match only or partial-credit-adjacent
-
-### Why this phase matters
-
-- strengthens real-world credibility
-- supports RL reward quality with evidence
-- helps avoid arbitrary or over-fuzzy scorer changes
-
-## Phase 4: Packaging, Deployment, And Judge-Facing Polish
-
-**Window:** April 6 to April 7
-
-**Goal:** close the submission-readiness gaps surfaced in `analysis/competition_notes.md`.
-
-### Must produce
-
-- Hugging Face Spaces README frontmatter
-- `.openenvignore`
-- `openenv validate` evidence
-- Docker smoke evidence on the merged branch
-- one clean-copy rerun if possible
-- structured inference logging verified against the official format
-- a practical check that inference remains inside the official runtime envelope
-
-### Nice-to-have only if green
-
-- short TRL / GRPO example in `README.md`
-- concise note in docs that grading is deterministic, partially structured, and not purely fuzzy
-
-### Do not do here
-
-- no dataset expansion
-- no major inference rewrite
-- no architecture refactor
-
-## Phase 5: Freeze And Submit
-
-**Window:** April 8
-
-**Goal:** submit from a calm, validated repo state.
-
-### Final day rules
-
-- only typo-level, doc-level, or packaging-only fixes
-- no risky scorer changes
-- no runtime refactors
-- no dataset edits unless they fix a blocker
-- stop risky edits several hours before submission
-- if possible, run the official validator or the closest local equivalent before final push
-
-## Ownership From Now Until Submission
-
-### Roopal ownership
-
-Primary files:
-
-- `data/dataset.json`
-- `server/tasks.py`
-- `server/grader.py`
-- `README.md`
-- `KNOWLEDGE.md`
-
-Primary responsibilities:
-
-- scorer calibration and label quality
-- unit tests around grader / task rules / dataset invariants
-- real-world grounding audit
-- judge-facing explanation of deterministic scoring and real-world realism
-- safe reward-quality improvements only when grounded and tested
-
-Concrete deliverables:
-
-- grader unit tests
-- grounding mapping note
-- any similarity-matrix update, if justified
-- doc updates if benchmark numbers or scoring explanation change
-- README frontmatter and judge-facing clarity
-- official requirement compliance review through `required.md`
-
-### Suyash ownership
-
-Primary files:
-
-- `models.py`
-- `server/environment.py`
-- `server/app.py`
-- `server/reward.py`
-- `client.py`
-- `inference.py`
-- `openenv.yaml`
-- `server/Dockerfile`
-- `pyproject.toml`
-- `requirements.txt`
-
-Primary responsibilities:
-
-- smoke and integration tests
-- runtime stability
-- Docker and deployment readiness
-- inference reproducibility
-- clean rerun evidence
-- optional small RL-signal improvements on the runtime side
-
-Concrete deliverables:
-
-- env smoke tests
-- API integration tests
-- heuristic inference regression path
-- `.openenvignore`
-- Docker smoke confirmation
-- clean-copy rerun if possible
-- structured inference logging compliance
-
-### Shared responsibilities
-
-- do not rename schemas or vocabulary
-- rerun the benchmark after any behavior-affecting change
-- keep `PROJECT_STATUS.md` honest
-- use the GitHub Actions Docker smoke workflow when local Docker is blocked
-- review Codex-generated diffs before accepting them
-- freeze feature work by the end of April 7
-- do not casually change the `[START]`, `[STEP]`, `[END]` inference log format once implemented
-
-## Date-By-Date Execution Plan
-
-## April 3, 2026
-
-Primary goal:
-
-- lock the execution plan and begin test scaffolding immediately
-
-Roopal:
-
-- finalize the exact scorer behaviors that must be proven by tests
-- list the exact-match-only cases and intended partial-credit cases
-- begin grader and task-loader unit tests
-
-Suyash:
-
-- scaffold `tests/`
-- begin smoke tests for `reset()`, `step()`, `state()`, and deterministic seeded behavior
-- confirm how integration tests will hit the app cleanly
-- review `required.md` and identify the exact official validation items still not reflected in runtime / inference behavior
-
-Shared checkpoint:
-
-- test strategy is agreed
-- file ownership is clear
-- no one is making unscoped runtime changes yet
-
-## April 4, 2026
-
-Primary goal:
-
-- land the first complete test layer
-
-Roopal:
-
-- complete grader, task, and dataset unit tests
-- add explicit tests showing where fuzziness is allowed and where it is not
-
-Suyash:
-
-- complete smoke tests
-- add first-pass integration tests for `/health`, `/tasks`, `/reset`, and `/step`
-- begin checking how current `inference.py` differs from the official structured logging requirement
-
-Shared checkpoint:
-
-- checked-in tests exist
-- the repo can prove deterministic scoring and score bounds
-- any failing behavior is triaged before adding improvements
-
-## April 5, 2026
-
-Primary goal:
-
-- improve RL usefulness safely
-
-Roopal:
-
-- start the grounding audit using the selected public datasets
-- decide whether any additional similarity pairs are truly defensible
-
-Suyash:
-
-- add integration coverage for full seeded episode flow and `state()`
-- add a light heuristic regression path for `inference.py`
-- optionally enrich observation history if tests are already green
-- bring `inference.py` closer to official structured logging format if the change can be done safely
-
-Shared checkpoint:
-
-- tests are stable
-- any RL-oriented change is small and justified
-- no baseline drift goes unexplained
-
-## April 6, 2026
-
-Primary goal:
-
-- finish grounding evidence and close packaging gaps
-
-Roopal:
-
-- finish grounding audit note
-- land only the scorer adjustments supported by audit evidence, if any
-- update docs to reflect deterministic, grounded scoring
-
-Suyash:
-
-- add `.openenvignore`
-- verify Docker smoke workflow on the merged branch
-- check deployment assumptions around `app_port`, `/docs`, `/health`, `/ws`, and `/web`
-- run `openenv validate` or the closest available validation path
-- verify structured inference logging and runtime-envelope expectations
-
-Shared checkpoint:
-
-- grounding evidence exists
-- packaging gaps are closed or explicitly blocked
-- benchmark references are still current
-
-## April 7, 2026
-
-Primary goal:
-
-- freeze on a green, submission-ready repo
-
-Roopal:
-
-- final docs consistency pass across `README.md` and `KNOWLEDGE.md`
-- add a short TRL / GRPO usage example only if everything else is already green
-
-Suyash:
-
-- do a clean-copy install-and-run pass if possible
-- rerun heuristic baseline if any runtime-side change landed
-- freeze runtime files by end of day
-
-Shared checkpoint:
-
-- tests are green
-- Docker evidence exists
-- docs, metadata, and runtime tell the same story
-- feature work stops unless the gated competitive-hardening window below is explicitly activated after all required checks are already green
-
-## After April 7 If Green: Competitive Hardening Window
-
-**Window:** late April 7 to early April 8 only if all required gates are already green
-
-**Goal:** improve the repo's competitive position against the strongest submissions by winning on reliability, validation quality, RL usefulness, and judge readability rather than by trying to match their architecture complexity.
-
-### Activation rule
-
-Activate this block only if all of the following are already true:
-
-- smoke, unit, and integration tests are green
-- Docker evidence exists or the blocker is clearly external
-- `openenv validate` has passed or the closest available validator path is already recorded
-- structured inference logging is already compliant or one tiny remaining fix is clearly isolated
-- the benchmark is stable and any behavior-changing diff can still be rerun safely
-
-If any of those are not true, skip this entire block and proceed directly to freeze / submission.
-
-### Allowed competitive upgrades
-
-- strengthen validation proof:
-  - add or tighten environment smoke tests
-  - add or tighten API integration tests
-  - add one lightweight heuristic regression check for `inference.py`
-- strengthen deployment proof:
-  - record `openenv validate` evidence
-  - record Docker smoke evidence
-  - record deployment-assumption checks for `app_port`, `/health`, `/docs`, `/ws`, and `/web`
-  - record one clean-copy rerun if practical
-- add only tiny RL-signal improvements if fully tested and benchmark-stable:
-  - enrich `history` with ticket title and predicted fields
-  - add `queue_size` as a reset kwarg only if the change remains small, bounded, and fully tested
-- add final judge-facing polish only after runtime proof is green:
-  - short TRL / GRPO README example
-  - concise README note on why our dense deterministic reward is more RL-friendly than binary-only grading
-
-### Hard limits
-
-- do not add MCP
-- do not add a simulator layer
-- do not add browser or multimodal features
-- do not expand the runtime dataset
-- do not make broad inference rewrites
-- do not stack multiple behavior changes without rerunning the benchmark
-
-### Decision rule
-
-- if a competitive-hardening change is tiny, tested, and clearly improves trust or judge readability, it is allowed
-- if it adds architectural ambition at the expense of stability, skip it
-- if it causes unexplained baseline drift, revert to the last green state and submit
-
-### Ownership
-
-Roopal:
-
-- final judge-facing README / KNOWLEDGE / `required.md` polish
-- RL-justification wording around deterministic partial credit
-- TRL / GRPO example only after all runtime proof is green
-
-Suyash:
-
-- validation evidence
-- deployment proof
-- tiny runtime-side RL-signal improvements only if fully tested
-
-Shared checkpoint:
-
-- the repo is already submission-safe before this block starts
-- every change in this block is optional
-- if time gets tight, cut this whole block first
-
-## April 8, 2026
-
-Primary goal:
-
-- submit early from a calm repo state
-
-Morning:
-
-- if the repo is already fully green, optionally activate the competitive-hardening window above for one last small, tested improvement
-- run final smoke / test slice on the submission branch
-- verify required files are present
-- verify README and metadata are current
-- run the final validation checklist from `required.md`
-
-Afternoon:
-
-- only typo-level or packaging-only fixes
-- no risky code changes
-
-Final rule:
-
-- stop risky edits several hours before 11:59 PM IST
-- submit as soon as the repo is clearly green
-
-## Cut Order If Time Gets Tight
-
-Cut these first:
-
-1. the entire competitive-hardening window after April 7
-1. `queue_size` reset kwarg
-2. richer `history`
-3. TRL / GRPO README example
-4. any optional similarity expansion beyond the most defensible cases
-
-Do not cut these:
-
-1. tests
-2. scorer crispness checks
-3. Docker / deployment validation
-4. grounding audit evidence
-5. final benchmark sanity rerun if behavior changed
-6. official structured inference logging compliance
-
-## Definition Of Done
-
-The project is ready when:
-
-1. unit, smoke, and integration tests exist and cover the critical paths
-2. scoring is demonstrably deterministic and not fuzzy by default
-3. a grounding audit against real public support datasets exists
-4. the heuristic baseline still runs successfully
-5. the inference path is compliant with the official log format
-6. `openenv validate` and Docker checks are validated
-7. docs and metadata are current and judge-friendly
-8. the repo is frozen and submitted on time
-
-## Simple Rule To Remember
-
-Roopal owns the labels, scoring truth, grounding, and public clarity.
-Suyash owns the runtime, tests beyond unit scope, packaging, and reproducibility rails.
-Both of you should optimize for a clean, defensible, rerunnable submission rather than last-minute complexity.

analysis/competition_notes.md

@@ -1,87 +0,0 @@
-# Competition Notes
-
-> Internal-only competitive positioning and late-stage prioritization note.
-> Do not cite competitor repos in public-facing docs.
-
-## Summary
-
-Our strongest comparative advantages are:
-
-- a clear 3-task easy-to-hard ladder
-- deterministic, dense partial-credit reward
-- compact judge-friendly architecture
-- a strong heuristic baseline
-
-The strongest external competitor pattern is higher simulator depth or broader architecture ambition, especially in long-horizon environments. Our best response is reliability and clarity, not late complexity.
-
-## What Matters Most
-
-Judges are most likely to reward:
-
-1. correctness and rerunnability
-2. real-world domain quality
-3. task and grader quality
-4. reward usefulness for RL
-5. clean packaging and deployment
-6. baseline reproducibility
-
-## Key Competitive Read
-
-### Where we are strong
-
-- helpdesk routing is a real enterprise workflow
-- the task ladder is explicit and curriculum-friendly
-- dense deterministic scoring is more RL-friendly than binary-only grading
-- the repo is easier for judges to understand quickly than heavier simulator-style projects
-
-### Where strong competitors can beat us
-
-- simulator depth and richer state
-- long-horizon control realism
-- larger datasets or generated scenario breadth
-- broader tooling such as MCP integrations
-
-## Priority Responses
-
-The highest-value late-stage moves are:
-
-1. strengthen validation proof
-2. keep scorer crispness explicit and tested
-3. document grounded scoring clearly
-4. prove Docker and validator readiness
-5. avoid architecture churn
-
-## Late-Stage Rules
-
-- do not add MCP
-- do not do a reward-architecture refactor
-- do not expand the runtime dataset late
-- do not make broad inference changes
-- only add tiny RL-signal improvements if fully tested and benchmark-stable
-
-## Practical Action List
-
-### Must keep
-
-- unit, smoke, and integration tests
-- scorer crispness checks
-- grounding audit evidence
-- Docker smoke proof
-- `openenv validate` readiness
-- clean judge-facing docs
-
-### Nice to have only if fully green
-
-- richer history fields
-- `queue_size` reset kwarg
-- short TRL / GRPO README example
-
-## Competitor Snapshot
-
-The field includes:
-
-- simple reference environments that we clearly outperform on realism
-- strong but binary-reward environments where we win on RL signal quality
-- ambitious simulator-style environments that win on technical scope but are harder to judge quickly
-
-Our best positioning is not "most complex"; it is "most defensible, trainable, and rerunnable."

analysis/deep_competitive_gap_report.md

@@ -1,1374 +0,0 @@
-# Deep Codebase Comparison: OpenEnv Reference Environments vs This Helpdesk Project
-
-## Scope and Method
-
-This report was written from a direct code read, not from README-driven interpretation. I treated the `OpenEnv/envs` directory as the reference baseline you pointed to, and I compared it against the implementation that lives in this repository root plus `server/`.
-
-I focused on code that actually defines runtime behavior:
-
-- `models.py`
-- `inference.py`
-- `client.py`
-- `vocabulary.py`
-- `server/environment.py`
-- `server/tasks.py`
-- `server/grader.py`
-- `server/reward.py`
-- `server/app.py`
-- `tests/*.py`
-- `data/dataset.json`
-
-
-That reading set is enough to answer the question that matters: what design moves make the strongest reference environments hard to beat, where your project is currently thinner than it looks, and what concrete changes would make your environment competitive instead of merely correct.
-
-## Executive Verdict
-
-Your project is a clean, readable, deterministic mini-benchmark. It is not yet a high-ceiling agent benchmark.
-
-That sounds harsh, but it is also the clearest way to unlock the right next move. Right now your environment behaves much more like a structured multi-label classification task wrapped in OpenEnv than like the richer reference environments that expose hidden state, tool use, long-horizon consequences, multi-step reasoning, or grounded interaction with external systems. The code is good enough as a starter environment. It is not yet strong enough to beat the best reference projects on depth, realism, or benchmark credibility.
-
-The good news is that the codebase is small, coherent, and fixable. The bad news is that the gap is not a one-line polish gap. It is a benchmark design gap.
-
-The strongest OpenEnv reference environments win for one or more of these reasons:
-
-- they expose a real action surface, not just label prediction
-- they make the agent inspect state rather than infer everything from one text blob
-- they reward process, not only end labels
-- they support long-horizon or multi-step behavior
-- they are harder to brute-force with dataset-specific heuristics
-- they are backed by real engines, shells, browsers, tools, or stateful simulators
-- they treat evaluation as a first-class system, not as a tiny helper function
-
-Your project currently loses on most of those axes.
-
-At the same time, your project has an underrated advantage: the domain is practical, legible, and product-shaped. IT helpdesk routing is a great benchmark domain if you push it harder. It naturally supports ambiguity, policy lookup, account context, queue optimization, escalation rules, duplicates, follow-up chains, customer sentiment, service health, SLA clocks, and partial observability. In other words, the domain is better than the current implementation. The environment has room to grow into something much stronger without abandoning the idea.
-
-So the answer is not “throw this away and copy BrowserGym.” The answer is “turn this from a label benchmark into a realistic triage operations environment.”
-
-## What the Reference Environments Actually Do Better
-
-### 1. They expose richer action spaces
-
-The single biggest difference between your code and the strongest reference projects is that the agent in your environment does very little. In your environment, the step is basically “predict some labels for this ticket.” In the stronger reference environments, the agent interacts.
-
-`BrowserGymEnvironment` accepts an `action_str` and pushes it into a live browser benchmark. That means the benchmark difficulty comes from action selection in stateful UI space, not just from text classification. `OpenAppEnvironment` similarly supports `click`, `fill`, `select_option`, `goto`, `scroll`, and `send_keys`, and even mixes BrowserGym-style element IDs with raw Playwright CSS selectors for pragmatic reliability. `GitTaskEnvironment` supports clone, list, and git command execution against a Gitea-backed workspace. `Tbench2Environment` supports `exec`, `write`, `view`, `wait`, `kill`, `write_file`, and `evaluate`, which is much closer to real agent work. `FinQAEnvironment` turns the task into tool use over tables, SQL, and answer submission. `REPLEnvironment` exposes code execution with optional recursive LLM calls. `TextArenaEnvironment` takes natural-language moves and advances a game engine.
-
-Your environment exposes none of that. The agent does not gather missing evidence. It does not inspect a related ticket. It does not search a KB. It does not look up account tier. It does not check service health. It does not add an internal note. It does not choose between acknowledging first and escalating later. It does not defer. It does not ask for more information. It does not resolve duplicates. It does not manage a queue. It only emits one shot structured output.
-
-That makes the benchmark much easier to game, much easier to overfit, and much less diagnostic of real agent competence.
-
-### 2. They separate visible observation from hidden truth
-
-The strongest reference environments keep some truth state behind the curtain. The agent sees an observation. The environment owns more. That separation is what makes an environment feel like an environment instead of a dataframe with reward labels.
-
-In `ChessEnvironment`, the agent observes legal moves, FEN, checks, and result state, but the environment owns board progression, opponent strategy, and trajectory reward accumulation. In `MazeEnvironment`, the environment tracks maze status and legal movement dynamics. In `TextArenaEnvironment`, the wrapped engine owns turn state, raw logs, rewards, role mapping, and step info. In `FinQAEnvironment`, the agent sees the question and tools, but the hidden ground truth answer, question identity, and full structured table data live behind the environment. In `Tbench2Environment`, the hidden truth is in the task files and tests. In `BrowserGymEnvironment`, the browser session and benchmark internals are hidden behind the observation.
-
-Your environment has much less hidden truth than it should. The ticket label is hidden, yes, but the benchmark structure is shallow. More importantly, the code already hints at richer hidden structure and then fails to expose or exploit it. `HelpdeskTicketRecord` includes `ambiguity_note` and `related_ticket_id`, but `_build_observation()` throws both away and only exposes `ticket_id`, `title`, `requester`, and `description`. So even though the dataset contains follow-up relationships and ambiguity annotations, the environment does not actually let the agent work with them as structured state. That is a missed opportunity and a design leak at the same time.
-
-The dataset is telling you the domain wants threads, ambiguity, and context. The environment currently flattens it back into plain text.
-
-### 3. They reward more than a final label match
-
-The reference environments do not all have brilliant reward design, but the best ones take reward seriously.
-
-`REPLEnvironment` combines an outcome rubric with optional process reward. It can reward successful execution, penalize failures, and separately judge the final answer. `ChessEnvironment` uses a trajectory rubric with exponential discounting to assign credit across a game. `FinQAEnvironment` does robust answer normalization, including boxed answers, percentages, fractions, and multi-value comparisons. `TextArenaEnvironment` overlays auxiliary reward signals such as Wordle greens, yellows, repetitions, and correctness. `Tbench2Environment` evaluates by actually running tests, which is a grounded form of outcome reward.
-
-Your reward design is better than “exact match only,” but it is still thin. `grade_action()` uses one handcrafted issue similarity table, one handcrafted priority proximity table, and exact match for assignment group and resolution action. `compute_step_reward()` is just clamping. `compute_trajectory_reward()` averages scores and subtracts an overshoot penalty.
-
-That sounds reasonable until you inspect the runtime path. In practice, the overshoot penalty is effectively dead logic. `step()` increments the ticket index once per ticket and sets done when the index reaches queue length. A later `step()` call raises an error. That means `steps_taken` cannot exceed `queue_size` during normal episode execution, so the overshoot branch in `compute_trajectory_reward()` has no meaningful role in the current environment. The code suggests the benchmark penalizes wasteful action loops, but the environment does not actually allow them.
-
-The deeper issue is that the reward judges only final fields, not triage quality as a process. There is no penalty for unnecessary escalation unless the final field is wrong. There is no reward for correctly identifying a duplicate and linking it. There is no cost model for routing everything to security “just in case.” There is no SLA-aware penalty for under-prioritizing a time-sensitive issue that still happens to hit some partial-credit similarity. There is no queue-level reward. There is no explanation consistency. There is no tool efficiency score because there are no tools. There is no notion of customer harm, resolver cost, escalation burden, or backlog impact.
-
-The strongest environments earn their credibility by making reward a modeling decision. Your reward is still a convenience function.
-
-### 4. They support multi-step or long-horizon behavior
-
-Even the simpler reference environments tend to have longer horizon than your three-task ladder suggests.
-
-`ChessEnvironment` is naturally long horizon. `BrowserGymEnvironment` and `OpenAppEnvironment` are stepwise interactions. `TextArenaEnvironment` proceeds over turns. `Tbench2Environment` supports iterative shell work and explicit evaluation. `REPLEnvironment` supports repeated code execution over an evolving namespace. `FinQAEnvironment` allows repeated tool calls up to `max_steps` before submission. Even `ReasoningGymEnvironment`, which is single-step, supports parameterized dataset generation and configurable tasks.
-
-Your environment has multiple steps inside an episode, but they are just a queue of independent tickets. Each step is still one-shot labeling. Tickets do not affect each other. The queue order does not matter. There is no resource constraint. There is no carry-over state except a score list and counters. No later ticket depends on an earlier action. No policy evolves over the episode. No investigation outcome from step one informs step two.
-
-So while the environment is technically episodic, it is not operationally long horizon. It is batching.
-
-That difference matters. The best agents and best benchmarks separate “can classify one item” from “can operate over a process.” Right now your environment mainly measures the first.
-
-### 5. They parameterize tasks rather than freezing one tiny benchmark
-
-`ReasoningGymEnvironment` rebuilds datasets from `dataset_name`, `dataset_config`, `dataset_specs`, `seed`, and `size`. `BrowserGymEnvironment` can choose a benchmark and task. `Tbench2Environment` can resolve tasks by task ID or path, even downloading a repo cache if needed. `GitTaskEnvironment` supports task-specific base repo states. `REPLEnvironment` can accept context, task prompt, expected answer, recursion depth, and model parameters at reset. `FinQAEnvironment` iterates over a question bank with real data-backed tools.
-
-Your environment has three tasks, but they are not truly different environments. They are the same tickets with a different subset of fields exposed through `allowed_fields`. That is a very weak notion of task diversity. Task difficulty is not created by different data generating processes, different hidden state, different workflows, or different action surfaces. It is created by output dimensionality alone.
-
-That means the easy, medium, and hard tasks are less like three tasks and more like one task with three scoring schemas.
-
-### 6. They take concurrency and runtime isolation seriously
-
-Several reference environments explicitly set `SUPPORTS_CONCURRENT_SESSIONS = True`, including `REPLEnvironment`, `Tbench2Environment`, `ReasoningGymEnvironment`, `MazeEnvironment`, and some others. The framework core in `http_server.py` is built around WebSocket sessions, session capacity, session info, session factories, and asynchronous handling. `MCPEnvironment` has explicit async and sync step paths because the framework authors ran into real event-loop and deadlock issues. `Tbench2DockerEnvironment` handles Docker-in-Docker by copying task directories into containers rather than assuming host bind mounts. `Calendar` builds database sessions per tenant. `GitTaskEnvironment` assumes isolated workspaces. `BrowserGymEnvironment` does cleanup of resources.
-
-Your environment inherits some capability from OpenEnv, but your own code does not actually engage with that depth. The server is mostly a minimal `create_app()` call plus a `/tasks` endpoint. There is no custom metadata. No custom concurrency choices. No session isolation logic beyond what the base server gives you. No runtime cleanup concerns because the environment owns almost no external resources. That simplicity is pleasant, but it also means the project is not stress-tested as a real environment service.
-
-### 7. They integrate grounded external systems or simulators
-
-This is where the biggest credibility gap appears.
-
-`FinQAEnvironment` grounds answers in company tables and SQL. `GitTaskEnvironment` grounds tasks in actual repositories. `Tbench2Environment` grounds them in actual shell execution and tests. `BrowserGymEnvironment` grounds tasks in web environments. `TextArenaEnvironment` grounds them in game engines. `ChessEnvironment` grounds them in a real board state. `Calendar` grounds them in a stateful API-backed application.
-
-Your environment is grounded in a JSON dataset. That is fine for a prototype, but it is dramatically easier to shortcut. If the environment does not provide tools, latent objects, or stateful consequences, the fastest route to a good score is to learn the labeling policy over the text. That is exactly what your current `inference.py` is doing.
-
-If you want to beat more ambitious projects, you need to force the agent to do more than map n-grams to labels.
-
-## Deep Audit of Your Current Project
-
-### Overall strengths before the critique
-
-Before I get more surgical, it is worth naming what is already good:
-
-- The codebase is small enough to understand quickly.
-- The naming is clear and the domain is coherent.
-- Pydantic validation is used correctly in the core models.
-- The taxonomy in `vocabulary.py` is readable and operational.
-- The environment is deterministic given a seed.
-- The three-task ladder is a decent pedagogical introduction.
-- The tests, while limited, are not absent.
-- The dataset has at least some intentional ambiguity and follow-up cases.
-
-So this is not a bad project. It is a project that has not yet converted a good domain into a hard benchmark.
-
-### Domain model and task structure
-
-`vocabulary.py` defines a clean label space:
-
-- 9 issue types
-- 4 priorities
-- 6 assignment groups
-- 5 resolution actions
-- 3 task IDs
-
-The mapping dictionaries immediately reveal one important structural weakness: assignment group is fully determined by issue type. Every issue type maps to exactly one assignment group. That means the “assignment_group” prediction in task 3 is not an independent reasoning problem. Once the model gets issue type right, assignment group is a lookup. That collapses the apparent complexity of the hardest task.
-
-The same problem exists, though less absolutely, for resolution action. `ISSUE_TYPE_TO_RESOLUTION_ACTION` already maps every issue type to a default resolution action. The dataset confirms that several issue types only ever use one resolution action:
-
-- `feature_request -> acknowledge`
-- `general_inquiry -> acknowledge`
-- `onboarding -> fulfill`
-- `service_request -> assign`
-- `spam_phishing -> ignore`
-
-Only a subset of issue types vary their resolution action in practice. So task 3 looks like a four-field prediction problem, but much of it is structurally reducible to issue type plus a few keyword exceptions. That is not how hard triage environments should work if the goal is to test agentic reasoning.
-
-`server/tasks.py` compounds this by defining difficulty purely as output field count:
-
-- Task 1: issue type only
-- Task 2: issue type plus priority
-- Task 3: full routing
-
-The ticket pool is the same across tasks. There is no task-specific curation, no task-family-specific observation, no different process constraints, and no different control surface. The only thing that changes is what the grader will read from the submitted action.
-
-That means your easy-medium-hard ladder is mostly a scoring ladder, not an environment ladder.
-
-### Observation and state design
-
-`HelpdeskTicketObservation` contains:
-
-- task metadata
-- `allowed_fields`
-- `current_ticket`
-- queue counts
-- history
-
-`current_ticket` exposes only:
-
-- `ticket_id`
-- `title`
-- `requester`
-- `description`
-
-This is too little for a benchmark that wants to simulate real helpdesk operations, and it is oddly little given what your data already stores. `HelpdeskTicketRecord` also includes:
-
-- `ambiguity_note`
-- `related_ticket_id`
-
-Those two fields are exactly the sort of structured hints that could turn this from flat classification into contextual triage. Yet `_build_observation()` discards them. That means the dataset contains richer structure than the observation contract.
-
-The state is also minimal:
-
-- `current_task_id`
-- `seed`
-- `queue_ticket_ids`
-- `current_ticket_index`
-- `per_ticket_scores`
-- `total_reward`
-
-This is enough for bookkeeping, but not enough for operational simulation. There is no notion of:
-
-- queue ordering rationale
-- account status
-- customer tier
-- outage context
-- prior communication attempts
-- internal notes
-- pending escalations
-- workload or resolver capacity
-- elapsed time or SLA timers
-- deduplication chains
-- partial investigation state
-
-The result is that the environment never becomes more informative or more demanding as the episode progresses. The state is a score ledger, not a world model.
-
-Compare that with the stronger references:
-
-- `BrowserGymState` tracks benchmark, task, URL, goal, max steps, cumulative reward.
-- `REPLState` tracks context, prompt, iteration, namespace keys, final answer, total execution time.
-- `Tbench2State` tracks task, session, command history, terminal readiness, last output.
-- `TextArenaState` tracks turn, raw state, last reward, last info, environment identity.
-- `FinQAState` tracks current question, company, ground truth, question ID.
-
-Those states are not just counters. They represent the environment’s evolving operational memory. Yours mostly does not.
-
-### Environment lifecycle
-
-`HelpdeskTicketRoutingEnvironment.reset()` is straightforward:
-
-- coerce `seed`
-- get task definition
-- seed RNG
-- sample a queue size from 3 to 5
-- sample that many tickets from the fixed dataset
-- initialize state
-- return the first observation
-
-`step()`:
-
-- validates reset happened
-- grades action against current ticket
-- computes reward
-- advances to next ticket
-- if done, computes trajectory reward
-- otherwise returns immediate step reward
-
-This is tidy. It is also shallow.
-
-There is no environment mutation other than index movement. No internal state changes based on the chosen action. No branching. No action-dependent future ticket behavior. No queue reprioritization. No retries. No note writing. No escalation backlog. No “wrong earlier action causes downstream penalty.” The only environment response is score feedback.
-
-A benchmark like this can still be useful, but it sits much closer to supervised evaluation than to agentic interaction. That becomes a competitive problem when the reference set includes environments where actions actually transform the world.
-
-One subtle but important weakness is that `step()` does not enforce the task contract tightly. `HelpdeskTicketAction` allows all four fields to be present on any task, and `grade_action()` simply reads the fields relevant to the chosen `task_id`. Extra fields are ignored. That means the environment tells the agent “allowed_fields are X,” but it does not enforce “only X may be submitted.” It is not catastrophic, but it reflects a looser benchmark contract than the environment surface suggests.
-
-### Grader and reward design
-
-`server/grader.py` is the most benchmark-defining file in the project, and it currently underdelivers relative to its importance.
-
-What is good:
-
-- it has partial credit for issue-type confusions
-- it has proximity-based scoring for priority
-- task weights sum to 1
-- it is deterministic
-- it is easy to reason about
-
-What is weak:
-
-- the similarity tables are static, narrow, and handcrafted
-- assignment group and resolution action are exact-match only even though the environment does not expose enough context to make some distinctions fully grounded
-- there is no calibration check on over-escalation
-- there is no queue-level objective
-- there is no policy compliance signal
-- there is no explanation consistency
-- there is no distinction between “reasonable but conservative” and “reckless but lucky”
-
-The biggest conceptual weakness is that the reward is local and label-centric. A strong helpdesk environment should care about operational behavior, not just answer key overlap.
-
-For example, suppose two actions both get the final resolution action wrong:
-
-- one escalates a low-risk general inquiry to security
-- one acknowledges a critical account lockout without escalation
-
-Today those mistakes mostly show up as missed fields in a flat weighted sum. But in real operations they are qualitatively different failures. One wastes specialist capacity. The other is a dangerous underreaction. A competitive benchmark should encode that asymmetry.
-
-There is also a concrete implementation weakness in `compute_trajectory_reward()`. It computes:
-
-- average per-ticket score
-- minus `0.03 * overshoot`
-
-But `overshoot = max(0, steps_taken - queue_size)`, and the environment ends the episode when the current ticket index reaches queue length. After that point, further stepping raises an error. So in the normal execution path, overshoot is effectively always zero. The code suggests the environment cares about extra wasted steps, but the environment does not actually permit them. That means part of the trajectory logic is decorative rather than active.
-
-In strong benchmarks, reward code usually reveals the benchmark’s philosophy. In your project, the reward code mostly reveals the current label schema.
-
-### Dataset design
-
-`data/dataset.json` currently holds 45 tickets. The class distribution is not terrible for a prototype, but it is still small:
-
-- `application_support`: 9
-- `billing_license`: 7
-- `service_request`: 6
-- `security_compliance`: 5
-- `spam_phishing`: 5
-- `identity_access`: 4
-- `onboarding`: 4
-- `general_inquiry`: 3
-- `feature_request`: 2
-
-That is a tiny dataset for any benchmark that hopes to resist memorization or heuristic overfitting. The especially small classes are a concern. A benchmark with 2 feature requests and 3 general inquiries is not meaningfully testing generalization in those categories.
-
-The priority distribution is also limited:
-
-- critical: 9
-- high: 15
-- medium: 12
-- low: 9
-
-That is balanced enough to be usable, but not rich enough to encode the true structure of priority assignment. There is no obvious representation of customer segment, contractual urgency, outage blast radius, legal exposure, dependency graphs, or business calendar sensitivity. Priority is largely being inferred from words in the title and description, which is exactly what a heuristic baseline will exploit.
-
-The dataset does have four ambiguous records and three follow-up linked records. That is good. But because the environment does not structurally expose `ambiguity_note` or `related_ticket_id`, those richer cases do not actually become richer environment mechanics. They mostly remain hints for the benchmark designer, not tools for the agent.
-
-The follow-up handling is especially underused. Tickets like `ticket-038` and `ticket-045` clearly encode longitudinal customer frustration and repeated failure, which should change triage behavior. But the environment treats them like standalone text blobs. There is no action to inspect previous tickets. No thread retrieval. No stateful consequence from unresolved history. The environment has the seed of longitudinal realism and then does not build on it.
-
-There is also no train/eval split, no hidden split, no procedural generation, no adversarial generation, and no OOD slice. The same fixed dataset defines the universe. That is fine for unit tests. It is weak for a benchmark intended to compete.
-
-### Inference baseline and benchmark leakage
-
-`inference.py` is more important than it may look, because it tells you how easy the benchmark is to shortcut.
-
-The heuristic path:
-
-- scans ticket text for fixed issue-type keywords in fixed order
-- assigns priority from small keyword buckets
-- assigns resolution action from issue type plus a few escalation and fulfillment keywords
-- assigns assignment group from issue type mapping
-
-That baseline is not merely a harmless example. It is a diagnostic of benchmark leakage. The easier it is to hand-author a ruleset that tracks your label policy, the less benchmark headroom you have.
-
-And in this codebase, the baseline is not just simple. It is tightly coupled to the environment’s ontology:
-
-- it uses the exact taxonomy constants
-- it exploits the one-to-one issue-to-assignment mapping
-- it exploits mostly deterministic issue-to-resolution defaults
-- it assumes priority is keyword-addressable from the visible text alone
-
-That means the benchmark currently invites ontology-driven shortcutting.
-
-There is an even more concerning signal. The tests describe a heuristic baseline around `0.9400`, but a local code-faithful replay of the rule ordering in PowerShell over the full `data/dataset.json` gives a much weaker picture:
-
-- issue type exact accuracy: about `0.7333`
-- priority exact accuracy: about `0.3778`
-- assignment exact accuracy: about `0.7333`
-- resolution exact accuracy: about `0.6889`
-- full task-3 exact match: about `0.2444`
-- approximate weighted average score across tasks 1, 2, and 3: about `0.7344`
-
-The exact number is less important than what it implies: the benchmark narrative about heuristic strength and the actual rule behavior appear out of sync. That can happen for several reasons:
-
-- the tests are stale relative to current data
-- the claimed baseline was measured on sampled queues rather than the whole dataset
-- the heuristic ordering now creates more collisions than expected
-- the benchmark evolved without a full-baseline recomputation
-
-Whatever the cause, it is a warning sign. When benchmark claims and benchmark code diverge, trust in the environment falls.
-
-### Test strategy
-
-Your project has six test files. That is good relative to many small hackathon projects. But the content of the tests matters more than the count.
-
-The most important limitation is that multiple tests stub the OpenEnv types, interfaces, or `create_app()` implementation rather than exercising the real installed framework. `tests/openenv_test_stubs.py` injects fake `openenv.core.env_server.types`. `tests/test_environment_smoke.py` and `tests/test_api_integration.py` patch in a fake `Environment` base class. `tests/test_api_integration.py` also installs a stub `create_app` that returns a small FastAPI app with simplified routes.
-
-That means much of the test suite verifies your code against a locally simulated OpenEnv contract, not against the actual `openenv-core` dependency declared in `pyproject.toml`.
-
-This is a big competitive weakness because the reference repository’s core is full of behavior that your test harness never touches:
-
-- WebSocket `/ws` interactions
-- session handling
-- concurrency settings
-- serialization edge cases
-- metadata and schema endpoints
-- MCP endpoints
-- async step paths
-- actual `EnvClient` protocol semantics
-
-Your tests mostly prove that the environment behaves under your own simplified assumptions. That is useful, but it is not the same as proving robust OpenEnv integration.
-
-The other limitation is that the tests are mostly shallow-contract tests:
-
-- reset returns something valid
-- step increments counts
-- reward is in `[0, 1]`
-- task IDs are present
-- heuristic episodes do not error
-
-Those are necessary. They are not sufficient for a competitive benchmark.
-
-What is missing includes:
-
-- real WebSocket end-to-end tests
-- invalid action contract tests with actual framework validation
-- tests for extra fields on restricted tasks
-- concurrency tests
-- seed reproducibility tests across actual server sessions
-- golden regression tests on full-dataset benchmark score
-- hidden/eval split integrity tests
-- tests for ambiguity and follow-up handling
-- tests that verify the environment is hard in the intended way, not just runnable
-
-In short, the current test suite validates operability, not benchmark integrity.
-
-## Critical Gaps That Matter Most
-
-This section is the most actionable part of the report. If the goal is to beat stronger reference projects, these are the gaps that matter.
-
-### Gap 1: The project is benchmarked as an environment, but designed as a classifier
-
-The core problem is conceptual. Your code uses the OpenEnv interface, but the actual task shape is still mostly multi-label classification over short ticket text.
-
-The better reference environments are hard because the agent has to interact:
-
-- `BrowserGymEnvironment` asks the agent to act in a browser.
-- `FinQAEnvironment` asks the agent to inspect tools and query structured data.
-- `REPLEnvironment` asks the agent to iteratively execute code and decide when to finalize.
-- `Tbench2Environment` asks the agent to manipulate a terminal workspace and then survive evaluation.
-- `TextArenaEnvironment` asks the agent to play through game turns.
-
-Your environment asks the agent to emit labels. Even when multiple tickets appear in a queue, the agent is still doing the same one-shot operation repeatedly. It is not exploring, not investigating, not mutating meaningful state, not managing resources, and not making action-sequence tradeoffs.
-
-That difference is bigger than it looks. Once the benchmark is classifier-shaped, the fastest route to good performance is classifier-shaped too. The environment does not force the agent to behave like an operator. It only asks it to sound like one.
-
-That is why the next leap must be architectural, not cosmetic.
-
-### Gap 2: The hardest task is structurally easier than it claims
-
-Task 3 appears to be a four-field routing task, but the ontology collapses much of the difficulty.
-
-`ISSUE_TYPE_TO_ASSIGNMENT_GROUP` is one-to-one. If the agent gets issue type right, assignment group is already implied. That means one quarter of the task-3 score is mostly a lookup rather than a separate judgment call.
-
-Resolution action is not fully deterministic, but it is still heavily compressed by issue type defaults. Several issue types have only one action in practice across the dataset. Others vary under small numbers of recognizable phrases such as legal threat, follow-up pressure, or explicit request wording.
-
-So the “hard” task is closer to:
-
-- infer issue type
-- infer urgency from a few cues
-- apply one deterministic mapping
-- apply one mostly deterministic mapping with a few exceptions
-
-That is not trivial, but it is much less rich than real service-desk routing. Real hard cases exist when the same visible ticket text can map to different actions depending on hidden context such as account tier, live incident status, prior history, or internal policy. Your environment does not currently model those cases.
-
-### Gap 3: The environment underuses the best parts of its own data
-
-Your dataset is more interesting than your observation contract.
-
-`HelpdeskTicketRecord` contains `ambiguity_note` and `related_ticket_id`. Those are exactly the kinds of fields that could turn this into a stronger environment:
-
-- ambiguity makes decisions less keyword-deterministic
-- related ticket IDs create thread continuity
-- follow-ups create escalation pressure and temporal realism
-
-But `_build_observation()` discards them and only exposes the basic ticket text fields.
-
-That has two consequences:
-
-First, the richer authored structure is lost to the agent. Second, the benchmark stops short of the very complexity the dataset author was already beginning to encode.
-
-This is one of the clearest signs that the current project is a first version. The seeds of a deeper environment are already present in the data model. The runtime contract just does not use them.
-
-### Gap 4: There is no investigation loop
-
-In real helpdesk operations, the visible complaint is rarely the whole decision problem.
-
-An operator often needs to know:
-
-- whether the requester is on an enterprise contract
-- whether the problem aligns with an active outage
-- whether the user is an admin
-- whether prior tickets already established a root cause
-- whether a security signal exists on the account
-- whether a compliance deadline is legally binding
-- whether the request is actually a duplicate
-
-Your environment has no tool loop for this. The agent sees a title, requester, and description, then is expected to decide everything directly.
-
-That makes the environment much easier to brute-force and much less realistic than the domains represented by the best reference projects. `FinQAEnvironment` does not ask the model to guess answers from wording alone; it gives tools. `GitTaskEnvironment` gives a repo. `Tbench2Environment` gives a terminal. `BrowserGymEnvironment` gives a browser. Your helpdesk environment gives a paragraph.
-
-The fastest path to a stronger benchmark is to add internal tools and make the hardest scenarios impossible to solve reliably without using them.
-
-### Gap 5: There is almost no internal economics
-
-A good environment usually has some notion of tradeoff or cost even if it is not expressed as money.
-
-In your environment:
-
-- there is no time budget
-- there is no backlog pressure
-- there is no penalty for over-escalating except field mismatch
-- there is no cost for routing everything to the safest specialist
-- there is no consequence for queue ordering
-- there is no tension between fast response and careful investigation
-
-The queue exists, but it is not an economy. It is just a list.
-
-That means the environment cannot really test operational judgment. It can only test whether the final labels match the benchmark designer’s answer key. Stronger environments force decisions under constraints. Your current implementation mostly scores unconstrained annotation.
-
-### Gap 6: The reward story is thinner than the benchmark story
-
-`grade_action()` is neat and deterministic, but it still mainly scores label overlap. It does not score operator quality.
-
-There is no difference between:
-
-- a cautious but slightly conservative routing choice
-- a reckless underreaction that happens to get some partial credit
-- an unnecessary escalation that wastes the security team
-- a smart intermediate step that gathers evidence before final routing
-
-Those distinctions do not exist because the action surface does not allow them and the reward design does not look for them.
-
-There is also a direct implementation issue: `compute_trajectory_reward()` includes an overshoot penalty, but because the environment ends when the queue is exhausted and refuses later steps, overshoot does not really happen in the normal path. So part of the trajectory logic looks more meaningful than it actually is.
-
-When reward code contains dead or decorative logic, trust in the benchmark drops.
-
-### Gap 7: The current benchmark is highly vulnerable to ontology memorization
-
-The more the task can be solved by memorizing your ontology and keyword policy, the lower the ceiling of the benchmark.
-
-Right now the environment is vulnerable because:
-
-- the dataset is small
-- the label space is public and fixed
-- some output fields are deterministic functions of others
-- the observation is a short text blob
-- the heuristic baseline directly encodes the ontology
-- there is no hidden split or generator-based variation
-
-The current inference script is a warning sign here. It is not just a demo baseline. It is evidence that a carefully chosen keyword system can cover a large fraction of the problem structure because the problem structure is currently that compressible.
-
-If you want to build something harder to game, the benchmark must stop being reducible to a keyword policy plus a few ontology tables.
-
-### Gap 8: The tests are too synthetic for the actual risk profile
-
-The test suite checks that the environment is runnable. It does not yet prove that the benchmark is trustworthy.
-
-The biggest limitation is the heavy use of stubs around the OpenEnv dependency boundary. Several tests replace the real OpenEnv types, interfaces, or `create_app()` implementation. That helps local testability, but it means the suite is not validating actual WebSocket session behavior, actual framework serialization, actual schema generation, or actual concurrency handling.
-
-That is a serious gap if the environment is meant to compete with stronger projects. Reference environments are embedded in a framework that supports:
-
-- WebSocket sessions
-- session capacity and session info
-- schema endpoints
-- metadata endpoints
-- MCP endpoints
-- sync and async execution paths
-
-Your current tests mostly validate business logic under a simplified local harness. That is still useful. It is just not enough to prove benchmark robustness.
-
-There is also no strong integrity suite around the benchmark itself. Missing pieces include:
-
-- full-dataset regression scoring
-- hidden split integrity
-- adversarial edge-case suites
-- benchmark versioning checks
-- ambiguity and follow-up behavior tests
-- contract tests that verify the hard task is genuinely hard in the intended way
-
-If you want the project to be taken seriously, the environment and the benchmark need separate test surfaces.
-
-### Gap 9: The benchmark narrative and executable reality are drifting apart
-
-A benchmark becomes fragile when people cannot tell which number to trust.
-
-Your tests imply a strong heuristic baseline. The environment code and local replay of the actual heuristic rules over the dataset suggest a weaker story. That discrepancy may be caused by stale thresholds, changed data, queue sampling effects, or unrefreshed benchmark assumptions. Whatever the reason, it is not a small issue.
-
-Strong benchmarks need executable answers to simple questions:
-
-- what is the official baseline?
-- how is it measured?
-- on which split?
-- with what seeds?
-- on which version of the data?
-- under which scenario families?
-
-Right now those answers are not fully stabilized in code. The result is that the benchmark is harder to trust than it should be.
-
-That may sound administrative, but it is actually competitive. A benchmark that feels ad hoc will lose to a benchmark that feels governed, even if both are interesting.
-
-### Gap 10: The project does not yet have a competitive moat
-
-The strongest environments in the reference set each have a clear identity:
-
-- BrowserGym: browser-native multimodal interaction
-- FinQA: tool-mediated reasoning over structured finance data
-- REPL: iterative code execution and rubric-based finalization
-- TBench2: terminal tasks grounded by executable evaluation
-- Calendar: stateful tool ecosystem over application APIs
-- Chess: adversarial long-horizon board play
-
-Your current identity is “helpdesk routing from short ticket text.” That is useful, but not yet distinctive enough to dominate.
-
-The domain itself can support a much stronger identity:
-
-- service desk triage under partial observability
-- enterprise support operations with tool use and policy constraints
-- multi-ticket queue management under SLA and escalation economics
-
-That is the moat you should build. The domain is good enough. The current benchmark shape is not yet deep enough to own it.
-
-## What Specific Reference Environments Teach You
-
-### BrowserGym: rich observations create real decision space
-
-`BrowserGymObservation` includes text, URL, optional screenshot, goal, accessibility tree text, pruned HTML, error strings, and action-error flags. `BrowserGymEnvironment` carefully converts raw benchmark objects into those modalities and preserves additional metadata while filtering large raw fields.
-
-The lesson is not “copy browser features.” The lesson is that an observation should support several reasoning strategies at once. Strong environments do not force everything through one narrow channel if the domain can naturally expose more.
-
-Your helpdesk environment should likely move from a plain ticket view to a mixed observation view that includes structured context, queue state, optional note previews, and pointers to retrievable evidence. A stronger observation contract makes the environment harder to solve with surface heuristics and easier to use for real agent development.
-
-### FinQA: tool use transforms a QA task into an environment
-
-`FinQAEnvironment` is one of the most relevant reference environments for your redesign. It takes a question-answering domain that could have been implemented as “read prompt, output answer” and instead builds a tool-mediated workflow:
-
-- list tools
-- inspect table descriptions
-- inspect table metadata
-- run SQL queries
-- submit final answer
-
-The ground truth is hidden. The agent has to do work. The reward system then normalizes answer formats so the benchmark is measuring reasoning rather than answer string quirks.
-
-Your helpdesk project should follow that pattern. The hard task should not be “read ticket and guess routing.” It should be “use service desk tools to investigate and then submit routing.” That would immediately raise the benchmark ceiling.
-
-### REPL: process reward and outcome reward should be separate
-
-`REPLEnvironment` is instructive because it distinguishes execution quality from final answer quality. The environment tracks iterations, namespace state, execution results, and finalization patterns. The rubric layer then separates outcome reward from process reward.
-
-That is directly applicable to helpdesk operations. A strong service desk environment should separately measure:
-
-- whether the final routing/action was correct
-- whether the agent investigated responsibly
-- whether the agent made avoidable operational mistakes
-- whether the agent wasted steps or overused escalation
-
-Without that split, you cannot tell the difference between good operations and lucky guessing.
-
-### TBench2: grounded evaluation is a moat
-
-`Tbench2Environment` is powerful because success is not a declared label. It is an executable check. The agent can manipulate a workspace and then call `evaluate`, which runs tests. That style of evaluation is very hard to fake and very easy to defend.
-
-Helpdesk will not use pytest in the same way, but the principle transfers cleanly. A stronger helpdesk benchmark should evaluate against hidden operational truth and downstream effects, not just a visible label table. If the environment can compute whether the chosen action violated SLA policy, ignored an active incident, or misrouted a duplicate chain, then benchmark credibility goes up immediately.
-
-### Calendar MCP: tool ecosystems can scale if the boundary is clean
-
-The Calendar stack shows how a domain can become more realistic without exploding the action schema. The environment exposes tools, request context, user context, and database-backed state. Tool handlers are generic where possible and dynamic routing does a lot of the heavy lifting.
-
-For your domain, that is a strong hint that helpdesk should probably become tool-centric. Instead of stuffing everything into one giant action object, expose a small set of operational tools. This will scale better, feel more realistic, and let you design harder scenarios without turning the action model into a kitchen sink.
-
-### GitTask: reproducible scenario resets matter
-
-`GitTaskEnvironment` is not the most feature-rich environment in the set, but it gets one important thing right: reproducible task state. Reset means something concrete. The environment can put you back into a known repo state efficiently.
-
-You need the same discipline in scenario design. Instead of sampling any 3 to 5 tickets from one public pool, define reproducible episode families:
-
-- urgent outage follow-up
-- mixed billing queue
-- false-positive security scare
-- onboarding plus access control bundle
-- executive escalation chain
-
-Once episodes become scenario-driven rather than ticket-sampled, the benchmark will feel much more intentional.
-
-### Chess and TextArena: delayed reward and auxiliary signals are valuable
-
-`ChessEnvironment` plus `ChessWinLossRubric` shows how delayed reward can be modeled cleanly across a trajectory. `TextArenaEnvironment` plus its reward providers shows how auxiliary signals can coexist with the main reward without replacing it. Those patterns matter because helpdesk operations are not fully one-shot even when the final routing choice is what gets judged.
-
-In a stronger version of your environment, you could preserve a main final reward while also emitting auxiliary channels such as:
-
-- evidence quality
-- duplicate-handling quality
-- escalation efficiency
-- SLA awareness
-- customer experience quality
-- policy compliance
-
-Even if you keep one main scalar reward for training or evaluation, those auxiliary signals would make the benchmark much more diagnosable.
-
-### ReasoningGym and Maze: simplicity is fine if it is honest
-
-`ReasoningGymEnvironment` is a simple parameterized single-step environment. `MazeEnvironment` is a simple gridworld. Neither one pretends to be deeper than it is. That honesty is useful as a design lesson.
-
-If you want to keep a light version of your current project, that is perfectly reasonable. But then it should be presented as a starter triage benchmark, not as a fully realized agentic operations environment. If you want to claim higher competitive value, the environment itself needs to support that claim with deeper mechanics.
-
-## A Concrete Design for Beating the Stronger Projects
-
-The right goal is not to imitate the broadest reference project. The right goal is to go much deeper in one domain you already own.
-
-You do not need to out-BrowserGym BrowserGym. You do not need to out-TBench2 TBench2. You need to become clearly better at service desk operations simulation than the reference set is today.
-
-### North star: build a service operations simulator
-
-The strongest future version of this project looks more like an IT service desk simulator than a label prediction benchmark.
-
-Core properties of that simulator should be:
-
-- partially observed ticket and account state
-- internal tools for investigation
-- scenario families rather than one static pool
-- multi-step resolution workflows
-- queue-level tradeoffs
-- policy-aware reward
-- hidden evaluation truth
-
-If you hit those properties, you will not just be polishing the current environment. You will be changing the category of the benchmark.
-
-### Proposed visible entities
-
-The agent should see richer but still realistic objects, for example:
-
-- ticket thread summary
-- current requester details
-- account/org summary
-- queue overview
-- recent internal note previews
-- live incident banner or incident tool access
-- available tools
-- allowed actions
-- task budget and SLA hints
-
-That does not mean every observation must be huge. It means the visible world should make the agent reason like an operator instead of like a labeler.
-
-### Proposed hidden entities
-
-The environment should own hidden state that determines the correct policy:
-
-- canonical root-cause category
-- customer tier
-- resolver ownership
-- actual business impact
-- active incident linkage
-- prior unresolved duplicates
-- whether manual escalation is necessary or wasteful
-- whether policy requires a specific handling path
-- whether the ticket is self-servable by documented guidance
-
-These hidden variables are what create genuinely hard cases. Two tickets that look similar on the surface should sometimes route differently because the hidden state differs.
-
-### Proposed action surface
-
-I would split the action space into investigation actions and commitment actions.
-
-Investigation actions:
-
-- `lookup_requester`
-- `get_account_plan`
-- `get_related_tickets`
-- `check_service_health`
-- `search_kb`
-- `inspect_internal_notes`
-- `get_security_signals`
-- `get_asset_or_license_state`
-
-Operational actions:
-
-- `add_internal_note`
-- `request_more_info`
-- `merge_duplicate`
-- `set_priority`
-- `assign_group`
-- `escalate`
-- `acknowledge`
-- `submit_final_decision`
-
-This preserves your current routing taxonomy while forcing the agent to earn the final answer through interaction.
-
-### Proposed task families
-
-Replace the current output-field ladder with scenario families.
-
-1. **Baseline classification**
-   Keep a simple version of the current task for calibration.
-
-2. **Priority under operational context**
-   Add visible account metadata and SLA hints.
-
-3. **Tool-assisted routing**
-   Hard cases require evidence retrieval.
-
-4. **Follow-up chain handling**
-   Correct routing depends on thread history and prior failures.
-
-5. **Duplicate resolution**
-   The agent must detect and merge with existing tickets or note the linkage.
-
-6. **Queue management**
-   Multiple tickets compete for limited steps or limited escalation budget.
-
-7. **Incident-aware triage**
-   Correct behavior depends on checking active incident state.
-
-8. **Policy-constrained operations**
-   Compliance, security, or executive-account policies change what the correct action is.
-
-Now difficulty comes from task structure, not just output dimensionality.
-
-### Proposed reward design
-
-A strong reward design for this domain should likely have four layers.
-
-Layer 1: **final outcome correctness**
-
-- correct issue family
-- correct priority
-- correct resolver team
-- correct action
-
-Layer 2: **operational policy correctness**
-
-- no violation of mandatory escalation rules
-- no unjustified critical priority
-- no missed compliance deadlines
-- no unsupported closure
-
-Layer 3: **process quality**
-
-- useful tool use
-- correct duplicate inspection
-- efficient evidence gathering
-- no unnecessary specialist escalation
-
-Layer 4: **episode economics**
-
-- queue-wide quality
-- backlog harm
-- escalation cost
-- SLA miss cost
-
-That may sound like a lot, but you do not need to expose all of it as one scalar at once. Some of it can be stored as metadata or auxiliary reward channels first.
-
-### Proposed data strategy
-
-Do not try to hand-author ten thousand fully custom tickets from scratch. Instead, build a layered data strategy.
-
-Layer A: curated seed cases
-
-- your best handcrafted exemplars
-- ambiguous pairs
-- follow-up chains
-- adversarial near-neighbors
-
-Layer B: templated scenario generation
-
-- same underlying issue with different requester tiers
-- same wording with different hidden incident context
-- duplicate vs non-duplicate versions
-- billing dispute with and without outage linkage
-
-Layer C: hidden benchmark splits
-
-- development split
-- public validation split
-- private evaluation split
-
-Layer D: scenario tagging
-
-- issue family
-- ambiguity level
-- investigation depth required
-- tool requirement
-- risk class
-- queue pressure
-
-This approach gives you scale without giving up control.
-
-## File-by-File Improvement Plan for This Repository
-
-This section ties the redesign back to the actual code you already have. The point is to show how the current repo can evolve into the stronger benchmark rather than be abandoned.
-
-### `models.py`
-
-Right now the models encode the benchmark as a label submission problem. That is fine for version one and too restrictive for version two.
-
-I would keep the existing validation patterns, but I would expand the schema into typed action families and typed observation payloads.
-
-Recommended direction:
-
-- keep `HelpdeskTicketRecord`, but add typed visible vs hidden fields
-- replace the loose `current_ticket: Optional[dict[str, str]]` with a ticket-view model
-- split actions into investigation actions and final submission actions
-- add typed structures for tool results, notes, queue items, and thread previews
-- enrich state with scenario metadata, action audit trail, and resource counters
-
-Why this matters:
-
-As long as the schema itself says “the agent submits optional routing fields,” every other part of the environment will naturally stay classifier-shaped. Schema is architecture. If you want the environment to feel agentic, the models have to make agentic behavior first-class.
-
-### `server/environment.py`
-
-This file is currently the main reason the benchmark feels thin. It is clean, but it is clean because it has very little world logic.
-
-I would evolve it in stages.
-
-Stage 1:
-
-- expose structured thread/follow-up information
-- enforce task contracts more tightly
-- store full action history, not just scores
-- make scenario metadata visible
-
-Stage 2:
-
-- add tool dispatch for investigation actions
-- maintain scenario-local hidden state
-- let actions mutate environment state
-- support final decision submission separately from intermediate investigation
-
-Stage 3:
-
-- add queue-level episodes with budget constraints
-- let earlier choices affect later ticket handling
-- introduce scenario-specific logic for duplicates, incidents, and policy constraints
-
-Why this matters:
-
-This file should become the simulator, not just the grader entrypoint.
-
-### `server/tasks.py`
-
-This file needs the most conceptual change after the environment itself.
-
-The current task list is:
-
-- task 1: issue type only
-- task 2: issue type plus priority
-- task 3: full routing
-
-That is too narrow. I would turn `tasks.py` into a scenario-family registry instead.
-
-For example:
-
-- `single_ticket_classification`
-- `priority_under_sla`
-- `tool_assisted_routing`
-- `duplicate_chain_resolution`
-- `incident_aware_triage`
-- `queue_optimization`
-- `policy_constrained_security_case`
-
-Each task family should define:
-
-- visible observation contract
-- allowed actions
-- hidden truth generator
-- episode budget
-- reward composition
-- benchmark split membership
-
-Why this matters:
-
-Right now tasks differ by scoring columns. A strong benchmark needs tasks that differ by problem structure.
-
-### `server/grader.py`
-
-This file should stop being only a lookup-based scorer and become the place where service-desk policy is encoded.
-
-I would keep the basic idea of partial credit, but move from a pure field-overlap worldview to a policy-and-outcome worldview.
-
-Examples of richer scoring logic:
-
-- small penalty for unnecessary escalation
-- strong penalty for under-prioritizing active access outages
-- reward for correctly linking duplicates
-- reward for choosing acknowledgment before final resolution when that is the right workflow
-- penalty for routing compliance work to general support
-- scenario-aware scoring where the same visible ticket can score differently depending on retrieved evidence
-
-Why this matters:
-
-The grader is the actual benchmark. It should reflect operational quality, not only taxonomy overlap.
-
-### `server/reward.py`
-
-This file is a good place to simplify and then rebuild.
-
-First, remove or redesign logic that is not meaningfully active, such as the current overshoot penalty that normal episode flow does not really trigger.
-
-Then add reward layers deliberately:
-
-- final decision score
-- process score
-- economics score
-- optional auxiliary diagnostics
-
-Why this matters:
-
-A benchmark becomes much easier to improve if the reward code honestly reflects what is being optimized.
-
-### `server/app.py`
-
-This file is currently fine for a minimal environment, but it should grow once the environment grows.
-
-Recommended additions:
-
-- environment metadata endpoint support if you want richer UI or benchmark introspection
-- possibly custom routes for benchmark info, scenario families, or baseline metadata
-- cleaner packaging around path setup once the project stabilizes
-
-Why this matters:
-
-This is not the highest-priority file, but stronger benchmark ergonomics do help credibility and usability.
-
-### `data/dataset.json`
-
-This file should evolve from “the benchmark” into “part of the benchmark.”
-
-Keep a curated hand-authored slice, but do not let one public JSON file define the whole environment forever.
-
-Recommended evolution:
-
-- expand the dataset substantially
-- add many more feature request and general inquiry cases
-- add multiple duplicate chains
-- add hidden context fields
-- add templated variants of existing scenarios
-- create a private evaluation bank
-
-Why this matters:
-
-A tiny fixed public dataset makes memorization too easy and benchmark claims too brittle.
-
-### `inference.py`
-
-This file is useful, but it currently plays several roles at once:
-
-- demo script
-- heuristic baseline
-- optional LLM runner
-- environment smoke path
-
-I would separate those responsibilities.
-
-Recommended structure:
-
-- one official deterministic baseline runner
-- one optional tool-using baseline runner once tools exist
-- one separate example script for simple local usage
-- one benchmark harness that records split, seed, scenario family, and version
-
-Why this matters:
-
-Benchmarks need reproducible baselines more than they need convenient demos.
-
-### `tests/`
-
-The most important change after environment design is testing philosophy.
-
-I would split tests into at least four groups:
-
-1. **unit tests**
-   Validation, scoring primitives, dataset loaders, tool helpers.
-
-2. **real integration tests**
-   Actual OpenEnv app, actual serialization, actual WebSocket interactions.
-
-3. **benchmark regression tests**
-   Fixed scenario suites, stable baseline scores, hidden split checks.
-
-4. **integrity tests**
-   No task leakage, no duplicate split contamination, no benchmark version drift.
-
-Why this matters:
-
-A serious benchmark is a data product, an environment product, and an evaluation product. The tests should reflect all three.
-
-## Practical Roadmap
-
-### Phase 1: Make the current environment honest and sturdier
-
-This is the fastest and cheapest improvement phase. Do this even if you are not ready for a full redesign.
-
-Goals:
-
-- expose thread/follow-up structure
-- tighten task contracts
-- recompute and stabilize baseline measurements
-- add a hidden evaluation split
-- remove decorative reward logic
-- improve test realism
-
-Deliverables:
-
-- stronger observation model
-- benchmark regression script
-- real integration tests
-- scenario-family-aware tasks, even if still text-only
-
-This phase will not yet make the environment winner-beating, but it will make it much more defensible.
-
-### Phase 2: Add tool-assisted investigation
-
-This is the highest-return phase because it changes the category of the benchmark.
-
-Minimum viable tool set:
-
-- requester/account lookup
-- related-ticket retrieval
-- service health lookup
-- KB search
-- final decision submission
-
-Once those exist, create scenario families where the visible ticket text is insufficient without tool use. That immediately raises the benchmark ceiling and reduces shortcutability.
-
-### Phase 3: Add operational economics and queue-level behavior
-
-After tool use works, add:
-
-- queue-wide episodes
-- time or action budgets
-- escalation cost
-- SLA miss cost
-- duplicate-handling benefit
-- specialist-capacity awareness
-
-This turns the environment from a case-by-case annotation task into an operational management task.
-
-### Phase 4: Add benchmark governance
-
-At this point you should formalize:
-
-- public vs private splits
-- scenario-family tags
-- official baselines
-- benchmark versioning
-- scorecards by scenario family
-- release notes for benchmark changes
-
-This is what makes the project not just interesting, but trustworthy.
-
-## Prioritized Recommendation List
-
-If I had to choose only ten improvements, in order, I would choose these:
-
-1. Stop defining difficulty only by `allowed_fields`.
-2. Add investigation tools and final submission as separate actions.
-3. Break the deterministic issue-type-to-assignment shortcut.
-4. Make resolution depend on hidden operational context more often.
-5. Surface follow-up and related-ticket structure.
-6. Expand data and add hidden eval splits.
-7. Add process-aware reward and remove dead trajectory logic.
-8. Add queue-level economics and limited budgets.
-9. Replace stub-heavy integration tests with real framework tests.
-10. Publish a stable benchmark harness and official baseline measurement.
-
-## Final Assessment
-
-After a deep code read, my conclusion is simple:
-
-Your project is promising, readable, and based on a very strong domain. But in its current form it is still a compact routing benchmark, not yet a high-ceiling service-operations environment.
-
-The better reference environments in `OpenEnv/envs` are better not because they are bigger for the sake of being bigger, but because they force the agent to operate inside state, tools, or consequences that cannot be collapsed into label mapping so easily.
-
-The encouraging part is that your domain can support exactly that kind of benchmark. IT helpdesk operations naturally contain ambiguity, hidden context, tool use, policy constraints, long threads, queue pressure, and downstream costs. Very few toy domains offer that combination so cleanly.
-
-So the right move is not to abandon the project. The right move is to evolve it.
-
-If you keep the current shape and only add more tickets, you will get a better classifier benchmark. That may be useful, but it probably will not beat the strongest reference projects.
-
-If you turn this into a tool-assisted, partially observed, multi-step service-operations simulator with stronger reward design and stronger benchmark governance, then you can absolutely build something more compelling than many of the reference environments, because your domain has the right raw material for a benchmark that is both realistic and highly evaluable.
-
-The domain is already winner material.
-
-The current implementation is starter material.
-
-The opportunity is to close that gap deliberately.
-
-## Appendix A: Comparative Scorecard
-
-The table below is not a scientific benchmark. It is a code-read scorecard based on the implementations reviewed in this report. The goal is to make the gap tangible.
-
-| Dimension | Your project now | Strong reference environments |
-| --- | --- | --- |
-| Action richness | Low | Medium to very high |
-| Hidden state depth | Low | Medium to high |
-| Tool use | None | Present in FinQA, Calendar, TBench2, Git, REPL |
-| Multistep interaction | Low-medium | Medium to high |
-| Queue/process economics | Very low | Medium in some envs, high in operational ones |
-| Reward sophistication | Low-medium | Medium to high |
-| Benchmark anti-overfitting | Low | Medium |
-| Runtime realism | Low | Medium to high |
-| Testing depth | Low-medium | Medium to high at repo scale |
-| Domain relevance | High | Varies by env |
-| Potential ceiling | High | Already demonstrated in several envs |
-
-The most important row here is the last one. Your current implementation is not yet at the same level as the strongest references, but the domain ceiling is absolutely high enough to catch up and possibly surpass them if you execute the redesign well.
-
-## Appendix B: What You Should Preserve
-
-When teams hear “major redesign,” they often accidentally throw away the parts that were already working. I do not recommend that here.
-
-The current project has several strengths that should be preserved as you expand it:
-
-### 1. Preserve the compactness of the taxonomy
-
-The label space in `vocabulary.py` is clear and product-shaped. It is not bloated. Even when the environment becomes tool-based and stateful, keep the routing ontology understandable. The problem with the current benchmark is not that the taxonomy is wrong. The problem is that the environment around the taxonomy is too thin.
-
-### 2. Preserve deterministic core scoring where possible
-
-Even after you add process reward and hidden context, keep as much deterministic scoring as possible. One reason your current project is easy to debug is that the grader is inspectable. Do not replace everything with opaque LLM judging if you can avoid it. Use explicit hidden truth and rule-based evaluation for most of the benchmark, and reserve softer judging only for areas that truly need it.
-
-### 3. Preserve readability
-
-The current codebase is easy to onboard into. That is an asset. Several bigger reference environments are strong, but also much harder to reason about quickly because they wrap external systems or broad framework machinery. As you deepen this project, keep modules well-separated:
-
-- models
-- scenario generation
-- environment runtime
-- tools
-- scoring
-- reward composition
-- benchmark harness
-
-That separation will make future iteration much faster.
-
-### 4. Preserve seeded reproducibility
-
-Your existing environment is deterministic under a seed, and that is worth keeping. Stronger benchmarks become much easier to trust when a given scenario family plus seed reproduces the same world state. As you add hidden context and generators, make seed behavior even more explicit instead of less.
-
-### 5. Preserve explicit validation
-
-The Pydantic validation in the current models is a quiet strength. Keep that discipline. As the action surface grows, validation becomes more important, not less. Tools and action types should reject malformed inputs cleanly so that environment failures are informative rather than muddy.
-
-## Appendix C: Example Scenario Families for Version 2
-
-To make the redesign more concrete, here are example scenario families that would feel much closer to a winner-level helpdesk benchmark.
-
-### Scenario Family 1: Access outage with incident ambiguity
-
-Visible state:
-
-- multiple users report being locked out
-- one requester sounds urgent
-- another sounds like a normal password reset
-
-Hidden state:
-
-- there is an active identity provider outage
-- some tickets are duplicate symptoms of the same incident
-
-Tools needed:
-
-- `check_service_health`
-- `get_related_tickets`
-- `lookup_requester_role`
-
-What this tests:
-
-- whether the agent distinguishes isolated access issues from systemic incidents
-- whether it avoids handling every case as an independent ticket
-- whether it correctly prioritizes executive or admin users without overreacting on every case
-
-### Scenario Family 2: Billing dispute tied to product defect
-
-Visible state:
-
-- customer says they were charged incorrectly
-- another case mentions checkout failures
-
-Hidden state:
-
-- the billing dispute is caused by a known application defect that duplicated transactions
-
-Tools needed:
-
-- `search_related_tickets`
-- `check_service_health`
-- `read_internal_incident_note`
-
-What this tests:
-
-- whether the agent routes based on real causal structure rather than superficial department ownership
-- whether it recognizes that pure billing handling is insufficient because engineering is involved
-
-### Scenario Family 3: Compliance deadline with account-context twist
-
-Visible state:
-
-- requester references GDPR or legal obligation
-
-Hidden state:
-
-- some requests are legitimate deletion requests
-- some are actually admin-level data export requests misphrased as deletion
-- some belong to customers on contracts with defined response obligations
-
-Tools needed:
-
-- `lookup_contract_tier`
-- `retrieve_policy_snippet`
-- `get_account_data_scope`
-
-What this tests:
-
-- whether the agent can combine legal wording with account and policy context
-- whether it overroutes all legal-sounding tickets to the same team
-
-### Scenario Family 4: Duplicate-heavy queue optimization
-
-Visible state:
-
-- ten tickets in a queue
-- several appear to be related
-
-Hidden state:
-
-- six are duplicates of two underlying issues
-- one low-volume ticket is actually the most SLA-critical
-
-Tools needed:
-
-- `search_related_tickets`
-- `merge_duplicate`
-- `set_priority`
-- `submit_queue_plan`
-
-What this tests:
-
-- whether the agent can manage a queue as a system
-- whether it reduces work through linkage
-- whether it balances urgency against volume
-
-### Scenario Family 5: Feature request versus broken workflow
-
-Visible state:
-
-- customer asks for export filters or better reporting
-
-Hidden state:
-
-- in some scenarios the feature genuinely does not exist
-- in others the feature exists but the customer lacks permissions or is using the wrong path
-
-Tools needed:
-
-- `search_kb`
-- `lookup_plan_features`
-- `inspect_recent_product_change`
-
-What this tests:
-
-- whether the agent treats every request for missing functionality as a feature request
-- whether it can separate education/support from roadmap input
-
-## Appendix D: Red Flags to Avoid During the Redesign
-
-There are a few ways a redesign like this can go wrong. Avoid these.
-
-### 1. Do not add tools that are merely decorative
-
-If a hard task can still be solved reliably without using the tools, then the tool surface is just benchmark theater. The hard scenario families should be designed so that retrieved evidence actually changes the correct answer.
-
-### 2. Do not make every scenario gigantic
-
-Richer does not mean bloated. Some scenarios should stay compact. The goal is meaningful hidden context, not maximum token count.
-
-### 3. Do not replace all scoring with LLM judging
-
-Use explicit hidden truth and deterministic scoring wherever possible. Opaque judging should be a last resort, not a default.
-
-### 4. Do not let the ontology become a maze
-
-Your current taxonomy is pleasantly clean. Keep it that way. More realism should come from state and evidence, not from exploding the label space into dozens of nearly indistinguishable categories.
-
-### 5. Do not forget benchmark governance
-
-If you add scenario generation but do not formalize splits, baselines, and versioning, you will create a cooler environment without creating a more trustworthy benchmark.

analysis/grounding_audit.md

@@ -1,77 +0,0 @@
-# Grounding Audit For Taxonomy And Similarity Decisions
-
-> Internal note for the roadmap work originally planned for April 5, 2026.
-> Reviewed on April 3, 2026 and pulled forward ahead of schedule.
-
-## Goal
-
-Ground the current ticket taxonomy and the limited partial-credit policy against real public IT-support data without turning external datasets into a runtime dependency.
-
-## Sources Reviewed
-
-1. [Classification of IT Support Tickets](https://zenodo.org/records/7648117)
-   - Zenodo dataset with 2,229 manually classified support tickets.
-   - Dataset description says the tickets were classified by three IT support professionals.
-   - The public preview exposes seven coarse categories: `Fileservice`, `Support general`, `Software`, `O365`, `Active Directory`, `Computer-Services`, and `EOL`.
-
-2. [Semantic Similarity of IT Support Tickets](https://zenodo.org/records/7426225)
-   - Zenodo dataset with 300 ticket pairs manually labeled for semantic similarity.
-   - The description says three IT support professionals performed the labeling.
-   - This is the best direct grounding for keeping similarity explicit and limited instead of treating the whole label space as fuzzy.
-
-3. [MSDialog dataset page](https://ciir.cs.umass.edu/downloads/msdialog/)
-   - Technical-support dialog corpus drawn from Microsoft Community.
-   - The site reports 35,000 dialogs in `MSDialog-Complete` and 2,199 labeled dialogs with 10,020 utterances in `MSDialog-Intent`.
-   - This grounds our use of follow-up cases, clarification-heavy threads, and helpdesk-style conversational language.
-
-## Mapping Principle
-
-The external datasets validate that real IT support traffic mixes access problems, software incidents, generic support questions, procurement-like requests, and multi-turn follow-ups. Our label set is more operational than the public category sets, so the mappings below are judgment calls based on source descriptions and public previews rather than exact label equivalence.
-
-## Grounding Examples
-
-1. Active Directory lockout, MFA trouble, or password reset -> `identity_access` -> exact-match dominant, with `onboarding` as the only defensible adjacent label when the request is really about new-user provisioning.
-2. New hire account setup or contractor access provisioning -> `onboarding` -> partial-credit adjacent to `identity_access`, because both can surface as account enablement work before ownership is fully resolved.
-3. Office or application crash, timeout, webhook failure, or migration-script breakage -> `application_support` -> partial-credit adjacent to `feature_request` only when the report reads like a capability gap rather than a break/fix issue.
-4. Feature wishlist or export-format enhancement request -> `feature_request` -> partial-credit adjacent to `application_support` only when the user reports the missing capability as if it were a defect.
-5. Vendor-evaluation question, demo request, or quote request -> `service_request` -> partial-credit adjacent to `general_inquiry` when the request is still exploratory rather than a committed operational action.
-6. Seat expansion or provisioning-style commercial request -> `service_request` -> partial-credit adjacent to `billing_license` when procurement and account-admin signals are mixed in the same ticket.
-7. Refund, invoice discrepancy, subscription cancellation, or payment-admin issue -> `billing_license` -> partial-credit adjacent to `service_request` only in commercial admin cases that overlap with a procurement or seat-change request.
-8. Broad capability question or lightweight product clarification -> `general_inquiry` -> partial-credit adjacent to `service_request` or `feature_request` when the request is vague enough to look like either evaluation or roadmap feedback.
-9. Spam lure or credential-phishing message sent to the inbox -> `spam_phishing` -> partial-credit adjacent to `security_compliance` only for security-themed inbound items, not for normal access or software tickets.
-10. GDPR deletion request, DPA request, audit finding, or mandatory MFA policy notice -> `security_compliance` -> exact-match dominant, with very limited adjacency to `spam_phishing` for suspicious security reports and a low-confidence edge to `billing_license` only in contractual paperwork contexts.
-11. Reopened outage thread or repeated bug report escalation -> `application_support` -> exact-match dominant; the main change across turns is usually `priority`, not `issue_type`.
-12. Repeated lockout complaint or suspension follow-up -> `identity_access` -> exact-match dominant; follow-up behavior is grounded by MSDialog-style multi-turn support flow rather than by adding new label fuzziness.
-
-## Review Of Current Similarity Pairs
-
-The current `ISSUE_TYPE_SIMILARITY` map stays intentionally small. The defensible themes are:
-
-- `billing_license` <-> `service_request`: commercial admin and procurement requests can overlap before the owning team is clear.
-- `application_support` <-> `identity_access`: SSO and login failures can initially look like either app failure or access failure.
-- `application_support` <-> `feature_request`: some users describe missing functionality in bug-report language.
-- `onboarding` <-> `identity_access`: provisioning and account enablement are adjacent in real helpdesk traffic.
-- `general_inquiry` <-> `feature_request`: vague product questions can blur into roadmap requests.
-- `general_inquiry` <-> `service_request`: vendor-evaluation and exploratory capability questions often overlap.
-- `spam_phishing` <-> `security_compliance`: both are security-facing, but they should stay separate from normal access or app-routing labels.
-- `security_compliance` <-> `billing_license`: kept only as a very low-score edge for contract and paperwork overlap; this is the weakest current pair and should not be expanded further without ticket-level evidence.
-
-## Candidate Expansions Reviewed And Rejected
-
-These pairs were reviewed during the April 5 roadmap pass and are intentionally not being added:
-
-- `onboarding` <-> `service_request`: both can involve setup, but the owning teams and next actions diverge too quickly.
-- `feature_request` <-> `service_request`: roadmap asks and procurement actions are operationally different.
-- `security_compliance` <-> `identity_access`: policy obligations may mention accounts, but the compliance workflow is distinct from user access support.
-- `billing_license` <-> `identity_access`: nonpayment or suspension can mention lockout symptoms, but the root-cause owner is different.
-- `application_support` <-> `billing_license`: mixed commercial and outage narratives exist, but broad partial credit here would blur incident handling too much.
-
-## Decision
-
-No new issue-type similarity pairs should be added from this review.
-
-The safest grounded position is:
-
-- keep the current limited similarity map,
-- rely on exact-match scoring for most wrong labels,
-- let `priority`, `assignment_group`, and `resolution_action` keep the hard-task routing signal crisp.

analysis/scoring_contract.md

@@ -1,71 +0,0 @@
-# Scoring Contract
-
-> Internal note for test design and scorer review
-
-## Goal
-
-Make the helpdesk grader deterministic, defensible, and only fuzzy where we can explain why.
-
-## Exact-Match-Only Fields
-
-These fields should never receive partial credit:
-
-- `assignment_group`
-- `resolution_action`
-
-If either is wrong, the field score should be exactly `0.0`.
-
-## Limited Partial-Credit Fields
-
-### `issue_type`
-
-`issue_type` can receive partial credit only for explicitly listed near-miss pairs in `server/grader.py`.
-
-Implications:
-
-- exact match = `1.0`
-- listed near miss = configured partial score
-- unlisted wrong label = `0.0`
-
-There should be no hidden semantic fuzziness beyond the declared similarity map.
-
-### `priority`
-
-`priority` can receive partial credit only for explicitly listed adjacency / proximity pairs in `server/grader.py`.
-
-Implications:
-
-- exact match = `1.0`
-- defined nearby priority = configured partial score
-- undefined mismatch = `0.0`
-
-## Task Weight Contract
-
-- Task 1: `issue_type` only
-- Task 2: `issue_type` 60%, `priority` 40%
-- Task 3:
-  - `issue_type` 35%
-  - `priority` 20%
-  - `assignment_group` 25%
-  - `resolution_action` 20%
-
-The weighted score should always stay in `[0.0, 1.0]`.
-
-## What The Tests Must Prove
-
-1. exact matches score `1.0`
-2. unsupported task IDs fail clearly
-3. only intended issue-type pairs get partial credit
-4. unrelated issue types get `0.0`
-5. priority proximity follows the declared table exactly
-6. assignment group and resolution action remain exact-only
-7. task weights apply exactly as documented
-8. dataset loading stays robust, including UTF-8 BOM handling
-
-## Review Rule
-
-Before adding any new similarity pair:
-
-1. justify it with a real-world ticket ambiguity
-2. make sure it does not blur clearly distinct operational actions
-3. add or update a test that proves the intended behavior

gaps.md

@@ -1,146 +0,0 @@
-# Gap Analysis — IT Helpdesk Ticket Routing OpenEnv
-
-Deep cross-reference of the codebase against every concrete mentor statement from the bootcamp transcript and Discord Q&A.
-
----
-
-## GAP 1 — CRITICAL: `inference.py` runs all 3 tasks in one invocation
-
-**Mentor (4/1/26, 9:48 PM, confirmed twice):**
-> "inference.py should execute a single task per run and emit exactly one [START] … [END] block. The evaluation system handles running across multiple tasks, so batching all tasks in one invocation is not expected."
-
-**Your code in `inference.py`:**
-```python
-TASKS = list(TASK_IDS)  # [1, 2, 3]
-for task_id in TASKS:   # loops all 3
-    emit_log("START", ...)
-    ...
-    emit_log("END", ...)
-emit_log("END", overall_avg=...)  # second END
-```
-
-The evaluator calls `inference.py` once per task. Your script ignores that and runs all 3 itself, emitting 3 `[START]`/`[END]` pairs. The evaluator expects exactly one. There is no `TASK_ID` env var read anywhere.
-
----
-
-## GAP 2 — CRITICAL: `state()` response is missing `reward` and `done` fields
-
-**Mentor (4/1/26, 9:33 PM):**
-> "state() must return minimum: `{ 'observation': ..., 'reward': last_step_reward, 'done': True/False }`"
-
-**Your `HelpdeskTicketState` model:**
-```python
-class HelpdeskTicketState(State):
-    current_task_id: Optional[int] = None
-    seed: Optional[int] = None
-    queue_ticket_ids: list[str]
-    current_ticket_index: int = 0
-    per_ticket_scores: list[float]
-    total_reward: float = 0.0
-    # NO reward field (last step reward)
-    # NO done field
-```
-
-`GET /state` returns this model directly. The evaluator checking `state()` for `reward` and `done` will find neither. `total_reward` is the accumulated reward, not the last step reward — which the mentor explicitly said NOT to return.
-
----
-
-## GAP 3 — MEDIUM: `history` in observation is too sparse for RL usefulness
-
-**Ben (YouTube bootcamp, ~00:31:07):**
-> "process supervision... give these more detailed rewards... enrich history with ticket title, predicted fields"
-
-**Your `_build_observation` history:**
-```python
-history.append({"step": i + 1, "score": s})
-# final entry gets: {"step": N, "ticket_id": ..., "score": ..., "breakdown": ...}
-```
-
-Non-final history entries only have `step` and `score`. No ticket title, no predicted action fields. The agent cannot learn from history because it cannot see what it predicted or what the ticket was. This directly weakens RL signal quality.
-
----
-
-## GAP 4 — MEDIUM: No milestone/delta reward shaping — flat score passthrough
-
-**Mentor (4/1/26, 9:34 PM):**
-> "A deterministic terminal grader with partial credit is valid, but it's better to include some intermediate (non-terminal) reward signals as well so the environment provides step-wise feedback. Milestone-based shaping is preferred over dense per-action rewards."
-
-**Your `step()` in `environment.py`:**
-```python
-if is_done:
-    final_reward = traj_reward   # trajectory reward only at end
-else:
-    final_reward = step_reward   # per-ticket score for non-final steps
-```
-
-You do return `step_reward` on non-final steps, which is correct. But `step_reward` is just `compute_step_reward(score)` which is `max(0.0, min(1.0, score))` — identical to the raw score. There is no shaping, no milestone signal, no delta-based signal. This is a quality gap, not a blocker.
-
----
-
-## GAP 5 — MEDIUM: `observation.history` doesn't include the predicted action
-
-**Your `_build_observation`:**
-```python
-history_entry = {
-    "ticket_id": current_ticket.ticket_id,
-    "score": score,
-    "breakdown": breakdown,
-}
-```
-
-The agent's own predicted action is never stored in history. When the agent looks at history to decide its next action, it cannot see what it previously predicted. This is a real RL signal gap — the agent has no memory of its own decisions.
-
----
-
-## GAP 6 — LOW: `tickets_remaining` semantics slightly ambiguous
-
-**Your `_build_observation`:**
-```python
-tickets_remaining=max(0, queue_size - idx),
-```
-
-`idx` is `current_ticket_index` which has already been incremented by `step()` before `_build_observation` is called. During the episode, `tickets_remaining` counts the current ticket as "remaining" even though it is being processed. Minor but could confuse an LLM agent reading the observation.
-
----
-
-## GAP 7 — LOW: `openenv.yaml` `entry_point` vs `pyproject.toml` `server` script mismatch
-
-**Mentor (3/31/26, 11:27 PM):**
-> "The validator is checking for a specific callable entrypoint. In some setups, it expects a main() function instead of an app object."
-
-**Your `pyproject.toml`:**
-```toml
-[project.scripts]
-server = "server.app:main"
-```
-
-**Your `openenv.yaml`:**
-```yaml
-entry_point: server.environment:HelpdeskTicketRoutingEnvironment
-```
-
-These point to different things. The validator may check `entry_point` in `openenv.yaml` and expect it to match `[project.scripts] server`. This inconsistency could cause validation confusion.
-
----
-
-## GAP 8 — LOW: No `/web` UI endpoint — blank HF Space page
-
-**Ben (YouTube, ~00:45:08):**
-> "They're small apps and they're based as spaces. So they're deployed with a UI and an API."
-
-The echo env example had `/web` for the UI. Your app has no `/web` route. The mentor said UI is optional and not scored, but the HF Space will show a blank page with no UI, which looks unpolished to judges doing Phase 3 human review.
-
----
-
-## Summary
-
-| # | Gap | Severity | File(s) |
-|---|-----|----------|---------|
-| 1 | `inference.py` runs all 3 tasks, evaluator expects 1 per run | CRITICAL | `inference.py` |
-| 2 | `GET /state` missing `reward` (last step) and `done` fields | CRITICAL | `models.py`, `environment.py` |
-| 3 | `history` missing predicted action — agent has no memory of decisions | MEDIUM | `environment.py` |
-| 4 | No milestone/delta reward shaping — flat score passthrough | MEDIUM | `reward.py` |
-| 5 | `history` non-final entries missing ticket title | MEDIUM | `environment.py` |
-| 6 | `tickets_remaining` semantics slightly ambiguous | LOW | `environment.py` |
-| 7 | `openenv.yaml` `entry_point` vs `pyproject.toml` `server` script mismatch | LOW | `openenv.yaml`, `pyproject.toml` |
-| 8 | No `/web` UI — blank HF Space page | LOW | `server/app.py` |

required.md

@@ -354,7 +354,7 @@ The project is ready when:
 
 ## Current Compliance Snapshot
 
-As of April 7, 2026, the roadmap gates through the end of the freeze window are in place:
+As of April 8, 2026, the core submission requirements and the major benchmark upgrades are in place:
 
 - real-world task definition is clear and stable
 - typed models, `reset()`, `step()`, `state()`, and `openenv.yaml` are present in the repo
@@ -365,16 +365,17 @@ As of April 7, 2026, the roadmap gates through the end of the freeze window are
 - integration tests now cover `/health`, `/tasks`, `/reset`, `/step`, `/state`, full seeded episodes, and heuristic regression
 - baseline heuristic results are recorded in the docs
 - the README now includes Hugging Face Spaces frontmatter and a judge-facing grounded-scoring explanation
-- an internal grounding audit exists in `analysis/grounding_audit.md`
+- the label space and partial-credit policy were reviewed against public IT-support references during development
 - `.openenvignore` is present
 - Docker smoke coverage exists through the checked-in GitHub Actions workflow and recorded April 6 run
 - `inference.py` structured `[START]`, `[STEP]`, and `[END]` logging is verified
 - `uv.lock` is checked in and `openenv validate` now passes on the current repo state
 - a clean-copy install-and-run pass has been completed
 
-The remaining April 8 work is operational rather than implementation-heavy:
+The remaining work is optional benchmark expansion rather than submission readiness work:
 
-- Hugging Face deployment ping and reset verification
-- the final submission-branch sanity rerun before push if any last-minute packaging-only change lands
+- make the simulator even more emergent instead of partially authored
+- broaden the data distribution further
+- replace the local policy search loop with a more training-oriented learning setup if needed later
 
-The roadmap's short TRL / GRPO README example remains optional and is still deferred because it is not required for submission readiness.
+The short TRL / GRPO README example remains optional and is still deferred because it is not required for this project to be understandable, runnable, or judgeable.