hackathon_changes.md

TalkingHeadBench — Hackathon Compliance Change Plan

A file-by-file breakdown of every change needed to fully comply with the OpenEnv hackathon requirements.

---

1. inference.py (ROOT) — MISSING, MUST CREATE

Status: Does not exist. This is an immediate disqualification condition.

What it needs to do

• Connect to the running TalkingHeadBench environment at a configurable URL.
• Run a full benchmark episode across all 3 task tiers (image audit, clip audit, weight audit) using test-set cases loaded from tests/test_set/.
• Use the OpenAI Python client (from openai import OpenAI) for LLM calls — not the current raw urllib/httpx approach.
• Pull LLM config exclusively from the three required environment variables:
  • API_BASE_URL
  • MODEL_NAME
  • HF_TOKEN
• Print a structured score report at the end: per-sub-env scores and the weighted final score.
• Complete in under 20 minutes on 2 vCPU / 8 GB RAM.

Required structure (pseudocode)

inference.py
├── Load API_BASE_URL, MODEL_NAME, HF_TOKEN from os.environ
├── Instantiate: client = OpenAI(api_key=HF_TOKEN, base_url=API_BASE_URL)
├── For each task: "image", "clips", "weights"
│   ├── Call env.reset(mode=<task>)   — via HTTP to the running server
│   ├── Read observation, construct LLM prompt
│   ├── Call client.chat.completions.create(model=MODEL_NAME, ...)
│   ├── Parse structured JSON action
│   ├── Call env.step(action)
│   └── Record reward / scores
└── Print final score report

Why examples/simple_agent.py does NOT qualify

• Uses raw httpx and manually constructed HF Inference Router calls instead of the OpenAI Python client.
• Reads API key from HF_API_KEY / HUGGINGFACEHUB_API_TOKEN, not from the required HF_TOKEN / API_BASE_URL / MODEL_NAME variables.
• Is not named inference.py and is not in the root directory.
• Does not iterate over all 3 task tiers with per-sub-env scoring output.

---

2. server/llm_adapter.py — LLM Client Replace

Status: Uses raw urllib.request for HTTP calls — no OpenAI client.

Lines affected

• _call_openai() (L627–667): uses _http_post_json() via urllib. Must be replaced with client.chat.completions.create(...).
• _call_anthropic() (L669–697): same issue.
• _call_huggingface() (L700–776): same issue — currently uses urllib directly to router.huggingface.co.
• All calls to _http_post_json() from provider functions should be replaced.

Changes needed

1. Add from openai import OpenAI at the top.
2. Replace _call_openai() to use the OpenAI client:

   client = OpenAI(api_key=api_key, base_url=base_url or "https://api.openai.com/v1")
   response = client.chat.completions.create(model=model_id, messages=..., ...)
   return response.choices[0].message.content
   

3. Replace _call_huggingface() to also use OpenAI client pointed at https://router.huggingface.co/v1:

   client = OpenAI(api_key=api_key, base_url="https://router.huggingface.co/v1")
   

4. Replace _call_anthropic() — either use the Anthropic SDK or the OpenAI client via the Anthropic OpenAI-compatible endpoint.
5. Add openai>=1.0 to server/requirements.txt and root requirements.txt.

Why this matters

The hackathon rules explicitly state: "Participants must use OpenAI Client for all LLM calls". Using raw urllib.request violates this even if the wire protocol is identical.

---

3. server/requirements.txt — ADD openai

Status: Missing openai package.

  openenv-core[core]>=0.2.2,<0.3
  fastapi>=0.115.0
  pydantic>=2.0
  uvicorn>=0.24.0
+ openai>=1.0
  numpy
  torch
  scipy
  safetensors
  opencv-python
  mediapipe

---

4. requirements.txt (ROOT) — ADD openai

Status: Missing openai package.

  numpy
  torch
  pydantic>=2.0
  scipy
  safetensors
  opencv-python
  mediapipe>=0.10.33
+ openai>=1.0
  pytest

---

5. pyproject.toml — ADD openai to dependencies

Status: openai is absent from the package's dependency list.

  dependencies = [
      "openenv-core[core]>=0.2.2,<0.3",
      "fastapi>=0.115.0",
      "pydantic>=2.0",
      "uvicorn>=0.24.0",
      "numpy",
      "torch",
      "scipy",
      "safetensors",
      "opencv-python",
      "mediapipe",
+     "openai>=1.0",
  ]

---

6. server/Dockerfile — VALIDATE ENV VAR PASSTHROUGH

Status: Partially compliant. Does not explicitly declare required env vars.

The Dockerfile currently just copies files and runs uvicorn. No issue with the build itself, but the following should be verified / added:

# Add explicit ARG/ENV declarations so evaluators can see the required
# environment variables are expected and forwarded at runtime.
ENV API_BASE_URL=""
ENV MODEL_NAME=""
ENV HF_TOKEN=""

Also confirm the image actually builds without network access to pip:

• torch is very heavy. Consider using torch --index-url https://download.pytorch.org/whl/cpu for a CPU-only build since the infrastructure constraint is 2 vCPU / 8 GB RAM (no GPU). This will also substantially reduce image size.

- RUN pip install --no-cache-dir -r /app/requirements.txt
+ RUN pip install --no-cache-dir -r /app/requirements.txt \
+     --extra-index-url https://download.pytorch.org/whl/cpu

---

7. openenv.yaml — OPTIONALLY CLEAN UP CUSTOM VARS

Status: Mostly fine. Contains two non-standard env vars that may confuse validators.

# Current:
env_vars:
  - API_BASE_URL
  - MODEL_NAME
  - HF_TOKEN
  - THB_ALLOW_CUSTOM_BASE_URLS      # <-- custom, not required by spec
  - THB_ALLOWED_BASE_URL_PREFIXES   # <-- custom, not required by spec

These two extra env vars are not harmful, but if the validator script only checks for the three required ones, they may be flagged as unexpected. Consider moving them to a comment block or keeping them with a clear inline explanation.

---

8. examples/simple_agent.py — DOES NOT USE OPENAI CLIENT

Status: Uses raw httpx + custom HF URL. Needs to be converted but is NOT the inference.py required by the hackathon.

• This file uses httpx.Client with a manually constructed HF router URL.
• It reads from HF_API_KEY / HUGGINGFACEHUB_API_TOKEN instead of the required variable names.

This file should either:

1. Be left as-is (fine, it's an example), but inference.py at root level must be written from scratch using the OpenAI client and the correct env var names (API_BASE_URL, MODEL_NAME, HF_TOKEN).
2. Or be upgraded to use the OpenAI client as well to avoid confusion.

---

9. server/talking_head_environment.py — COMBINED EPISODE REWARD FORMULA

Status: The per-episode reward formula for individual task modes is correct. However, the COMBINED reward formula hardcoded in _handle_node8_action() does NOT match the REWARD_LOGIC.md documentation.

The discrepancy

• REWARD_LOGIC.md (and models.py docstring, line 12) states the final reward across ALL three sub-envs is:

  final_reward = 0.25 * subenv1 + 0.35 * subenv2 + 0.40 * subenv3
  

• But _handle_node8_action() (L590–591) uses:

  final_score = 0.50 * s2 + 0.50 * s3
  

  …for clips_and_weights mode, which is a different formula and excludes subenv1.

If the hackathon evaluators run a full-episode combined audit and compare reported scores against your REWARD_LOGIC.md, this inconsistency will be flagged.

Fix needed

Either:

• Update REWARD_LOGIC.md to explicitly document the per-mode formulas (it currently only documents the "global" combined formula), OR
• Implement a proper 3-way episode type that runs subenv1 + subenv2 + subenv3 with the full 0.25/0.35/0.40 weighting.

---

10. server/app.py — MODEL_NAME / API_BASE_URL NOT USED BY SERVER ITSELF

Status: The server reads LLM settings from the AnalyzeIngestionRequest body (per-request model override), not from the required standard env vars.

The hackathon spec requires API_BASE_URL, MODEL_NAME, and HF_TOKEN to be defined in the environment configuration and used for all LLM calls. Currently:

• API_BASE_URL and MODEL_NAME are declared in openenv.yaml as expected vars but the server's analyze_ingestion endpoint ignores them — it reads request.model_id and request.base_url from the request body instead.
• The env vars are never fetched with os.environ.get("MODEL_NAME") etc.

Fix needed

In server/app.py (or server/llm_adapter.py), the analyze_ingested_bundle function should default to the env vars when the request body fields are absent:

import os

model_id = request.model_id or os.environ.get("MODEL_NAME")
api_key  = request.api_key  or os.environ.get("HF_TOKEN")
base_url = request.base_url or os.environ.get("API_BASE_URL")

---

11. README.md — MISSING ACTION/OBSERVATION SPACE TABLE

Status: The README documents architecture, scoring, and deployment. However, it lacks a dedicated, explicit Action Space / Observation Space section.

The hackathon spec requires:

"README with environment description, action/observation spaces, setup instructions"

The current README has architecture diagrams and episode flow tables but no clean, dedicated section labelled "Action Space" / "Observation Space" that clearly lists all fields, types, and ranges in a format a new agent developer could immediately use.

Add a section like:

## Action & Observation Spaces

### Observation Space (reset output per mode)
| Mode    | Schema                      | Key Fields                              |
|---------|-----------------------------|-----------------------------------------|
| image   | ImageDiagnosticsObservation | face_occupancy_ratio, yaw_degrees, ...  |
| clips   | ClipDispositionObservation  | evidence_dossier, marginal_drift, ...   |
| weights | PhonemeRiskObservation      | layer_entropy, rank_utilization, ...    |

### Action Space (step input per mode)
| Mode    | Schema                  | Key Fields                                   |
|---------|-------------------------|----------------------------------------------|
| image   | ImageDiagnosticsAction  | regime_classification, risk_factors, score   |
| clips   | ClipDispositionAction   | disposition, fix_instructions, override      |
| weights | PhonemeRiskAction       | phoneme_risk_ranking, mitigation_recs, ...   |

All numeric reward outputs are bounded in [0.0, 1.0].

---

12. examples/simple_agent.py — ESCAPE SEQUENCES BUG

Status: Minor but causes broken prompts at runtime.

Lines 282–288: The build_user_prompt() function uses Python literal \\n (escaped backslash + n) inside a regular string, which means the actual prompt string will contain the two characters \n instead of a real newline:

# Current (BROKEN — sends literal \n characters):
f"Environment step index: {step_index}\\n"

# Should be:
f"Environment step index: {step_index}\n"

This affects all agent prompts and will cause the LLM to receive mangled single-line instructions rather than well-structured multi-line prompts.

---

13. server/llm_adapter.py — _call_local() FALLS THROUGH TO OLLAMA FORMAT

Status: When no base_url is provided for the local provider, it sends requests to the Ollama native API format (/api/generate) with a prompt key rather than the OpenAI-compatible chat completions format.

This means if evaluators provide API_BASE_URL pointing to any OpenAI-compatible local server (vLLM, LM Studio, etc.), the request format will be wrong.

Fix needed

_call_local() should always use the OpenAI-compatible endpoint:

client = OpenAI(api_key="local", base_url=base_url or "http://localhost:11434/v1")
response = client.chat.completions.create(...)

---

Summary Priority Table

Priority	File	Change
🚨 CRITICAL	inference.py	CREATE from scratch using OpenAI client + correct env vars
🚨 CRITICAL	server/llm_adapter.py	Replace all urllib/httpx calls with OpenAI Python client
🚨 CRITICAL	server/requirements.txt	Add openai>=1.0
🚨 CRITICAL	requirements.txt	Add openai>=1.0
🚨 CRITICAL	pyproject.toml	Add openai>=1.0 to dependencies
🔴 HIGH	server/app.py	Default model_id, api_key, base_url from env vars
🔴 HIGH	README.md	Add explicit Action/Observation Space section
🟡 MEDIUM	server/Dockerfile	Declare ENV vars; switch to CPU torch wheel
🟡 MEDIUM	server/talking_head_environment.py	Clarify/fix reward formula discrepancy vs REWARD_LOGIC.md
🟡 MEDIUM	examples/simple_agent.py	Fix \\n escape sequence bug in prompt builder
🟢 LOW	openenv.yaml	Annotate/clean up non-standard env vars
🟢 LOW	examples/simple_agent.py	Switch to OpenAI client + correct env var names