# TalkingHeadBench — Hackathon Compliance Change Plan

A file-by-file breakdown of every change needed to fully comply with the
[OpenEnv hackathon requirements](./hackathon_requirements.md).

---

## 1. `inference.py` (ROOT) — MISSING, MUST CREATE

**Status: Does not exist.** This is an immediate disqualification condition.

### What it needs to do
- Connect to the running TalkingHeadBench environment at a configurable URL.
- Run a full benchmark episode across **all 3 task tiers** (image audit, clip audit,
  weight audit) using test-set cases loaded from `tests/test_set/`.
- Use the **OpenAI Python client** (`from openai import OpenAI`) for LLM calls
  — not the current raw `urllib`/`httpx` approach.
- Pull LLM config exclusively from the three required environment variables:
  - `API_BASE_URL`
  - `MODEL_NAME`
  - `HF_TOKEN`
- Print a structured score report at the end: per-sub-env scores and the
  weighted final score.
- Complete in under **20 minutes** on 2 vCPU / 8 GB RAM.

### Required structure (pseudocode)
```
inference.py
├── Load API_BASE_URL, MODEL_NAME, HF_TOKEN from os.environ
├── Instantiate: client = OpenAI(api_key=HF_TOKEN, base_url=API_BASE_URL)
├── For each task: "image", "clips", "weights"
│   ├── Call env.reset(mode=<task>)   — via HTTP to the running server
│   ├── Read observation, construct LLM prompt
│   ├── Call client.chat.completions.create(model=MODEL_NAME, ...)
│   ├── Parse structured JSON action
│   ├── Call env.step(action)
│   └── Record reward / scores
└── Print final score report
```

### Why `examples/simple_agent.py` does NOT qualify
- Uses raw `httpx` and manually constructed HF Inference Router calls instead of
  the OpenAI Python client.
- Reads API key from `HF_API_KEY` / `HUGGINGFACEHUB_API_TOKEN`, not from the
  required `HF_TOKEN` / `API_BASE_URL` / `MODEL_NAME` variables.
- Is not named `inference.py` and is not in the root directory.
- Does not iterate over all 3 task tiers with per-sub-env scoring output.

---

## 2. `server/llm_adapter.py` — LLM Client Replace

**Status: Uses raw `urllib.request` for HTTP calls — no OpenAI client.**

### Lines affected
- `_call_openai()` (L627–667): uses `_http_post_json()` via `urllib`. Must be
  replaced with `client.chat.completions.create(...)`.
- `_call_anthropic()` (L669–697): same issue.
- `_call_huggingface()` (L700–776): same issue — currently uses `urllib`
  directly to `router.huggingface.co`.
- All calls to `_http_post_json()` from provider functions should be replaced.

### Changes needed
1. Add `from openai import OpenAI` at the top.
2. Replace `_call_openai()` to use the OpenAI client:
   ```python
   client = OpenAI(api_key=api_key, base_url=base_url or "https://api.openai.com/v1")
   response = client.chat.completions.create(model=model_id, messages=..., ...)
   return response.choices[0].message.content
   ```
3. Replace `_call_huggingface()` to also use OpenAI client pointed at
   `https://router.huggingface.co/v1`:
   ```python
   client = OpenAI(api_key=api_key, base_url="https://router.huggingface.co/v1")
   ```
4. Replace `_call_anthropic()` — either use the Anthropic SDK or the OpenAI
   client via the Anthropic OpenAI-compatible endpoint.
5. Add `openai>=1.0` to `server/requirements.txt` and root `requirements.txt`.

### Why this matters
The hackathon rules explicitly state: **"Participants must use OpenAI Client for
all LLM calls"**. Using raw `urllib.request` violates this even if the wire
protocol is identical.

---

## 3. `server/requirements.txt` — ADD `openai`

**Status: Missing `openai` package.**

```diff
  openenv-core[core]>=0.2.2,<0.3
  fastapi>=0.115.0
  pydantic>=2.0
  uvicorn>=0.24.0
+ openai>=1.0
  numpy
  torch
  scipy
  safetensors
  opencv-python
  mediapipe
```

---

## 4. `requirements.txt` (ROOT) — ADD `openai`

**Status: Missing `openai` package.**

```diff
  numpy
  torch
  pydantic>=2.0
  scipy
  safetensors
  opencv-python
  mediapipe>=0.10.33
+ openai>=1.0
  pytest
```

---

## 5. `pyproject.toml` — ADD `openai` to dependencies

**Status: `openai` is absent from the package's dependency list.**

```diff
  dependencies = [
      "openenv-core[core]>=0.2.2,<0.3",
      "fastapi>=0.115.0",
      "pydantic>=2.0",
      "uvicorn>=0.24.0",
      "numpy",
      "torch",
      "scipy",
      "safetensors",
      "opencv-python",
      "mediapipe",
+     "openai>=1.0",
  ]
```

---

## 6. `server/Dockerfile` — VALIDATE ENV VAR PASSTHROUGH

**Status: Partially compliant. Does not explicitly declare required env vars.**

The Dockerfile currently just copies files and runs uvicorn. No issue with the
build itself, but the following should be verified / added:

```dockerfile
# Add explicit ARG/ENV declarations so evaluators can see the required
# environment variables are expected and forwarded at runtime.
ENV API_BASE_URL=""
ENV MODEL_NAME=""
ENV HF_TOKEN=""
```

Also confirm the image actually builds without network access to pip:
- `torch` is very heavy. Consider using `torch --index-url
  https://download.pytorch.org/whl/cpu` for a CPU-only build since the
  infrastructure constraint is 2 vCPU / 8 GB RAM (no GPU). This will also
  substantially reduce image size.

```diff
- RUN pip install --no-cache-dir -r /app/requirements.txt
+ RUN pip install --no-cache-dir -r /app/requirements.txt \
+     --extra-index-url https://download.pytorch.org/whl/cpu
```

---

## 7. `openenv.yaml` — OPTIONALLY CLEAN UP CUSTOM VARS

**Status: Mostly fine. Contains two non-standard env vars that may confuse validators.**

```yaml
# Current:
env_vars:
  - API_BASE_URL
  - MODEL_NAME
  - HF_TOKEN
  - THB_ALLOW_CUSTOM_BASE_URLS      # <-- custom, not required by spec
  - THB_ALLOWED_BASE_URL_PREFIXES   # <-- custom, not required by spec
```

These two extra env vars are not harmful, but if the validator script only
checks for the three required ones, they may be flagged as unexpected. Consider
moving them to a comment block or keeping them with a clear inline explanation.

---

## 8. `examples/simple_agent.py` — DOES NOT USE OPENAI CLIENT

**Status: Uses raw `httpx` + custom HF URL. Needs to be converted but is NOT
the `inference.py` required by the hackathon.**

- This file uses `httpx.Client` with a manually constructed HF router URL.
- It reads from `HF_API_KEY` / `HUGGINGFACEHUB_API_TOKEN` instead of the
  required variable names.

This file should either:
1. Be **left as-is** (fine, it's an example), but `inference.py` at root
   level must be written from scratch using the OpenAI client **and** the
   correct env var names (`API_BASE_URL`, `MODEL_NAME`, `HF_TOKEN`).
2. Or be **upgraded** to use the OpenAI client as well to avoid confusion.

---

## 9. `server/talking_head_environment.py` — COMBINED EPISODE REWARD FORMULA

**Status: The per-episode reward formula for individual task modes is correct.
However, the COMBINED reward formula hardcoded in `_handle_node8_action()` does
NOT match the REWARD_LOGIC.md documentation.**

### The discrepancy
- `REWARD_LOGIC.md` (and `models.py` docstring, line 12) states the final reward
  across ALL three sub-envs is:
  ```
  final_reward = 0.25 * subenv1 + 0.35 * subenv2 + 0.40 * subenv3
  ```
- But `_handle_node8_action()` (L590–591) uses:
  ```python
  final_score = 0.50 * s2 + 0.50 * s3
  ```
  …for `clips_and_weights` mode, which is a different formula and excludes subenv1.

If the hackathon evaluators run a full-episode combined audit and compare
reported scores against your REWARD_LOGIC.md, this inconsistency will be
flagged.

### Fix needed
Either:
- Update `REWARD_LOGIC.md` to explicitly document the per-mode formulas (it
  currently only documents the "global" combined formula), OR
- Implement a proper 3-way episode type that runs `subenv1 + subenv2 + subenv3`
  with the full `0.25/0.35/0.40` weighting.

---

## 10. `server/app.py` — `MODEL_NAME` / `API_BASE_URL` NOT USED BY SERVER ITSELF

**Status: The server reads LLM settings from the `AnalyzeIngestionRequest`
body (per-request model override), not from the required standard env vars.**

The hackathon spec requires `API_BASE_URL`, `MODEL_NAME`, and `HF_TOKEN` to be
defined in the environment configuration and used for all LLM calls. Currently:

- `API_BASE_URL` and `MODEL_NAME` are declared in `openenv.yaml` as expected vars
  but the server's `analyze_ingestion` endpoint ignores them — it reads
  `request.model_id` and `request.base_url` from the request body instead.
- The env vars are never fetched with `os.environ.get("MODEL_NAME")` etc.

### Fix needed
In `server/app.py` (or `server/llm_adapter.py`), the `analyze_ingested_bundle`
function should default to the env vars when the request body fields are absent:

```python
import os

model_id = request.model_id or os.environ.get("MODEL_NAME")
api_key  = request.api_key  or os.environ.get("HF_TOKEN")
base_url = request.base_url or os.environ.get("API_BASE_URL")
```

---

## 11. README.md — MISSING ACTION/OBSERVATION SPACE TABLE

**Status: The README documents architecture, scoring, and deployment. However,
it lacks a dedicated, explicit Action Space / Observation Space section.**

The hackathon spec requires:
> "README with environment description, action/observation spaces, setup instructions"

The current README has architecture diagrams and episode flow tables but no
clean, dedicated section labelled "Action Space" / "Observation Space" that
clearly lists all fields, types, and ranges in a format a new agent developer
could immediately use.

### Add a section like:
```markdown
## Action & Observation Spaces

### Observation Space (reset output per mode)
| Mode    | Schema                      | Key Fields                              |
|---------|-----------------------------|-----------------------------------------|
| image   | ImageDiagnosticsObservation | face_occupancy_ratio, yaw_degrees, ...  |
| clips   | ClipDispositionObservation  | evidence_dossier, marginal_drift, ...   |
| weights | PhonemeRiskObservation      | layer_entropy, rank_utilization, ...    |

### Action Space (step input per mode)
| Mode    | Schema                  | Key Fields                                   |
|---------|-------------------------|----------------------------------------------|
| image   | ImageDiagnosticsAction  | regime_classification, risk_factors, score   |
| clips   | ClipDispositionAction   | disposition, fix_instructions, override      |
| weights | PhonemeRiskAction       | phoneme_risk_ranking, mitigation_recs, ...   |

All numeric reward outputs are bounded in [0.0, 1.0].
```

---

## 12. `examples/simple_agent.py` — ESCAPE SEQUENCES BUG

**Status: Minor but causes broken prompts at runtime.**

Lines 282–288: The `build_user_prompt()` function uses Python literal `\\n`
(escaped backslash + n) inside a regular string, which means the actual prompt
string will contain the two characters `\n` instead of a real newline:

```python
# Current (BROKEN — sends literal \n characters):
f"Environment step index: {step_index}\\n"

# Should be:
f"Environment step index: {step_index}\n"
```

This affects all agent prompts and will cause the LLM to receive mangled
single-line instructions rather than well-structured multi-line prompts.

---

## 13. `server/llm_adapter.py` — `_call_local()` FALLS THROUGH TO OLLAMA FORMAT

**Status: When no `base_url` is provided for the `local` provider, it sends
requests to the Ollama native API format (`/api/generate`) with a `prompt` key
rather than the OpenAI-compatible chat completions format.**

This means if evaluators provide `API_BASE_URL` pointing to any OpenAI-compatible
local server (vLLM, LM Studio, etc.), the request format will be wrong.

### Fix needed
`_call_local()` should always use the OpenAI-compatible endpoint:
```python
client = OpenAI(api_key="local", base_url=base_url or "http://localhost:11434/v1")
response = client.chat.completions.create(...)
```

---

## Summary Priority Table

| Priority | File | Change |
|----------|------|--------|
| 🚨 CRITICAL | `inference.py` | CREATE from scratch using OpenAI client + correct env vars |
| 🚨 CRITICAL | `server/llm_adapter.py` | Replace all `urllib`/`httpx` calls with OpenAI Python client |
| 🚨 CRITICAL | `server/requirements.txt` | Add `openai>=1.0` |
| 🚨 CRITICAL | `requirements.txt` | Add `openai>=1.0` |
| 🚨 CRITICAL | `pyproject.toml` | Add `openai>=1.0` to dependencies |
| 🔴 HIGH | `server/app.py` | Default `model_id`, `api_key`, `base_url` from env vars |
| 🔴 HIGH | `README.md` | Add explicit Action/Observation Space section |
| 🟡 MEDIUM | `server/Dockerfile` | Declare ENV vars; switch to CPU torch wheel |
| 🟡 MEDIUM | `server/talking_head_environment.py` | Clarify/fix reward formula discrepancy vs REWARD_LOGIC.md |
| 🟡 MEDIUM | `examples/simple_agent.py` | Fix `\\n` escape sequence bug in prompt builder |
| 🟢 LOW | `openenv.yaml` | Annotate/clean up non-standard env vars |
| 🟢 LOW | `examples/simple_agent.py` | Switch to OpenAI client + correct env var names |