EVAL_REPORT.md

# Evaluation Report: Project Polymath

## 1. Executive Summary
Project Polymath addresses the "Sycophancy Gap" in current Large Language Models (LLMs)—the tendency to prioritize polite, generic prose over strict adherence to conflicting stakeholder constraints. 

To bridge this gap, we utilize a **two-stage curriculum** on the **OpenEnv** framework:
1.  **Stage 1 (Easy):** Efficiency in hidden constraint discovery (Research).
2.  **Stage 2 (Medium):** Balanced synthesis using a **Harmonic Mean Reward** (Decision Making).

---

## 2. Curriculum Stage 1: Research Stability
We validated the environment using a **Scripted Oracle** to ensure the task is solvable and the rewards are deterministic. We then ran a baseline using the untrained LLM.

### Easy Mode: Performance Metrics
| Metric | Oracle Scripted | Base LLM (Pre-Training) | Delta |
| :--- | :---: | :---: | :---: |
| **Completion Rate** | 1.00 | 1.00 | -- |
| **Avg. Cumulative Reward** | 0.99 | 0.825 | -0.165 |
| **Avg. Final Step Reward** | 0.33 | 0.264 | -0.066 |
| **Avg. Turns Completed** | 3.0 | 3.2 | +0.2 turns |
| **Constraint Discovery Rate**| 100% | 100% | -- |

### The "Policy Efficiency" Gap
While the Base LLM successfully discovers the constraints, it demonstrates **sloppy policy logic**:
* **Redundancy:** Episodes 2 and 4 showed the agent asking the same expert identical questions multiple times.
* **Shortcut Misuse:** Episode 5 showed the agent using `target="All"` to broadcast, resulting in a **0.0 reward** due to environment-enforced privacy/discipline penalties.

---

## 3. Curriculum Stage 2: Synthesis & The "Final Boss"
In Medium Mode, we shift the reward signal from "Discovery" to "Synthesis." The model must now generate a PRD that satisfies all three constraints simultaneously.

### Medium Mode Baseline (The Problem)
| Metric | Base LLM (Pre-Training) | Target (Post-Training) |
| :--- | :--- | :--- |
| **Avg. Final Reward** | **0.00** | **> 0.90** |
| **Synthesis Accuracy** | **Low** | **High** |

**Observation:**
Even when the Base LLM knows the constraints (e.g., $50k budget), it often omits them in the final PRD in favor of professional-sounding "filler" text. Because we use a **Harmonic Mean Reward**, failing to satisfy even one stakeholder (Finance, Security, or UX) results in a total reward collapse to **0.0**.

---

## 4. Training Roadmap (Onsite 36-Hour Sprint)
Our objective is to close the gap between the **Base LLM** and the **Oracle**.

### Training Targets
| Focus Area | Metric | Baseline | Target |
| :--- | :--- | :---: | :---: |
| **Policy** | Broadcast (`target="All"`) Usage | Present | **0%** |
| **Policy** | Repeated Query Rate | Present | **< 5%** |
| **Logic** | Multi-Constraint Synthesis | 0% | **> 90%** |

---

## 5. Judging Narrative & Pitch
> "Our evaluation proves that LLMs don't fail because they are 'uninformed'; they fail because their default policy is inefficient and sycophantic. We built a stable, verifiable gym where the Oracle proves perfection is possible. Our GRPO training will move the agent from a **0.825** sloppy researcher to a **0.99** disciplined negotiator and bridge the **0.0 to 0.9** gap in multi-stakeholder synthesis."

---

## 6. Failure Breakdown (Pre-Training)
| Failure Type | Count | Interpretation |
| :--- | :---: | :--- |
| Policy Loops | 2/10 | Asked the same question 3 times in one episode. |
| Broadcast Penalty | 2/10 | Tried to message 'All' to skip individual negotiation. |
| Synthesis Failure | 10/10 | Failed to include all 3 exact patterns in the final PRD (Medium). |



The Story This Tells the Judges:

If you walk into the judging room with a model that already scores a $1.0$, the judges will say, "You didn't need RL for this. Just use a better prompt."By showing them this baseline, you are saying:"Prompt engineering isn't enough for complex, multi-agent negotiation. The base model gets distracted (Episode 5), hallucinates JSON (Episode 3), and fails to synthesize all constraints into the final document (Episode 2). It's a sycophant that tries to please the last person it talked to. We need Reinforcement Learning to fix this."