Upload folder using huggingface_hub

.gitattributes

@@ -49,3 +49,9 @@ wheels/urllib3-2.6.3-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
 wheels/uvloop-0.22.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl filter=lfs diff=lfs merge=lfs -text
 wheels/watchfiles-1.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl filter=lfs diff=lfs merge=lfs -text
 wheels/websockets-16.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl filter=lfs diff=lfs merge=lfs -text
+README.pdf filter=lfs diff=lfs merge=lfs -text
+assets/formulas-1.png filter=lfs diff=lfs merge=lfs -text
+assets/hybrid.png filter=lfs diff=lfs merge=lfs -text
+assets/reflexion.png filter=lfs diff=lfs merge=lfs -text
+assets/sys[[:space:]]arch.png filter=lfs diff=lfs merge=lfs -text
+images/big.png filter=lfs diff=lfs merge=lfs -text

PIPELINE.md

@@ -0,0 +1,301 @@
+# GraphStrike — End-to-End Pipeline
+
+A complete walkthrough of how GraphStrike works, from data generation to scoring.
+
+---
+
+## Pipeline Overview
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                        BUILD TIME (Docker)                         │
+│                                                                     │
+│  generator.py ──► 150 episode JSONs (50 per difficulty tier)       │
+│       │              each with 10 gang + N real + decoys            │
+│       │              deterministic by seed                          │
+│       ▼                                                             │
+│  episodes/easy_000.json ... hard_049.json                          │
+└─────────────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│                     RUNTIME (Environment Server)                    │
+│                                                                     │
+│  FastAPI + Gradio @ port 7860                                      │
+│                                                                     │
+│  POST /reset ──► Load episode, init graph, reveal visible IDs      │
+│  POST /step  ──► Execute action, update state, return observation  │
+│  GET  /grader ──► Compute final score after SUBMIT                 │
+│  POST /baseline ──► Run rule-based agent on all 3 tasks            │
+└─────────────────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────────────┐
+│                      INFERENCE (Agent / LLM)                        │
+│                                                                     │
+│  inference.py connects to server via HTTP                          │
+│  Uses OpenAI-compatible client to call LLM                         │
+│  Emits structured logs: [START] [STEP] [END]                      │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Stage 1: Synthetic Data Generation
+
+**File:** `server/generator.py`
+
+```
+Input:  seed (int) + difficulty tier (easy/medium/hard)
+Output: episode JSON with full social graph
+```
+
+### What gets generated per episode:
+
+| Component | Easy | Medium | Hard |
+|-----------|------|--------|------|
+| Total accounts | 50 | 200 | 1000 |
+| Gang members (target) | 10 | 10 | 10 |
+| Decoy accounts | 0 | 20 | 50 |
+| Celebrity accounts | 2 | 2 | 2 |
+| Zero-edge accounts | 2 | 2 | 2 |
+| Max steps | 30 | 50 | 80 |
+| Evasion events | None | At step 20 | Recurring |
+
+### Gang member signals (coordinated):
+- `ip_cluster_id = "ip_gang_{seed}"` — all 10 share one IP
+- `shared_ip_count = 9` — each sees 9 others on same IP
+- `photo_reuse_score ∈ [0.70, 0.95]` — stolen celebrity photos
+- `bio_template_score ∈ [0.60, 0.90]` — copy-paste bios
+- `comment_repeat_score ∈ [0.60, 0.90]` — spam comments
+- `avg_post_hour` clustered within 2h window — coordinated posting
+
+### Real account signals (independent):
+- Unique IP per account
+- `photo_reuse_score ∈ [0.0, 0.05]`
+- `bio_template_score ∈ [0.0, 0.08]`
+- `comment_repeat_score ∈ [0.0, 0.08]`
+
+---
+
+## Stage 2: Environment State Machine
+
+**File:** `server/environment.py`
+
+```
+              POST /reset
+                  │
+                  ▼
+        ┌─────────────────┐
+        │   EPISODE INIT   │
+        │  Load JSON graph │
+        │  Reveal visible  │
+        │  IDs to agent    │
+        └────────┬────────┘
+                 │
+                 ▼
+        ┌─────────────────┐     POST /step
+        │   AGENT LOOP     │◄────────────────┐
+        │                  │                  │
+        │  Observation:    │    ┌��─────────┐  │
+        │  - visible_ids   │    │  ACTION   │  │
+        │  - profiles      │    │          │  │
+        │  - flagged_ids   │──►│ INSPECT   │──┘
+        │  - suspect_ids   │    │ INVEST_NET│
+        │  - steps_remain  │    │ FLAG      │
+        │  - message/hints │    │ UNFLAG    │
+        └────────┬────────┘    │ SUBMIT    │
+                 │              └──────────┘
+                 │ (SUBMIT or steps=0)
+                 ▼
+        ┌─────────────────┐
+        │   EPISODE END    │
+        │  Compute grader  │
+        │  score via       │
+        │  scoring.py      │
+        └─────────────────┘
+                 │
+                 ▼
+          GET /grader
+          → { score: 0.0-1.0 }
+```
+
+### Action Details
+
+| Action | Step Cost | Effect |
+|--------|-----------|--------|
+| **INSPECT** | 1 | Reveals full profile + risk scores for one account |
+| **INVESTIGATE_NETWORK** | 1 | 2-hop bidirectional expansion, reveals neighbor IDs |
+| **FLAG** | 0 (free) | Marks account as fake, triggers dual cascade |
+| **UNFLAG** | 0 (free) | Removes flag from account |
+| **SUBMIT** | 0 | Ends episode, triggers grading |
+
+### Dual Cascade on FLAG
+
+```
+FLAG acc_0012
+    │
+    ├──► Follow-graph cascade
+    │    For each neighbor of acc_0012:
+    │      if visible AND status=NORMAL → set SUSPECT
+    │
+    └──► IP cluster cascade
+         Find all accounts with same ip_cluster_id:
+           if visible AND status=NORMAL → set SUSPECT
+```
+
+---
+
+## Stage 3: Risk Scoring Engine
+
+**File:** `server/scoring.py`
+
+All scores computed at INSPECT time. Stateless pure functions.
+
+```
+                    ┌─────────────┐
+                    │ Raw Features │
+                    │ from profile │
+                    └──────┬──────┘
+                           │
+              ┌────────────┼────────────┐
+              ▼            ▼            ▼
+        ┌──────────┐ ┌──────────┐ ┌──────────┐
+        │ Node Risk│ │Behavior  │ │Graph Risk│
+        │ 0.6×photo│ │Risk      │ │0.45×flag │
+        │+0.4×bio  │ │0.55×age  │ │ _nbr_rat │
+        └────┬─────┘ │+0.45×hr  │ │+0.35×mut │
+             │       │ cluster  │ │+0.20×nbr │
+             │       └────┬─────┘ │ photo    │
+             │            │       └────┬─────┘
+             │            │            │
+             ▼            ▼            ▼
+        ┌──────────────────────────────────┐    ┌──────────────┐
+        │         fake_risk_score           │◄───│Hub Legitimacy│
+        │ 0.30×node + 0.25×beh + 0.45×graph│    │(discount for │
+        │         − 0.25×hub               │    │ celebrities) │
+        │         clamp [0.0, 1.0]         │    └──────────────┘
+        └──────────────────────────────────┘
+```
+
+### Grader Formula (after SUBMIT)
+
+```
+If recall ≥ 0.8 AND precision ≥ 0.7:
+    score = 0.55 + 0.20×recall + 0.15×precision + 0.10×efficiency
+
+Else (partial credit):
+    score = 0.30×recall + 0.10×precision
+
+Where:
+    recall     = TP / 10
+    precision  = TP / (TP + FP)
+    efficiency = (max_steps − steps_used) / max_steps
+```
+
+---
+
+## Stage 4: Inference Pipeline
+
+**File:** `inference.py`
+
+### Rule-Based Agent (baseline)
+
+```
+while not done:
+    1. INSPECT suspects first (auto-elevated by cascade)
+    2. FLAG any inspected account with fake_risk ≥ threshold
+       (easy: 0.60, medium: 0.50, hard: 0.45)
+    3. INSPECT highest-risk uninspected account
+    4. SUBMIT when 10 flagged or steps running low
+```
+
+### LLM Agent (hybrid policy)
+
+```
+┌──────────────┐     ┌───────────────┐     ┌──────────────┐
+│ Observation   │────►│ Rule Engine   │────►│   α blend    │
+│ (from server) │     │ (get_rule_    │     │              │
+│               │     │  action +     │     │ if conf>thr: │
+│               │────►│  confidence)  │     │   use rule   │
+│               │     └───────────────┘     │ else:        │
+│               │                           │   query LLM  │
+│               │     ┌───────────────┐     │   blend      │
+│               │────►│ LLM (OpenAI   │────►│   actions    │
+│               │     │  client)      │     └──────┬───────┘
+│               │     └───────────────┘            │
+└──────────────┘                                   ▼
+                                            ┌──────────────┐
+                                            │ Final Action │
+                                            └──────────────┘
+
+α caps: easy ≤ 0.50 | medium ≤ 0.70 | hard ≤ 0.85
+(rule engine retains veto power on high-confidence decisions)
+```
+
+### Structured Log Format
+
+```
+[START] task=easy env=graphstrike model=Qwen/Qwen2.5-72B-Instruct
+[STEP] step=1 action=inspect:acc_0012 reward=0.00 done=false error=null
+[STEP] step=2 action=flag:acc_0012 reward=1.20 done=false error=null
+...
+[STEP] step=15 action=submit reward=12.40 done=true error=null
+[END] success=true steps=15 score=0.910 rewards=0.00,1.20,...,12.40
+```
+
+---
+
+## Stage 5: Deployment
+
+```
+┌─────────────────────────────────────────┐
+│          Hugging Face Spaces             │
+│                                          │
+│  Docker container @ port 7860            │
+│  ┌────────────────────────────────────┐  │
+│  │ FastAPI (API endpoints)            │  │
+│  │   /health /tasks /reset /step      │  │
+│  │   /state /grader /baseline         │  │
+│  │   /metadata /schema /mcp           │  │
+│  ├────────────────────────────────────┤  │
+│  │ Gradio UI (mounted at /)           │  │
+│  │   Manual testing interface         │  │
+│  │   Reset / Step / Grader buttons    │  │
+│  └────────────────────────────────────┘  │
+│                                          │
+│  /web → redirect to / (HF probe compat) │
+└─────────────────────────────────────────┘
+          │                    ▲
+          │ HTTP               │ HTTP
+          ▼                    │
+┌─────────────────────┐  ┌────────────┐
+│  inference.py       │  │  openenv   │
+│  (runs externally)  │  │  validate  │
+│  LLM ←→ Server      │  │  (judging) │
+└─────────────────────┘  └────────────┘
+```
+
+---
+
+## Quick Start
+
+```bash
+# 1. Build & run the environment server
+docker build -f Dockerfile -t graphstrike .
+docker run -p 7860:7860 graphstrike
+
+# 2. Verify endpoints
+curl http://localhost:7860/health
+curl http://localhost:7860/tasks
+curl -X POST http://localhost:7860/baseline
+
+# 3. Run LLM inference (separate terminal)
+export API_KEY="your-hf-token"
+export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
+python3 inference.py --url http://localhost:7860 --all-tasks
+
+# 4. Validate submission
+openenv validate
+openenv validate --url http://localhost:7860
+```

README.md

@@ -15,783 +15,255 @@ tags:
   - llm-agent
 base_path: /web
 ---
+<br>
 
-# GraphStrike : Coordinated Fake Account Ring Detection
+<p align="center">
+<img src="assets/logo.png" width="600"/>
+</p>
 
-> **OpenEnv Hackathon × SCALER School of Technology**
-> Live deployment: [huggingface.co/spaces/Pandago/graphstrike](https://huggingface.co/spaces/Pandago/graphstrike)
+<br>
 
-An OpenEnv-compatible reinforcement learning environment where an LLM agent
-must identify all 10 members of a coordinated fake account ring hidden
-inside a synthetic social network. The agent learns via **Reflexion** and a
-**dynamic hybrid rule/LLM policy** , not via gradient updates or fine-tuning.
+<p align="center">
+  <img src="https://img.shields.io/badge/Hugging%20Face-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black"/>
+  <img src="https://img.shields.io/badge/HF%20Spaces-FFBF00?style=for-the-badge&logo=huggingface&logoColor=black"/>
+  <img src="https://img.shields.io/badge/FastAPI-009688?style=for-the-badge&logo=fastapi&logoColor=white"/>
+  <img src="https://img.shields.io/badge/Docker-2496ED?style=for-the-badge&logo=docker&logoColor=white"/>
+  <img src="https://img.shields.io/badge/Gradio-F97316?style=for-the-badge&logo=gradio&logoColor=white"/>
+  <img src="https://img.shields.io/badge/OpenEnv-4B5563?style=for-the-badge&logo=envato&logoColor=white"/>
+  <img src="https://img.shields.io/badge/Amazon%20Bedrock-FF9900?style=for-the-badge&logo=amazonaws&logoColor=white"/>
+</p>
+<br>
 
----
+<h1 align="center">
+</h1>
+  <p align="center">
+    An OpenEnv-compatible reinforcement learning environment where an LLM agent must identify all 10 members of a coordinated fake account network hidden inside a synthetic social network. The agent learns via Reflexion and a dynamic hybrid rule/LLM policy , not via gradient updates or fine-tuning.
+    <br />
+    </p>
+</p>
 
-## Table of Contents
+<br>
 
-1. [What This Is](#1-what-this-is)
-2. [Repository Layout](#2-repository-layout)
-3. [The Problem: How Fake Detection Actually Works](#3-the-problem-how-fake-detection-actually-works)
-4. [Synthetic Data Generation](#4-synthetic-data-generation)
-5. [Data Model — Every Field Explained](#5-data-model--every-field-explained)
-6. [The RL Environment](#6-the-rl-environment)
-7. [Risk Scoring Mathematics](#7-risk-scoring-mathematics)
-8. [Account Status State Machine](#8-account-status-state-machine)
-9. [The LLM Policy (Qwen3 via Bedrock)](#9-the-llm-policy-qwen3-via-bedrock)
-10. [Reflexion — How the Agent Learns](#10-reflexion--how-the-agent-learns)
-11. [Hybrid Policy — The Novel Contribution](#11-hybrid-policy--the-novel-contribution)
-12. [Training Loop End-to-End](#12-training-loop-end-to-end)
-13. [API Reference](#13-api-reference)
-14. [Docker Deployment](#14-docker-deployment)
-15. [Submission Requirements](#15-submission-requirements)
-16. [Verification &amp; Validation](#16-verification--validation)
+## Theme
 
----
+**SUPPORT**
 
-## 1. What This Is
+### Customer Service Agents
 
-This is an **OpenEnv hackathon** submission. OpenEnv is a framework for building
-reinforcement learning environments with a standard microservice interface
-(`/reset`, `/step`, `/state`) so that any agent implementation can plug in.
+Complex environment where agents resolve multi-step queries using external tools and APIs.
 
-**The task:** A social network contains fake accounts organised into a
-single coordinated ring of 10. The ring behaves in a coordinated way — same posting hour,
-same IP subnet, stolen celebrity photos, copy-paste bios. The agent must find
-all 10 by navigating a limited step budget, inspecting accounts, and flagging suspects.
+## Problem Statement
 
-**What makes this non-trivial:**
+**The task:** A social network contains fake accounts organised into a single coordinated ring of 10. The ring behaves in a coordinated way — same posting hour, same IP subnet, stolen celebrity photos, copy-paste bios. The agent must find all 10 by navigating a limited step budget, inspecting accounts, and flagging suspects.
 
-- The network is large (50–1000 accounts depending on difficulty).
-- Fake accounts are mixed with innocent high-signal "decoy" accounts.
-- In hard mode, the gang actively evades — dropping intra-gang follows,
-  renaming profiles — while the agent is mid-investigation.
-- The agent cannot see the full network upfront: it must explore via INSPECT and
-  INVESTIGATE_NETWORK actions, spending steps to reveal information.
-
-**What makes the learning novel:**
+## Proposed Solution
 
-- The LLM (Qwen3-80B via AWS Bedrock) cannot be fine-tuned — it is a black-box API.
-- The agent learns via **Reflexion**: post-episode lessons are written back into
-  memory and injected into every future prompt.
-- A **dynamic hybrid policy** (α-weighted) blends the LLM with a deterministic
-  rule engine, with the blend weight α updating based on recent win rate.
-  Rules dominate early; the LLM takes over as it proves itself.
+An OpenEnv-compatible reinforcement learning environment where an LLM agent must identify all 10 members of a coordinated fake account ring hidden inside a synthetic social network. The agent learns via **Reflexion** and a **dynamic hybrid rule/LLM policy** — not via gradient updates or fine-tuning.
 
 ---
-
-## 2. Repository Layout
-
-```
-fake_gang_env/
-│
-├── models.py                  # All Pydantic types: Action, Observation, State, Profile
-├── bedrock_model.py           # AWS Bedrock client — invoke_qwen()
-├── client.py                  # HTTP client for talking to the running server
-├── inference.py               # Submission: rule-based baseline runner + HTTP client mode
-├── validate.py                # Submission: pre-submission validator (24 checks)
-├── train.py                   # Main training loop (curriculum + hybrid policy)
-├── run.sh                     # Docker entrypoint: episodes → server → training
-├── requirements.txt           # Python dependencies
-│
-├── server/
-│   ├── app.py                 # FastAPI server: /reset /step /state /health /tasks /grader /baseline
-│   ├── environment.py         # Core RL environment — FakeGangEnvironment class
-│   ├── generator.py           # Synthetic episode generator (50 per task × 3 tasks = 150 files)
-│   ├── scoring.py             # Pure-math risk formula engine (stateless functions)
-│   ├── Dockerfile             # Offline pip install via pre-downloaded wheels
-│   └── .dockerignore          # Excludes episodes/, memory/, runs/ from build context
-│
-├── agent/
-│   ├── policy.py              # LLM policy: formats obs → calls Qwen → parses <action> tag
-│   ├── hybrid_policy.py       # Hybrid policy: blends rules + LLM via dynamic α
-│   ├── memory.py              # Disk-backed memory: reflections, trajectories, win history, α
-│   └── reflection.py          # Post-episode reflection generator (also calls Qwen)
-│
-├── episodes/                  # 150 pre-generated JSON episode files (excluded from Docker build)
-├── memory/                    # Docker volume: reflections, trajectories, α values (persists)
-└── runs/                      # Docker volume: per-episode metrics JSONL (persists)
-```
-
+## Novelty Highlights
+
+- **Adaptive Hybrid Intelligence (Rules + LLM):** Unlike static ensembles, GraphStrike dynamically blends deterministic rules and LLM reasoning using a trust gate, shifting control as performance improves.
+- **Learning Without Fine-Tuning:** Instead of updating model weights, the agent learns through Reflexion lessons and best-trajectory memory injected into future prompts.
+- **Graph-First Detection Pipeline:** Detection is not account-by-account only; it uses cascade effects, neighbor propagation, and multi-hop graph expansion to uncover coordinated rings.
+- **Math-Grounded Decision Control:** Risk composition, trust calibration, and grader alignment are formula-driven, making behavior interpretable and reproducible.
+- **Adversarial Evasion Benchmarking:** Hard-mode includes timed evasion events, so success reflects robustness under disruption rather than overfitting to static patterns.
+- **Safety-Net by Design:** High-confidence rule overrides prevent catastrophic LLM errors while preserving LLM flexibility for strategic exploration.
 ---
 
-## 3. The Problem: How Fake Detection Actually Works
-
-A real-world fake account detector does **not** read post content. Content is
-expensive to process, multilingual, and easily spoofed. Instead, detection relies
-on three categories of signals that are computed from metadata:
-
-### 3.1 Node Signals (per-account features)
-
-These are pre-computed by a content pipeline before the agent ever sees the account:
+## Performance Summary
 
-| Feature                  | What it measures                                                             | Fake range | Real range |
-| ------------------------ | ---------------------------------------------------------------------------- | ---------- | ---------- |
-| `photo_reuse_score`    | Fraction of posts using stolen celebrity photos (pHash fingerprint matching) | 0.30–0.95 | 0.00–0.15 |
-| `bio_template_score`   | Cosine similarity of bio text to known fake-account bio templates            | 0.20–0.90 | 0.00–0.12 |
-| `comment_repeat_score` | Fraction of comments that are copy-pasted across accounts (spam pattern)     | 0.60–0.90 | 0.00–0.08 |
+We evaluate GraphStrike's hybrid rule/LLM policy across multiple *frontier models to measure how well each model handles the investigation task. All runs use
+the same inference pipeline (`inference.py`) with identical system prompts and structured logging. Each model ran: (1) seed=0 on all 3 tasks, and
+(2) seeds 0-2 on all 3 tasks for variance measurement.*
 
-The agent receives these as numbers (not the raw content). The content pipeline
-runs offline; the agent operates on numeric features only.
+**Seed=0 scores (single episode per task):**
 
-### 3.2 Behavioral Signals (temporal + device)
+<p align="center">
+  <img src="images/table1.png" alt="Model Performance Table" width="1600"/>
+</p>
+<br>
 
-| Feature              | What it measures                                 | Fake pattern                                                     |
-| -------------------- | ------------------------------------------------ | ---------------------------------------------------------------- |
-| `avg_post_hour`    | Mean hour of day for all posts (0–23)           | Gang members cluster within ±0.5 hours (coordinated scheduling) |
-| `account_age_days` | Days since account creation                      | Gang created within same week (base_age ± 7 days)               |
-| `shared_ip_count`  | How many other accounts share the same IP subnet | 9 (all 10 gang members share one IP)                             |
-| `ip_cluster_id`    | Identifier for the IP subnet                     | All fakes:`ip_gang_{seed}`                                     |
+**3-seed variance scores (mean across seeds 0, 1, 2):**
 
-### 3.3 Structural / Graph Signals (derived at INSPECT time)
+<p align="center">
+  <img src="images/table2.png" alt="Model Performance Table" width="1600"/>
+</p>
+<br>
 
-These cannot be pre-computed because they depend on **which accounts the agent
-has already flagged** — they change dynamically as the investigation progresses:
+ **Rule-Based Baseline (no LLM, deterministic)**
 
-| Feature                      | What it measures                                                  | Fake pattern                                        |
-| ---------------------------- | ----------------------------------------------------------------- | --------------------------------------------------- |
-| `mutual_follow_rate`       | Fraction of this account's follows that also follow back          | Gang mutually follow each other heavily (0.6–0.9)  |
-| `flagged_neighbor_count`   | How many of this account's follows are currently flagged          | High → embedded in the cluster you're tracking     |
-| `avg_neighbor_photo_reuse` | Mean photo_reuse_score of already-inspected follows               | Gang members cluster; inspecting one reveals others |
-| `post_hour_cluster_score`  | How closely posting hour matches the flagged accounts' mean       | High → same operation schedule                     |
-| `suspicious_mutual_ratio`  | Fraction of suspicious follows that also follow this account back | Used in hub legitimacy computation                  |
-
-**Why graph signals are the most powerful:** A gang member always has high
-`mutual_follow_rate` and grows `flagged_neighbor_count` as the investigation
-proceeds. A celebrity who is *followed by* fakes but does not *follow them back*
-will have low `mutual_follow_rate` — this is how the hub legitimacy score
-protects against false positives.
+<p align="center">
+  <img src="images/table3.png" alt="Model Performance Table" width="1600"/>
+</p>
+<br>
 
 ---
+## Table of Contents
 
-## 4. Synthetic Data Generation
-
-**File:** `server/generator.py`
-
-Episodes are generated deterministically by seed. 150 episodes are pre-generated
-(50 per task) and cached as JSON files in `episodes/`.
+1. [What This Is](#1-what-this-is)
+2. [The Problem: How Fake Detection Actually Works](#2-the-problem-how-fake-detection-actually-works)
+3. [Synthetic Data Generation](#3-synthetic-data-generation)
+4. [Data Model](#4-data-model)
+5. [The RL Environment](#5-the-rl-environment)
+6. [Risk Scoring Mathematics](#6-risk-scoring-mathematics)
+8. [The LLM Policy (Qwen3 via Bedrock)](#8-the-llm-policy-qwen3-via-bedrock)
+9. [Reflexion — How the Agent Learns](#9-reflexion--how-the-agent-learns)
+10. [Hybrid Policy — The Novel Contribution](#10-hybrid-policy--the-novel-contribution)
+11. [Training Loop End-to-End](#11-training-loop-end-to-end)
+12. [API Reference](#12-api-reference)
+13. [Docker Deployment](#13-docker-deployment)
+14. [Submission Requirements](#14-submission-requirements)
+15. [Verification & Validation](#15-verification--validation)
 
-### 4.1 Network Composition
+---
 
-| Task   | Network size | Gang | Decoys | Real | Max steps |
-| ------ | ------------ | ---- | ------ | ---- | --------- |
-| easy   | 50           | 10   | 0      | 40   | 30        |
-| medium | 200          | 10   | 20     | 170  | 50        |
-| hard   | 1000         | 10   | 50     | 940  | 80        |
+## 1. What is this !?
 
-**Gang accounts:** All 10 share:
+This is an **OpenEnv hackathon** submission. OpenEnv is a framework for building RL environments with a standard microservice interface (`/reset`, `/step`, `/state`) so that any agent implementation can plug in.
 
-- A `base_age` (same creation week): `account_age_days = base_age + randint(0, 7)`
-- Tightly clustered `avg_post_hour`: sampled from `gauss(14.0, std)` where std=0.5/1.5/2.5 by task
-- High `photo_reuse_score` and `bio_template_score` (ranges defined per task)
-- `comment_repeat_score` in [0.60, 0.90]
-- `ip_cluster_id = "ip_gang_{seed}"` and `shared_ip_count = 9`
-- Dense intra-gang follow edges (density 0.60–0.80 by task)
+**What makes this non-trivial:**
 
-**Real accounts:** Drawn from log-normal distributions mimicking real Instagram stats:
+- The network is large (50–1000 accounts depending on difficulty).
+- Fake accounts are mixed with innocent high-signal "decoy" accounts.
+- In hard mode, the gang actively evades — dropping intra-gang follows, renaming profiles — while the agent is mid-investigation.
+- The agent cannot see the full network upfront: it must explore via INSPECT and INVESTIGATE_NETWORK actions, spending steps to reveal information.
 
-- `follower_count`: log-normal(μ=6.0, σ=1.5), clipped to [10, 100,000]
-- `comment_repeat_score`: Beta(1, 20) ≈ mostly 0.00–0.08
-- Each has a unique `ip_cluster_id`
+**What makes the learning novel:**
 
-**Decoy accounts** (medium/hard only): Real accounts with elevated fraud scores:
+- The LLM (inference via AWS Bedrock) cannot be fine-tuned — it is a black-box API.
+- The agent learns via **Reflexion**: post-episode lessons are written back into memory and injected into every future prompt.
+- A **dynamic hybrid policy** (α-weighted) blends the LLM with a deterministic rule engine, with the blend weight α updating based on recent win rate. Rules dominate early; the LLM takes over as it proves itself.
 
-- `photo_reuse_score` in [0.20, 0.40], `bio_template_score` in [0.20, 0.40]
-- `comment_repeat_score` in [0.10, 0.30]
-- They look suspicious but are NOT gang members — they penalise reckless flagging.
+### System Architecture
 
-**Celebrity accounts** (2 per episode): Very high follower counts, very low fake scores:
+![System Architecture](assets/sys%20arch.png)
 
-- `follower_count` in [100,000, 5,000,000]
-- `photo_reuse_score` in [0.00, 0.05], `bio_template_score` in [0.00, 0.05]
-- High `hub_legitimacy_score` → the formula protects them from false positives.
+---
 
-**Zero-edge isolates** (2 per episode): Real accounts with `follower_count=0`,
-`following_count=0`, no graph edges. They test whether the agent wastes steps
-on disconnected nodes.
+## 2. The Problem: How Fake Detection Actually Works
 
-### 4.2 Edge Generation
+A real-world fake account detector does **not** read post content. Detection relies on three categories of signals computed from metadata:
 
-Intra-gang edges are added first with high density:
+### Signal Hierarchy (Node -> Behavioral -> Graph)
 
-```
-for g in gang_ids:
-    for h in gang_ids:
-        if g != h and random() < density:
-            g follows h
-```
+![Signal Hierarchy](assets/gs.png)
 
-Real and decoy accounts get sparse preferential-attachment edges: each follows
-5–50 random other accounts. This creates a realistic social graph where gang
-members are much more tightly interconnected than real users.
-
-### 4.3 Episode JSON Schema
-
-```json
-{
-  "episode_id": "uuid4",
-  "task": "easy",
-  "seed": 0,
-  "max_steps": 30,
-  "win_recall": 0.8,
-  "win_precision": 0.7,
-  "starting_visible": ["acc_0012", "acc_0037", ...],
-  "gang_member_ids": ["acc_0003", "acc_0017", ...],
-  "decoy_ids": [],
-  "celeb_ids": ["acc_0048", "acc_0049"],
-  "zero_edge_ids": ["acc_0046", "acc_0047"],
-  "network": {
-    "accounts": [
-      {
-        "id": "acc_0003",
-        "is_fake": true,
-        "gang_id": "gang_A",
-        "features": {
-          "follower_count": 3421,
-          "following_count": 847,
-          "post_count": 214,
-          "avg_post_hour": 14.23,
-          "photo_reuse_score": 0.8712,
-          "bio_template_score": 0.7403,
-          "account_age_days": 67,
-          "comment_repeat_score": 0.7831,
-          "ip_cluster_id": "ip_gang_0",
-          "shared_ip_count": 9,
-          "name_change_count": 0
-        },
-        "true_edges": {
-          "follows": ["acc_0017", "acc_0029", ...],
-          "followed_by": ["acc_0017", "acc_0008", ...]
-        }
-      }
-    ]
-  },
-  "evasion_schedule": []
-}
-```
+- **Node signals (offline):** content fingerprints like photo reuse, bio-template similarity, and comment repetition provide the first suspicion layer.
+- **Behavioral signals (temporal/device):** coordinated posting hour, account-age clustering, and shared IP subnet add stronger gang-level evidence.
+- **Graph signals (live at INSPECT):** mutual follows, flagged-neighbor growth, and cluster alignment are hardest to evade, so they carry the highest weight in risk scoring.
+- **False-positive control:** high-legitimacy hubs (for example celebrities) are down-weighted through hub-legitimacy discounting.
 
 ---
 
-## 5. Data Model — Every Field Explained
+## 3. Synthetic Data Generation
 
-**File:** `models.py`
-
-### 5.1 ActionType (enum)
-
-| Value                   | Cost    | Effect                                                                                   |
-| ----------------------- | ------- | ---------------------------------------------------------------------------------------- |
-| `inspect`             | 1 step  | Reveals full `AccountProfile` + follow list; adds neighbors to `visible_account_ids` |
-| `investigate_network` | 2 steps | Expands 2 hops from account; only reveals account IDs (no profiles)                      |
-| `flag`                | 0 steps | Marks account as gang member; triggers SUSPECT cascade to visible neighbors              |
-| `unflag`              | 0 steps | Removes flag; clears CONFIRMED_FAKE status                                               |
-| `submit`              | 0 steps | Ends episode; triggers scoring                                                           |
-
-### 5.2 AccountStatus (enum)
+**File:** `server/generator.py`
 
-```
-NORMAL          → no signal or formula risk < 0.35
-SUSPECT         → auto-elevated via dual cascade:
-                  (1) a flagged account follows this account, OR
-                  (2) this account shares ip_cluster_id with a flagged account
-CONFIRMED_FAKE  → agent explicitly flagged this account
-```
+Episodes are generated deterministically by seed. 150 episodes are pre-generated (50 per task) and cached as JSON files in `episodes/`.
 
-Transitions are one-directional except UNFLAG which clears CONFIRMED_FAKE.
-SUSPECT is set automatically — the agent never sets it manually.
+### Network Composition
 
-### 5.3 AccountProfile — all 22 fields
+| Task | Network size | Gang | Decoys | Real | Max steps |
+|---|---|---|---|---|---|
+| easy | 50 | 10 | 0 | 40 | 30 |
+| medium | 200 | 10 | 20 | 170 | 50 |
+| hard | 1000 | 10 | 50 | 940 | 80 |
 
-```python
-account_id: str               # "acc_0042"
+- **Gang accounts:** All 10 share `base_age` (same creation week), tightly clustered `avg_post_hour`, high `photo_reuse_score`/`bio_template_score`, `comment_repeat_score` in [0.60, 0.90], `ip_cluster_id = "ip_gang_{seed}"`, and dense intra-gang follow edges (density 0.60–0.80).
+- **Real accounts:** Log-normal follower distributions, unique IP clusters, low fake scores.
+- **Decoy accounts** (medium/hard): Real accounts with elevated fraud scores (0.20–0.40 range) — they look suspicious but are NOT gang members and penalise reckless flagging.
+- **Celebrity accounts** (2 per episode): 100k–5M followers, very low fake scores, high `hub_legitimacy_score`.
+- **Zero-edge isolates** (2 per episode): No edges — test whether the agent wastes steps on disconnected nodes.
 
-# Raw counts
-follower_count: int           # followers (fakes: 1k-8k, celebs: 100k-5M)
-following_count: int          # accounts followed (fakes: 200-2000)
-post_count: int               # total posts (fakes: 50-500)
+---
 
-# Temporal
-avg_post_hour: float          # mean posting hour 0-23 (gang: tightly clustered)
-account_age_days: int         # days since creation (gang: same week, within 7 days)
+## 4. Data Model
 
-# Content pipeline scores (pre-computed, 0-1)
-photo_reuse_score: float      # pHash stolen-photo detection
-bio_template_score: float     # cosine similarity to fake bio templates
-comment_repeat_score: float   # copy-paste spam comment fraction
+**File:** `models.py`
 
-# IP / device
-shared_ip_count: int          # accounts sharing same IP subnet (gang: 9)
+### ActionType
 
-# Graph features (computed live at INSPECT time)
-mutual_follow_rate: float     # fraction of follows that follow back (fakes: 0.6-0.9)
-flagged_neighbor_count: int   # follows already flagged (grows as investigation progresses)
-avg_neighbor_photo_reuse: float   # mean photo_reuse of inspected follows
-inspected_neighbor_count: int     # denominator for flagged_neighbor_ratio
-post_hour_cluster_score: float    # hour alignment to flagged cluster mean (0-1)
-suspicious_mutual_ratio: float    # mutual rate among suspicious follows (for hub legitimacy)
+| Value | Cost | Effect |
+|---|---|---|
+| `inspect` | 1 step | Reveals full `AccountProfile` + follow list |
+| `investigate_network` | 2 steps | Expands 2 hops; reveals account IDs only |
+| `flag` | 0 steps | Marks account as gang member; triggers SUSPECT cascade |
+| `unflag` | 0 steps | Removes flag; clears CONFIRMED_FAKE status |
+| `submit` | 0 steps | Ends episode; triggers scoring |
 
-# Risk breakdown (computed via scoring.py at INSPECT time)
-fake_risk_score: float        # composite 0-1 (main decision signal)
-node_risk: float              # from photo_reuse + bio_template
-behavior_risk: float          # from account_age + post_hour_cluster
-graph_risk: float             # from flagged_neighbor_ratio + mutual + avg_neighbor
-hub_legitimacy_score: float   # celebrity/hub discount
+### AccountProfile — key fields
 
-# Evasion tracking
-name_change_count: int        # incremented by hard-mode evasion events
+| Category | Fields |
+|---|---|
+| Raw counts | `follower_count`, `following_count`, `post_count` |
+| Temporal | `avg_post_hour`, `account_age_days` |
+| Content pipeline (0–1) | `photo_reuse_score`, `bio_template_score`, `comment_repeat_score` |
+| IP/device | `shared_ip_count`, `ip_cluster_id` |
+| Graph (live at INSPECT) | `mutual_follow_rate`, `flagged_neighbor_count`, `avg_neighbor_photo_reuse`, `post_hour_cluster_score` |
+| Risk breakdown | `fake_risk_score`, `node_risk`, `behavior_risk`, `graph_risk`, `hub_legitimacy_score` |
+| Evasion/status | `name_change_count`, `status` (NORMAL/SUSPECT/CONFIRMED_FAKE) |
 
-# Status
-status: AccountStatus         # NORMAL / SUSPECT / CONFIRMED_FAKE
-visible_follows: List[str]    # follow list revealed by INSPECT
-```
+### FakeGangObservation — what the agent sees each step
 
-### 5.4 FakeGangObservation — what the agent sees each step
-
-```python
-done: bool                    # episode over?
-reward: Optional[float]       # only set on terminal step
-visible_accounts: List[AccountProfile]   # fully profiled (inspected) accounts
-visible_account_ids: List[str]           # all known account IDs (profiled + seen)
-flagged_ids: List[str]        # currently flagged by agent
-inspected_ids: List[str]      # accounts with full profiles revealed
-suspect_ids: List[str]        # auto-elevated SUSPECT accounts (uninspected cascade)
-graph_edges: Dict[str, List[str]]        # follow lists for inspected accounts
-steps_remaining: int          # budget left
-evasion_triggered: bool       # was evasion active this episode?
-evasion_count: int            # how many evasion events have fired
-task: str                     # "easy" / "medium" / "hard"
-message: str                  # human-readable result / status message
-```
+`done`, `reward`, `visible_accounts`, `visible_account_ids`, `flagged_ids`, `inspected_ids`, `suspect_ids`, `graph_edges`, `steps_remaining`, `evasion_triggered`, `evasion_count`, `task`, `message`
 
 ---
 
-## 6. The RL Environment
+## 5. The RL Environment
 
 **File:** `server/environment.py`
 
-### 6.1 Episode Lifecycle
+### Episode Lifecycle & Action Mechanics
 
-```
-reset(task, seed)
-    └── loads JSON episode file (or generates on the fly)
-    └── initialises _visible_ids with starting_visible accounts
-    └── returns initial observation (no profiles yet)
-
-step(action)  [called repeatedly]
-    └── INSPECT  → _do_inspect()  → reveals profile + neighbors
-    └── FLAG     → _do_flag()     → cascades SUSPECT to visible neighbors
-    └── UNFLAG   → _do_unflag()   → clears status
-    └── INVESTIGATE_NETWORK → _do_investigate() → reveals 2-hop IDs
-    └── SUBMIT   → _do_submit()   → scores and ends episode
-
-    If step_count >= max_steps → forced submit (penalty -2.0)
-```
-
-### 6.2 Action Mechanics in Detail
-
-**INSPECT (1 step):**
-
-1. Adds account to `_inspected`
-2. Calls `_build_profile(acc_id)` — computes all 22 features dynamically
-3. Adds all accounts this account follows to `_visible_ids`
-4. Returns updated observation
-
-**INVESTIGATE_NETWORK (2 steps):**
+![Episode Flow](assets/episode.png)
 
-1. Adds account to `_inspected` (counts it as seen)
-2. **Bidirectional 2-hop expansion:** Traverses both `_live_edges` (outgoing follows)
-   AND `_reverse_edges` (incoming followers) for the target and each 1-hop neighbor.
-   This means the expansion covers:
-   - Outgoing: `acc → follows → their follows` AND `acc → follows → their followers`
-   - Incoming: `acc ← followers → their follows` AND `acc ← followers ← their followers`
-3. Adds all new account IDs to `_visible_ids` (no full profiles — IDs only)
-4. **Re-cascades SUSPECT** to newly visible accounts via two signals:
-   - *Follow-graph cascade:* any newly visible account followed by a flagged account → SUSPECT
-   - *IP cluster cascade:* any newly visible account sharing `ip_cluster_id` with a flagged account → SUSPECT (zero false positives — gang shares one IP; real accounts have unique IPs)
-5. Cost: 2 steps, -0.02 score. Returns count of newly discovered IDs.
+**FLAG cascade (dual):** When FLAG(X) is called — (1) every visible account that X follows becomes SUSPECT via the follow-graph, and (2) every visible account sharing X's `ip_cluster_id` becomes SUSPECT. Gang members share `ip_gang_{seed}`; real accounts have unique IPs → zero false positives.
 
-**FLAG (free):**
-
-1. Adds account to `_flagged`
-2. Sets `_account_statuses[acc_id] = "confirmed_fake"`
-3. **Dual cascade** to SUSPECT:
-   - *Cascade 1 — Follow-graph:* For every neighbor in `_live_edges[acc_id]`
-     (accounts the flagged user follows), if the neighbor is visible and NORMAL → SUSPECT.
-     Gang members follow each other at density 0.70+, so this is high-precision.
-   - *Cascade 2 — IP cluster:* Any visible account sharing the same `ip_cluster_id`
-     as the flagged account → SUSPECT. Gang members all share `ip_gang_{seed}`;
-     real and decoy accounts each have a unique IP cluster. Zero false positives.
-4. Refreshes all already-inspected accounts that follow `acc_id`
-   (their `flagged_neighbor_count` just increased, so risk scores change)
-
-**SUBMIT:**
-Computes final scores (see §6.3).
-
-### 6.3 Reward Function
+### Reward Function
 
 ```
-tp = len(gang_ids ∩ flagged_ids)     # true positives
-fp = len(flagged_ids - gang_ids)     # false positives
-fn = len(gang_ids - flagged_ids)     # false negatives
-
 base_reward = tp×1.0 − fp×0.5 − fn×0.3
 
-Win condition (task-dependent thresholds):
+Win condition:
   easy/medium:  recall ≥ 0.8 AND precision ≥ 0.7
   hard:         recall ≥ 0.9 AND precision ≥ 0.8
 
-If WIN:
-  +5.0          # full win bonus
-  +3.0          # if perfect recall (tp == len(gang_ids))
-elif recall ≥ win_recall:
-  +2.0          # partial win (high recall, low precision)
-
-Efficiency bonus (if SUBMIT called voluntarily with ≥50% steps remaining):
-  +1.0
-
-Hard mode evasion penalty:
-  −1.0 × evasion_count
-
-Forced submit (ran out of steps):
-  −2.0
-
-Final score = base_reward + all bonuses/penalties
-```
-
-**Example:** Easy task, found 9/10 gang members, flagged 2 innocent accounts,
-30 steps used, submitted voluntarily with 5 steps left (< 50%):
-
-```
-tp=9, fp=2, fn=1
-base = 9×1.0 − 2×0.5 − 1×0.3 = 9 − 1 − 0.3 = 7.7
-recall = 9/10 = 0.90 ≥ 0.8 ✓  precision = 9/11 = 0.82 ≥ 0.7 ✓
-+5.0 win bonus
-0 efficiency bonus (steps_left=5 < 30×0.5=15)
-total = 7.7 + 5.0 = 12.7
+Bonuses:
+  +5.0   full win
+  +3.0   perfect recall
+  +2.0   partial win (high recall, low precision)
+  +1.0   efficiency (SUBMIT with ≥50% steps remaining)
+  −1.0   per evasion event (hard mode)
+  −2.0   forced submit (ran out of steps)
 ```
 
-### 6.4 Evasion (hard mode)
-
-The `evasion_schedule` in each episode defines trigger points. When
-`step_count >= event["step"]` and the event hasn't fired yet:
+### Evasion (hard mode)
 
-**`unfollow_intragang`:** A fraction (`drop_rate=0.3`) of intra-gang edges are
-randomly removed from `_live_edges`. This destroys the graph signal mid-investigation.
-The agent sees `mutual_follow_rate` and `flagged_neighbor_count` drop on
-re-inspection. Hard mode fires this 4 times (steps 15, 30, 45, 60).
-
-**`rename_count`:** A random subset of gang members get `name_change_count += 1`.
-This is a visual signal — the agent should notice accounts that have changed
-their name multiple times.
+- **`unfollow_intragang`:** 30% of intra-gang edges randomly removed mid-investigation — destroys graph signal. Fires 4 times (steps 15, 30, 45, 60).
+- **`rename_count`:** Random gang members get `name_change_count += 1` — a visual evasion signal.
 
 ---
 
-## 7. Risk Scoring Mathematics
-
-**File:** `server/scoring.py`
+## 6. Risk Scoring Mathematics
 
-All five functions are **stateless and deterministic** — no side effects, no
-global state. They are called inside `_build_profile()` every time an account
-is inspected or a neighbor is re-profiled after a FLAG.
+**File:** `server/scoring.py` — all functions are stateless and deterministic.
 
-### 7.1 Node Risk
+![pipw](images/big.png)
 
-Captures content-based fakeness signals:
+![Risk Scoring Formulas 1](assets/formulas-1.png)
 
-```
-node_risk = 0.60 × photo_reuse_score + 0.40 × bio_template_score
-```
-
-Photo reuse gets 60% weight because it is harder to spoof (requires actual
-pHash fingerprint matching against a celebrity photo database).
-
-**Example:** Gang member with `photo_reuse=0.87`, `bio_template=0.74`:
-
-```
-node_risk = 0.60 × 0.87 + 0.40 × 0.74 = 0.522 + 0.296 = 0.818
-```
-
-### 7.2 Behavior Risk
-
-Captures temporal anomalies:
-
-```
-age_norm = min(1.0, account_age_days / 365.0)
-behavior_risk = 0.55 × (1 − age_norm) + 0.45 × post_hour_cluster_score
-```
-
-`(1 − age_norm)` is high for newly created accounts (fakes are created right
-before the operation starts). `post_hour_cluster_score` measures alignment with
-the flagged cluster's mean posting hour (see §7.5).
-
-**Example:** Gang member, `account_age_days=67`, `post_hour_cluster_score=0.91`:
-
-```
-age_norm = 67/365 = 0.184
-behavior_risk = 0.55×(1−0.184) + 0.45×0.91 = 0.55×0.816 + 0.4095
-             = 0.449 + 0.410 = 0.859
-```
-
-### 7.3 Graph Risk
-
-The most predictive signal once the investigation has started:
-
-```
-flagged_neighbor_ratio = flagged_neighbor_count / max(inspected_neighbor_count, 1)
-graph_risk = 0.45 × flagged_neighbor_ratio
-           + 0.35 × mutual_follow_rate
-           + 0.20 × avg_neighbor_photo_reuse
-```
-
-`flagged_neighbor_ratio` gets 45% weight — if several of this account's friends
-are already confirmed fakes, this account is very likely fake too.
-
-**Example:** After 3 gang members flagged; inspecting a 4th gang member:
-
-```
-flagged_neighbor_count = 3 (3 already-flagged accounts in its follow list)
-inspected_neighbor_count = 4 (total inspected follows)
-mutual_follow_rate = 0.78 (gang mutually follow heavily)
-avg_neighbor_photo_reuse = 0.81
-
-flagged_neighbor_ratio = 3/4 = 0.75
-graph_risk = 0.45×0.75 + 0.35×0.78 + 0.20×0.81
-           = 0.338 + 0.273 + 0.162 = 0.773
-```
-
-### 7.4 Hub Legitimacy
-
-Protects celebrities and legitimate large accounts from false positives:
-
-```
-F_MAX = 1,000,000
-followers_norm = min(1.0, log(1+follower_count) / log(1+F_MAX))
-follow_ratio_norm = min(1.0, (following_count / max(follower_count, 1)) / 5.0)
-age_norm = min(1.0, account_age_days / 365.0)
-
-hub_legitimacy = 0.45 × followers_norm
-               + 0.25 × (1 − follow_ratio_norm)
-               + 0.20 × age_norm
-               + 0.10 × (1 − suspicious_mutual_ratio)
-```
-
-Four signals of legitimacy:
-
-- Large log-scaled follower count (0.45 weight) — genuine celebrities have
-  millions; fake accounts peak at ~8,000
-- Low follow-to-follower ratio (0.25 weight) — celebs follow few, are followed
-  by many; fakes follow aggressively
-- Old account (0.20 weight) — real celebrities have accounts years old
-- Not mutually following suspicious accounts (0.10 weight) — a celeb being
-  *followed by* fakes doesn't make the celeb fake
-
-**Example — Celebrity with 2,000,000 followers:**
-
-```
-followers_norm = log(2,000,001) / log(1,000,001) = 14.509/13.816 = 1.0 (capped)
-follow_ratio_norm = (200 / 2,000,000) / 5.0 = 0.00002 ≈ 0.0
-age_norm = min(1.0, 2000/365) = 1.0
-
-hub_legitimacy = 0.45×1.0 + 0.25×(1−0.0) + 0.20×1.0 + 0.10×1.0 = 1.00
-```
-
-**Example — Gang member:**
-
-```
-followers_norm = log(3422) / log(1,000,001) = 8.138/13.816 = 0.589
-follow_ratio_norm = min(1.0, (847/3422)/5.0) = 0.0495
-age_norm = 67/365 = 0.184
-
-hub_legitimacy = 0.45×0.589 + 0.25×(1−0.0495) + 0.20×0.184 + 0.10×0.9
-              = 0.265 + 0.238 + 0.037 + 0.090 = 0.630
-```
-
-### 7.5 Post-Hour Cluster Score
-
-Computed dynamically inside `environment.py`, not in `scoring.py`:
-
-```
-mean_h = average avg_post_hour across all currently flagged accounts
-diff = min(|acc_hour − mean_h|, 24 − |acc_hour − mean_h|)  # wrap-around
-post_hour_cluster_score = max(0.0, 1.0 − diff / 6.0)
-```
-
-The wrap-around handles the midnight boundary (e.g., 23:00 and 01:00 are 2 hours
-apart, not 22). A score of 1.0 means posting at exactly the same hour as the
-flagged cluster. A score of 0.0 means ≥6 hours away.
-
-**Why 6.0 as the divisor:** 6 hours is a generous "different time zone" threshold.
-If you post within 6 hours of the gang's schedule, you get partial credit.
-
-**Example:** Gang posts at mean=14.0. Inspecting an account posting at 14.3:
-
-```
-diff = |14.3 − 14.0| = 0.3
-post_hour_cluster_score = 1.0 − 0.3/6.0 = 0.950
-```
-
-### 7.6 Composite Fake Risk
-
-```
-fake_risk = clip(
-    0.30 × node_risk
-  + 0.25 × behavior_risk
-  + 0.45 × graph_risk
-  − 0.25 × hub_legitimacy,
-  0.0, 1.0
-)
-```
-
-Weight rationale:
-
-- **Graph risk 0.45** — structural signals are hardest for fakes to hide.
-  Mutual follow density requires real coordination; once you find one member,
-  the whole cluster lights up.
-- **Node risk 0.30** — content signals are strong but can appear on decoys.
-- **Behavior risk 0.25** — temporal clustering is a reliable early signal,
-  especially before any flags are set.
-- **Hub legitimacy −0.25** — subtractive discount. A celebrity with 5M followers
-  has hub_legitimacy ≈ 1.0, so even if gang members follow them, their risk
-  formula produces: `0.30×0.02 + 0.25×0.05 + 0.45×0.10 − 0.25×1.0 ≈ −0.17 → clipped to 0.0`
-
-**Full gang member example** (after 3 flags set):
-
-```
-node_risk     = 0.818  (photo=0.87, bio=0.74)
-behavior_risk = 0.859  (age=67d, cluster_score=0.91)
-graph_risk    = 0.773  (ratio=0.75, mutual=0.78, nbr_photo=0.81)
-hub_legitimacy= 0.630  (3k followers, 1y old, no celeb)
-
-fake_risk = 0.30×0.818 + 0.25×0.859 + 0.45×0.773 − 0.25×0.630
-          = 0.245 + 0.215 + 0.348 − 0.158
-          = 0.650
-```
-
-### 7.7 Risk Classification
-
-```
-fake_risk < 0.35    → "normal"
-0.35 ≤ risk < 0.60 → "suspect"
-risk ≥ 0.60        → "confirmed_fake"   (formula-level; explicit flag overrides)
-```
-
-### 7.8 Grader Score (Submission Metric)
-
-This normalised [0.0, 1.0] score is returned by the `/grader` endpoint:
-
-```
-recall    = tp / 10
-precision = tp / max(tp + fp, 1)
-efficiency = max(0.0, (max_steps − steps_used) / max_steps)
-
-if recall ≥ 0.8 AND precision ≥ 0.7:
-    score = 0.55 + 0.20×recall + 0.15×precision + 0.10×efficiency
-else:
-    score = 0.30×recall + 0.10×precision
-```
-
-**Maximum possible score:** `0.55 + 0.20×1.0 + 0.15×1.0 + 0.10×1.0 = 1.00`
-(requires all 10 found, no false positives, and 0 steps used — perfect play)
-
-**Win threshold score:** `0.55 + 0.20×0.8 + 0.15×0.7 + 0.10×0 = 0.55 + 0.16 + 0.105 = 0.815`
-
-**Partial credit examples:**
-
-- Found 6/10, no false positives: `0.30×0.6 + 0.10×1.0 = 0.18 + 0.10 = 0.28`
-- Found 9/10, 3 false positives: recall=0.9, precision=9/12=0.75 → win: `0.55 + 0.18 + 0.113 = 0.843`
-
----
-
-## 8. Account Status State Machine
-
-```
-                  ┌──────────────────────────────────────┐
-                  │                                      │
-             INSPECT                               INSPECT
-                  │                                      │
-                  ▼                                      ▼
-         ┌──────────────┐    FLAG cascade     ┌──────────────────┐
-         │    NORMAL    │ ─────────────────►  │     SUSPECT      │
-         └──────────────┘   (neighbor of       └──────────────────┘
-                │              flagged)                 │
-                │                                       │
-          FLAG(account)                          FLAG(account)
-                │                                       │
-                ▼                                       ▼
-         ┌──────────────────────────────────────────────────┐
-         │                 CONFIRMED_FAKE                   │
-         └──────────────────────────────────────────────────┘
-                │
-          UNFLAG(account)
-                │
-                ▼
-         (status cleared → NORMAL)
-```
-
-**When FLAG(X) is called:**
-
-1. X → CONFIRMED_FAKE
-2. **Dual SUSPECT cascade:**
-   - *Follow-graph:* For every account Y that X follows (`_live_edges[X]`):
-     if Y is visible AND Y is NORMAL → Y becomes SUSPECT
-   - *IP cluster:* For every visible account Z sharing X's `ip_cluster_id`:
-     if Z is not flagged AND Z is NORMAL → Z becomes SUSPECT
-     (gang members share `ip_gang_{seed}`; real accounts have unique IPs → zero false positives)
-3. All already-inspected accounts that follow X are re-profiled
-   (their `flagged_neighbor_count` increases, which raises their `fake_risk_score`)
-
-**Why SUSPECT matters:**
-
-- The `suspect_ids` field in the observation lists all SUSPECT accounts not yet inspected
-- Both the rule engine and the LLM treat these as highest priority for the next INSPECT
-- This creates an efficient cascade: flag one → inspect suspects → some are gang
-  → flag them → more suspects appear → repeat until cluster is exhausted
-
-**Example cascade on easy task:**
-
-```
-Step 1: INSPECT acc_0003 (gang member) → no flags yet, fake_risk ≈ 0.45
-Step 2: FLAG acc_0003
-         → acc_0017, acc_0029, acc_0041 become SUSPECT (they follow acc_0003)
-         → obs.suspect_ids = ["acc_0017", "acc_0029", "acc_0041"]
-Step 3: INSPECT acc_0017 (gang member) → fake_risk now 0.72 (flagged_neighbor_count=1)
-Step 4: FLAG acc_0017
-         → acc_0003 (already flagged), acc_0029, acc_0041, acc_0055 get SUSPECT
-         → acc_0003, acc_0017 profiles refreshed (their mutual flags increased)
-Step 5: INSPECT acc_0029 → fake_risk = 0.81 (flagged_neighbor_count=2)
-...
-```
-
-Each FLAG makes the next gang member easier to find because their risk score rises.
+![Risk Scoring Formulas 2](assets/formulas-2.png)
 
 ---
 
-## 9. The LLM Policy (Qwen3 via Bedrock)
+## 8. The LLM Policy (Qwen3 via Bedrock)
 
 **File:** `agent/policy.py`
 
-### 9.1 Model
+**Model:** `qwen.qwen3-next-80b-a3b` via AWS Bedrock Converse API (`maxTokens=512, temperature=0.4`)
 
-**Qwen3-Next-80B** accessed via AWS Bedrock Marketplace:
-
-```python
-MODEL_ID = "qwen.qwen3-next-80b-a3b"
-```
-
-Called via the Bedrock Converse API:
-
-```python
-client.converse(
-    modelId=MODEL_ID,
-    messages=[{"role": "user", "content": [{"text": prompt}]}],
-    system=[{"text": SYSTEM_PROMPT}],
-    inferenceConfig={"maxTokens": 512, "temperature": 0.4}
-)
-```
-
-Temperature 0.4 is low enough for consistent action format but high enough to
-avoid degenerate repetition.
-
-### 9.2 Prompt Construction
+### Prompt Structure
 
 Every step, the policy builds a prompt from three components:
 
@@ -803,612 +275,178 @@ Every step, the policy builds a prompt from three components:
 What is your next action?
 ```
 
-### 9.3 Observation Formatting
-
-The `_format_observation()` function converts the typed `FakeGangObservation`
-into a text block. Accounts are **sorted by `fake_risk_score` descending**,
-with status badges prepended:
-
-```
-TASK: EASY | Steps remaining: 22
-Currently flagged (3/10): acc_0003, acc_0017, acc_0029
-Suspects not yet inspected (4): acc_0041, acc_0055, acc_0062, acc_0078
-
-PROFILED ACCOUNTS (sorted by fake_risk_score — highest first):
-  [status | risk | node beh graph hub | photo bio mutual | comment ip_count]
-  CONFIRMED_FAKE acc_0029 ◀ FLAGGED: risk=0.821 | node=0.82 beh=0.77 graph=0.86 hub=0.63
-  SUSPECT        acc_0041: risk=0.714 | node=0.79 beh=0.81 graph=0.74 hub=0.65 fnbr=3(!)
-  SUSPECT        acc_0055: risk=0.681 | node=0.71 beh=0.74 graph=0.69 hub=0.67 fnbr=2(!)
-  NORMAL         acc_0022: risk=0.121 | node=0.09 beh=0.31 graph=0.03 hub=0.84 [HUB?]
-  ...
-
-KNOWN UNINSPECTED IDs: acc_0062, acc_0078, acc_0091, ...
-
-Environment message: Flagged acc_0029 as suspected fake.
-```
-
-Key formatting choices:
+Accounts in the observation are **sorted by `fake_risk_score` descending**, with status badges prepended. `fnbr=N(!)` highlights when `flagged_neighbor_count > 0`; `[HUB?]` warns the LLM not to flag high-legitimacy accounts.
 
-- `fnbr=N(!)` highlights when `flagged_neighbor_count > 0` — this is the most
-  actionable graph signal
-- `[HUB?]` appears when `hub_legitimacy_score > 0.70` — warns the LLM not to flag
-- Status badge width is fixed (13 chars) for visual alignment
-
-### 9.4 Required Response Format
+### Required Response Format
 
 ```xml
 <thinking>
-Your reasoning here — which account is most suspicious and why,
-what signal you're acting on, what your next move is.
+Reasoning — which account is most suspicious and why.
 </thinking>
 <action>
 INSPECT acc_0041
 </action>
 ```
 
-The parser (`_parse_action`) extracts the content inside `<action>...</action>`
-using regex, then matches to action types. If parsing fails entirely, the
-fallback inspects the highest-scored uninspected account.
-
-### 9.5 Retry Logic
-
-```python
-for attempt in range(3):
-    try:
-        raw = invoke_qwen(...)
-        action = _parse_action(raw, obs)
-        return action, raw
-    except Exception as exc:
-        wait = 2 ** attempt  # 1s, 2s, 4s
-        time.sleep(wait)
-
-# All retries failed → heuristic fallback
-return _heuristic_fallback(obs), "[FALLBACK]"
-```
+If parsing fails, a heuristic fallback inspects the highest-scored uninspected account. Retries use exponential backoff (1s, 2s, 4s) up to 3 attempts.
 
 ---
 
-## 10. Reflexion — How the Agent Learns
+## 9. Reflexion — How the Agent Learns
 
 **Files:** `agent/reflection.py`, `agent/memory.py`
 
-The agent **cannot** update Qwen3's weights — Bedrock is a black-box API.
-Instead, it learns via **Reflexion**: post-episode lessons are written as text
-and injected into future prompts.
+The agent **cannot** update Qwen3's weights — Bedrock is a black-box API. Instead, it learns via **Reflexion**: post-episode lessons are written as text and injected into future prompts.
 
-### 10.1 Learning Loop
+### Reflexion Learning Loop
 
-```
-Episode N
-  1. LLM acts using: system_prompt + reflections[1..4] + best_trajectory
-  2. Episode ends → WIN or LOSS
-  3. Post-episode learning:
-     If LOSS:
-       → generate_reflection(action_log, outcome) → Qwen writes a lesson
-       → lesson stored to memory/reflections_easy.jsonl
-     If WIN:
-       → save trajectory to memory/best_trajectory_easy.json (if better reward)
-       → generate_success_reflection() → Qwen writes what worked
-       → stored to reflections
-
-Episode N+1
-  → get_reflections("easy", n=4) returns last 4 lessons
-  → get_best_trajectory("easy") returns best win as few-shot example
-  → both injected into prompt → LLM has learned from its past
-```
+![Reflexion Learning Loop](assets/reflexion.png)
 
-### 10.2 Reflection Generation
 
-A separate Qwen3 call is made after each episode with this prompt:
 
 ```
-CASE DEBRIEF — Episode 12
-Task difficulty: MEDIUM
-Outcome: FAILURE
-Steps used: 50/50
-Result: [LOSS] TP=6 FP=3 FN=4 Recall=0.60 Precision=0.67
-
-INVESTIGATION LOG:
-  1. INSPECT acc_0022
-  2. INSPECT acc_0037
-  ...
-  20. SUBMIT
-
-Write a 2-3 sentence lesson for your future self based on this case.
-```
-
-**Example generated reflection:**
-
-> "The starting accounts were all real; I wasted 8 steps inspecting low-signal nodes
-> before pivoting. When photo_reuse and bio_template are both below 0.3 after 3 inspections,
-> immediately use INVESTIGATE_NETWORK to jump to a different graph region.
-> Once I found the first gang member at step 14, I should have cascaded faster
-> via SUSPECT accounts rather than continuing to inspect unknown IDs."
-
-This lesson is stored and appears in Episode 13's prompt, causing the agent to
-pivot earlier and follow the cascade more aggressively.
-
-### 10.3 Best Trajectory (Few-Shot Example)
-
-The first episode that wins is saved as a few-shot example. Every subsequent win
-replaces it only if the reward is higher. The trajectory appears in the prompt as:
+Episode N:
+  1. LLM acts using: system_prompt + reflections[last 4] + best_trajectory
+  2. Episode ends → WIN or LOSS
+  3. Post-episode:
+     LOSS → generate_reflection(action_log, outcome) → lesson stored
+     WIN  → save trajectory if better reward + generate_success_reflection
 
+Episode N+1:
+  → last 4 reflections + best win trajectory injected into prompt
+  → LLM has learned from its past
 ```
-━━━ EXAMPLE SUCCESSFUL CASE (task=easy, reward=+14.20) ━━━
-  1. INSPECT acc_0012
-  2. INSPECT acc_0037
-  3. FLAG acc_0037
-  4. INSPECT acc_0041    (suspect — cascaded from acc_0037)
-  5. FLAG acc_0041
-  ...
-  → [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00
-```
-
-The LLM sees a concrete example of the exact pattern that leads to a perfect win,
-and mirrors this strategy.
-
-### 10.4 Memory Persistence
 
-All memory is stored in `memory/` as flat files:
-
-```
-memory/
-├── reflections_easy.jsonl      # one JSON entry per reflection
-├── reflections_medium.jsonl
-├── reflections_hard.jsonl
-├── best_trajectory_easy.json   # single best win per task
-├── best_trajectory_medium.json
-├── best_trajectory_hard.json
-├── wins_easy.jsonl             # episode-level win history (for alpha)
-├── wins_medium.jsonl
-├── wins_hard.jsonl
-├── alpha_easy.json             # current α for this task
-├── alpha_medium.json
-└── alpha_hard.json
-```
+**Example generated reflection:**
+> *"The starting accounts were all real; I wasted 8 steps inspecting low-signal nodes before pivoting. When photo_reuse and bio_template are both below 0.3 after 3 inspections, immediately use INVESTIGATE_NETWORK to jump to a different graph region."*
 
-The `memory/` directory is a Docker volume (`VOLUME ["/app/memory"]`), so all
-learning persists across container restarts and redeployments.
+All memory persists in a Docker volume (`memory/`) across container restarts — reflections, best trajectories, win history, and α values per task.
 
 ---
 
-## 11. Hybrid Policy — The Novel Contribution
+## 10. Hybrid Policy — The Novel Contribution
 
 **File:** `agent/hybrid_policy.py`
 
-The key insight: **a new LLM agent starts dumb but improves over time. A rule
-engine is always consistent but cannot adapt.** The hybrid policy exploits both:
-rules provide a safety net early while the LLM builds its track record; once the
-LLM proves itself, rules step back.
-
-### 11.1 The Problem with Pure LLM
-
-In the first few episodes:
+**Key insight:** A new LLM agent starts dumb but improves over time. A rule engine is always consistent but cannot adapt. The hybrid policy exploits both — rules provide a safety net early while the LLM builds its track record; once the LLM proves itself, rules step back.
 
-- No reflections have been generated yet
-- No successful trajectory to use as a few-shot example
-- The LLM is essentially guessing based only on the system prompt
-- Win rate on `easy` episodes ≈ 30% at episode 1 (single-digit recall)
+### Architecture
 
-A deterministic rule engine using `fake_risk_score` thresholds would achieve
-~60% win rate on `easy` from episode 1, with zero learning overhead.
+![Hybrid Policy Architecture](assets/hybrid.png)
 
-### 11.2 The Problem with Pure Rules
+### Alpha (α): The Trust Weight
 
-Rules use fixed thresholds. They cannot:
-
-- Adapt to the evasion events in hard mode
-- Prioritise which SUSPECT to inspect based on context
-- Recognise unusual configurations (e.g., decoys clustered near gang members)
-- Balance exploration vs. exploitation optimally
-
-The LLM, given enough reflections, learns these nuances.
-
-### 11.3 Alpha: The Trust Weight
-
-α (alpha) is a per-task value in [0.20, cap] representing the agent's current
-trust in the LLM:
+α is a per-task value in [0.20, cap] representing current trust in the LLM:
 
 ```
 reflection_factor = min(1.0, n_reflections / 4.0)
 raw = 0.20 + reflection_factor × (0.80 × recent_win_rate + 0.12)
 α = clamp(raw, 0.20, cap)
-
-where:
-  recent_win_rate = wins in last 10 episodes for this task
-  reflection_factor = min(1.0, n_reflections / 4.0)
 ```
 
-**Per-task alpha caps** prevent α from climbing so high that the LLM overrides
-correct high-confidence rule-engine decisions (e.g., Priority 2 INSPECT SUSPECT
-at confidence=0.95):
-
-| Task   | α cap | Rationale                                                          |
-| ------ | ------ | ------------------------------------------------------------------ |
-| easy   | 0.50   | Rule engine alone achieves ~91% — LLM should assist, not override |
-| medium | 0.70   | Decoys require some LLM judgment, but cascade must stay            |
-| hard   | 0.85   | LLM needs latitude for evasion adaptation, but safety rules remain |
-
-`reflection_factor` ensures the LLM must accumulate at least **4 reflections**
-before it can reach meaningful trust — pure win rate is not enough, because the LLM
-needs to have demonstrably learned from failures.
+| Task | α cap | Rationale |
+|---|---|---|
+| easy | 0.50 | Rule engine alone achieves ~91% — LLM should assist, not override |
+| medium | 0.70 | Decoys require some LLM judgment, but cascade must stay |
+| hard | 0.85 | LLM needs latitude for evasion adaptation, but safety rules remain |
 
 **Alpha trajectory over training (easy task, cap=0.50):**
 
-| Episode | Wins (last 10) | Reflections | reflection_factor | raw  | α (capped)    |
-| ------- | -------------- | ----------- | ----------------- | ---- | -------------- |
-| 1       | 0/0 → wr=0%   | 0           | 0.00              | 0.20 | 0.20           |
-| 5       | 1/5 → wr=20%  | 4           | 1.00              | 0.48 | 0.48           |
-| 10      | 5/10 → wr=50% | 9           | 1.00              | 0.72 | **0.50** |
-| 20      | 8/10 → wr=80% | 19          | 1.00              | 0.96 | **0.50** |
-
-α starts at 0.20 (rules dominate) and climbs toward the task-specific cap as
-the LLM wins consistently and accumulates lessons. The cap ensures the rule
-engine retains veto power over high-confidence structural decisions.
-
-### 11.4 Rule Action + Confidence
-
-`get_rule_action(obs)` returns `(FakeGangAction, float)` where the float is
-the rule's confidence in its own decision:
+| Episode | Win rate | Reflections | α (capped) |
+|---|---|---|---|
+| 1 | 0% | 0 | 0.20 |
+| 5 | 20% | 4 | 0.48 |
+| 10 | 50% | 9 | **0.50** |
+| 20 | 80% | 19 | **0.50** |
 
-| Situation                                         | Action                | Confidence                         |
-| ------------------------------------------------- | --------------------- | ---------------------------------- |
-| Steps remaining = 0                               | SUBMIT                | 1.00                               |
-| Uninspected SUSPECT accounts exist                | INSPECT suspects[0]   | 0.95                               |
-| Inspected account: fake_risk ≥ 0.85              | FLAG that account     | 0.95                               |
-| Inspected account: fake_risk in [threshold, 0.85) | FLAG that account     | 0.70 + (risk − threshold) × 0.60 |
-| 10 accounts already flagged                       | SUBMIT                | 0.85                               |
-| Steps remaining ≤ 3                              | SUBMIT                | 0.90                               |
-| Uninspected accounts available                    | INSPECT top candidate | 0.30                               |
-| Nothing to do                                     | SUBMIT                | 0.75                               |
+<br>
 
-Confidence values are calibrated such that:
+![System Architecture](images/plot.png)
 
-- Structural/safety decisions (out of steps, cascade suspects) have confidence ≥ 0.90
-- Direct flag decisions have confidence ≥ 0.70
-- Exploratory decisions have confidence 0.30 (the rule is just suggesting, not insisting)
+### Rule Confidence Levels
 
-### 11.5 Blending Decision
-
-```python
-rule_action, rule_conf = get_rule_action(obs)
-llm_action, raw_llm   = get_action(obs, reflections, few_shot, temperature)
-
-if rule_action == llm_action:          # same type AND same account_id
-    mode = "agree"
-    final = llm_action
-
-elif rule_conf >= alpha:               # rule is confident enough to override
-    mode = f"rule_override(c={rule_conf:.2f},α={alpha:.2f})"
-    final = rule_action
-
-else:                                  # LLM is trusted; rule doesn't insist
-    mode = f"llm(c={rule_conf:.2f}<α={alpha:.2f})"
-    final = llm_action
-```
+| Situation | Action | Confidence |
+|---|---|---|
+| Steps remaining = 0 | SUBMIT | 1.00 |
+| Uninspected SUSPECT accounts exist | INSPECT suspects[0] | 0.95 |
+| `fake_risk ≥ 0.85` | FLAG that account | 0.95 |
+| `fake_risk` in [threshold, 0.85) | FLAG that account | 0.70+ |
+| 10 accounts already flagged | SUBMIT | 0.85 |
+| Steps remaining ≤ 3 | SUBMIT | 0.90 |
+| Uninspected accounts available | INSPECT top candidate | 0.30 |
 
-**Why this works mathematically:**
+At **α=0.20** (early): rules dominate (~90% of decisions). At **α=0.50** (moderate): LLM controls exploration; rules control safety. At **α=0.85** (high): LLM controls most decisions; rules only override forced submits and uninspected suspects.
 
-The condition `rule_conf >= alpha` creates a natural threshold system:
-
-- At **α=0.20** (early training, no history):
-
-  - Rules win whenever confidence ≥ 0.20
-  - The only exploratory INSPECT (confidence=0.30) still beats α=0.20
-  - So rules dominate: ~90% of decisions are rule-driven
-  - Effectively acts like the rule-based baseline agent
-- At **α=0.50** (moderate trust, mixed results):
-
-  - Rules win when confidence ≥ 0.50
-  - Safety decisions (suspects, forced submit) still override: conf=0.95 > 0.50
-  - Exploratory decisions (conf=0.30) now go to LLM: 0.30 < 0.50
-  - The LLM controls exploration; rules control safety
-- At **α=0.84** (high trust, consistent wins):
-
-  - Rules win only when confidence ≥ 0.84
-  - Only the two highest-confidence situations still override: forced submit
-    (1.00) and uninspected suspects (0.95)
-  - Everything else goes to the LLM, including direct flag decisions
-- At **α=cap** (maximum trust for the task):
-
-  - On easy (cap=0.50): rules still override for suspects (0.95), flags (0.70+),
-    and forced submits (1.00) — only exploratory INSPECTs (0.30) go to LLM
-  - On hard (cap=0.85): rules only override the highest-confidence situations
-    (suspects, forced submit); LLM controls flag and exploration decisions
-
-### 11.6 Disagreement Examples
-
-**Example 1 — Early training (α=0.25), LLM exploring, rule insisting on suspect:**
-
-```
-Rule:  INSPECT acc_0041  (SUSPECT account)  confidence=0.95
-LLM:   INSPECT acc_0099  (random exploration)
-Rule wins: 0.95 ≥ 0.25 → INSPECT acc_0041
-mode = "rule_override(c=0.95,α=0.25)"
-```
-
-**Example 2 — Mid training (α=0.60), LLM flags a high-risk account:**
-
-```
-Rule:  INSPECT acc_0041  (uninspected suspect)  confidence=0.95
-LLM:   FLAG acc_0055  (fake_risk=0.79, already inspected)
-Rule wins: 0.95 ≥ 0.60 → INSPECT acc_0041
-mode = "rule_override(c=0.95,α=0.60)"
-```
-
-*(Both actions are useful; the rule correctly prioritises cascade suspects
-before random flags)*
-
-**Example 3 — High trust (α=0.85), LLM has learned to prioritise smarter:**
-
-```
-Rule:  INSPECT acc_0041  (exploratory, conf=0.30)
-LLM:   FLAG acc_0055  (fake_risk=0.88, very high confidence)
-LLM wins: 0.30 < 0.85 → FLAG acc_0055
-mode = "llm(c=0.30<α=0.85)"
-```
-
-**Example 4 — Both agree (most common case in late training):**
-
-```
-Rule:  INSPECT acc_0041  (SUSPECT, conf=0.95)
-LLM:   INSPECT acc_0041  (LLM also noticed the suspect badge)
-mode = "agree"
-```
-
-### 11.7 Alpha Persistence
-
-After every episode, `train.py` does:
-
-```python
-# Record outcome
-memory.record_win(task, won, episode_num)
-
-# Recompute alpha with updated win history (per-task cap applied)
-new_wr = memory.recent_win_rate(task, n=10)
-new_alpha = compute_alpha(new_wr, n_reflections, task=current_task)
-
-# Save for next run (even if container restarts)
-memory.save_alpha(task, new_alpha)
-```
-
-Alpha is stored in `memory/alpha_{task}.json` and loaded at the start of each
-training run. This means the agent's trust level is preserved across Docker
-restarts — it doesn't reset to 0.20 every time.
-
-### 11.8 Mode Logging
-
-Every episode's metrics include a mode breakdown:
-
-```json
-{
-  "alpha_used": 0.42,
-  "mode_agree": 11,
-  "mode_rule": 7,
-  "mode_llm": 4
-}
-```
-
-The training printer shows this per episode:
-
-```
-Ep  12 | easy   | WIN  | reward= +12.40 | recall=1.00 prec=0.91 | steps=21 | wr=60% | α=0.42 | agree=11 rule=7 llm=4
-```
-
-You can watch the transition: early episodes have high `rule` counts; later
-episodes have high `agree` counts (LLM learned to make the same decisions as
-the rules, but also brings in strategic reasoning the rules can't).
+α is saved to `memory/alpha_{task}.json` and persists across Docker restarts — the agent doesn't reset to 0.20 every time.
 
 ---
 
-## 12. Training Loop End-to-End
+## 11. Training Loop End-to-End
 
 **File:** `train.py`
 
-### 12.1 Curriculum
+### Curriculum
 
-| Phase | Episodes | Task   | Goal                                                   |
-| ----- | -------- | ------ | ------------------------------------------------------ |
-| 1     | 1–20    | easy   | Learn basic signal thresholds, build first reflections |
-| 2     | 21–35   | medium | Handle decoys, learn evasion response                  |
-| 3     | 36–50   | hard   | Feature-only detection, persistent evasion             |
+| Phase | Episodes | Task | Goal |
+|---|---|---|---|
+| 1 | 1–20 | easy | Learn basic signal thresholds, build first reflections |
+| 2 | 21–35 | medium | Handle decoys, learn evasion response |
+| 3 | 36–50 | hard | Feature-only detection, persistent evasion |
 
 Seeds rotate deterministically: `seed = (episode_num + task_offset) % 50`
-so the agent sees all 50 pre-generated episodes before revisiting any.
 
-### 12.2 Per-Episode Flow
+### Per-Episode Flow
 
 ```
 for ep in range(n_episodes):
 
-  1. DETERMINE TASK
-     current_task = curriculum_task(ep) or fixed task
-
-  2. COMPUTE ALPHA (per-task cap applied)
-     n_refs = memory.reflection_count(current_task)
-     wr = memory.recent_win_rate(current_task, n=10)
-     alpha = compute_alpha(wr, n_refs, task=current_task)  # capped per task
-
-  3. LOAD CONTEXT
-     reflections = memory.get_reflections(task, n=4)   # last 4 lessons
-     few_shot = memory.get_best_trajectory(task)        # best win so far
-
-  4. RUN EPISODE (hybrid policy)
-     obs = env.reset(task, seed)
-     while not obs.done:
-         rule_action, rule_conf = get_rule_action(obs)
-         llm_action, raw_llm = get_action(obs, reflections, few_shot, α, temperature)
-         final = blend(rule_action, llm_action, rule_conf, alpha)
-         obs = env.step(final)
-
-  5. POST-EPISODE LEARNING
-     memory.record_win(task, won, ep)
-     new_alpha = compute_alpha(updated_wr, n_refs)
-     memory.save_alpha(task, new_alpha)
-
-     if won:
-         memory.add_trajectory(task, action_log, final_msg, reward, ep)
-         if new_best_or_no_refs:
-             reflection = generate_success_reflection(...)
-             memory.add_reflection(task, reflection, ep, reward)
-     else:
-         reflection = generate_reflection(task, action_log, final_msg, ...)
-         memory.add_reflection(task, reflection, ep, reward)
-
-  6. LOG
-     print per-episode stats: task, win/loss, reward, recall, precision,
-                               steps, win_rate, α, mode breakdown
+  1. DETERMINE TASK      curriculum_task(ep) or fixed task
+  2. COMPUTE ALPHA       compute_alpha(win_rate, n_reflections, task)
+  3. LOAD CONTEXT        last 4 reflections + best win trajectory
+  4. RUN EPISODE         while not obs.done:
+                           blend(rule_action, llm_action, rule_conf, α)
+                           → obs = env.step(final)
+  5. POST-EPISODE        record_win → update α → generate reflection
+  6. LOG                 task | win/loss | reward | recall | precision | α | modes
 ```
 
-### 12.3 Metrics Saved
-
-Every 5 episodes, metrics are flushed to `runs/metrics.jsonl`:
-
-```json
-{
-  "episode": 15,
-  "task": "easy",
-  "seed": 14,
-  "won": true,
-  "reward": 13.20,
-  "steps_used": 23,
-  "recall": 1.00,
-  "precision": 0.91,
-  "action_log": ["INSPECT acc_0022", "INSPECT acc_0037", ...],
-  "final_message": "[WIN] TP=10 FP=1 FN=0 ...",
-  "n_reflections_used": 4,
-  "had_few_shot": true,
-  "alpha_used": 0.52,
-  "mode_agree": 13,
-  "mode_rule": 6,
-  "mode_llm": 4,
-  "timestamp": "2026-04-01T10:23:41"
-}
-```
+Episode metrics (flushed to `runs/metrics.jsonl` every 5 episodes) include: `episode`, `task`, `won`, `reward`, `recall`, `precision`, `steps_used`, `alpha_used`, `mode_agree`, `mode_rule`, `mode_llm`, `n_reflections_used`.
+
+You can watch the transition: early episodes have high `rule` counts; later episodes have high `agree` counts (LLM learned to make the same decisions as the rules, but also brings strategic reasoning the rules can't).
 
 ---
 
-## 13. API Reference
+## 12. API Reference
 
 **File:** `server/app.py`
 
-### GET /health
+| Endpoint | Method | Description |
+|---|---|---|
+| `/health` | GET | `{"status": "healthy"}` |
+| `/tasks` | GET | Task list + `action_schema` + `score_range: [0.0, 1.0]` |
+| `/reset` | POST | Accepts `{task, seed}` → returns initial observation |
+| `/step` | POST | Accepts any `FakeGangAction` → returns updated observation |
+| `/state` | GET | Current episode metadata (step count, task, score) |
+| `/grader` | GET | Normalised [0.0, 1.0] score after SUBMIT |
+| `/baseline` | POST | Runs rule-based agent on all 3 tasks, returns scores |
 
-```json
-{"status": "healthy"}
-```
+**Baseline performance:**
 
-### GET /tasks
-
-```json
-{
-  "tasks": ["easy", "medium", "hard"],
-  "descriptions": {
-    "easy":   "50 accounts, 10 fakes, no evasion, 30 steps",
-    "medium": "200 accounts, 10 fakes + 20 decoys, evasion at step 20, 50 steps",
-    "hard":   "1000 accounts, 10 fakes + 50 decoys, recurring evasion, 80 steps"
-  },
-  "action_schema": {
-    "action_type": ["inspect", "investigate_network", "flag", "unflag", "submit"],
-    "account_id": "string (required for all actions except submit)"
-  },
-  "score_range": [0.0, 1.0]
-}
-```
-
-### POST /reset
-
-Request:
-
-```json
-{"task": "easy", "seed": 0}
-```
-
-Response: `StepResponse` with initial observation.
-
-### POST /step
-
-Request: Any `FakeGangAction`:
-
-```json
-{"action_type": "inspect", "account_id": "acc_0042"}
-{"action_type": "flag", "account_id": "acc_0017"}
-{"action_type": "submit"}
-```
-
-Response: `StepResponse` with updated observation, done flag, and reward.
-
-### GET /state
-
-Returns current episode metadata:
-
-```json
-{
-  "episode_id": "uuid",
-  "step_count": 12,
-  "task": "easy",
-  "score_so_far": -0.12,
-  "evasion_count": 0,
-  "network_size": 50,
-  "gang_size": 10,
-  "episode_seed": 0
-}
-```
-
-### GET /grader
-
-Returns the normalised grader score after SUBMIT. Error 400 if episode not done.
-
-```json
-{"score": 0.91, "task": "easy", "episode_id": "uuid"}
-```
-
-### POST /baseline
-
-Runs the rule-based agent on all three tasks (seed=0) and returns scores:
-
-```json
-{
-  "scores": {"easy": 0.91, "medium": 0.906, "hard": 0.9038},
-  "agent": "rule_based"
-}
-```
-
-**Baseline performance across 50 seeds:**
-
-| Task   | Seed=0 score | Win rate (50 seeds) | Mean score (50 seeds) |
-| ------ | ------------ | ------------------- | --------------------- |
-| easy   | 0.91         | 100%                | ~0.91                 |
-| medium | 0.906        | 84%                 | ~0.77                 |
-| hard   | 0.9038       | 52%                 | ~0.47                 |
-
-The baseline is a deterministic rule-based agent — no LLM, no learning. The
-difficulty scaling is designed so that easy is consistently solvable, medium
-requires some luck, and hard genuinely challenges frontier LLM agents via
-evasion events that destroy graph signals mid-investigation.
+| Task | Seed=0 score | Win rate (50 seeds) | Mean score (50 seeds) |
+|---|---|---|---|
+| easy | 0.91 | 100% | ~0.91 |
+| medium | 0.906 | 84% | ~0.77 |
+| hard | 0.9038 | 52% | ~0.47 |
 
 ---
 
-## 14. Docker Deployment
-
-**File:** `server/Dockerfile`
-
-### 14.1 Build
+## 13. Docker Deployment
 
 ```bash
-cd fake_gang_env
+# Build
 docker build -f server/Dockerfile -t graphstrike .
-```
-
-Build takes ~10 seconds because:
-
-- The `.dockerignore` excludes `episodes/` (109 MB), `memory/`, `runs/`
-- Python wheels are pre-downloaded to `wheels/` — no network access during `pip install`
-- No `apt-get` installs needed (everything is pure Python)
-
-### 14.2 Run
 
-```bash
+# Run
 docker run -it \
   -e AWS_ACCESS_KEY_ID=your_key \
   -e AWS_SECRET_ACCESS_KEY=your_secret \
@@ -1418,239 +456,67 @@ docker run -it \
   graphstrike
 ```
 
-The volumes preserve all learning between runs. When you restart the container,
-the agent continues from where it left off (α values, reflections, best trajectories).
+The `memory/` and `runs/` volumes preserve all learning between container restarts.
 
-### 14.3 Environment Variables
+### Environment Variables
 
-| Variable                  | Default         | Description                           |
-| ------------------------- | --------------- | ------------------------------------- |
-| `AWS_ACCESS_KEY_ID`     | (required)      | For Bedrock/Qwen3 access              |
-| `AWS_SECRET_ACCESS_KEY` | (required)      | For Bedrock/Qwen3 access              |
-| `AWS_DEFAULT_REGION`    | `us-east-1`   | Bedrock region                        |
-| `TRAIN_TASK`            | `` (curriculum) | Fix to `easy`/`medium`/`hard`   |
-| `TRAIN_EPISODES`        | `50`          | Total training episodes               |
-| `TRAIN_TEMP`            | `0.4`         | LLM sampling temperature              |
-| `TRAIN_VERBOSE`         | `0`           | Set `1` for per-step action logging |
-| `SERVER_PORT`           | `8000`        | FastAPI port                          |
+| Variable | Default | Description |
+|---|---|---|
+| `AWS_ACCESS_KEY_ID` | (required) | For Bedrock/Qwen3 access |
+| `AWS_SECRET_ACCESS_KEY` | (required) | For Bedrock/Qwen3 access |
+| `AWS_DEFAULT_REGION` | `us-east-1` | Bedrock region |
+| `TRAIN_TASK` | (curriculum) | Fix to `easy`/`medium`/`hard` |
+| `TRAIN_EPISODES` | `50` | Total training episodes |
+| `TRAIN_TEMP` | `0.4` | LLM sampling temperature |
+| `TRAIN_VERBOSE` | `0` | Set `1` for per-step action logging |
+| `SERVER_PORT` | `8000` | FastAPI port |
 
-### 14.4 Startup Sequence (run.sh)
+### Startup Sequence (`run.sh`)
 
 ```
-1. Validate AWS credentials (exits if missing)
-2. python server/generator.py    → generates/overwrites 150 episode JSON files (~1s)
+1. Validate AWS credentials
+2. python server/generator.py    → generates 150 episode JSON files
 3. uvicorn server.app:app        → starts the environment server
-4. Python urllib health check    → polls /health until ready (no curl needed)
+4. Health check polling          → waits until /health responds
 5. python train.py               → runs the full training loop
 ```
 
 ---
 
-## 15. Submission Requirements
-
-All submission requirements are satisfied. The environment is deployed at
-[huggingface.co/spaces/Pandago/graphstrike](https://huggingface.co/spaces/Pandago/graphstrike).
-
-### 15.1 Required Endpoints
-
-| Endpoint      | Method | Status | Description                                              |
-| ------------- | ------ | ------ | -------------------------------------------------------- |
-| `/health`   | GET    | ✅     | Returns `{"status": "healthy"}`                        |
-| `/tasks`    | GET    | ✅     | 3 tasks +`action_schema` + `score_range: [0.0, 1.0]` |
-| `/reset`    | POST   | ✅     | Accepts `{task, seed}`, returns initial observation    |
-| `/step`     | POST   | ✅     | Accepts any valid action, returns updated observation    |
-| `/state`    | GET    | ✅     | Returns episode metadata (step count, task, score)       |
-| `/grader`   | GET    | ✅     | Returns normalised [0.0, 1.0] score after SUBMIT         |
-| `/baseline` | POST   | ✅     | Runs rule-based agent on all 3 tasks, returns scores     |
-
-### 15.2 /tasks with action_schema
-
-The `/tasks` endpoint returns the `action_schema` dict listing all valid
-`action_type` values and the `account_id` field description. Graders can
-discover the full action space without reading code.
-
-### 15.3 /grader — Normalised Scoring
-
-After calling `SUBMIT` (via `/step`), call `GET /grader` to retrieve the
-normalised [0.0, 1.0] grader score. Returns 400 if the episode is not yet done.
-
-The score formula (see §7.8) rewards recall, precision, and efficiency.
-Maximum score 1.0 requires finding all 10 gang members with no false positives
-and using no steps. The grader is **deterministic** — same actions produce same score.
-
-### 15.4 /baseline — Reproducible Baseline Agent
-
-`POST /baseline` imports `inference.py`'s `run_rule_based_episode` and runs it
-on all three tasks with seed=0. Returns:
-
-```json
-{"scores": {"easy": 0.91, "medium": 0.906, "hard": 0.9038}, "agent": "rule_based"}
-```
-
-**Reproducibility:** The baseline is fully deterministic — no randomness, no LLM calls.
-Calling `/baseline` 3+ times in succession produces **identical scores** every time.
-The evasion flags (`_fired_*` attributes) are properly cleared on `reset()`,
-ensuring episodes replay identically across runs.
-
-### 15.5 inference.py
-
-**Library mode** (used by `/baseline`):
-
-```python
-from inference import run_rule_based_episode
-score = run_rule_based_episode(env, task="easy", seed=0)
-# Returns float in [0.0, 1.0]
-```
-
-**CLI mode** (connect to running server):
-
-```bash
-python inference.py --url http://localhost:8000
-# → {"scores": {"easy": 0.91, "medium": 0.906, "hard": 0.9038}, "agent": "rule_based"}
-```
-
-**CLI mode** (local, no server needed):
-
-```bash
-python inference.py --local
-```
-
-The rule-based strategy:
-
-1. If SUSPECT accounts are uninspected → INSPECT highest suspect
-2. If any inspected account has `fake_risk_score ≥ threshold` and not flagged → FLAG it
-3. If no immediate flag or suspect → INSPECT highest-risk uninspected account
-4. If steps ≤ 3 or 10 flags placed → SUBMIT
-
-Thresholds by task: easy=0.60, medium=0.50, hard=0.45.
-
-### 15.6 validate.py — 24-Point Pre-Submission Validator
-
-Runs 24 checks split between local (no server) and HTTP:
-
-```bash
-python validate.py --local            # 9 local checks only
-python validate.py --url http://...   # all 24 checks (requires running server)
-```
-
-Checks include:
-
-- scoring.py math correctness (gang risk ≥ 0.60, celebrity risk < 0.20, perfect score = 1.00)
-- models.py has all new fields (fake_risk_score, suspect_ids, AccountStatus)
-- environment.py SUSPECT cascade triggers after FLAG
-- inference.py runs without error and returns [0,1] float
-- episodes have new features (comment_repeat_score, shared_ip_count, celeb_ids)
-- /health reachable
-- /tasks has action_schema and score_range
-- /reset works for all three tasks
-- /step supports INSPECT, FLAG, SUBMIT
-- /grader returns [0,1] float after SUBMIT
-- /baseline returns 3 valid scores
-
-**All 24/24 checks pass.**
-
-### 15.7 Judging Criteria Alignment
-
-| Criterion                    | Weight | How GraphStrike addresses it                                                                                                                                     |
-| ---------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **Domain quality**     | 30%    | Real-world fraud detection domain; signals modelled on actual Instagram fake-account patterns (IP clustering, photo reuse, bio templates, temporal coordination) |
-| **Task & grader**      | 25%    | 3 difficulty tiers with clear win conditions; grader formula rewards recall, precision, and efficiency; partial credit for incomplete investigations             |
-| **Environment design** | 20%    | Bidirectional graph, dual cascade (follow + IP), evasion events that destroy signals mid-investigation, decoy accounts that penalise reckless flagging           |
-| **Code quality**       | 15%    | Typed Pydantic models, stateless scoring functions, 24-point validator, deterministic episode generation by seed                                                 |
-| **Creativity**         | 10%    | Hybrid rule/LLM policy with dynamic α caps, Reflexion-based learning without fine-tuning, IP cluster cascade as evasion-resistant signal                        |
-
----
-
-## 16. Verification & Validation
-
-### Quick smoke test
-
-```bash
-cd fake_gang_env
-
-# Test scoring math
-python3 -c "
-import sys; sys.path.insert(0,'server')
-from scoring import compute_fake_risk, compute_hub_legitimacy, grader_score
-
-gang_r = compute_fake_risk(0.75, 0.65, 0.85, 0.10)
-hub    = compute_hub_legitimacy(2_000_000, 200, 2000, 0.05)
-celeb  = compute_fake_risk(0.02, 0.02, 0.10, hub)
-assert gang_r >= 0.60,  f'Gang risk too low: {gang_r}'
-assert celeb  <  0.20,  f'Celebrity risk too high: {celeb}'
-assert grader_score(10, 0, 0, 0, 30) == 1.0
-print(f'Gang risk={gang_r}  Celeb risk={celeb}  Perfect score=1.0  OK')
-"
-
-# Test hybrid policy + cascade
-python3 -c "
-import sys, json; sys.path.insert(0,'server')
-from models import FakeGangAction, ActionType
-from environment import FakeGangEnvironment
-from agent.hybrid_policy import get_rule_action, compute_alpha
-
-env = FakeGangEnvironment()
-obs = env.reset(task='easy', seed=0)
-gang = json.loads(open('episodes/easy_000.json').read())['gang_member_ids']
-obs = env.step(FakeGangAction(action_type=ActionType.INSPECT, account_id=gang[0]))
-obs = env.step(FakeGangAction(action_type=ActionType.FLAG, account_id=gang[0]))
-assert len(obs.suspect_ids) > 0, 'Cascade failed'
-action, conf = get_rule_action(obs)
-assert action.account_id in obs.suspect_ids, 'Rule not prioritising suspects'
-print(f'Cascade OK: {len(obs.suspect_ids)} suspects. Rule → INSPECT {action.account_id} (conf={conf:.2f})')
-a0 = compute_alpha(0, 0, 'easy')
-a1 = compute_alpha(0.5, 2, 'easy')
-a2 = compute_alpha(1.0, 4, 'easy')
-print(f'Alpha (easy, cap=0.50): min={a0} mid={a1} max={a2}')
-"
-
-# Full local validate
-python3 validate.py --local
-```
 
-### Full HTTP validation (requires running server)
+### Full HTTP validation
 
 ```bash
 python3 -m uvicorn server.app:app --port 8001 &
 sleep 3
 python3 validate.py --url http://localhost:8001
+# Expected: Results: 24/24 passed — all OK
 ```
 
-Expected output: `Results: 24/24 passed — all OK`
-
 ### Deployed Endpoint Verification
 
-The live environment at [huggingface.co/spaces/Pandago/graphstrike](https://huggingface.co/spaces/Pandago/graphstrike)
-responds to all standard OpenEnv endpoints:
-
 ```bash
-# Health check
 curl https://pandago-graphstrike.hf.space/health
 # → {"status": "healthy"}
 
-# Task discovery
 curl https://pandago-graphstrike.hf.space/tasks
 # → {"tasks": ["easy","medium","hard"], "action_schema": {...}, "score_range": [0.0, 1.0]}
 
-# Baseline (deterministic, reproducible)
 curl -X POST https://pandago-graphstrike.hf.space/baseline
 # → {"scores": {"easy": 0.91, "medium": 0.906, "hard": 0.9038}, "agent": "rule_based"}
 ```
 
 ---
 
-
-
 ![Material wave loading](https://github.com/user-attachments/assets/a08255eb-9647-471d-9881-61871332249f)
 
 ## Developed with ❤️ by Team ComputeXOR
 
-
 ### {
 
-### [Sai Nivedh](https://github.com/SaiNivedh26) , 
+### [Sai Nivedh](https://github.com/SaiNivedh26) ,
 
-### [Chaaruvarthan](https://github.com/Charuvarthan-T) , 
+### [Charuvarthan](https://github.com/Charuvarthan-T) ,
 
 ### [Sajeev](https://github.com/SajeevSenthil)
 

README.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1723ffdcb7e36d47ff546500f86b30ebbb40af3a8616e1818798aaf144e0f5fc
+size 1448893

docs.md

@@ -0,0 +1,312 @@
+---
+title: GraphStrike
+emoji: 🕵️
+colorFrom: blue
+colorTo: indigo
+sdk: docker
+app_port: 7860
+pinned: false
+license: mit
+tags:
+  - reinforcement-learning
+  - social-network
+  - fraud-detection
+  - openenv
+  - llm-agent
+---
+<br>
+
+<p align="center">
+<img src="images/logo.png" width="600"/>
+</p>
+
+<br>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/Hugging%20Face-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black"/>
+  <img src="https://img.shields.io/badge/HF%20Spaces-FFBF00?style=for-the-badge&logo=huggingface&logoColor=black"/>
+  <img src="https://img.shields.io/badge/FastAPI-009688?style=for-the-badge&logo=fastapi&logoColor=white"/>
+  <img src="https://img.shields.io/badge/Docker-2496ED?style=for-the-badge&logo=docker&logoColor=white"/>
+  <img src="https://img.shields.io/badge/Gradio-F97316?style=for-the-badge&logo=gradio&logoColor=white"/>
+  <img src="https://img.shields.io/badge/OpenEnv-4B5563?style=for-the-badge&logo=envato&logoColor=white"/>
+  <img src="https://img.shields.io/badge/Amazon%20Bedrock-FF9900?style=for-the-badge&logo=amazonaws&logoColor=white"/>
+</p>
+<br>
+
+<h1 align="center">
+</h1>
+  <p align="center">
+    An OpenEnv-compatible reinforcement learning environment where an LLM agent must identify all 10 members of a coordinated fake account network hidden inside a synthetic social network. The agent learns via Reflexion and a dynamic hybrid rule/LLM policy , not via gradient updates or fine-tuning.
+    <br />
+    </p>
+</p>
+
+<br>
+<br>
+
+### *Deployed Endpoint Verification*
+
+The live environment at [huggingface.co/spaces/Pandago/graphstrike](https://huggingface.co/spaces/Pandago/graphstrike)
+responds to all standard OpenEnv endpoints:
+
+```bash
+# Health check
+curl https://pandago-graphstrike.hf.space/health
+# → {"status": "healthy"}
+
+# Task discovery
+curl https://pandago-graphstrike.hf.space/tasks
+# → {"tasks": ["easy","medium","hard"], "action_schema": {...}, "score_range": [0.0, 1.0]}
+
+# Baseline (deterministic, reproducible)
+curl -X POST https://pandago-graphstrike.hf.space/baseline
+# → {"scores": {"easy": 0.91, "medium": 0.906, "hard": 0.9038}, "agent": "rule_based"}
+```
+
+---
+
+<br>
+
+We evaluate GraphStrike's hybrid rule/LLM policy across multiple *frontier models to measure how well each model handles the investigation task. All runs use
+the same inference pipeline (`inference.py`) with identical system prompts and structured logging. Each model ran: (1) seed=0 on all 3 tasks, and
+(2) seeds 0-2 on all 3 tasks for variance measurement.*
+
+<br>
+
+**Seed=0 scores (single episode per task):**
+
+<p align="center">
+  <img src="images/table1.png" alt="Model Performance Table" width="1600"/>
+</p>
+<br>
+
+**3-seed variance scores (mean across seeds 0, 1, 2):**
+
+<p align="center">
+  <img src="images/table2.png" alt="Model Performance Table" width="1600"/>
+</p>
+<br>
+
+ **Rule-Based Baseline (no LLM, deterministic)**
+
+<p align="center">
+  <img src="images/table3.png" alt="Model Performance Table" width="1600"/>
+</p>
+<br>
+
+---
+
+**The task:** A social network contains fake accounts organised into a
+single coordinated network of 10. The network behaves in a coordinated way — same posting hour,
+same IP subnet, stolen celebrity photos, copy-paste bios. The agent must find
+all 10 by navigating a limited step budget, inspecting accounts, and flagging suspects.
+
+**What makes this non-trivial:** The network is large (50–1000 accounts depending on difficulty). Fake accounts are mixed with innocent high-signal "decoy" accounts.In hard mode, the fake accounts actively evades — dropping intra-account follows, renaming profiles — while the agent is mid-investigation.The agent cannot see the full network upfront: it must explore via INSPECT and INVESTIGATE_NETWORK actions, spending steps to reveal information.
+
+**What makes the learning novel:** The LInference LLM (via aws bedrock) cannot be fine-tuned.it's a black-box API. The agent learns via Reflexion i.e., post-episode lessons are written back into memory and injected into every future prompt. A dynamic hybrid policy (α-weighted) blends the LLM with a deterministic rule engine, with the blend weight α updating based on recent win rate. Rules dominate early; the LLM takes over as it proves itself.
+
+---
+
+## Detection Signals
+
+Detection operates entirely on numeric metadata — no content processing. Three signal categories, computed at different points:
+
+**Node signals** (pre-computed by content pipeline, static per account):
+
+| Feature | Fake range | Real range | Notes |
+|---|---|---|---|
+| `photo_reuse_score` | 0.30–0.95 | 0.00–0.15 | pHash fingerprint match against celebrity photo DB |
+| `bio_template_score` | 0.20–0.90 | 0.00–0.12 | Cosine sim to known fake bio templates |
+| `comment_repeat_score` | 0.60–0.90 | 0.00–0.08 | Copy-paste spam fraction across accounts |
+
+**Behavioral signals** (static, from account metadata):
+
+| Feature | Fake pattern |
+|---|---|
+| `avg_post_hour` | All 10 gang members cluster within ±0.5 hours — coordinated scheduling |
+| `account_age_days` | Created within the same week (`base_age ± 7 days`) |
+| `shared_ip_count` | 9 — all 10 share one IP subnet (`ip_gang_{seed}`) |
+
+**Graph signals** (dynamic — computed at INSPECT time, shift as investigation progresses):
+
+| Feature | Why it matters |
+|---|---|
+| `mutual_follow_rate` | Gang members mutually follow each other at 0.6–0.9 density; legitimate hubs don't follow back |
+| `flagged_neighbor_count` | Grows as more gang members are flagged — the cascade signal |
+| `post_hour_cluster_score` | Alignment to mean posting hour of currently-flagged accounts (wrap-around aware) |
+| `suspicious_mutual_ratio` | Used to compute hub legitimacy — protects celebrities from false positives |
+
+Graph signals are the most powerful: once one gang member is flagged, `flagged_neighbor_count` rises for all connected members, compounding with each subsequent flag.
+
+---
+
+## Synthetic Network Composition
+
+150 episodes pre-generated deterministically (50 per task). Each episode is a JSON file (`episodes/{task}_{seed:03d}.json`).
+
+| Task | Accounts | Gang | Decoys | Max steps | Evasion |
+|---|---|---|---|---|---|
+| easy | 50 | 10 | 0 | 30 | None |
+| medium | 200 | 10 | 20 | 50 | Step 20 (once) |
+| hard | 1000 | 10 | 50 | 80 | Steps 15/30/45/60 |
+
+- **Gang:** Dense intra-follow graph (density 0.60–0.80), same IP subnet, tightly clustered post hours (std 0.5/1.5/2.5 by task).
+- **Decoys** (medium/hard only): Real accounts with elevated `photo_reuse` and `bio_template` scores (0.20–0.40). They score as suspicious but are not gang members — they penalise reckless flagging.
+- **Celebrities** (2 per episode): 100k–5M followers, near-zero fake scores. Hub legitimacy formula protects them.
+- **Zero-edge isolates** (2 per episode): `follower_count=0`, no edges. Test whether the agent wastes steps on disconnected nodes.
+
+---
+
+## Actions
+
+| Action | Cost | Effect |
+|---|---|---|
+| `inspect` | 1 step | Reveals full `AccountProfile` (all 22 features), adds neighbors to visible set |
+| `investigate_network` | 2 steps | Bidirectional 2-hop expansion — reveals account IDs only (no profiles); re-cascades SUSPECT |
+| `flag` | 0 steps | Marks account CONFIRMED_FAKE; dual cascade: follow-graph + IP cluster |
+| `unflag` | 0 steps | Clears CONFIRMED_FAKE status |
+| `submit` | 0 steps | Ends episode, triggers scoring |
+
+**Dual SUSPECT cascade on FLAG:**
+1. *Follow-graph:* Every visible account that the flagged account follows → SUSPECT (high precision: gang follow density 0.70+).
+2. *IP cluster:* Every visible account sharing the same `ip_cluster_id` → SUSPECT (zero false positives: real accounts each have a unique IP; gang shares `ip_gang_{seed}`).
+
+Both mechanisms surface in `obs.suspect_ids` — the agent's highest-priority INSPECT targets.
+
+---
+
+## Risk Scoring (`server/scoring.py`)
+
+All functions are stateless, called inside `_build_profile()` at INSPECT time and on re-profiling after each FLAG.
+
+```
+node_risk     = 0.60 × photo_reuse + 0.40 × bio_template
+
+age_norm      = min(1.0, account_age_days / 365)
+behavior_risk = 0.55 × (1 − age_norm) + 0.45 × post_hour_cluster_score
+
+flagged_ratio = flagged_neighbor_count / max(inspected_neighbor_count, 1)
+graph_risk    = 0.45 × flagged_ratio + 0.35 × mutual_follow_rate + 0.20 × avg_neighbor_photo_reuse
+
+hub_legitimacy = 0.45 × log(1+followers)/log(1+1M)
+              + 0.25 × (1 − follow_ratio_norm)
+              + 0.20 × age_norm
+              + 0.10 × (1 − suspicious_mutual_ratio)
+
+fake_risk = clip(0.30×node_risk + 0.25×behavior_risk + 0.45×graph_risk − 0.25×hub_legitimacy, 0, 1)
+```
+
+**Weight rationale:** Graph risk (0.45) is dominant — structural signals are hardest to fake and compound across the investigation. Hub legitimacy is subtractive — a celebrity with 5M followers produces `hub_legitimacy ≈ 1.0`, making their fake_risk near zero even if gang members follow them.
+
+**Classification thresholds:**
+- `fake_risk < 0.35` → normal
+- `0.35 ≤ fake_risk < 0.60` → suspect
+- `fake_risk ≥ 0.60` → confirmed_fake (formula-level; explicit FLAG overrides)
+
+**Grader score** (normalised [0.0, 1.0], returned by `/grader`):
+```
+recall    = tp / 10
+precision = tp / max(tp + fp, 1)
+efficiency = max(0, (max_steps − steps_used) / max_steps)
+
+if recall ≥ 0.8 AND precision ≥ 0.7:
+    score = 0.55 + 0.20×recall + 0.15×precision + 0.10×efficiency
+else:
+    score = 0.30×recall + 0.10×precision
+```
+Maximum 1.0 (all 10 found, zero false positives, zero steps used). Win threshold ≈ 0.815.
+
+---
+
+## Hybrid Policy (`agent/hybrid_policy.py`)
+
+The agent blends a deterministic rule engine with Qwen3-Next-80B (via AWS Bedrock) using a per-task trust weight α.
+
+**Alpha update** (per episode, after win/loss recorded):
+```
+reflection_factor = min(1.0, n_reflections / 4.0)
+raw   = 0.20 + reflection_factor × (0.80 × recent_win_rate + 0.12)
+alpha = clamp(raw, 0.20, task_cap)
+```
+
+| Task | α cap | Rationale |
+|---|---|---|
+| easy | 0.50 | Rule engine alone hits ~91% — LLM assists, doesn't override |
+| medium | 0.70 | Decoys require LLM judgment, but cascade must stay |
+| hard | 0.85 | LLM needs latitude for evasion adaptation |
+
+`reflection_factor` gates α: the LLM must accumulate ≥4 post-episode lessons before reaching meaningful trust, regardless of raw win rate.
+
+**Blending decision:**
+```python
+rule_action, rule_conf = get_rule_action(obs)   # deterministic, with confidence score
+llm_action,  _        = get_action(obs, ...)    # Qwen3 via Bedrock
+
+if rule_action == llm_action:   final = llm_action     # agree
+elif rule_conf >= alpha:        final = rule_action     # rule overrides
+else:                           final = llm_action      # LLM trusted
+```
+
+Rule confidences: SUBMIT-forced=1.00, INSPECT-suspect=0.95, FLAG-high-risk=0.95, FLAG-threshold=0.70+, INSPECT-explore=0.30. At `α=0.50` (easy cap), safety decisions (suspects, forced submit) always override; exploration goes to the LLM.
+
+**Reflexion learning:** After each episode, Qwen3 generates a 2–3 sentence lesson from the action log and outcome. Lessons are stored in `memory/reflections_{task}.jsonl` and injected into every future prompt (last 4 lessons + best winning trajectory as few-shot example). Memory persists across container restarts via Docker volume.
+
+---
+
+## API Reference
+
+| Endpoint | Method | Description |
+|---|---|---|
+| `/health` | GET | `{"status": "healthy"}` |
+| `/tasks` | GET | Task list + `action_schema` + `score_range: [0.0, 1.0]` |
+| `/reset` | POST | `{task, seed}` → initial observation |
+| `/step` | POST | `{action_type, account_id?}` → updated observation |
+| `/state` | GET | Episode metadata (step count, task, score, evasion count) |
+| `/grader` | GET | Normalised [0.0, 1.0] score after SUBMIT (400 if not done) |
+| `/baseline` | POST | Runs rule-based agent on all 3 tasks, seed=0 |
+| `/metadata` | GET | OpenEnv metadata block |
+| `/schema` | GET | Full JSON schema for actions and observations |
+| `/mcp` | POST | JSON-RPC 2.0 tool discovery (Model Context Protocol) |
+
+Live: `https://pandago-graphstrike.hf.space`
+
+---
+
+## File Structure
+
+```
+server/
+  app.py          — FastAPI + Gradio UI (gr.mount_gradio_app)
+  environment.py  — Episode lifecycle, action mechanics, cascade logic
+  generator.py    — Deterministic episode generation (150 JSON files)
+  scoring.py      — Stateless risk formula functions
+  models.py       — Pydantic models: AccountProfile, FakeGangObservation, ActionType
+
+agent/
+  policy.py       — Qwen3 prompt construction + action parsing
+  hybrid_policy.py — Alpha blending, rule engine with confidence scores
+  reflection.py   — Post-episode lesson generation
+  memory.py       — JSONL persistence for reflections, trajectories, alpha
+
+inference.py      — Submission entrypoint: [START]/[STEP]/[END] structured logs, OpenAI client
+validate.py       — 24-point pre-submission validator (local + HTTP)
+train.py          — Full training loop with curriculum
+episodes/         — 150 pre-generated JSON episode files (baked into Docker image)
+memory/           — Docker volume: reflections, win history, alpha values
+```
+
+---
+
+## Baseline Scores
+
+| Task | Seed=0 | Win rate (50 seeds) | Mean (50 seeds) |
+|---|---|---|---|
+| easy | 0.910 | 100% | ~0.91 |
+| medium | 0.906 | 84% | ~0.77 |
+| hard | 0.9038 | 52% | ~0.47 |
+
+The rule-based baseline (no LLM) is competitive on easy/medium. Hard is the real differentiator — evasion events drop intra-gang edges mid-investigation, destroying graph signals. Frontier LLM agents with accumulated reflections adapt; the rule engine degrades.
+
+---
+
+*Built by team computeXor*

eval-models/deepseek_test_judge_eval.py

@@ -0,0 +1,478 @@
+#!/usr/bin/env python3
+"""
+Judge Evaluation Simulator
+==========================
+
+Simulates EXACTLY how hackathon judges will evaluate your environment:
+
+1. Baseline re-run: POST /baseline → verify scores are stable
+2. Standard Open LLM agent: Run an LLM (via HF router) against all 3 tasks
+3. Score variance check: Run same task multiple seeds, check variance
+
+USAGE:
+    # Against live HF Space (requires HF_TOKEN):
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Against local server:
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url http://localhost:7860
+
+    # Choose model (default: Qwen/Qwen2.5-72B-Instruct):
+    export MODEL_NAME="nvidia/Llama-3.1-Nemotron-70B-Instruct-HF"
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Just test endpoints (no LLM needed):
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space --endpoints-only
+"""
+
+import json
+import os
+import sys
+import time
+import urllib.request
+from typing import Dict, List, Optional
+
+# ---------------------------------------------------------------------------
+# Config from env vars (same as judges will set)
+# ---------------------------------------------------------------------------
+
+HF_TOKEN = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+MODEL_NAME = os.getenv("MODEL_NAME", "deepseek.v3.2")
+
+# AWS Bedrock config (use --bedrock flag)
+LLM_BACKEND = "hf"  # "hf" or "bedrock"
+BEDROCK_MODEL_ID = os.getenv("BEDROCK_MODEL_ID", "deepseek.v3.2")
+
+SYSTEM_PROMPT = """You are an AI detective finding 10 coordinated fake accounts in a social network.
+
+ACTIONS (reply with exactly ONE line):
+- INSPECT acc_XXXX  — reveal profile (costs 1 step)
+- FLAG acc_XXXX     — mark as fake (FREE, no step cost, triggers suspect cascade)
+- SUBMIT            — end episode, get scored
+
+DECISION RULES (apply top-to-bottom, first match wins):
+1. If suspect_ids lists accounts you haven't inspected → INSPECT the first one
+2. If ANY profiled account has shared_ip_count >= 5 and is NOT flagged → FLAG it immediately
+3. If ANY profiled account has photo_reuse >= 0.50 AND bio_template >= 0.40 and hub < 0.70 and NOT flagged → FLAG it
+4. If ANY profiled account has fake_risk_score >= 0.30 and hub < 0.70 and NOT flagged → FLAG it
+5. If there are uninspected visible accounts and steps > 3 → INSPECT the next one
+6. If you have 10 flags OR steps <= 3 → SUBMIT
+
+IMPORTANT:
+- FLAG is FREE (costs 0 steps) — flag aggressively when you see suspicious signals
+- After each FLAG, new suspects appear — always inspect suspects before other accounts
+- hub_legitimacy_score > 0.70 means celebrity — do NOT flag
+- shared_ip_count >= 5 is the strongest gang signal (all 10 share one IP)
+- Do NOT re-inspect already inspected accounts
+
+Reply with EXACTLY one line, nothing else:
+FLAG acc_XXXX
+INSPECT acc_XXXX
+SUBMIT"""
+
+
+# ---------------------------------------------------------------------------
+# HTTP helpers
+# ---------------------------------------------------------------------------
+
+def _retry(fn, retries=3, backoff=3):
+    """Retry a function on network errors."""
+    for attempt in range(retries):
+        try:
+            return fn()
+        except OSError as e:
+            if attempt == retries - 1:
+                raise
+            wait = backoff * (attempt + 1)
+            print(f"    [RETRY] Network error: {e} — retrying in {wait}s ({attempt+1}/{retries})")
+            time.sleep(wait)
+
+
+def http_post(url: str, body: Optional[dict] = None) -> dict:
+    def _do():
+        data = json.dumps(body or {}).encode()
+        req = urllib.request.Request(
+            url, data=data,
+            headers={"Content-Type": "application/json"},
+            method="POST"
+        )
+        with urllib.request.urlopen(req, timeout=120) as resp:
+            return json.loads(resp.read())
+    return _retry(_do)
+
+
+def http_get(url: str, expect_json: bool = True) -> dict:
+    def _do():
+        with urllib.request.urlopen(url, timeout=120) as resp:
+            body = resp.read()
+            if not expect_json:
+                return {"_status": resp.status, "_body_len": len(body)}
+            return json.loads(body)
+    return _retry(_do)
+
+
+# ---------------------------------------------------------------------------
+# LLM call via OpenAI-compatible API
+# ---------------------------------------------------------------------------
+
+def _call_hf(prompt: str) -> str:
+    """Call LLM via HF router (OpenAI-compatible)."""
+    from openai import OpenAI
+    client = OpenAI(base_url=API_BASE_URL, api_key=HF_TOKEN)
+    resp = client.chat.completions.create(
+        model=MODEL_NAME,
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        temperature=0.3,
+        max_tokens=256,
+    )
+    return (resp.choices[0].message.content or "").strip()
+
+
+def _call_bedrock(prompt: str) -> str:
+    """Call LLM via AWS Bedrock. Tries converse() first, falls back to invoke_model()."""
+    import boto3
+    client = boto3.client(
+        service_name="bedrock-runtime",
+        region_name=os.getenv("AWS_DEFAULT_REGION", "us-east-1"),
+        aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
+        aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
+    )
+    # Try converse API first (boto3 >= 1.34.x)
+    if hasattr(client, "converse"):
+        resp = client.converse(
+            modelId=BEDROCK_MODEL_ID,
+            messages=[{"role": "user", "content": [{"text": prompt}]}],
+            system=[{"text": SYSTEM_PROMPT}],
+            inferenceConfig={"maxTokens": 256, "temperature": 0.3},
+        )
+        return resp["output"]["message"]["content"][0]["text"].strip()
+    # Fallback: invoke_model (works with all boto3 versions)
+    body = json.dumps({
+        "messages": [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        "max_tokens": 256,
+        "temperature": 0.3,
+    })
+    resp = client.invoke_model(
+        modelId=BEDROCK_MODEL_ID,
+        contentType="application/json",
+        accept="application/json",
+        body=body,
+    )
+    result = json.loads(resp["body"].read())
+    # Handle both OpenAI-style and Bedrock-native response formats
+    if "choices" in result:
+        return result["choices"][0]["message"]["content"].strip()
+    if "content" in result:
+        content = result["content"]
+        if isinstance(content, list):
+            return content[0].get("text", "").strip()
+        return str(content).strip()
+    if "output" in result:
+        return result["output"].get("text", "").strip()
+    return str(result).strip()
+
+
+def call_llm(prompt: str) -> str:
+    """Call LLM with retries. Uses HF router or Bedrock based on LLM_BACKEND."""
+    fn = _call_bedrock if LLM_BACKEND == "bedrock" else _call_hf
+    for attempt in range(3):
+        try:
+            raw = fn(prompt)
+            if os.getenv("DEBUG_LLM"):
+                print(f"    [LLM RAW] {raw[:200]}")
+            # Strip Qwen3 <think>...</think> reasoning blocks
+            import re
+            cleaned = re.sub(r"<think>.*?</think>", "", raw, flags=re.DOTALL).strip()
+            return cleaned if cleaned else raw
+        except Exception as e:
+            if attempt == 2:
+                print(f"    [LLM ERROR] {e} (gave up after 3 attempts)")
+                return ""
+            wait = 3 * (attempt + 1)
+            print(f"    [LLM RETRY] {e} — retrying in {wait}s")
+            time.sleep(wait)
+    return ""
+
+
+def format_obs(obs: dict) -> str:
+    """Format observation as text for LLM — shows raw signals prominently."""
+    lines = []
+    lines.append(f"TASK: {obs.get('task','?').upper()} | Steps remaining: {obs.get('steps_remaining','?')}")
+
+    flagged = obs.get("flagged_ids", [])
+    lines.append(f"Flagged ({len(flagged)}/10): {', '.join(flagged) if flagged else 'none'}")
+
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    uninspected_suspects = [s for s in suspects if s not in inspected]
+    if uninspected_suspects:
+        lines.append(f"*** SUSPECTS (uninspected) → INSPECT THESE FIRST: {', '.join(uninspected_suspects)} ***")
+
+    accounts = obs.get("visible_accounts", [])
+    if accounts:
+        # Split: unflagged accounts that should be flagged vs rest
+        unflagged_suspicious = []
+        flagged_accs = []
+        clean_accs = []
+        for a in sorted(accounts, key=lambda x: x.get("fake_risk_score", 0), reverse=True):
+            aid = a.get("account_id", "?")
+            if aid in flagged:
+                flagged_accs.append(a)
+            elif (a.get("shared_ip_count", 0) >= 5 or
+                  (a.get("photo_reuse_score", 0) >= 0.50 and a.get("bio_template_score", 0) >= 0.40)):
+                unflagged_suspicious.append(a)
+            else:
+                clean_accs.append(a)
+
+        if unflagged_suspicious:
+            lines.append(f"\n!!! ACTION NEEDED — FLAG THESE ({len(unflagged_suspicious)} accounts with strong fake signals):")
+            for a in unflagged_suspicious:
+                aid = a.get("account_id", "?")
+                lines.append(f"  → FLAG {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} ip_shared={a.get('shared_ip_count',0)} hub={a.get('hub_legitimacy_score',0):.2f}")
+
+        if flagged_accs:
+            lines.append(f"\nALREADY FLAGGED ({len(flagged_accs)}):")
+            for a in flagged_accs[:5]:
+                lines.append(f"  ✓ {a.get('account_id','?')}")
+
+        if clean_accs:
+            lines.append(f"\nCLEAN ACCOUNTS ({len(clean_accs)}):")
+            for a in clean_accs[:5]:
+                aid = a.get("account_id", "?")
+                hub = a.get("hub_legitimacy_score", 0)
+                hub_mark = " [CELEBRITY]" if hub > 0.70 else ""
+                lines.append(f"  {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} hub={hub:.2f}{hub_mark}")
+
+    visible = obs.get("visible_account_ids", [])
+    uninspected = [i for i in visible if i not in inspected]
+    if uninspected:
+        lines.append(f"\nUninspected IDs ({len(uninspected)}): {', '.join(uninspected[:8])}{'...' if len(uninspected) > 8 else ''}")
+
+    lines.append(f"\nMessage: {obs.get('message', '')}")
+    return "\n".join(lines)
+
+
+def parse_action(llm_text: str, obs: dict) -> dict:
+    """Parse LLM output to action dict."""
+    for line in llm_text.split("\n"):
+        line = line.strip()
+        upper = line.upper()
+        if upper.startswith("INSPECT ") or upper.startswith("FLAG ") or upper.startswith("INVESTIGATE_NETWORK ") or upper.startswith("UNFLAG "):
+            parts = line.split(maxsplit=1)
+            return {"action_type": parts[0].lower(), "account_id": parts[1].lower() if len(parts) > 1 else None}
+        if upper == "SUBMIT":
+            return {"action_type": "submit"}
+
+    # Fallback: inspect first uninspected suspect
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    for s in suspects:
+        if s not in inspected:
+            return {"action_type": "inspect", "account_id": s}
+    visible = obs.get("visible_account_ids", [])
+    for v in visible:
+        if v not in inspected:
+            return {"action_type": "inspect", "account_id": v}
+    return {"action_type": "submit"}
+
+
+# ---------------------------------------------------------------------------
+# Test phases
+# ---------------------------------------------------------------------------
+
+def test_endpoints(base_url: str) -> bool:
+    """Phase 0: Verify all required endpoints respond correctly."""
+    print("\n" + "="*60)
+    print("PHASE 0: Endpoint Verification")
+    print("="*60)
+
+    checks = [
+        ("GET",  "/health",   None,  True),
+        ("GET",  "/tasks",    None,  True),
+        ("GET",  "/metadata", None,  True),
+        ("GET",  "/schema",   None,  True),
+        ("GET",  "/web",      None,  False),   # returns HTML, not JSON
+        ("POST", "/reset",    {"task": "easy", "seed": 0}, True),
+        ("GET",  "/state",    None,  True),
+        ("POST", "/step",     {"action_type": "inspect", "account_id": "acc_0000"}, True),
+        ("POST", "/step",     {"action_type": "submit"}, True),
+        ("GET",  "/grader",   None,  True),
+        ("POST", "/mcp",      {"jsonrpc": "2.0", "method": "tools/list", "id": 1}, True),
+        ("POST", "/baseline", None,  True),
+    ]
+
+    all_ok = True
+    for method, path, body, expect_json in checks:
+        try:
+            if method == "GET":
+                http_get(f"{base_url}{path}", expect_json=expect_json)
+            else:
+                http_post(f"{base_url}{path}", body)
+            print(f"  ✓ {method} {path}")
+        except Exception as e:
+            print(f"  ✗ {method} {path} — {e}")
+            all_ok = False
+
+    return all_ok
+
+
+def test_baseline_stability(base_url: str) -> bool:
+    """Phase 1: Baseline re-run (must produce identical scores)."""
+    print("\n" + "="*60)
+    print("PHASE 1: Baseline Stability (3 runs)")
+    print("="*60)
+
+    scores_list = []
+    for i in range(3):
+        r = http_post(f"{base_url}/baseline")
+        scores = r["scores"]
+        scores_list.append(scores)
+        print(f"  Run {i+1}: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+
+    # Check all identical
+    stable = all(s == scores_list[0] for s in scores_list)
+    if stable:
+        print("  ✓ All 3 runs identical — baseline is deterministic")
+    else:
+        print("  ✗ SCORES DIFFER — baseline is non-deterministic!")
+    return stable
+
+
+def test_llm_agent(base_url: str, task: str, seed: int = 0) -> float:
+    """Phase 2: Run an LLM agent against one task (simulates judge's Nemotron run)."""
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"\n  --- LLM Agent: task={task}, seed={seed}, model={_model} ---")
+
+    # Reset
+    reset_resp = http_post(f"{base_url}/reset", {"task": task, "seed": seed})
+    obs = reset_resp.get("observation", reset_resp)
+    done = reset_resp.get("done", False)
+
+    step_num = 0
+    while not done:
+        step_num += 1
+        prompt = format_obs(obs)
+        llm_text = call_llm(prompt)
+        action = parse_action(llm_text, obs)
+
+        action_str = f"{action['action_type'].upper()} {action.get('account_id', '')}".strip()
+
+        step_resp = http_post(f"{base_url}/step", action)
+        obs = step_resp.get("observation", step_resp)
+        done = step_resp.get("done", False)
+        reward = step_resp.get("reward")
+
+        flagged_n = len(obs.get("flagged_ids", []))
+        suspects_n = len(obs.get("suspect_ids", []))
+        steps_left = obs.get("steps_remaining", "?")
+
+        print(f"    Step {step_num:2d}: {action_str:35s} flagged={flagged_n}/10  suspects={suspects_n}  steps_left={steps_left}")
+
+        if done and reward is not None:
+            msg = step_resp.get("message", obs.get("message", ""))
+            print(f"    → Episode ended: {msg[:100]}")
+
+    # Get grader score
+    grader = http_get(f"{base_url}/grader")
+    score = grader["score"]
+    print(f"    ★ GRADER SCORE: {score:.4f}")
+    return score
+
+
+def test_llm_all_tasks(base_url: str) -> Dict[str, float]:
+    """Phase 2: Run LLM agent on all 3 tasks."""
+    print("\n" + "="*60)
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"PHASE 2: LLM Agent Evaluation (model={_model})")
+    print("="*60)
+
+    scores = {}
+    for task in ["easy", "medium", "hard"]:
+        scores[task] = test_llm_agent(base_url, task=task, seed=0)
+
+    print(f"\n  Summary: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+    return scores
+
+
+def test_variance(base_url: str, seeds: List[int] = [0, 1, 2, 3, 4]) -> None:
+    """Phase 3: Score variance check (multiple seeds per task)."""
+    print("\n" + "="*60)
+    print(f"PHASE 3: Score Variance (seeds={seeds})")
+    print("="*60)
+
+    for task in ["easy", "medium", "hard"]:
+        task_scores = []
+        for seed in seeds:
+            score = test_llm_agent(base_url, task=task, seed=seed)
+            task_scores.append(score)
+
+        mean = sum(task_scores) / len(task_scores)
+        variance = sum((s - mean) ** 2 for s in task_scores) / len(task_scores)
+        print(f"\n  {task}: scores={[f'{s:.3f}' for s in task_scores]}  mean={mean:.4f}  var={variance:.6f}")
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Judge Evaluation Simulator for GraphStrike")
+    parser.add_argument("--url", required=True, help="Environment server URL")
+    parser.add_argument("--bedrock", action="store_true", help="Use AWS Bedrock instead of HF router")
+    parser.add_argument("--endpoints-only", action="store_true", help="Only test endpoints (no LLM)")
+    parser.add_argument("--skip-variance", action="store_true", help="Skip variance check (faster)")
+    parser.add_argument("--seeds", type=int, default=3, help="Number of seeds for variance check")
+    args = parser.parse_args()
+
+    if args.bedrock:
+        LLM_BACKEND = "bedrock"
+
+    base = args.url.rstrip("/")
+    model_display = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"GraphStrike Judge Evaluation Simulator")
+    print(f"Target:  {base}")
+    print(f"Backend: {LLM_BACKEND}")
+    print(f"Model:   {model_display}")
+    print(f"Token:   {'set' if (HF_TOKEN or os.getenv('AWS_ACCESS_KEY_ID')) else 'NOT SET'}")
+
+    # Phase 0: Endpoints
+    if not test_endpoints(base):
+        print("\n✗ Endpoint check failed. Fix before proceeding.")
+        sys.exit(1)
+
+    # Phase 1: Baseline stability
+    test_baseline_stability(base)
+
+    if args.endpoints_only:
+        print("\n✓ Endpoint-only mode — skipping LLM tests.")
+        sys.exit(0)
+
+    if LLM_BACKEND == "bedrock":
+        if not os.getenv("AWS_ACCESS_KEY_ID"):
+            print("\n✗ AWS_ACCESS_KEY_ID not set. Cannot run Bedrock LLM tests.")
+            sys.exit(1)
+    elif not HF_TOKEN:
+        print("\n✗ HF_TOKEN not set. Cannot run LLM agent tests.")
+        print("  export HF_TOKEN='hf_...'  OR  use --bedrock with AWS creds")
+        sys.exit(1)
+
+    # Phase 2: LLM on all tasks
+    scores = test_llm_all_tasks(base)
+
+    # Phase 3: Variance
+    if not args.skip_variance:
+        test_variance(base, seeds=list(range(args.seeds)))
+
+    print("\n" + "="*60)
+    print("EVALUATION COMPLETE")
+    print("="*60)

eval-models/gemma_test_judge_eval.py

@@ -0,0 +1,478 @@
+#!/usr/bin/env python3
+"""
+Judge Evaluation Simulator
+==========================
+
+Simulates EXACTLY how hackathon judges will evaluate your environment:
+
+1. Baseline re-run: POST /baseline → verify scores are stable
+2. Standard Open LLM agent: Run an LLM (via HF router) against all 3 tasks
+3. Score variance check: Run same task multiple seeds, check variance
+
+USAGE:
+    # Against live HF Space (requires HF_TOKEN):
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Against local server:
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url http://localhost:7860
+
+    # Choose model (default: Qwen/Qwen2.5-72B-Instruct):
+    export MODEL_NAME="nvidia/Llama-3.1-Nemotron-70B-Instruct-HF"
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Just test endpoints (no LLM needed):
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space --endpoints-only
+"""
+
+import json
+import os
+import sys
+import time
+import urllib.request
+from typing import Dict, List, Optional
+
+# ---------------------------------------------------------------------------
+# Config from env vars (same as judges will set)
+# ---------------------------------------------------------------------------
+
+HF_TOKEN = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+MODEL_NAME = os.getenv("MODEL_NAME", "google.gemma-3-12b-it")
+
+# AWS Bedrock config (use --bedrock flag)
+LLM_BACKEND = "hf"  # "hf" or "bedrock"
+BEDROCK_MODEL_ID = os.getenv("BEDROCK_MODEL_ID", "google.gemma-3-12b-it")
+
+SYSTEM_PROMPT = """You are an AI detective finding 10 coordinated fake accounts in a social network.
+
+ACTIONS (reply with exactly ONE line):
+- INSPECT acc_XXXX  — reveal profile (costs 1 step)
+- FLAG acc_XXXX     — mark as fake (FREE, no step cost, triggers suspect cascade)
+- SUBMIT            — end episode, get scored
+
+DECISION RULES (apply top-to-bottom, first match wins):
+1. If suspect_ids lists accounts you haven't inspected → INSPECT the first one
+2. If ANY profiled account has shared_ip_count >= 5 and is NOT flagged → FLAG it immediately
+3. If ANY profiled account has photo_reuse >= 0.50 AND bio_template >= 0.40 and hub < 0.70 and NOT flagged → FLAG it
+4. If ANY profiled account has fake_risk_score >= 0.30 and hub < 0.70 and NOT flagged → FLAG it
+5. If there are uninspected visible accounts and steps > 3 → INSPECT the next one
+6. If you have 10 flags OR steps <= 3 → SUBMIT
+
+IMPORTANT:
+- FLAG is FREE (costs 0 steps) — flag aggressively when you see suspicious signals
+- After each FLAG, new suspects appear — always inspect suspects before other accounts
+- hub_legitimacy_score > 0.70 means celebrity — do NOT flag
+- shared_ip_count >= 5 is the strongest gang signal (all 10 share one IP)
+- Do NOT re-inspect already inspected accounts
+
+Reply with EXACTLY one line, nothing else:
+FLAG acc_XXXX
+INSPECT acc_XXXX
+SUBMIT"""
+
+
+# ---------------------------------------------------------------------------
+# HTTP helpers
+# ---------------------------------------------------------------------------
+
+def _retry(fn, retries=3, backoff=3):
+    """Retry a function on network errors."""
+    for attempt in range(retries):
+        try:
+            return fn()
+        except OSError as e:
+            if attempt == retries - 1:
+                raise
+            wait = backoff * (attempt + 1)
+            print(f"    [RETRY] Network error: {e} — retrying in {wait}s ({attempt+1}/{retries})")
+            time.sleep(wait)
+
+
+def http_post(url: str, body: Optional[dict] = None) -> dict:
+    def _do():
+        data = json.dumps(body or {}).encode()
+        req = urllib.request.Request(
+            url, data=data,
+            headers={"Content-Type": "application/json"},
+            method="POST"
+        )
+        with urllib.request.urlopen(req, timeout=120) as resp:
+            return json.loads(resp.read())
+    return _retry(_do)
+
+
+def http_get(url: str, expect_json: bool = True) -> dict:
+    def _do():
+        with urllib.request.urlopen(url, timeout=120) as resp:
+            body = resp.read()
+            if not expect_json:
+                return {"_status": resp.status, "_body_len": len(body)}
+            return json.loads(body)
+    return _retry(_do)
+
+
+# ---------------------------------------------------------------------------
+# LLM call via OpenAI-compatible API
+# ---------------------------------------------------------------------------
+
+def _call_hf(prompt: str) -> str:
+    """Call LLM via HF router (OpenAI-compatible)."""
+    from openai import OpenAI
+    client = OpenAI(base_url=API_BASE_URL, api_key=HF_TOKEN)
+    resp = client.chat.completions.create(
+        model=MODEL_NAME,
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        temperature=0.3,
+        max_tokens=256,
+    )
+    return (resp.choices[0].message.content or "").strip()
+
+
+def _call_bedrock(prompt: str) -> str:
+    """Call LLM via AWS Bedrock. Tries converse() first, falls back to invoke_model()."""
+    import boto3
+    client = boto3.client(
+        service_name="bedrock-runtime",
+        region_name=os.getenv("AWS_DEFAULT_REGION", "us-east-1"),
+        aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
+        aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
+    )
+    # Try converse API first (boto3 >= 1.34.x)
+    if hasattr(client, "converse"):
+        resp = client.converse(
+            modelId=BEDROCK_MODEL_ID,
+            messages=[{"role": "user", "content": [{"text": prompt}]}],
+            system=[{"text": SYSTEM_PROMPT}],
+            inferenceConfig={"maxTokens": 256, "temperature": 0.3},
+        )
+        return resp["output"]["message"]["content"][0]["text"].strip()
+    # Fallback: invoke_model (works with all boto3 versions)
+    body = json.dumps({
+        "messages": [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        "max_tokens": 256,
+        "temperature": 0.3,
+    })
+    resp = client.invoke_model(
+        modelId=BEDROCK_MODEL_ID,
+        contentType="application/json",
+        accept="application/json",
+        body=body,
+    )
+    result = json.loads(resp["body"].read())
+    # Handle both OpenAI-style and Bedrock-native response formats
+    if "choices" in result:
+        return result["choices"][0]["message"]["content"].strip()
+    if "content" in result:
+        content = result["content"]
+        if isinstance(content, list):
+            return content[0].get("text", "").strip()
+        return str(content).strip()
+    if "output" in result:
+        return result["output"].get("text", "").strip()
+    return str(result).strip()
+
+
+def call_llm(prompt: str) -> str:
+    """Call LLM with retries. Uses HF router or Bedrock based on LLM_BACKEND."""
+    fn = _call_bedrock if LLM_BACKEND == "bedrock" else _call_hf
+    for attempt in range(3):
+        try:
+            raw = fn(prompt)
+            if os.getenv("DEBUG_LLM"):
+                print(f"    [LLM RAW] {raw[:200]}")
+            # Strip Qwen3 <think>...</think> reasoning blocks
+            import re
+            cleaned = re.sub(r"<think>.*?</think>", "", raw, flags=re.DOTALL).strip()
+            return cleaned if cleaned else raw
+        except Exception as e:
+            if attempt == 2:
+                print(f"    [LLM ERROR] {e} (gave up after 3 attempts)")
+                return ""
+            wait = 3 * (attempt + 1)
+            print(f"    [LLM RETRY] {e} — retrying in {wait}s")
+            time.sleep(wait)
+    return ""
+
+
+def format_obs(obs: dict) -> str:
+    """Format observation as text for LLM — shows raw signals prominently."""
+    lines = []
+    lines.append(f"TASK: {obs.get('task','?').upper()} | Steps remaining: {obs.get('steps_remaining','?')}")
+
+    flagged = obs.get("flagged_ids", [])
+    lines.append(f"Flagged ({len(flagged)}/10): {', '.join(flagged) if flagged else 'none'}")
+
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    uninspected_suspects = [s for s in suspects if s not in inspected]
+    if uninspected_suspects:
+        lines.append(f"*** SUSPECTS (uninspected) → INSPECT THESE FIRST: {', '.join(uninspected_suspects)} ***")
+
+    accounts = obs.get("visible_accounts", [])
+    if accounts:
+        # Split: unflagged accounts that should be flagged vs rest
+        unflagged_suspicious = []
+        flagged_accs = []
+        clean_accs = []
+        for a in sorted(accounts, key=lambda x: x.get("fake_risk_score", 0), reverse=True):
+            aid = a.get("account_id", "?")
+            if aid in flagged:
+                flagged_accs.append(a)
+            elif (a.get("shared_ip_count", 0) >= 5 or
+                  (a.get("photo_reuse_score", 0) >= 0.50 and a.get("bio_template_score", 0) >= 0.40)):
+                unflagged_suspicious.append(a)
+            else:
+                clean_accs.append(a)
+
+        if unflagged_suspicious:
+            lines.append(f"\n!!! ACTION NEEDED — FLAG THESE ({len(unflagged_suspicious)} accounts with strong fake signals):")
+            for a in unflagged_suspicious:
+                aid = a.get("account_id", "?")
+                lines.append(f"  → FLAG {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} ip_shared={a.get('shared_ip_count',0)} hub={a.get('hub_legitimacy_score',0):.2f}")
+
+        if flagged_accs:
+            lines.append(f"\nALREADY FLAGGED ({len(flagged_accs)}):")
+            for a in flagged_accs[:5]:
+                lines.append(f"  ✓ {a.get('account_id','?')}")
+
+        if clean_accs:
+            lines.append(f"\nCLEAN ACCOUNTS ({len(clean_accs)}):")
+            for a in clean_accs[:5]:
+                aid = a.get("account_id", "?")
+                hub = a.get("hub_legitimacy_score", 0)
+                hub_mark = " [CELEBRITY]" if hub > 0.70 else ""
+                lines.append(f"  {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} hub={hub:.2f}{hub_mark}")
+
+    visible = obs.get("visible_account_ids", [])
+    uninspected = [i for i in visible if i not in inspected]
+    if uninspected:
+        lines.append(f"\nUninspected IDs ({len(uninspected)}): {', '.join(uninspected[:8])}{'...' if len(uninspected) > 8 else ''}")
+
+    lines.append(f"\nMessage: {obs.get('message', '')}")
+    return "\n".join(lines)
+
+
+def parse_action(llm_text: str, obs: dict) -> dict:
+    """Parse LLM output to action dict."""
+    for line in llm_text.split("\n"):
+        line = line.strip()
+        upper = line.upper()
+        if upper.startswith("INSPECT ") or upper.startswith("FLAG ") or upper.startswith("INVESTIGATE_NETWORK ") or upper.startswith("UNFLAG "):
+            parts = line.split(maxsplit=1)
+            return {"action_type": parts[0].lower(), "account_id": parts[1].lower() if len(parts) > 1 else None}
+        if upper == "SUBMIT":
+            return {"action_type": "submit"}
+
+    # Fallback: inspect first uninspected suspect
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    for s in suspects:
+        if s not in inspected:
+            return {"action_type": "inspect", "account_id": s}
+    visible = obs.get("visible_account_ids", [])
+    for v in visible:
+        if v not in inspected:
+            return {"action_type": "inspect", "account_id": v}
+    return {"action_type": "submit"}
+
+
+# ---------------------------------------------------------------------------
+# Test phases
+# ---------------------------------------------------------------------------
+
+def test_endpoints(base_url: str) -> bool:
+    """Phase 0: Verify all required endpoints respond correctly."""
+    print("\n" + "="*60)
+    print("PHASE 0: Endpoint Verification")
+    print("="*60)
+
+    checks = [
+        ("GET",  "/health",   None,  True),
+        ("GET",  "/tasks",    None,  True),
+        ("GET",  "/metadata", None,  True),
+        ("GET",  "/schema",   None,  True),
+        ("GET",  "/web",      None,  False),   # returns HTML, not JSON
+        ("POST", "/reset",    {"task": "easy", "seed": 0}, True),
+        ("GET",  "/state",    None,  True),
+        ("POST", "/step",     {"action_type": "inspect", "account_id": "acc_0000"}, True),
+        ("POST", "/step",     {"action_type": "submit"}, True),
+        ("GET",  "/grader",   None,  True),
+        ("POST", "/mcp",      {"jsonrpc": "2.0", "method": "tools/list", "id": 1}, True),
+        ("POST", "/baseline", None,  True),
+    ]
+
+    all_ok = True
+    for method, path, body, expect_json in checks:
+        try:
+            if method == "GET":
+                http_get(f"{base_url}{path}", expect_json=expect_json)
+            else:
+                http_post(f"{base_url}{path}", body)
+            print(f"  ✓ {method} {path}")
+        except Exception as e:
+            print(f"  ✗ {method} {path} — {e}")
+            all_ok = False
+
+    return all_ok
+
+
+def test_baseline_stability(base_url: str) -> bool:
+    """Phase 1: Baseline re-run (must produce identical scores)."""
+    print("\n" + "="*60)
+    print("PHASE 1: Baseline Stability (3 runs)")
+    print("="*60)
+
+    scores_list = []
+    for i in range(3):
+        r = http_post(f"{base_url}/baseline")
+        scores = r["scores"]
+        scores_list.append(scores)
+        print(f"  Run {i+1}: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+
+    # Check all identical
+    stable = all(s == scores_list[0] for s in scores_list)
+    if stable:
+        print("  ✓ All 3 runs identical — baseline is deterministic")
+    else:
+        print("  ✗ SCORES DIFFER — baseline is non-deterministic!")
+    return stable
+
+
+def test_llm_agent(base_url: str, task: str, seed: int = 0) -> float:
+    """Phase 2: Run an LLM agent against one task (simulates judge's Nemotron run)."""
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"\n  --- LLM Agent: task={task}, seed={seed}, model={_model} ---")
+
+    # Reset
+    reset_resp = http_post(f"{base_url}/reset", {"task": task, "seed": seed})
+    obs = reset_resp.get("observation", reset_resp)
+    done = reset_resp.get("done", False)
+
+    step_num = 0
+    while not done:
+        step_num += 1
+        prompt = format_obs(obs)
+        llm_text = call_llm(prompt)
+        action = parse_action(llm_text, obs)
+
+        action_str = f"{action['action_type'].upper()} {action.get('account_id', '')}".strip()
+
+        step_resp = http_post(f"{base_url}/step", action)
+        obs = step_resp.get("observation", step_resp)
+        done = step_resp.get("done", False)
+        reward = step_resp.get("reward")
+
+        flagged_n = len(obs.get("flagged_ids", []))
+        suspects_n = len(obs.get("suspect_ids", []))
+        steps_left = obs.get("steps_remaining", "?")
+
+        print(f"    Step {step_num:2d}: {action_str:35s} flagged={flagged_n}/10  suspects={suspects_n}  steps_left={steps_left}")
+
+        if done and reward is not None:
+            msg = step_resp.get("message", obs.get("message", ""))
+            print(f"    → Episode ended: {msg[:100]}")
+
+    # Get grader score
+    grader = http_get(f"{base_url}/grader")
+    score = grader["score"]
+    print(f"    ★ GRADER SCORE: {score:.4f}")
+    return score
+
+
+def test_llm_all_tasks(base_url: str) -> Dict[str, float]:
+    """Phase 2: Run LLM agent on all 3 tasks."""
+    print("\n" + "="*60)
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"PHASE 2: LLM Agent Evaluation (model={_model})")
+    print("="*60)
+
+    scores = {}
+    for task in ["easy", "medium", "hard"]:
+        scores[task] = test_llm_agent(base_url, task=task, seed=0)
+
+    print(f"\n  Summary: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+    return scores
+
+
+def test_variance(base_url: str, seeds: List[int] = [0, 1, 2, 3, 4]) -> None:
+    """Phase 3: Score variance check (multiple seeds per task)."""
+    print("\n" + "="*60)
+    print(f"PHASE 3: Score Variance (seeds={seeds})")
+    print("="*60)
+
+    for task in ["easy", "medium", "hard"]:
+        task_scores = []
+        for seed in seeds:
+            score = test_llm_agent(base_url, task=task, seed=seed)
+            task_scores.append(score)
+
+        mean = sum(task_scores) / len(task_scores)
+        variance = sum((s - mean) ** 2 for s in task_scores) / len(task_scores)
+        print(f"\n  {task}: scores={[f'{s:.3f}' for s in task_scores]}  mean={mean:.4f}  var={variance:.6f}")
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Judge Evaluation Simulator for GraphStrike")
+    parser.add_argument("--url", required=True, help="Environment server URL")
+    parser.add_argument("--bedrock", action="store_true", help="Use AWS Bedrock instead of HF router")
+    parser.add_argument("--endpoints-only", action="store_true", help="Only test endpoints (no LLM)")
+    parser.add_argument("--skip-variance", action="store_true", help="Skip variance check (faster)")
+    parser.add_argument("--seeds", type=int, default=3, help="Number of seeds for variance check")
+    args = parser.parse_args()
+
+    if args.bedrock:
+        LLM_BACKEND = "bedrock"
+
+    base = args.url.rstrip("/")
+    model_display = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"GraphStrike Judge Evaluation Simulator")
+    print(f"Target:  {base}")
+    print(f"Backend: {LLM_BACKEND}")
+    print(f"Model:   {model_display}")
+    print(f"Token:   {'set' if (HF_TOKEN or os.getenv('AWS_ACCESS_KEY_ID')) else 'NOT SET'}")
+
+    # Phase 0: Endpoints
+    if not test_endpoints(base):
+        print("\n✗ Endpoint check failed. Fix before proceeding.")
+        sys.exit(1)
+
+    # Phase 1: Baseline stability
+    test_baseline_stability(base)
+
+    if args.endpoints_only:
+        print("\n✓ Endpoint-only mode — skipping LLM tests.")
+        sys.exit(0)
+
+    if LLM_BACKEND == "bedrock":
+        if not os.getenv("AWS_ACCESS_KEY_ID"):
+            print("\n✗ AWS_ACCESS_KEY_ID not set. Cannot run Bedrock LLM tests.")
+            sys.exit(1)
+    elif not HF_TOKEN:
+        print("\n✗ HF_TOKEN not set. Cannot run LLM agent tests.")
+        print("  export HF_TOKEN='hf_...'  OR  use --bedrock with AWS creds")
+        sys.exit(1)
+
+    # Phase 2: LLM on all tasks
+    scores = test_llm_all_tasks(base)
+
+    # Phase 3: Variance
+    if not args.skip_variance:
+        test_variance(base, seeds=list(range(args.seeds)))
+
+    print("\n" + "="*60)
+    print("EVALUATION COMPLETE")
+    print("="*60)

eval-models/llama_test_judge_eval.py

@@ -0,0 +1,478 @@
+#!/usr/bin/env python3
+"""
+Judge Evaluation Simulator
+==========================
+
+Simulates EXACTLY how hackathon judges will evaluate your environment:
+
+1. Baseline re-run: POST /baseline → verify scores are stable
+2. Standard Open LLM agent: Run an LLM (via HF router) against all 3 tasks
+3. Score variance check: Run same task multiple seeds, check variance
+
+USAGE:
+    # Against live HF Space (requires HF_TOKEN):
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Against local server:
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url http://localhost:7860
+
+    # Choose model (default: Qwen/Qwen2.5-72B-Instruct):
+    export MODEL_NAME="nvidia/Llama-3.1-Nemotron-70B-Instruct-HF"
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Just test endpoints (no LLM needed):
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space --endpoints-only
+"""
+
+import json
+import os
+import sys
+import time
+import urllib.request
+from typing import Dict, List, Optional
+
+# ---------------------------------------------------------------------------
+# Config from env vars (same as judges will set)
+# ---------------------------------------------------------------------------
+
+HF_TOKEN = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+MODEL_NAME = os.getenv("MODEL_NAME", "meta.llama4-scout-17b-instruct-v1:0")
+
+# AWS Bedrock config (use --bedrock flag)
+LLM_BACKEND = "hf"  # "hf" or "bedrock"
+BEDROCK_MODEL_ID = os.getenv("BEDROCK_MODEL_ID", "meta.llama4-scout-17b-instruct-v1:0")
+
+SYSTEM_PROMPT = """You are an AI detective finding 10 coordinated fake accounts in a social network.
+
+ACTIONS (reply with exactly ONE line):
+- INSPECT acc_XXXX  — reveal profile (costs 1 step)
+- FLAG acc_XXXX     — mark as fake (FREE, no step cost, triggers suspect cascade)
+- SUBMIT            — end episode, get scored
+
+DECISION RULES (apply top-to-bottom, first match wins):
+1. If suspect_ids lists accounts you haven't inspected → INSPECT the first one
+2. If ANY profiled account has shared_ip_count >= 5 and is NOT flagged → FLAG it immediately
+3. If ANY profiled account has photo_reuse >= 0.50 AND bio_template >= 0.40 and hub < 0.70 and NOT flagged → FLAG it
+4. If ANY profiled account has fake_risk_score >= 0.30 and hub < 0.70 and NOT flagged → FLAG it
+5. If there are uninspected visible accounts and steps > 3 → INSPECT the next one
+6. If you have 10 flags OR steps <= 3 → SUBMIT
+
+IMPORTANT:
+- FLAG is FREE (costs 0 steps) — flag aggressively when you see suspicious signals
+- After each FLAG, new suspects appear — always inspect suspects before other accounts
+- hub_legitimacy_score > 0.70 means celebrity — do NOT flag
+- shared_ip_count >= 5 is the strongest gang signal (all 10 share one IP)
+- Do NOT re-inspect already inspected accounts
+
+Reply with EXACTLY one line, nothing else:
+FLAG acc_XXXX
+INSPECT acc_XXXX
+SUBMIT"""
+
+
+# ---------------------------------------------------------------------------
+# HTTP helpers
+# ---------------------------------------------------------------------------
+
+def _retry(fn, retries=3, backoff=3):
+    """Retry a function on network errors."""
+    for attempt in range(retries):
+        try:
+            return fn()
+        except OSError as e:
+            if attempt == retries - 1:
+                raise
+            wait = backoff * (attempt + 1)
+            print(f"    [RETRY] Network error: {e} — retrying in {wait}s ({attempt+1}/{retries})")
+            time.sleep(wait)
+
+
+def http_post(url: str, body: Optional[dict] = None) -> dict:
+    def _do():
+        data = json.dumps(body or {}).encode()
+        req = urllib.request.Request(
+            url, data=data,
+            headers={"Content-Type": "application/json"},
+            method="POST"
+        )
+        with urllib.request.urlopen(req, timeout=120) as resp:
+            return json.loads(resp.read())
+    return _retry(_do)
+
+
+def http_get(url: str, expect_json: bool = True) -> dict:
+    def _do():
+        with urllib.request.urlopen(url, timeout=120) as resp:
+            body = resp.read()
+            if not expect_json:
+                return {"_status": resp.status, "_body_len": len(body)}
+            return json.loads(body)
+    return _retry(_do)
+
+
+# ---------------------------------------------------------------------------
+# LLM call via OpenAI-compatible API
+# ---------------------------------------------------------------------------
+
+def _call_hf(prompt: str) -> str:
+    """Call LLM via HF router (OpenAI-compatible)."""
+    from openai import OpenAI
+    client = OpenAI(base_url=API_BASE_URL, api_key=HF_TOKEN)
+    resp = client.chat.completions.create(
+        model=MODEL_NAME,
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        temperature=0.3,
+        max_tokens=256,
+    )
+    return (resp.choices[0].message.content or "").strip()
+
+
+def _call_bedrock(prompt: str) -> str:
+    """Call LLM via AWS Bedrock. Tries converse() first, falls back to invoke_model()."""
+    import boto3
+    client = boto3.client(
+        service_name="bedrock-runtime",
+        region_name=os.getenv("AWS_DEFAULT_REGION", "us-east-1"),
+        aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
+        aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
+    )
+    # Try converse API first (boto3 >= 1.34.x)
+    if hasattr(client, "converse"):
+        resp = client.converse(
+            modelId=BEDROCK_MODEL_ID,
+            messages=[{"role": "user", "content": [{"text": prompt}]}],
+            system=[{"text": SYSTEM_PROMPT}],
+            inferenceConfig={"maxTokens": 256, "temperature": 0.3},
+        )
+        return resp["output"]["message"]["content"][0]["text"].strip()
+    # Fallback: invoke_model (works with all boto3 versions)
+    body = json.dumps({
+        "messages": [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        "max_tokens": 256,
+        "temperature": 0.3,
+    })
+    resp = client.invoke_model(
+        modelId=BEDROCK_MODEL_ID,
+        contentType="application/json",
+        accept="application/json",
+        body=body,
+    )
+    result = json.loads(resp["body"].read())
+    # Handle both OpenAI-style and Bedrock-native response formats
+    if "choices" in result:
+        return result["choices"][0]["message"]["content"].strip()
+    if "content" in result:
+        content = result["content"]
+        if isinstance(content, list):
+            return content[0].get("text", "").strip()
+        return str(content).strip()
+    if "output" in result:
+        return result["output"].get("text", "").strip()
+    return str(result).strip()
+
+
+def call_llm(prompt: str) -> str:
+    """Call LLM with retries. Uses HF router or Bedrock based on LLM_BACKEND."""
+    fn = _call_bedrock if LLM_BACKEND == "bedrock" else _call_hf
+    for attempt in range(3):
+        try:
+            raw = fn(prompt)
+            if os.getenv("DEBUG_LLM"):
+                print(f"    [LLM RAW] {raw[:200]}")
+            # Strip Qwen3 <think>...</think> reasoning blocks
+            import re
+            cleaned = re.sub(r"<think>.*?</think>", "", raw, flags=re.DOTALL).strip()
+            return cleaned if cleaned else raw
+        except Exception as e:
+            if attempt == 2:
+                print(f"    [LLM ERROR] {e} (gave up after 3 attempts)")
+                return ""
+            wait = 3 * (attempt + 1)
+            print(f"    [LLM RETRY] {e} — retrying in {wait}s")
+            time.sleep(wait)
+    return ""
+
+
+def format_obs(obs: dict) -> str:
+    """Format observation as text for LLM — shows raw signals prominently."""
+    lines = []
+    lines.append(f"TASK: {obs.get('task','?').upper()} | Steps remaining: {obs.get('steps_remaining','?')}")
+
+    flagged = obs.get("flagged_ids", [])
+    lines.append(f"Flagged ({len(flagged)}/10): {', '.join(flagged) if flagged else 'none'}")
+
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    uninspected_suspects = [s for s in suspects if s not in inspected]
+    if uninspected_suspects:
+        lines.append(f"*** SUSPECTS (uninspected) → INSPECT THESE FIRST: {', '.join(uninspected_suspects)} ***")
+
+    accounts = obs.get("visible_accounts", [])
+    if accounts:
+        # Split: unflagged accounts that should be flagged vs rest
+        unflagged_suspicious = []
+        flagged_accs = []
+        clean_accs = []
+        for a in sorted(accounts, key=lambda x: x.get("fake_risk_score", 0), reverse=True):
+            aid = a.get("account_id", "?")
+            if aid in flagged:
+                flagged_accs.append(a)
+            elif (a.get("shared_ip_count", 0) >= 5 or
+                  (a.get("photo_reuse_score", 0) >= 0.50 and a.get("bio_template_score", 0) >= 0.40)):
+                unflagged_suspicious.append(a)
+            else:
+                clean_accs.append(a)
+
+        if unflagged_suspicious:
+            lines.append(f"\n!!! ACTION NEEDED — FLAG THESE ({len(unflagged_suspicious)} accounts with strong fake signals):")
+            for a in unflagged_suspicious:
+                aid = a.get("account_id", "?")
+                lines.append(f"  → FLAG {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} ip_shared={a.get('shared_ip_count',0)} hub={a.get('hub_legitimacy_score',0):.2f}")
+
+        if flagged_accs:
+            lines.append(f"\nALREADY FLAGGED ({len(flagged_accs)}):")
+            for a in flagged_accs[:5]:
+                lines.append(f"  ✓ {a.get('account_id','?')}")
+
+        if clean_accs:
+            lines.append(f"\nCLEAN ACCOUNTS ({len(clean_accs)}):")
+            for a in clean_accs[:5]:
+                aid = a.get("account_id", "?")
+                hub = a.get("hub_legitimacy_score", 0)
+                hub_mark = " [CELEBRITY]" if hub > 0.70 else ""
+                lines.append(f"  {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} hub={hub:.2f}{hub_mark}")
+
+    visible = obs.get("visible_account_ids", [])
+    uninspected = [i for i in visible if i not in inspected]
+    if uninspected:
+        lines.append(f"\nUninspected IDs ({len(uninspected)}): {', '.join(uninspected[:8])}{'...' if len(uninspected) > 8 else ''}")
+
+    lines.append(f"\nMessage: {obs.get('message', '')}")
+    return "\n".join(lines)
+
+
+def parse_action(llm_text: str, obs: dict) -> dict:
+    """Parse LLM output to action dict."""
+    for line in llm_text.split("\n"):
+        line = line.strip()
+        upper = line.upper()
+        if upper.startswith("INSPECT ") or upper.startswith("FLAG ") or upper.startswith("INVESTIGATE_NETWORK ") or upper.startswith("UNFLAG "):
+            parts = line.split(maxsplit=1)
+            return {"action_type": parts[0].lower(), "account_id": parts[1].lower() if len(parts) > 1 else None}
+        if upper == "SUBMIT":
+            return {"action_type": "submit"}
+
+    # Fallback: inspect first uninspected suspect
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    for s in suspects:
+        if s not in inspected:
+            return {"action_type": "inspect", "account_id": s}
+    visible = obs.get("visible_account_ids", [])
+    for v in visible:
+        if v not in inspected:
+            return {"action_type": "inspect", "account_id": v}
+    return {"action_type": "submit"}
+
+
+# ---------------------------------------------------------------------------
+# Test phases
+# ---------------------------------------------------------------------------
+
+def test_endpoints(base_url: str) -> bool:
+    """Phase 0: Verify all required endpoints respond correctly."""
+    print("\n" + "="*60)
+    print("PHASE 0: Endpoint Verification")
+    print("="*60)
+
+    checks = [
+        ("GET",  "/health",   None,  True),
+        ("GET",  "/tasks",    None,  True),
+        ("GET",  "/metadata", None,  True),
+        ("GET",  "/schema",   None,  True),
+        ("GET",  "/web",      None,  False),   # returns HTML, not JSON
+        ("POST", "/reset",    {"task": "easy", "seed": 0}, True),
+        ("GET",  "/state",    None,  True),
+        ("POST", "/step",     {"action_type": "inspect", "account_id": "acc_0000"}, True),
+        ("POST", "/step",     {"action_type": "submit"}, True),
+        ("GET",  "/grader",   None,  True),
+        ("POST", "/mcp",      {"jsonrpc": "2.0", "method": "tools/list", "id": 1}, True),
+        ("POST", "/baseline", None,  True),
+    ]
+
+    all_ok = True
+    for method, path, body, expect_json in checks:
+        try:
+            if method == "GET":
+                http_get(f"{base_url}{path}", expect_json=expect_json)
+            else:
+                http_post(f"{base_url}{path}", body)
+            print(f"  ✓ {method} {path}")
+        except Exception as e:
+            print(f"  ✗ {method} {path} — {e}")
+            all_ok = False
+
+    return all_ok
+
+
+def test_baseline_stability(base_url: str) -> bool:
+    """Phase 1: Baseline re-run (must produce identical scores)."""
+    print("\n" + "="*60)
+    print("PHASE 1: Baseline Stability (3 runs)")
+    print("="*60)
+
+    scores_list = []
+    for i in range(3):
+        r = http_post(f"{base_url}/baseline")
+        scores = r["scores"]
+        scores_list.append(scores)
+        print(f"  Run {i+1}: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+
+    # Check all identical
+    stable = all(s == scores_list[0] for s in scores_list)
+    if stable:
+        print("  ✓ All 3 runs identical — baseline is deterministic")
+    else:
+        print("  ✗ SCORES DIFFER — baseline is non-deterministic!")
+    return stable
+
+
+def test_llm_agent(base_url: str, task: str, seed: int = 0) -> float:
+    """Phase 2: Run an LLM agent against one task (simulates judge's Nemotron run)."""
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"\n  --- LLM Agent: task={task}, seed={seed}, model={_model} ---")
+
+    # Reset
+    reset_resp = http_post(f"{base_url}/reset", {"task": task, "seed": seed})
+    obs = reset_resp.get("observation", reset_resp)
+    done = reset_resp.get("done", False)
+
+    step_num = 0
+    while not done:
+        step_num += 1
+        prompt = format_obs(obs)
+        llm_text = call_llm(prompt)
+        action = parse_action(llm_text, obs)
+
+        action_str = f"{action['action_type'].upper()} {action.get('account_id', '')}".strip()
+
+        step_resp = http_post(f"{base_url}/step", action)
+        obs = step_resp.get("observation", step_resp)
+        done = step_resp.get("done", False)
+        reward = step_resp.get("reward")
+
+        flagged_n = len(obs.get("flagged_ids", []))
+        suspects_n = len(obs.get("suspect_ids", []))
+        steps_left = obs.get("steps_remaining", "?")
+
+        print(f"    Step {step_num:2d}: {action_str:35s} flagged={flagged_n}/10  suspects={suspects_n}  steps_left={steps_left}")
+
+        if done and reward is not None:
+            msg = step_resp.get("message", obs.get("message", ""))
+            print(f"    → Episode ended: {msg[:100]}")
+
+    # Get grader score
+    grader = http_get(f"{base_url}/grader")
+    score = grader["score"]
+    print(f"    ★ GRADER SCORE: {score:.4f}")
+    return score
+
+
+def test_llm_all_tasks(base_url: str) -> Dict[str, float]:
+    """Phase 2: Run LLM agent on all 3 tasks."""
+    print("\n" + "="*60)
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"PHASE 2: LLM Agent Evaluation (model={_model})")
+    print("="*60)
+
+    scores = {}
+    for task in ["easy", "medium", "hard"]:
+        scores[task] = test_llm_agent(base_url, task=task, seed=0)
+
+    print(f"\n  Summary: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+    return scores
+
+
+def test_variance(base_url: str, seeds: List[int] = [0, 1, 2, 3, 4]) -> None:
+    """Phase 3: Score variance check (multiple seeds per task)."""
+    print("\n" + "="*60)
+    print(f"PHASE 3: Score Variance (seeds={seeds})")
+    print("="*60)
+
+    for task in ["easy", "medium", "hard"]:
+        task_scores = []
+        for seed in seeds:
+            score = test_llm_agent(base_url, task=task, seed=seed)
+            task_scores.append(score)
+
+        mean = sum(task_scores) / len(task_scores)
+        variance = sum((s - mean) ** 2 for s in task_scores) / len(task_scores)
+        print(f"\n  {task}: scores={[f'{s:.3f}' for s in task_scores]}  mean={mean:.4f}  var={variance:.6f}")
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Judge Evaluation Simulator for GraphStrike")
+    parser.add_argument("--url", required=True, help="Environment server URL")
+    parser.add_argument("--bedrock", action="store_true", help="Use AWS Bedrock instead of HF router")
+    parser.add_argument("--endpoints-only", action="store_true", help="Only test endpoints (no LLM)")
+    parser.add_argument("--skip-variance", action="store_true", help="Skip variance check (faster)")
+    parser.add_argument("--seeds", type=int, default=3, help="Number of seeds for variance check")
+    args = parser.parse_args()
+
+    if args.bedrock:
+        LLM_BACKEND = "bedrock"
+
+    base = args.url.rstrip("/")
+    model_display = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"GraphStrike Judge Evaluation Simulator")
+    print(f"Target:  {base}")
+    print(f"Backend: {LLM_BACKEND}")
+    print(f"Model:   {model_display}")
+    print(f"Token:   {'set' if (HF_TOKEN or os.getenv('AWS_ACCESS_KEY_ID')) else 'NOT SET'}")
+
+    # Phase 0: Endpoints
+    if not test_endpoints(base):
+        print("\n✗ Endpoint check failed. Fix before proceeding.")
+        sys.exit(1)
+
+    # Phase 1: Baseline stability
+    test_baseline_stability(base)
+
+    if args.endpoints_only:
+        print("\n✓ Endpoint-only mode — skipping LLM tests.")
+        sys.exit(0)
+
+    if LLM_BACKEND == "bedrock":
+        if not os.getenv("AWS_ACCESS_KEY_ID"):
+            print("\n✗ AWS_ACCESS_KEY_ID not set. Cannot run Bedrock LLM tests.")
+            sys.exit(1)
+    elif not HF_TOKEN:
+        print("\n✗ HF_TOKEN not set. Cannot run LLM agent tests.")
+        print("  export HF_TOKEN='hf_...'  OR  use --bedrock with AWS creds")
+        sys.exit(1)
+
+    # Phase 2: LLM on all tasks
+    scores = test_llm_all_tasks(base)
+
+    # Phase 3: Variance
+    if not args.skip_variance:
+        test_variance(base, seeds=list(range(args.seeds)))
+
+    print("\n" + "="*60)
+    print("EVALUATION COMPLETE")
+    print("="*60)

eval-models/mistral_test_judge_eval.py

@@ -0,0 +1,478 @@
+#!/usr/bin/env python3
+"""
+Judge Evaluation Simulator
+==========================
+
+Simulates EXACTLY how hackathon judges will evaluate your environment:
+
+1. Baseline re-run: POST /baseline → verify scores are stable
+2. Standard Open LLM agent: Run an LLM (via HF router) against all 3 tasks
+3. Score variance check: Run same task multiple seeds, check variance
+
+USAGE:
+    # Against live HF Space (requires HF_TOKEN):
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Against local server:
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url http://localhost:7860
+
+    # Choose model (default: Qwen/Qwen2.5-72B-Instruct):
+    export MODEL_NAME="nvidia/Llama-3.1-Nemotron-70B-Instruct-HF"
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Just test endpoints (no LLM needed):
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space --endpoints-only
+"""
+
+import json
+import os
+import sys
+import time
+import urllib.request
+from typing import Dict, List, Optional
+
+# ---------------------------------------------------------------------------
+# Config from env vars (same as judges will set)
+# ---------------------------------------------------------------------------
+
+HF_TOKEN = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+MODEL_NAME = os.getenv("MODEL_NAME", "mistral.ministral-3-8b-instruct")
+
+# AWS Bedrock config (use --bedrock flag)
+LLM_BACKEND = "hf"  # "hf" or "bedrock"
+BEDROCK_MODEL_ID = os.getenv("BEDROCK_MODEL_ID", "mistral.ministral-3-8b-instruct")
+
+SYSTEM_PROMPT = """You are an AI detective finding 10 coordinated fake accounts in a social network.
+
+ACTIONS (reply with exactly ONE line):
+- INSPECT acc_XXXX  — reveal profile (costs 1 step)
+- FLAG acc_XXXX     — mark as fake (FREE, no step cost, triggers suspect cascade)
+- SUBMIT            — end episode, get scored
+
+DECISION RULES (apply top-to-bottom, first match wins):
+1. If suspect_ids lists accounts you haven't inspected → INSPECT the first one
+2. If ANY profiled account has shared_ip_count >= 5 and is NOT flagged → FLAG it immediately
+3. If ANY profiled account has photo_reuse >= 0.50 AND bio_template >= 0.40 and hub < 0.70 and NOT flagged → FLAG it
+4. If ANY profiled account has fake_risk_score >= 0.30 and hub < 0.70 and NOT flagged → FLAG it
+5. If there are uninspected visible accounts and steps > 3 → INSPECT the next one
+6. If you have 10 flags OR steps <= 3 → SUBMIT
+
+IMPORTANT:
+- FLAG is FREE (costs 0 steps) — flag aggressively when you see suspicious signals
+- After each FLAG, new suspects appear — always inspect suspects before other accounts
+- hub_legitimacy_score > 0.70 means celebrity — do NOT flag
+- shared_ip_count >= 5 is the strongest gang signal (all 10 share one IP)
+- Do NOT re-inspect already inspected accounts
+
+Reply with EXACTLY one line, nothing else:
+FLAG acc_XXXX
+INSPECT acc_XXXX
+SUBMIT"""
+
+
+# ---------------------------------------------------------------------------
+# HTTP helpers
+# ---------------------------------------------------------------------------
+
+def _retry(fn, retries=3, backoff=3):
+    """Retry a function on network errors."""
+    for attempt in range(retries):
+        try:
+            return fn()
+        except OSError as e:
+            if attempt == retries - 1:
+                raise
+            wait = backoff * (attempt + 1)
+            print(f"    [RETRY] Network error: {e} — retrying in {wait}s ({attempt+1}/{retries})")
+            time.sleep(wait)
+
+
+def http_post(url: str, body: Optional[dict] = None) -> dict:
+    def _do():
+        data = json.dumps(body or {}).encode()
+        req = urllib.request.Request(
+            url, data=data,
+            headers={"Content-Type": "application/json"},
+            method="POST"
+        )
+        with urllib.request.urlopen(req, timeout=120) as resp:
+            return json.loads(resp.read())
+    return _retry(_do)
+
+
+def http_get(url: str, expect_json: bool = True) -> dict:
+    def _do():
+        with urllib.request.urlopen(url, timeout=120) as resp:
+            body = resp.read()
+            if not expect_json:
+                return {"_status": resp.status, "_body_len": len(body)}
+            return json.loads(body)
+    return _retry(_do)
+
+
+# ---------------------------------------------------------------------------
+# LLM call via OpenAI-compatible API
+# ---------------------------------------------------------------------------
+
+def _call_hf(prompt: str) -> str:
+    """Call LLM via HF router (OpenAI-compatible)."""
+    from openai import OpenAI
+    client = OpenAI(base_url=API_BASE_URL, api_key=HF_TOKEN)
+    resp = client.chat.completions.create(
+        model=MODEL_NAME,
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        temperature=0.3,
+        max_tokens=256,
+    )
+    return (resp.choices[0].message.content or "").strip()
+
+
+def _call_bedrock(prompt: str) -> str:
+    """Call LLM via AWS Bedrock. Tries converse() first, falls back to invoke_model()."""
+    import boto3
+    client = boto3.client(
+        service_name="bedrock-runtime",
+        region_name=os.getenv("AWS_DEFAULT_REGION", "us-east-1"),
+        aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
+        aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
+    )
+    # Try converse API first (boto3 >= 1.34.x)
+    if hasattr(client, "converse"):
+        resp = client.converse(
+            modelId=BEDROCK_MODEL_ID,
+            messages=[{"role": "user", "content": [{"text": prompt}]}],
+            system=[{"text": SYSTEM_PROMPT}],
+            inferenceConfig={"maxTokens": 256, "temperature": 0.3},
+        )
+        return resp["output"]["message"]["content"][0]["text"].strip()
+    # Fallback: invoke_model (works with all boto3 versions)
+    body = json.dumps({
+        "messages": [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        "max_tokens": 256,
+        "temperature": 0.3,
+    })
+    resp = client.invoke_model(
+        modelId=BEDROCK_MODEL_ID,
+        contentType="application/json",
+        accept="application/json",
+        body=body,
+    )
+    result = json.loads(resp["body"].read())
+    # Handle both OpenAI-style and Bedrock-native response formats
+    if "choices" in result:
+        return result["choices"][0]["message"]["content"].strip()
+    if "content" in result:
+        content = result["content"]
+        if isinstance(content, list):
+            return content[0].get("text", "").strip()
+        return str(content).strip()
+    if "output" in result:
+        return result["output"].get("text", "").strip()
+    return str(result).strip()
+
+
+def call_llm(prompt: str) -> str:
+    """Call LLM with retries. Uses HF router or Bedrock based on LLM_BACKEND."""
+    fn = _call_bedrock if LLM_BACKEND == "bedrock" else _call_hf
+    for attempt in range(3):
+        try:
+            raw = fn(prompt)
+            if os.getenv("DEBUG_LLM"):
+                print(f"    [LLM RAW] {raw[:200]}")
+            # Strip Qwen3 <think>...</think> reasoning blocks
+            import re
+            cleaned = re.sub(r"<think>.*?</think>", "", raw, flags=re.DOTALL).strip()
+            return cleaned if cleaned else raw
+        except Exception as e:
+            if attempt == 2:
+                print(f"    [LLM ERROR] {e} (gave up after 3 attempts)")
+                return ""
+            wait = 3 * (attempt + 1)
+            print(f"    [LLM RETRY] {e} — retrying in {wait}s")
+            time.sleep(wait)
+    return ""
+
+
+def format_obs(obs: dict) -> str:
+    """Format observation as text for LLM — shows raw signals prominently."""
+    lines = []
+    lines.append(f"TASK: {obs.get('task','?').upper()} | Steps remaining: {obs.get('steps_remaining','?')}")
+
+    flagged = obs.get("flagged_ids", [])
+    lines.append(f"Flagged ({len(flagged)}/10): {', '.join(flagged) if flagged else 'none'}")
+
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    uninspected_suspects = [s for s in suspects if s not in inspected]
+    if uninspected_suspects:
+        lines.append(f"*** SUSPECTS (uninspected) → INSPECT THESE FIRST: {', '.join(uninspected_suspects)} ***")
+
+    accounts = obs.get("visible_accounts", [])
+    if accounts:
+        # Split: unflagged accounts that should be flagged vs rest
+        unflagged_suspicious = []
+        flagged_accs = []
+        clean_accs = []
+        for a in sorted(accounts, key=lambda x: x.get("fake_risk_score", 0), reverse=True):
+            aid = a.get("account_id", "?")
+            if aid in flagged:
+                flagged_accs.append(a)
+            elif (a.get("shared_ip_count", 0) >= 5 or
+                  (a.get("photo_reuse_score", 0) >= 0.50 and a.get("bio_template_score", 0) >= 0.40)):
+                unflagged_suspicious.append(a)
+            else:
+                clean_accs.append(a)
+
+        if unflagged_suspicious:
+            lines.append(f"\n!!! ACTION NEEDED — FLAG THESE ({len(unflagged_suspicious)} accounts with strong fake signals):")
+            for a in unflagged_suspicious:
+                aid = a.get("account_id", "?")
+                lines.append(f"  → FLAG {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} ip_shared={a.get('shared_ip_count',0)} hub={a.get('hub_legitimacy_score',0):.2f}")
+
+        if flagged_accs:
+            lines.append(f"\nALREADY FLAGGED ({len(flagged_accs)}):")
+            for a in flagged_accs[:5]:
+                lines.append(f"  ✓ {a.get('account_id','?')}")
+
+        if clean_accs:
+            lines.append(f"\nCLEAN ACCOUNTS ({len(clean_accs)}):")
+            for a in clean_accs[:5]:
+                aid = a.get("account_id", "?")
+                hub = a.get("hub_legitimacy_score", 0)
+                hub_mark = " [CELEBRITY]" if hub > 0.70 else ""
+                lines.append(f"  {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} hub={hub:.2f}{hub_mark}")
+
+    visible = obs.get("visible_account_ids", [])
+    uninspected = [i for i in visible if i not in inspected]
+    if uninspected:
+        lines.append(f"\nUninspected IDs ({len(uninspected)}): {', '.join(uninspected[:8])}{'...' if len(uninspected) > 8 else ''}")
+
+    lines.append(f"\nMessage: {obs.get('message', '')}")
+    return "\n".join(lines)
+
+
+def parse_action(llm_text: str, obs: dict) -> dict:
+    """Parse LLM output to action dict."""
+    for line in llm_text.split("\n"):
+        line = line.strip()
+        upper = line.upper()
+        if upper.startswith("INSPECT ") or upper.startswith("FLAG ") or upper.startswith("INVESTIGATE_NETWORK ") or upper.startswith("UNFLAG "):
+            parts = line.split(maxsplit=1)
+            return {"action_type": parts[0].lower(), "account_id": parts[1].lower() if len(parts) > 1 else None}
+        if upper == "SUBMIT":
+            return {"action_type": "submit"}
+
+    # Fallback: inspect first uninspected suspect
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    for s in suspects:
+        if s not in inspected:
+            return {"action_type": "inspect", "account_id": s}
+    visible = obs.get("visible_account_ids", [])
+    for v in visible:
+        if v not in inspected:
+            return {"action_type": "inspect", "account_id": v}
+    return {"action_type": "submit"}
+
+
+# ---------------------------------------------------------------------------
+# Test phases
+# ---------------------------------------------------------------------------
+
+def test_endpoints(base_url: str) -> bool:
+    """Phase 0: Verify all required endpoints respond correctly."""
+    print("\n" + "="*60)
+    print("PHASE 0: Endpoint Verification")
+    print("="*60)
+
+    checks = [
+        ("GET",  "/health",   None,  True),
+        ("GET",  "/tasks",    None,  True),
+        ("GET",  "/metadata", None,  True),
+        ("GET",  "/schema",   None,  True),
+        ("GET",  "/web",      None,  False),   # returns HTML, not JSON
+        ("POST", "/reset",    {"task": "easy", "seed": 0}, True),
+        ("GET",  "/state",    None,  True),
+        ("POST", "/step",     {"action_type": "inspect", "account_id": "acc_0000"}, True),
+        ("POST", "/step",     {"action_type": "submit"}, True),
+        ("GET",  "/grader",   None,  True),
+        ("POST", "/mcp",      {"jsonrpc": "2.0", "method": "tools/list", "id": 1}, True),
+        ("POST", "/baseline", None,  True),
+    ]
+
+    all_ok = True
+    for method, path, body, expect_json in checks:
+        try:
+            if method == "GET":
+                http_get(f"{base_url}{path}", expect_json=expect_json)
+            else:
+                http_post(f"{base_url}{path}", body)
+            print(f"  ✓ {method} {path}")
+        except Exception as e:
+            print(f"  ✗ {method} {path} — {e}")
+            all_ok = False
+
+    return all_ok
+
+
+def test_baseline_stability(base_url: str) -> bool:
+    """Phase 1: Baseline re-run (must produce identical scores)."""
+    print("\n" + "="*60)
+    print("PHASE 1: Baseline Stability (3 runs)")
+    print("="*60)
+
+    scores_list = []
+    for i in range(3):
+        r = http_post(f"{base_url}/baseline")
+        scores = r["scores"]
+        scores_list.append(scores)
+        print(f"  Run {i+1}: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+
+    # Check all identical
+    stable = all(s == scores_list[0] for s in scores_list)
+    if stable:
+        print("  ✓ All 3 runs identical — baseline is deterministic")
+    else:
+        print("  ✗ SCORES DIFFER — baseline is non-deterministic!")
+    return stable
+
+
+def test_llm_agent(base_url: str, task: str, seed: int = 0) -> float:
+    """Phase 2: Run an LLM agent against one task (simulates judge's Nemotron run)."""
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"\n  --- LLM Agent: task={task}, seed={seed}, model={_model} ---")
+
+    # Reset
+    reset_resp = http_post(f"{base_url}/reset", {"task": task, "seed": seed})
+    obs = reset_resp.get("observation", reset_resp)
+    done = reset_resp.get("done", False)
+
+    step_num = 0
+    while not done:
+        step_num += 1
+        prompt = format_obs(obs)
+        llm_text = call_llm(prompt)
+        action = parse_action(llm_text, obs)
+
+        action_str = f"{action['action_type'].upper()} {action.get('account_id', '')}".strip()
+
+        step_resp = http_post(f"{base_url}/step", action)
+        obs = step_resp.get("observation", step_resp)
+        done = step_resp.get("done", False)
+        reward = step_resp.get("reward")
+
+        flagged_n = len(obs.get("flagged_ids", []))
+        suspects_n = len(obs.get("suspect_ids", []))
+        steps_left = obs.get("steps_remaining", "?")
+
+        print(f"    Step {step_num:2d}: {action_str:35s} flagged={flagged_n}/10  suspects={suspects_n}  steps_left={steps_left}")
+
+        if done and reward is not None:
+            msg = step_resp.get("message", obs.get("message", ""))
+            print(f"    → Episode ended: {msg[:100]}")
+
+    # Get grader score
+    grader = http_get(f"{base_url}/grader")
+    score = grader["score"]
+    print(f"    ★ GRADER SCORE: {score:.4f}")
+    return score
+
+
+def test_llm_all_tasks(base_url: str) -> Dict[str, float]:
+    """Phase 2: Run LLM agent on all 3 tasks."""
+    print("\n" + "="*60)
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"PHASE 2: LLM Agent Evaluation (model={_model})")
+    print("="*60)
+
+    scores = {}
+    for task in ["easy", "medium", "hard"]:
+        scores[task] = test_llm_agent(base_url, task=task, seed=0)
+
+    print(f"\n  Summary: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+    return scores
+
+
+def test_variance(base_url: str, seeds: List[int] = [0, 1, 2, 3, 4]) -> None:
+    """Phase 3: Score variance check (multiple seeds per task)."""
+    print("\n" + "="*60)
+    print(f"PHASE 3: Score Variance (seeds={seeds})")
+    print("="*60)
+
+    for task in ["easy", "medium", "hard"]:
+        task_scores = []
+        for seed in seeds:
+            score = test_llm_agent(base_url, task=task, seed=seed)
+            task_scores.append(score)
+
+        mean = sum(task_scores) / len(task_scores)
+        variance = sum((s - mean) ** 2 for s in task_scores) / len(task_scores)
+        print(f"\n  {task}: scores={[f'{s:.3f}' for s in task_scores]}  mean={mean:.4f}  var={variance:.6f}")
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Judge Evaluation Simulator for GraphStrike")
+    parser.add_argument("--url", required=True, help="Environment server URL")
+    parser.add_argument("--bedrock", action="store_true", help="Use AWS Bedrock instead of HF router")
+    parser.add_argument("--endpoints-only", action="store_true", help="Only test endpoints (no LLM)")
+    parser.add_argument("--skip-variance", action="store_true", help="Skip variance check (faster)")
+    parser.add_argument("--seeds", type=int, default=3, help="Number of seeds for variance check")
+    args = parser.parse_args()
+
+    if args.bedrock:
+        LLM_BACKEND = "bedrock"
+
+    base = args.url.rstrip("/")
+    model_display = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"GraphStrike Judge Evaluation Simulator")
+    print(f"Target:  {base}")
+    print(f"Backend: {LLM_BACKEND}")
+    print(f"Model:   {model_display}")
+    print(f"Token:   {'set' if (HF_TOKEN or os.getenv('AWS_ACCESS_KEY_ID')) else 'NOT SET'}")
+
+    # Phase 0: Endpoints
+    if not test_endpoints(base):
+        print("\n✗ Endpoint check failed. Fix before proceeding.")
+        sys.exit(1)
+
+    # Phase 1: Baseline stability
+    test_baseline_stability(base)
+
+    if args.endpoints_only:
+        print("\n✓ Endpoint-only mode — skipping LLM tests.")
+        sys.exit(0)
+
+    if LLM_BACKEND == "bedrock":
+        if not os.getenv("AWS_ACCESS_KEY_ID"):
+            print("\n✗ AWS_ACCESS_KEY_ID not set. Cannot run Bedrock LLM tests.")
+            sys.exit(1)
+    elif not HF_TOKEN:
+        print("\n✗ HF_TOKEN not set. Cannot run LLM agent tests.")
+        print("  export HF_TOKEN='hf_...'  OR  use --bedrock with AWS creds")
+        sys.exit(1)
+
+    # Phase 2: LLM on all tasks
+    scores = test_llm_all_tasks(base)
+
+    # Phase 3: Variance
+    if not args.skip_variance:
+        test_variance(base, seeds=list(range(args.seeds)))
+
+    print("\n" + "="*60)
+    print("EVALUATION COMPLETE")
+    print("="*60)

eval-models/nvidia_test_judge_eval.py

@@ -0,0 +1,478 @@
+#!/usr/bin/env python3
+"""
+Judge Evaluation Simulator
+==========================
+
+Simulates EXACTLY how hackathon judges will evaluate your environment:
+
+1. Baseline re-run: POST /baseline → verify scores are stable
+2. Standard Open LLM agent: Run an LLM (via HF router) against all 3 tasks
+3. Score variance check: Run same task multiple seeds, check variance
+
+USAGE:
+    # Against live HF Space (requires HF_TOKEN):
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Against local server:
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url http://localhost:7860
+
+    # Choose model (default: Qwen/Qwen2.5-72B-Instruct):
+    export MODEL_NAME="nvidia/Llama-3.1-Nemotron-70B-Instruct-HF"
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Just test endpoints (no LLM needed):
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space --endpoints-only
+"""
+
+import json
+import os
+import sys
+import time
+import urllib.request
+from typing import Dict, List, Optional
+
+# ---------------------------------------------------------------------------
+# Config from env vars (same as judges will set)
+# ---------------------------------------------------------------------------
+
+HF_TOKEN = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+MODEL_NAME = os.getenv("MODEL_NAME", "nvidia.nemotron-super-3-120b")
+
+# AWS Bedrock config (use --bedrock flag)
+LLM_BACKEND = "hf"  # "hf" or "bedrock"
+BEDROCK_MODEL_ID = os.getenv("BEDROCK_MODEL_ID", "nvidia.nemotron-super-3-120b")
+
+SYSTEM_PROMPT = """You are an AI detective finding 10 coordinated fake accounts in a social network.
+
+ACTIONS (reply with exactly ONE line):
+- INSPECT acc_XXXX  — reveal profile (costs 1 step)
+- FLAG acc_XXXX     — mark as fake (FREE, no step cost, triggers suspect cascade)
+- SUBMIT            — end episode, get scored
+
+DECISION RULES (apply top-to-bottom, first match wins):
+1. If suspect_ids lists accounts you haven't inspected → INSPECT the first one
+2. If ANY profiled account has shared_ip_count >= 5 and is NOT flagged → FLAG it immediately
+3. If ANY profiled account has photo_reuse >= 0.50 AND bio_template >= 0.40 and hub < 0.70 and NOT flagged → FLAG it
+4. If ANY profiled account has fake_risk_score >= 0.30 and hub < 0.70 and NOT flagged → FLAG it
+5. If there are uninspected visible accounts and steps > 3 → INSPECT the next one
+6. If you have 10 flags OR steps <= 3 → SUBMIT
+
+IMPORTANT:
+- FLAG is FREE (costs 0 steps) — flag aggressively when you see suspicious signals
+- After each FLAG, new suspects appear — always inspect suspects before other accounts
+- hub_legitimacy_score > 0.70 means celebrity — do NOT flag
+- shared_ip_count >= 5 is the strongest gang signal (all 10 share one IP)
+- Do NOT re-inspect already inspected accounts
+
+Reply with EXACTLY one line, nothing else:
+FLAG acc_XXXX
+INSPECT acc_XXXX
+SUBMIT"""
+
+
+# ---------------------------------------------------------------------------
+# HTTP helpers
+# ---------------------------------------------------------------------------
+
+def _retry(fn, retries=3, backoff=3):
+    """Retry a function on network errors."""
+    for attempt in range(retries):
+        try:
+            return fn()
+        except OSError as e:
+            if attempt == retries - 1:
+                raise
+            wait = backoff * (attempt + 1)
+            print(f"    [RETRY] Network error: {e} — retrying in {wait}s ({attempt+1}/{retries})")
+            time.sleep(wait)
+
+
+def http_post(url: str, body: Optional[dict] = None) -> dict:
+    def _do():
+        data = json.dumps(body or {}).encode()
+        req = urllib.request.Request(
+            url, data=data,
+            headers={"Content-Type": "application/json"},
+            method="POST"
+        )
+        with urllib.request.urlopen(req, timeout=120) as resp:
+            return json.loads(resp.read())
+    return _retry(_do)
+
+
+def http_get(url: str, expect_json: bool = True) -> dict:
+    def _do():
+        with urllib.request.urlopen(url, timeout=120) as resp:
+            body = resp.read()
+            if not expect_json:
+                return {"_status": resp.status, "_body_len": len(body)}
+            return json.loads(body)
+    return _retry(_do)
+
+
+# ---------------------------------------------------------------------------
+# LLM call via OpenAI-compatible API
+# ---------------------------------------------------------------------------
+
+def _call_hf(prompt: str) -> str:
+    """Call LLM via HF router (OpenAI-compatible)."""
+    from openai import OpenAI
+    client = OpenAI(base_url=API_BASE_URL, api_key=HF_TOKEN)
+    resp = client.chat.completions.create(
+        model=MODEL_NAME,
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        temperature=0.3,
+        max_tokens=256,
+    )
+    return (resp.choices[0].message.content or "").strip()
+
+
+def _call_bedrock(prompt: str) -> str:
+    """Call LLM via AWS Bedrock. Tries converse() first, falls back to invoke_model()."""
+    import boto3
+    client = boto3.client(
+        service_name="bedrock-runtime",
+        region_name=os.getenv("AWS_DEFAULT_REGION", "us-east-1"),
+        aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
+        aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
+    )
+    # Try converse API first (boto3 >= 1.34.x)
+    if hasattr(client, "converse"):
+        resp = client.converse(
+            modelId=BEDROCK_MODEL_ID,
+            messages=[{"role": "user", "content": [{"text": prompt}]}],
+            system=[{"text": SYSTEM_PROMPT}],
+            inferenceConfig={"maxTokens": 256, "temperature": 0.3},
+        )
+        return resp["output"]["message"]["content"][0]["text"].strip()
+    # Fallback: invoke_model (works with all boto3 versions)
+    body = json.dumps({
+        "messages": [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        "max_tokens": 256,
+        "temperature": 0.3,
+    })
+    resp = client.invoke_model(
+        modelId=BEDROCK_MODEL_ID,
+        contentType="application/json",
+        accept="application/json",
+        body=body,
+    )
+    result = json.loads(resp["body"].read())
+    # Handle both OpenAI-style and Bedrock-native response formats
+    if "choices" in result:
+        return result["choices"][0]["message"]["content"].strip()
+    if "content" in result:
+        content = result["content"]
+        if isinstance(content, list):
+            return content[0].get("text", "").strip()
+        return str(content).strip()
+    if "output" in result:
+        return result["output"].get("text", "").strip()
+    return str(result).strip()
+
+
+def call_llm(prompt: str) -> str:
+    """Call LLM with retries. Uses HF router or Bedrock based on LLM_BACKEND."""
+    fn = _call_bedrock if LLM_BACKEND == "bedrock" else _call_hf
+    for attempt in range(3):
+        try:
+            raw = fn(prompt)
+            if os.getenv("DEBUG_LLM"):
+                print(f"    [LLM RAW] {raw[:200]}")
+            # Strip Qwen3 <think>...</think> reasoning blocks
+            import re
+            cleaned = re.sub(r"<think>.*?</think>", "", raw, flags=re.DOTALL).strip()
+            return cleaned if cleaned else raw
+        except Exception as e:
+            if attempt == 2:
+                print(f"    [LLM ERROR] {e} (gave up after 3 attempts)")
+                return ""
+            wait = 3 * (attempt + 1)
+            print(f"    [LLM RETRY] {e} — retrying in {wait}s")
+            time.sleep(wait)
+    return ""
+
+
+def format_obs(obs: dict) -> str:
+    """Format observation as text for LLM — shows raw signals prominently."""
+    lines = []
+    lines.append(f"TASK: {obs.get('task','?').upper()} | Steps remaining: {obs.get('steps_remaining','?')}")
+
+    flagged = obs.get("flagged_ids", [])
+    lines.append(f"Flagged ({len(flagged)}/10): {', '.join(flagged) if flagged else 'none'}")
+
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    uninspected_suspects = [s for s in suspects if s not in inspected]
+    if uninspected_suspects:
+        lines.append(f"*** SUSPECTS (uninspected) → INSPECT THESE FIRST: {', '.join(uninspected_suspects)} ***")
+
+    accounts = obs.get("visible_accounts", [])
+    if accounts:
+        # Split: unflagged accounts that should be flagged vs rest
+        unflagged_suspicious = []
+        flagged_accs = []
+        clean_accs = []
+        for a in sorted(accounts, key=lambda x: x.get("fake_risk_score", 0), reverse=True):
+            aid = a.get("account_id", "?")
+            if aid in flagged:
+                flagged_accs.append(a)
+            elif (a.get("shared_ip_count", 0) >= 5 or
+                  (a.get("photo_reuse_score", 0) >= 0.50 and a.get("bio_template_score", 0) >= 0.40)):
+                unflagged_suspicious.append(a)
+            else:
+                clean_accs.append(a)
+
+        if unflagged_suspicious:
+            lines.append(f"\n!!! ACTION NEEDED — FLAG THESE ({len(unflagged_suspicious)} accounts with strong fake signals):")
+            for a in unflagged_suspicious:
+                aid = a.get("account_id", "?")
+                lines.append(f"  → FLAG {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} ip_shared={a.get('shared_ip_count',0)} hub={a.get('hub_legitimacy_score',0):.2f}")
+
+        if flagged_accs:
+            lines.append(f"\nALREADY FLAGGED ({len(flagged_accs)}):")
+            for a in flagged_accs[:5]:
+                lines.append(f"  ✓ {a.get('account_id','?')}")
+
+        if clean_accs:
+            lines.append(f"\nCLEAN ACCOUNTS ({len(clean_accs)}):")
+            for a in clean_accs[:5]:
+                aid = a.get("account_id", "?")
+                hub = a.get("hub_legitimacy_score", 0)
+                hub_mark = " [CELEBRITY]" if hub > 0.70 else ""
+                lines.append(f"  {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} hub={hub:.2f}{hub_mark}")
+
+    visible = obs.get("visible_account_ids", [])
+    uninspected = [i for i in visible if i not in inspected]
+    if uninspected:
+        lines.append(f"\nUninspected IDs ({len(uninspected)}): {', '.join(uninspected[:8])}{'...' if len(uninspected) > 8 else ''}")
+
+    lines.append(f"\nMessage: {obs.get('message', '')}")
+    return "\n".join(lines)
+
+
+def parse_action(llm_text: str, obs: dict) -> dict:
+    """Parse LLM output to action dict."""
+    for line in llm_text.split("\n"):
+        line = line.strip()
+        upper = line.upper()
+        if upper.startswith("INSPECT ") or upper.startswith("FLAG ") or upper.startswith("INVESTIGATE_NETWORK ") or upper.startswith("UNFLAG "):
+            parts = line.split(maxsplit=1)
+            return {"action_type": parts[0].lower(), "account_id": parts[1].lower() if len(parts) > 1 else None}
+        if upper == "SUBMIT":
+            return {"action_type": "submit"}
+
+    # Fallback: inspect first uninspected suspect
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    for s in suspects:
+        if s not in inspected:
+            return {"action_type": "inspect", "account_id": s}
+    visible = obs.get("visible_account_ids", [])
+    for v in visible:
+        if v not in inspected:
+            return {"action_type": "inspect", "account_id": v}
+    return {"action_type": "submit"}
+
+
+# ---------------------------------------------------------------------------
+# Test phases
+# ---------------------------------------------------------------------------
+
+def test_endpoints(base_url: str) -> bool:
+    """Phase 0: Verify all required endpoints respond correctly."""
+    print("\n" + "="*60)
+    print("PHASE 0: Endpoint Verification")
+    print("="*60)
+
+    checks = [
+        ("GET",  "/health",   None,  True),
+        ("GET",  "/tasks",    None,  True),
+        ("GET",  "/metadata", None,  True),
+        ("GET",  "/schema",   None,  True),
+        ("GET",  "/web",      None,  False),   # returns HTML, not JSON
+        ("POST", "/reset",    {"task": "easy", "seed": 0}, True),
+        ("GET",  "/state",    None,  True),
+        ("POST", "/step",     {"action_type": "inspect", "account_id": "acc_0000"}, True),
+        ("POST", "/step",     {"action_type": "submit"}, True),
+        ("GET",  "/grader",   None,  True),
+        ("POST", "/mcp",      {"jsonrpc": "2.0", "method": "tools/list", "id": 1}, True),
+        ("POST", "/baseline", None,  True),
+    ]
+
+    all_ok = True
+    for method, path, body, expect_json in checks:
+        try:
+            if method == "GET":
+                http_get(f"{base_url}{path}", expect_json=expect_json)
+            else:
+                http_post(f"{base_url}{path}", body)
+            print(f"  ✓ {method} {path}")
+        except Exception as e:
+            print(f"  ✗ {method} {path} — {e}")
+            all_ok = False
+
+    return all_ok
+
+
+def test_baseline_stability(base_url: str) -> bool:
+    """Phase 1: Baseline re-run (must produce identical scores)."""
+    print("\n" + "="*60)
+    print("PHASE 1: Baseline Stability (3 runs)")
+    print("="*60)
+
+    scores_list = []
+    for i in range(3):
+        r = http_post(f"{base_url}/baseline")
+        scores = r["scores"]
+        scores_list.append(scores)
+        print(f"  Run {i+1}: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+
+    # Check all identical
+    stable = all(s == scores_list[0] for s in scores_list)
+    if stable:
+        print("  ✓ All 3 runs identical — baseline is deterministic")
+    else:
+        print("  ✗ SCORES DIFFER — baseline is non-deterministic!")
+    return stable
+
+
+def test_llm_agent(base_url: str, task: str, seed: int = 0) -> float:
+    """Phase 2: Run an LLM agent against one task (simulates judge's Nemotron run)."""
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"\n  --- LLM Agent: task={task}, seed={seed}, model={_model} ---")
+
+    # Reset
+    reset_resp = http_post(f"{base_url}/reset", {"task": task, "seed": seed})
+    obs = reset_resp.get("observation", reset_resp)
+    done = reset_resp.get("done", False)
+
+    step_num = 0
+    while not done:
+        step_num += 1
+        prompt = format_obs(obs)
+        llm_text = call_llm(prompt)
+        action = parse_action(llm_text, obs)
+
+        action_str = f"{action['action_type'].upper()} {action.get('account_id', '')}".strip()
+
+        step_resp = http_post(f"{base_url}/step", action)
+        obs = step_resp.get("observation", step_resp)
+        done = step_resp.get("done", False)
+        reward = step_resp.get("reward")
+
+        flagged_n = len(obs.get("flagged_ids", []))
+        suspects_n = len(obs.get("suspect_ids", []))
+        steps_left = obs.get("steps_remaining", "?")
+
+        print(f"    Step {step_num:2d}: {action_str:35s} flagged={flagged_n}/10  suspects={suspects_n}  steps_left={steps_left}")
+
+        if done and reward is not None:
+            msg = step_resp.get("message", obs.get("message", ""))
+            print(f"    → Episode ended: {msg[:100]}")
+
+    # Get grader score
+    grader = http_get(f"{base_url}/grader")
+    score = grader["score"]
+    print(f"    ★ GRADER SCORE: {score:.4f}")
+    return score
+
+
+def test_llm_all_tasks(base_url: str) -> Dict[str, float]:
+    """Phase 2: Run LLM agent on all 3 tasks."""
+    print("\n" + "="*60)
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"PHASE 2: LLM Agent Evaluation (model={_model})")
+    print("="*60)
+
+    scores = {}
+    for task in ["easy", "medium", "hard"]:
+        scores[task] = test_llm_agent(base_url, task=task, seed=0)
+
+    print(f"\n  Summary: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+    return scores
+
+
+def test_variance(base_url: str, seeds: List[int] = [0, 1, 2, 3, 4]) -> None:
+    """Phase 3: Score variance check (multiple seeds per task)."""
+    print("\n" + "="*60)
+    print(f"PHASE 3: Score Variance (seeds={seeds})")
+    print("="*60)
+
+    for task in ["easy", "medium", "hard"]:
+        task_scores = []
+        for seed in seeds:
+            score = test_llm_agent(base_url, task=task, seed=seed)
+            task_scores.append(score)
+
+        mean = sum(task_scores) / len(task_scores)
+        variance = sum((s - mean) ** 2 for s in task_scores) / len(task_scores)
+        print(f"\n  {task}: scores={[f'{s:.3f}' for s in task_scores]}  mean={mean:.4f}  var={variance:.6f}")
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Judge Evaluation Simulator for GraphStrike")
+    parser.add_argument("--url", required=True, help="Environment server URL")
+    parser.add_argument("--bedrock", action="store_true", help="Use AWS Bedrock instead of HF router")
+    parser.add_argument("--endpoints-only", action="store_true", help="Only test endpoints (no LLM)")
+    parser.add_argument("--skip-variance", action="store_true", help="Skip variance check (faster)")
+    parser.add_argument("--seeds", type=int, default=3, help="Number of seeds for variance check")
+    args = parser.parse_args()
+
+    if args.bedrock:
+        LLM_BACKEND = "bedrock"
+
+    base = args.url.rstrip("/")
+    model_display = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"GraphStrike Judge Evaluation Simulator")
+    print(f"Target:  {base}")
+    print(f"Backend: {LLM_BACKEND}")
+    print(f"Model:   {model_display}")
+    print(f"Token:   {'set' if (HF_TOKEN or os.getenv('AWS_ACCESS_KEY_ID')) else 'NOT SET'}")
+
+    # Phase 0: Endpoints
+    if not test_endpoints(base):
+        print("\n✗ Endpoint check failed. Fix before proceeding.")
+        sys.exit(1)
+
+    # Phase 1: Baseline stability
+    test_baseline_stability(base)
+
+    if args.endpoints_only:
+        print("\n✓ Endpoint-only mode — skipping LLM tests.")
+        sys.exit(0)
+
+    if LLM_BACKEND == "bedrock":
+        if not os.getenv("AWS_ACCESS_KEY_ID"):
+            print("\n✗ AWS_ACCESS_KEY_ID not set. Cannot run Bedrock LLM tests.")
+            sys.exit(1)
+    elif not HF_TOKEN:
+        print("\n✗ HF_TOKEN not set. Cannot run LLM agent tests.")
+        print("  export HF_TOKEN='hf_...'  OR  use --bedrock with AWS creds")
+        sys.exit(1)
+
+    # Phase 2: LLM on all tasks
+    scores = test_llm_all_tasks(base)
+
+    # Phase 3: Variance
+    if not args.skip_variance:
+        test_variance(base, seeds=list(range(args.seeds)))
+
+    print("\n" + "="*60)
+    print("EVALUATION COMPLETE")
+    print("="*60)

eval-models/qwen_test_judge_eval.py

@@ -0,0 +1,478 @@
+#!/usr/bin/env python3
+"""
+Judge Evaluation Simulator
+==========================
+
+Simulates EXACTLY how hackathon judges will evaluate your environment:
+
+1. Baseline re-run: POST /baseline → verify scores are stable
+2. Standard Open LLM agent: Run an LLM (via HF router) against all 3 tasks
+3. Score variance check: Run same task multiple seeds, check variance
+
+USAGE:
+    # Against live HF Space (requires HF_TOKEN):
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Against local server:
+    export HF_TOKEN="hf_..."
+    python test_judge_eval.py --url http://localhost:7860
+
+    # Choose model (default: Qwen/Qwen2.5-72B-Instruct):
+    export MODEL_NAME="nvidia/Llama-3.1-Nemotron-70B-Instruct-HF"
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space
+
+    # Just test endpoints (no LLM needed):
+    python test_judge_eval.py --url https://pandago-graphstrike.hf.space --endpoints-only
+"""
+
+import json
+import os
+import sys
+import time
+import urllib.request
+from typing import Dict, List, Optional
+
+# ---------------------------------------------------------------------------
+# Config from env vars (same as judges will set)
+# ---------------------------------------------------------------------------
+
+HF_TOKEN = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+MODEL_NAME = os.getenv("MODEL_NAME", "Qwen/Qwen2.5-72B-Instruct")
+
+# AWS Bedrock config (use --bedrock flag)
+LLM_BACKEND = "hf"  # "hf" or "bedrock"
+BEDROCK_MODEL_ID = os.getenv("BEDROCK_MODEL_ID", "qwen.qwen3-next-80b-a3b")
+
+SYSTEM_PROMPT = """You are an AI detective finding 10 coordinated fake accounts in a social network.
+
+ACTIONS (reply with exactly ONE line):
+- INSPECT acc_XXXX  — reveal profile (costs 1 step)
+- FLAG acc_XXXX     — mark as fake (FREE, no step cost, triggers suspect cascade)
+- SUBMIT            — end episode, get scored
+
+DECISION RULES (apply top-to-bottom, first match wins):
+1. If suspect_ids lists accounts you haven't inspected → INSPECT the first one
+2. If ANY profiled account has shared_ip_count >= 5 and is NOT flagged → FLAG it immediately
+3. If ANY profiled account has photo_reuse >= 0.50 AND bio_template >= 0.40 and hub < 0.70 and NOT flagged → FLAG it
+4. If ANY profiled account has fake_risk_score >= 0.30 and hub < 0.70 and NOT flagged → FLAG it
+5. If there are uninspected visible accounts and steps > 3 → INSPECT the next one
+6. If you have 10 flags OR steps <= 3 → SUBMIT
+
+IMPORTANT:
+- FLAG is FREE (costs 0 steps) — flag aggressively when you see suspicious signals
+- After each FLAG, new suspects appear — always inspect suspects before other accounts
+- hub_legitimacy_score > 0.70 means celebrity — do NOT flag
+- shared_ip_count >= 5 is the strongest gang signal (all 10 share one IP)
+- Do NOT re-inspect already inspected accounts
+
+Reply with EXACTLY one line, nothing else:
+FLAG acc_XXXX
+INSPECT acc_XXXX
+SUBMIT"""
+
+
+# ---------------------------------------------------------------------------
+# HTTP helpers
+# ---------------------------------------------------------------------------
+
+def _retry(fn, retries=3, backoff=3):
+    """Retry a function on network errors."""
+    for attempt in range(retries):
+        try:
+            return fn()
+        except OSError as e:
+            if attempt == retries - 1:
+                raise
+            wait = backoff * (attempt + 1)
+            print(f"    [RETRY] Network error: {e} — retrying in {wait}s ({attempt+1}/{retries})")
+            time.sleep(wait)
+
+
+def http_post(url: str, body: Optional[dict] = None) -> dict:
+    def _do():
+        data = json.dumps(body or {}).encode()
+        req = urllib.request.Request(
+            url, data=data,
+            headers={"Content-Type": "application/json"},
+            method="POST"
+        )
+        with urllib.request.urlopen(req, timeout=120) as resp:
+            return json.loads(resp.read())
+    return _retry(_do)
+
+
+def http_get(url: str, expect_json: bool = True) -> dict:
+    def _do():
+        with urllib.request.urlopen(url, timeout=120) as resp:
+            body = resp.read()
+            if not expect_json:
+                return {"_status": resp.status, "_body_len": len(body)}
+            return json.loads(body)
+    return _retry(_do)
+
+
+# ---------------------------------------------------------------------------
+# LLM call via OpenAI-compatible API
+# ---------------------------------------------------------------------------
+
+def _call_hf(prompt: str) -> str:
+    """Call LLM via HF router (OpenAI-compatible)."""
+    from openai import OpenAI
+    client = OpenAI(base_url=API_BASE_URL, api_key=HF_TOKEN)
+    resp = client.chat.completions.create(
+        model=MODEL_NAME,
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        temperature=0.3,
+        max_tokens=256,
+    )
+    return (resp.choices[0].message.content or "").strip()
+
+
+def _call_bedrock(prompt: str) -> str:
+    """Call LLM via AWS Bedrock. Tries converse() first, falls back to invoke_model()."""
+    import boto3
+    client = boto3.client(
+        service_name="bedrock-runtime",
+        region_name=os.getenv("AWS_DEFAULT_REGION", "us-east-1"),
+        aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
+        aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
+    )
+    # Try converse API first (boto3 >= 1.34.x)
+    if hasattr(client, "converse"):
+        resp = client.converse(
+            modelId=BEDROCK_MODEL_ID,
+            messages=[{"role": "user", "content": [{"text": prompt}]}],
+            system=[{"text": SYSTEM_PROMPT}],
+            inferenceConfig={"maxTokens": 256, "temperature": 0.3},
+        )
+        return resp["output"]["message"]["content"][0]["text"].strip()
+    # Fallback: invoke_model (works with all boto3 versions)
+    body = json.dumps({
+        "messages": [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": prompt},
+        ],
+        "max_tokens": 256,
+        "temperature": 0.3,
+    })
+    resp = client.invoke_model(
+        modelId=BEDROCK_MODEL_ID,
+        contentType="application/json",
+        accept="application/json",
+        body=body,
+    )
+    result = json.loads(resp["body"].read())
+    # Handle both OpenAI-style and Bedrock-native response formats
+    if "choices" in result:
+        return result["choices"][0]["message"]["content"].strip()
+    if "content" in result:
+        content = result["content"]
+        if isinstance(content, list):
+            return content[0].get("text", "").strip()
+        return str(content).strip()
+    if "output" in result:
+        return result["output"].get("text", "").strip()
+    return str(result).strip()
+
+
+def call_llm(prompt: str) -> str:
+    """Call LLM with retries. Uses HF router or Bedrock based on LLM_BACKEND."""
+    fn = _call_bedrock if LLM_BACKEND == "bedrock" else _call_hf
+    for attempt in range(3):
+        try:
+            raw = fn(prompt)
+            if os.getenv("DEBUG_LLM"):
+                print(f"    [LLM RAW] {raw[:200]}")
+            # Strip Qwen3 <think>...</think> reasoning blocks
+            import re
+            cleaned = re.sub(r"<think>.*?</think>", "", raw, flags=re.DOTALL).strip()
+            return cleaned if cleaned else raw
+        except Exception as e:
+            if attempt == 2:
+                print(f"    [LLM ERROR] {e} (gave up after 3 attempts)")
+                return ""
+            wait = 3 * (attempt + 1)
+            print(f"    [LLM RETRY] {e} — retrying in {wait}s")
+            time.sleep(wait)
+    return ""
+
+
+def format_obs(obs: dict) -> str:
+    """Format observation as text for LLM — shows raw signals prominently."""
+    lines = []
+    lines.append(f"TASK: {obs.get('task','?').upper()} | Steps remaining: {obs.get('steps_remaining','?')}")
+
+    flagged = obs.get("flagged_ids", [])
+    lines.append(f"Flagged ({len(flagged)}/10): {', '.join(flagged) if flagged else 'none'}")
+
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    uninspected_suspects = [s for s in suspects if s not in inspected]
+    if uninspected_suspects:
+        lines.append(f"*** SUSPECTS (uninspected) → INSPECT THESE FIRST: {', '.join(uninspected_suspects)} ***")
+
+    accounts = obs.get("visible_accounts", [])
+    if accounts:
+        # Split: unflagged accounts that should be flagged vs rest
+        unflagged_suspicious = []
+        flagged_accs = []
+        clean_accs = []
+        for a in sorted(accounts, key=lambda x: x.get("fake_risk_score", 0), reverse=True):
+            aid = a.get("account_id", "?")
+            if aid in flagged:
+                flagged_accs.append(a)
+            elif (a.get("shared_ip_count", 0) >= 5 or
+                  (a.get("photo_reuse_score", 0) >= 0.50 and a.get("bio_template_score", 0) >= 0.40)):
+                unflagged_suspicious.append(a)
+            else:
+                clean_accs.append(a)
+
+        if unflagged_suspicious:
+            lines.append(f"\n!!! ACTION NEEDED — FLAG THESE ({len(unflagged_suspicious)} accounts with strong fake signals):")
+            for a in unflagged_suspicious:
+                aid = a.get("account_id", "?")
+                lines.append(f"  → FLAG {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} ip_shared={a.get('shared_ip_count',0)} hub={a.get('hub_legitimacy_score',0):.2f}")
+
+        if flagged_accs:
+            lines.append(f"\nALREADY FLAGGED ({len(flagged_accs)}):")
+            for a in flagged_accs[:5]:
+                lines.append(f"  ✓ {a.get('account_id','?')}")
+
+        if clean_accs:
+            lines.append(f"\nCLEAN ACCOUNTS ({len(clean_accs)}):")
+            for a in clean_accs[:5]:
+                aid = a.get("account_id", "?")
+                hub = a.get("hub_legitimacy_score", 0)
+                hub_mark = " [CELEBRITY]" if hub > 0.70 else ""
+                lines.append(f"  {aid}: risk={a.get('fake_risk_score',0):.3f} photo={a.get('photo_reuse_score',0):.2f} bio={a.get('bio_template_score',0):.2f} hub={hub:.2f}{hub_mark}")
+
+    visible = obs.get("visible_account_ids", [])
+    uninspected = [i for i in visible if i not in inspected]
+    if uninspected:
+        lines.append(f"\nUninspected IDs ({len(uninspected)}): {', '.join(uninspected[:8])}{'...' if len(uninspected) > 8 else ''}")
+
+    lines.append(f"\nMessage: {obs.get('message', '')}")
+    return "\n".join(lines)
+
+
+def parse_action(llm_text: str, obs: dict) -> dict:
+    """Parse LLM output to action dict."""
+    for line in llm_text.split("\n"):
+        line = line.strip()
+        upper = line.upper()
+        if upper.startswith("INSPECT ") or upper.startswith("FLAG ") or upper.startswith("INVESTIGATE_NETWORK ") or upper.startswith("UNFLAG "):
+            parts = line.split(maxsplit=1)
+            return {"action_type": parts[0].lower(), "account_id": parts[1].lower() if len(parts) > 1 else None}
+        if upper == "SUBMIT":
+            return {"action_type": "submit"}
+
+    # Fallback: inspect first uninspected suspect
+    suspects = obs.get("suspect_ids", [])
+    inspected = obs.get("inspected_ids", [])
+    for s in suspects:
+        if s not in inspected:
+            return {"action_type": "inspect", "account_id": s}
+    visible = obs.get("visible_account_ids", [])
+    for v in visible:
+        if v not in inspected:
+            return {"action_type": "inspect", "account_id": v}
+    return {"action_type": "submit"}
+
+
+# ---------------------------------------------------------------------------
+# Test phases
+# ---------------------------------------------------------------------------
+
+def test_endpoints(base_url: str) -> bool:
+    """Phase 0: Verify all required endpoints respond correctly."""
+    print("\n" + "="*60)
+    print("PHASE 0: Endpoint Verification")
+    print("="*60)
+
+    checks = [
+        ("GET",  "/health",   None,  True),
+        ("GET",  "/tasks",    None,  True),
+        ("GET",  "/metadata", None,  True),
+        ("GET",  "/schema",   None,  True),
+        ("GET",  "/web",      None,  False),   # returns HTML, not JSON
+        ("POST", "/reset",    {"task": "easy", "seed": 0}, True),
+        ("GET",  "/state",    None,  True),
+        ("POST", "/step",     {"action_type": "inspect", "account_id": "acc_0000"}, True),
+        ("POST", "/step",     {"action_type": "submit"}, True),
+        ("GET",  "/grader",   None,  True),
+        ("POST", "/mcp",      {"jsonrpc": "2.0", "method": "tools/list", "id": 1}, True),
+        ("POST", "/baseline", None,  True),
+    ]
+
+    all_ok = True
+    for method, path, body, expect_json in checks:
+        try:
+            if method == "GET":
+                http_get(f"{base_url}{path}", expect_json=expect_json)
+            else:
+                http_post(f"{base_url}{path}", body)
+            print(f"  ✓ {method} {path}")
+        except Exception as e:
+            print(f"  ✗ {method} {path} — {e}")
+            all_ok = False
+
+    return all_ok
+
+
+def test_baseline_stability(base_url: str) -> bool:
+    """Phase 1: Baseline re-run (must produce identical scores)."""
+    print("\n" + "="*60)
+    print("PHASE 1: Baseline Stability (3 runs)")
+    print("="*60)
+
+    scores_list = []
+    for i in range(3):
+        r = http_post(f"{base_url}/baseline")
+        scores = r["scores"]
+        scores_list.append(scores)
+        print(f"  Run {i+1}: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+
+    # Check all identical
+    stable = all(s == scores_list[0] for s in scores_list)
+    if stable:
+        print("  ✓ All 3 runs identical — baseline is deterministic")
+    else:
+        print("  ✗ SCORES DIFFER — baseline is non-deterministic!")
+    return stable
+
+
+def test_llm_agent(base_url: str, task: str, seed: int = 0) -> float:
+    """Phase 2: Run an LLM agent against one task (simulates judge's Nemotron run)."""
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"\n  --- LLM Agent: task={task}, seed={seed}, model={_model} ---")
+
+    # Reset
+    reset_resp = http_post(f"{base_url}/reset", {"task": task, "seed": seed})
+    obs = reset_resp.get("observation", reset_resp)
+    done = reset_resp.get("done", False)
+
+    step_num = 0
+    while not done:
+        step_num += 1
+        prompt = format_obs(obs)
+        llm_text = call_llm(prompt)
+        action = parse_action(llm_text, obs)
+
+        action_str = f"{action['action_type'].upper()} {action.get('account_id', '')}".strip()
+
+        step_resp = http_post(f"{base_url}/step", action)
+        obs = step_resp.get("observation", step_resp)
+        done = step_resp.get("done", False)
+        reward = step_resp.get("reward")
+
+        flagged_n = len(obs.get("flagged_ids", []))
+        suspects_n = len(obs.get("suspect_ids", []))
+        steps_left = obs.get("steps_remaining", "?")
+
+        print(f"    Step {step_num:2d}: {action_str:35s} flagged={flagged_n}/10  suspects={suspects_n}  steps_left={steps_left}")
+
+        if done and reward is not None:
+            msg = step_resp.get("message", obs.get("message", ""))
+            print(f"    → Episode ended: {msg[:100]}")
+
+    # Get grader score
+    grader = http_get(f"{base_url}/grader")
+    score = grader["score"]
+    print(f"    ★ GRADER SCORE: {score:.4f}")
+    return score
+
+
+def test_llm_all_tasks(base_url: str) -> Dict[str, float]:
+    """Phase 2: Run LLM agent on all 3 tasks."""
+    print("\n" + "="*60)
+    _model = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"PHASE 2: LLM Agent Evaluation (model={_model})")
+    print("="*60)
+
+    scores = {}
+    for task in ["easy", "medium", "hard"]:
+        scores[task] = test_llm_agent(base_url, task=task, seed=0)
+
+    print(f"\n  Summary: easy={scores['easy']:.4f}  medium={scores['medium']:.4f}  hard={scores['hard']:.4f}")
+    return scores
+
+
+def test_variance(base_url: str, seeds: List[int] = [0, 1, 2, 3, 4]) -> None:
+    """Phase 3: Score variance check (multiple seeds per task)."""
+    print("\n" + "="*60)
+    print(f"PHASE 3: Score Variance (seeds={seeds})")
+    print("="*60)
+
+    for task in ["easy", "medium", "hard"]:
+        task_scores = []
+        for seed in seeds:
+            score = test_llm_agent(base_url, task=task, seed=seed)
+            task_scores.append(score)
+
+        mean = sum(task_scores) / len(task_scores)
+        variance = sum((s - mean) ** 2 for s in task_scores) / len(task_scores)
+        print(f"\n  {task}: scores={[f'{s:.3f}' for s in task_scores]}  mean={mean:.4f}  var={variance:.6f}")
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Judge Evaluation Simulator for GraphStrike")
+    parser.add_argument("--url", required=True, help="Environment server URL")
+    parser.add_argument("--bedrock", action="store_true", help="Use AWS Bedrock instead of HF router")
+    parser.add_argument("--endpoints-only", action="store_true", help="Only test endpoints (no LLM)")
+    parser.add_argument("--skip-variance", action="store_true", help="Skip variance check (faster)")
+    parser.add_argument("--seeds", type=int, default=3, help="Number of seeds for variance check")
+    args = parser.parse_args()
+
+    if args.bedrock:
+        LLM_BACKEND = "bedrock"
+
+    base = args.url.rstrip("/")
+    model_display = f"Bedrock/{BEDROCK_MODEL_ID}" if LLM_BACKEND == "bedrock" else MODEL_NAME
+    print(f"GraphStrike Judge Evaluation Simulator")
+    print(f"Target:  {base}")
+    print(f"Backend: {LLM_BACKEND}")
+    print(f"Model:   {model_display}")
+    print(f"Token:   {'set' if (HF_TOKEN or os.getenv('AWS_ACCESS_KEY_ID')) else 'NOT SET'}")
+
+    # Phase 0: Endpoints
+    if not test_endpoints(base):
+        print("\n✗ Endpoint check failed. Fix before proceeding.")
+        sys.exit(1)
+
+    # Phase 1: Baseline stability
+    test_baseline_stability(base)
+
+    if args.endpoints_only:
+        print("\n✓ Endpoint-only mode — skipping LLM tests.")
+        sys.exit(0)
+
+    if LLM_BACKEND == "bedrock":
+        if not os.getenv("AWS_ACCESS_KEY_ID"):
+            print("\n✗ AWS_ACCESS_KEY_ID not set. Cannot run Bedrock LLM tests.")
+            sys.exit(1)
+    elif not HF_TOKEN:
+        print("\n✗ HF_TOKEN not set. Cannot run LLM agent tests.")
+        print("  export HF_TOKEN='hf_...'  OR  use --bedrock with AWS creds")
+        sys.exit(1)
+
+    # Phase 2: LLM on all tasks
+    scores = test_llm_all_tasks(base)
+
+    # Phase 3: Variance
+    if not args.skip_variance:
+        test_variance(base, seeds=list(range(args.seeds)))
+
+    print("\n" + "="*60)
+    print("EVALUATION COMPLETE")
+    print("="*60)

judge_log.txt

@@ -0,0 +1,513 @@
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26$ python3 test_judge_eval.py --url https://pandago-graphstrike.hf.space --bedrock
+GraphStrike Judge Evaluation Simulator
+Target:  https://pandago-graphstrike.hf.space
+Backend: bedrock
+Model:   Bedrock/qwen.qwen3-next-80b-a3b
+Token:   set
+
+============================================================
+PHASE 0: Endpoint Verification
+============================================================
+  ✓ GET /health
+  ✓ GET /tasks
+  ✓ GET /metadata
+  ✓ GET /schema
+  ✓ GET /web
+  ✓ POST /reset
+  ✓ GET /state
+  ✓ POST /step
+  ✓ POST /step
+  ✓ GET /grader
+  ✓ POST /mcp
+  ✓ POST /baseline
+
+============================================================
+PHASE 1: Baseline Stability (3 runs)
+============================================================
+  Run 1: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 2: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 3: easy=0.9100  medium=0.9060  hard=0.9038
+  ✓ All 3 runs identical — baseline is deterministic
+
+============================================================
+PHASE 2: LLM Agent Evaluation (model=Bedrock/qwen.qwen3-next-80b-a3b)
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=29
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=28
+    Step  4: FLAG acc_0036                       flagged=2/10  suspects=8  steps_left=28
+    Step  5: INSPECT acc_0001                    flagged=2/10  suspects=8  steps_left=27
+    Step  6: FLAG acc_0001                       flagged=3/10  suspects=7  steps_left=27
+    Step  7: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=26
+    Step  8: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=26
+    Step  9: INSPECT acc_0012                    flagged=4/10  suspects=6  steps_left=25
+    Step 10: FLAG acc_0012                       flagged=5/10  suspects=5  steps_left=25
+    Step 11: INSPECT acc_0000                    flagged=5/10  suspects=5  steps_left=24
+    Step 12: FLAG acc_0000                       flagged=6/10  suspects=4  steps_left=24
+    Step 13: INSPECT acc_0027                    flagged=6/10  suspects=4  steps_left=23
+    Step 14: FLAG acc_0027                       flagged=7/10  suspects=3  steps_left=23
+    Step 15: INSPECT acc_0047                    flagged=7/10  suspects=3  steps_left=22
+    Step 16: FLAG acc_0047                       flagged=8/10  suspects=2  steps_left=22
+    Step 17: INSPECT acc_0007                    flagged=8/10  suspects=2  steps_left=21
+    Step 18: FLAG acc_0007                       flagged=9/10  suspects=1  steps_left=21
+    Step 19: INSPECT acc_0028                    flagged=9/10  suspects=1  steps_left=20
+    Step 20: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=20
+    Step 21: SUBMIT                              flagged=10/10  suspects=0  steps_left=20
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.90
+    ★ GRADER SCORE: 0.9667
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=41
+    Step 11: INSPECT acc_0181                    flagged=1/10  suspects=8  steps_left=40
+    Step 12: FLAG acc_0181                       flagged=2/10  suspects=8  steps_left=40
+    Step 13: INSPECT acc_0022                    flagged=2/10  suspects=8  steps_left=39
+    Step 14: FLAG acc_0022                       flagged=3/10  suspects=7  steps_left=39
+    Step 15: INSPECT acc_0092                    flagged=3/10  suspects=7  steps_left=38
+    Step 16: INSPECT acc_0097                    flagged=3/10  suspects=7  steps_left=37
+    Step 17: FLAG acc_0092                       flagged=4/10  suspects=6  steps_left=37
+    Step 18: FLAG acc_0097                       flagged=5/10  suspects=5  steps_left=37
+    Step 19: INSPECT acc_0187                    flagged=5/10  suspects=5  steps_left=36
+    Step 20: FLAG acc_0187                       flagged=6/10  suspects=4  steps_left=36
+    Step 21: INSPECT acc_0093                    flagged=6/10  suspects=4  steps_left=35
+    Step 22: FLAG acc_0093                       flagged=7/10  suspects=3  steps_left=35
+    Step 23: INSPECT acc_0172                    flagged=7/10  suspects=3  steps_left=34
+    Step 24: FLAG acc_0172                       flagged=8/10  suspects=2  steps_left=34
+    Step 25: INSPECT acc_0058                    flagged=8/10  suspects=2  steps_left=33
+    Step 26: FLAG acc_0058                       flagged=9/10  suspects=1  steps_left=33
+    Step 27: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=32
+    Step 28: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=32
+    Step 29: SUBMIT                              flagged=10/10  suspects=0  steps_left=32
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.82
+    ★ GRADER SCORE: 0.9640
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0441                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0871                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0070                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0443                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0037                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0438                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: FLAG acc_0439                       flagged=1/10  suspects=5  steps_left=60
+    Step 22: INSPECT acc_0237                    flagged=1/10  suspects=5  steps_left=59
+    Step 23: FLAG acc_0237                       flagged=2/10  suspects=6  steps_left=59
+    Step 24: INSPECT acc_0621                    flagged=2/10  suspects=6  steps_left=58
+    Step 25: FLAG acc_0621                       flagged=3/10  suspects=6  steps_left=58
+    Step 26: INSPECT acc_0389                    flagged=3/10  suspects=6  steps_left=57
+    Step 27: FLAG acc_0389                       flagged=4/10  suspects=6  steps_left=57
+    Step 28: INSPECT acc_0160                    flagged=4/10  suspects=6  steps_left=56
+    Step 29: FLAG acc_0160                       flagged=5/10  suspects=5  steps_left=56
+    Step 30: INSPECT acc_0549                    flagged=5/10  suspects=5  steps_left=55
+    Step 31: FLAG acc_0549                       flagged=6/10  suspects=4  steps_left=55
+    Step 32: INSPECT acc_0658                    flagged=6/10  suspects=4  steps_left=54
+    Step 33: FLAG acc_0658                       flagged=7/10  suspects=3  steps_left=54
+    Step 34: INSPECT acc_0290                    flagged=7/10  suspects=3  steps_left=53
+    Step 35: FLAG acc_0290                       flagged=8/10  suspects=2  steps_left=53
+    Step 36: INSPECT acc_0124                    flagged=8/10  suspects=2  steps_left=52
+    Step 37: INSPECT acc_0507                    flagged=8/10  suspects=2  steps_left=51
+    Step 38: FLAG acc_0124                       flagged=9/10  suspects=1  steps_left=51
+    Step 39: FLAG acc_0507                       flagged=10/10  suspects=0  steps_left=51
+    Step 40: SUBMIT                              flagged=10/10  suspects=0  steps_left=51
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.71
+    ★ GRADER SCORE: 0.9637
+
+  Summary: easy=0.9667  medium=0.9640  hard=0.9637
+
+============================================================
+PHASE 3: Score Variance (seeds=[0, 1, 2])
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=29
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=28
+    Step  4: FLAG acc_0036                       flagged=2/10  suspects=8  steps_left=28
+    Step  5: INSPECT acc_0001                    flagged=2/10  suspects=8  steps_left=27
+    Step  6: FLAG acc_0001                       flagged=3/10  suspects=7  steps_left=27
+    Step  7: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=26
+    Step  8: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=26
+    Step  9: INSPECT acc_0012                    flagged=4/10  suspects=6  steps_left=25
+    Step 10: FLAG acc_0012                       flagged=5/10  suspects=5  steps_left=25
+    Step 11: INSPECT acc_0000                    flagged=5/10  suspects=5  steps_left=24
+    Step 12: FLAG acc_0000                       flagged=6/10  suspects=4  steps_left=24
+    Step 13: INSPECT acc_0027                    flagged=6/10  suspects=4  steps_left=23
+    Step 14: FLAG acc_0027                       flagged=7/10  suspects=3  steps_left=23
+    Step 15: INSPECT acc_0047                    flagged=7/10  suspects=3  steps_left=22
+    Step 16: FLAG acc_0047                       flagged=8/10  suspects=2  steps_left=22
+    Step 17: INSPECT acc_0007                    flagged=8/10  suspects=2  steps_left=21
+    Step 18: FLAG acc_0007                       flagged=9/10  suspects=1  steps_left=21
+    Step 19: INSPECT acc_0028                    flagged=9/10  suspects=1  steps_left=20
+    Step 20: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=20
+    Step 21: SUBMIT                              flagged=10/10  suspects=0  steps_left=20
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.90
+    ★ GRADER SCORE: 0.9667
+
+  --- LLM Agent: task=easy, seed=1, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0034                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: INSPECT acc_0003                    flagged=0/10  suspects=0  steps_left=28
+    Step  3: INSPECT acc_0049                    flagged=0/10  suspects=0  steps_left=27
+    Step  4: INSPECT acc_0006                    flagged=0/10  suspects=0  steps_left=26
+    Step  5: INSPECT acc_0047                    flagged=0/10  suspects=0  steps_left=25
+    Step  6: FLAG acc_0047                       flagged=1/10  suspects=9  steps_left=25
+    Step  7: INSPECT acc_0009                    flagged=1/10  suspects=9  steps_left=24
+    Step  8: FLAG acc_0009                       flagged=2/10  suspects=8  steps_left=24
+    Step  9: INSPECT acc_0046                    flagged=2/10  suspects=8  steps_left=23
+    Step 10: FLAG acc_0046                       flagged=3/10  suspects=7  steps_left=23
+    Step 11: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=22
+    Step 12: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=22
+    Step 13: INSPECT acc_0021                    flagged=4/10  suspects=6  steps_left=21
+    Step 14: FLAG acc_0021                       flagged=5/10  suspects=5  steps_left=21
+    Step 15: INSPECT acc_0002                    flagged=5/10  suspects=5  steps_left=20
+    Step 16: FLAG acc_0002                       flagged=6/10  suspects=4  steps_left=20
+    Step 17: INSPECT acc_0048                    flagged=6/10  suspects=4  steps_left=19
+    Step 18: FLAG acc_0048                       flagged=7/10  suspects=3  steps_left=19
+    Step 19: INSPECT acc_0029                    flagged=7/10  suspects=3  steps_left=18
+    Step 20: FLAG acc_0029                       flagged=8/10  suspects=2  steps_left=18
+    Step 21: INSPECT acc_0015                    flagged=8/10  suspects=2  steps_left=17
+    Step 22: FLAG acc_0015                       flagged=9/10  suspects=1  steps_left=17
+    Step 23: INSPECT acc_0005                    flagged=9/10  suspects=1  steps_left=16
+    Step 24: FLAG acc_0005                       flagged=10/10  suspects=0  steps_left=16
+    Step 25: SUBMIT                              flagged=10/10  suspects=0  steps_left=16
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.86
+    ★ GRADER SCORE: 0.9533
+
+  --- LLM Agent: task=easy, seed=2, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0040                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: INSPECT acc_0017                    flagged=0/10  suspects=0  steps_left=28
+    Step  3: INSPECT acc_0025                    flagged=0/10  suspects=0  steps_left=27
+    Step  4: INSPECT acc_0026                    flagged=0/10  suspects=0  steps_left=26
+    Step  5: INSPECT acc_0038                    flagged=0/10  suspects=0  steps_left=25
+    Step  6: INSPECT acc_0029                    flagged=0/10  suspects=0  steps_left=24
+    Step  7: FLAG acc_0029                       flagged=1/10  suspects=9  steps_left=24
+    Step  8: INSPECT acc_0006                    flagged=1/10  suspects=9  steps_left=23
+    Step  9: FLAG acc_0006                       flagged=2/10  suspects=8  steps_left=23
+    Step 10: INSPECT acc_0033                    flagged=2/10  suspects=8  steps_left=22
+    Step 11: FLAG acc_0033                       flagged=3/10  suspects=7  steps_left=22
+    Step 12: INSPECT acc_0015                    flagged=3/10  suspects=7  steps_left=21
+    Step 13: FLAG acc_0015                       flagged=4/10  suspects=6  steps_left=21
+    Step 14: INSPECT acc_0022                    flagged=4/10  suspects=6  steps_left=20
+    Step 15: FLAG acc_0022                       flagged=5/10  suspects=5  steps_left=20
+    Step 16: INSPECT acc_0009                    flagged=5/10  suspects=5  steps_left=19
+    Step 17: FLAG acc_0009                       flagged=6/10  suspects=4  steps_left=19
+    Step 18: INSPECT acc_0004                    flagged=6/10  suspects=4  steps_left=18
+    Step 19: FLAG acc_0004                       flagged=7/10  suspects=3  steps_left=18
+    Step 20: INSPECT acc_0024                    flagged=7/10  suspects=3  steps_left=17
+    Step 21: FLAG acc_0024                       flagged=8/10  suspects=2  steps_left=17
+    Step 22: INSPECT acc_0049                    flagged=8/10  suspects=2  steps_left=16
+    Step 23: FLAG acc_0049                       flagged=9/10  suspects=1  steps_left=16
+    Step 24: INSPECT acc_0035                    flagged=9/10  suspects=1  steps_left=15
+    Step 25: FLAG acc_0035                       flagged=10/10  suspects=0  steps_left=15
+    Step 26: SUBMIT                              flagged=10/10  suspects=0  steps_left=15
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.85
+    ★ GRADER SCORE: 0.9500
+
+  easy: scores=['0.967', '0.953', '0.950']  mean=0.9567  var=0.000052
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=41
+    Step 11: INSPECT acc_0181                    flagged=1/10  suspects=8  steps_left=40
+    Step 12: FLAG acc_0181                       flagged=2/10  suspects=8  steps_left=40
+    Step 13: INSPECT acc_0022                    flagged=2/10  suspects=8  steps_left=39
+    Step 14: FLAG acc_0022                       flagged=3/10  suspects=7  steps_left=39
+    Step 15: INSPECT acc_0092                    flagged=3/10  suspects=7  steps_left=38
+    Step 16: FLAG acc_0092                       flagged=4/10  suspects=6  steps_left=38
+    Step 17: INSPECT acc_0097                    flagged=4/10  suspects=6  steps_left=37
+    Step 18: FLAG acc_0097                       flagged=5/10  suspects=5  steps_left=37
+    Step 19: INSPECT acc_0187                    flagged=5/10  suspects=5  steps_left=36
+    Step 20: FLAG acc_0187                       flagged=6/10  suspects=4  steps_left=36
+    Step 21: INSPECT acc_0093                    flagged=6/10  suspects=4  steps_left=35
+    Step 22: FLAG acc_0093                       flagged=7/10  suspects=3  steps_left=35
+    Step 23: INSPECT acc_0172                    flagged=7/10  suspects=3  steps_left=34
+    Step 24: FLAG acc_0172                       flagged=8/10  suspects=2  steps_left=34
+    Step 25: INSPECT acc_0058                    flagged=8/10  suspects=2  steps_left=33
+    Step 26: FLAG acc_0058                       flagged=9/10  suspects=1  steps_left=33
+    Step 27: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=32
+    Step 28: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=32
+    Step 29: SUBMIT                              flagged=10/10  suspects=0  steps_left=32
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.82
+    ★ GRADER SCORE: 0.9640
+
+  --- LLM Agent: task=medium, seed=1, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0171                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0099                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0152                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0092                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0078                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0112                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0012                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: FLAG acc_0012                       flagged=1/10  suspects=8  steps_left=43
+    Step  9: INSPECT acc_0033                    flagged=1/10  suspects=8  steps_left=42
+    Step 10: FLAG acc_0033                       flagged=2/10  suspects=8  steps_left=42
+    Step 11: INSPECT acc_0174                    flagged=2/10  suspects=8  steps_left=41
+    Step 12: FLAG acc_0174                       flagged=3/10  suspects=7  steps_left=41
+    Step 13: INSPECT acc_0187                    flagged=3/10  suspects=7  steps_left=40
+    Step 14: FLAG acc_0187                       flagged=4/10  suspects=6  steps_left=40
+    Step 15: INSPECT acc_0079                    flagged=4/10  suspects=6  steps_left=39
+    Step 16: FLAG acc_0079                       flagged=5/10  suspects=5  steps_left=39
+    Step 17: INSPECT acc_0032                    flagged=5/10  suspects=5  steps_left=38
+    Step 18: FLAG acc_0032                       flagged=6/10  suspects=4  steps_left=38
+    Step 19: INSPECT acc_0023                    flagged=6/10  suspects=4  steps_left=37
+    Step 20: FLAG acc_0023                       flagged=7/10  suspects=3  steps_left=37
+    Step 21: INSPECT acc_0146                    flagged=7/10  suspects=3  steps_left=36
+    Step 22: FLAG acc_0146                       flagged=8/10  suspects=2  steps_left=36
+    Step 23: INSPECT acc_0019                    flagged=8/10  suspects=2  steps_left=35
+    Step 24: FLAG acc_0019                       flagged=9/10  suspects=1  steps_left=35
+    Step 25: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=34
+    Step 26: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=34
+    Step 27: SUBMIT                              flagged=10/10  suspects=0  steps_left=34
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.84
+    ★ GRADER SCORE: 0.9680
+
+  --- LLM Agent: task=medium, seed=2, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0107                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0175                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0030                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0041                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0054                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0199                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0181                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0166                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: INSPECT acc_0098                    flagged=0/10  suspects=0  steps_left=40
+    Step 11: INSPECT acc_0121                    flagged=0/10  suspects=0  steps_left=39
+    Step 12: INSPECT acc_0053                    flagged=0/10  suspects=0  steps_left=38
+    Step 13: INSPECT acc_0103                    flagged=0/10  suspects=0  steps_left=37
+    Step 14: INSPECT acc_0000                    flagged=0/10  suspects=0  steps_left=36
+    Step 15: INSPECT acc_0168                    flagged=0/10  suspects=0  steps_left=35
+    Step 16: INSPECT acc_0040                    flagged=0/10  suspects=0  steps_left=34
+    Step 17: INSPECT acc_0149                    flagged=0/10  suspects=0  steps_left=33
+    Step 18: INSPECT acc_0064                    flagged=0/10  suspects=0  steps_left=32
+    Step 19: INSPECT acc_0016                    flagged=0/10  suspects=0  steps_left=31
+    Step 20: INSPECT acc_0105                    flagged=0/10  suspects=0  steps_left=30
+    Step 21: INSPECT acc_0035                    flagged=0/10  suspects=0  steps_left=29
+    Step 22: FLAG acc_0035                       flagged=1/10  suspects=9  steps_left=29
+    Step 23: INSPECT acc_0020                    flagged=1/10  suspects=9  steps_left=28
+    Step 24: FLAG acc_0020                       flagged=2/10  suspects=8  steps_left=28
+    Step 25: INSPECT acc_0036                    flagged=2/10  suspects=8  steps_left=27
+    Step 26: FLAG acc_0036                       flagged=3/10  suspects=7  steps_left=27
+    Step 27: INSPECT acc_0050                    flagged=3/10  suspects=7  steps_left=26
+    Step 28: FLAG acc_0050                       flagged=4/10  suspects=6  steps_left=26
+    Step 29: INSPECT acc_0051                    flagged=4/10  suspects=6  steps_left=25
+    Step 30: FLAG acc_0051                       flagged=5/10  suspects=5  steps_left=25
+    Step 31: INSPECT acc_0085                    flagged=5/10  suspects=5  steps_left=24
+    Step 32: FLAG acc_0085                       flagged=6/10  suspects=4  steps_left=24
+    Step 33: INSPECT acc_0177                    flagged=6/10  suspects=4  steps_left=23
+    Step 34: FLAG acc_0177                       flagged=7/10  suspects=3  steps_left=23
+    Step 35: INSPECT acc_0170                    flagged=7/10  suspects=3  steps_left=22
+    Step 36: FLAG acc_0170                       flagged=8/10  suspects=2  steps_left=22
+    Step 37: INSPECT acc_0055                    flagged=8/10  suspects=2  steps_left=21
+    Step 38: FLAG acc_0055                       flagged=9/10  suspects=1  steps_left=21
+    Step 39: INSPECT acc_0094                    flagged=9/10  suspects=1  steps_left=20
+    Step 40: FLAG acc_0094                       flagged=10/10  suspects=0  steps_left=20
+    Step 41: SUBMIT                              flagged=10/10  suspects=0  steps_left=20
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.70
+    ★ GRADER SCORE: 0.9400
+
+  medium: scores=['0.964', '0.968', '0.940']  mean=0.9573  var=0.000153
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0441                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0871                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0070                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0443                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0037                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0438                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: FLAG acc_0439                       flagged=1/10  suspects=5  steps_left=60
+    Step 22: INSPECT acc_0237                    flagged=1/10  suspects=5  steps_left=59
+    Step 23: FLAG acc_0237                       flagged=2/10  suspects=6  steps_left=59
+    Step 24: INSPECT acc_0621                    flagged=2/10  suspects=6  steps_left=58
+    Step 25: FLAG acc_0621                       flagged=3/10  suspects=6  steps_left=58
+    Step 26: INSPECT acc_0389                    flagged=3/10  suspects=6  steps_left=57
+    Step 27: INSPECT acc_0160                    flagged=3/10  suspects=6  steps_left=56
+    Step 28: FLAG acc_0389                       flagged=4/10  suspects=6  steps_left=56
+    Step 29: FLAG acc_0160                       flagged=5/10  suspects=5  steps_left=56
+    Step 30: INSPECT acc_0549                    flagged=5/10  suspects=5  steps_left=55
+    Step 31: FLAG acc_0549                       flagged=6/10  suspects=4  steps_left=55
+    Step 32: INSPECT acc_0658                    flagged=6/10  suspects=4  steps_left=54
+    Step 33: FLAG acc_0658                       flagged=7/10  suspects=3  steps_left=54
+    Step 34: INSPECT acc_0290                    flagged=7/10  suspects=3  steps_left=53
+    Step 35: FLAG acc_0290                       flagged=8/10  suspects=2  steps_left=53
+    Step 36: INSPECT acc_0124                    flagged=8/10  suspects=2  steps_left=52
+    Step 37: INSPECT acc_0507                    flagged=8/10  suspects=2  steps_left=51
+    Step 38: FLAG acc_0124                       flagged=9/10  suspects=1  steps_left=51
+    Step 39: FLAG acc_0507                       flagged=10/10  suspects=0  steps_left=51
+    Step 40: SUBMIT                              flagged=10/10  suspects=0  steps_left=51
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.71
+    ★ GRADER SCORE: 0.9637
+
+  --- LLM Agent: task=hard, seed=1, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0014                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0835                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0855                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0930                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0336                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0929                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0076                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0543                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0590                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0401                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0322                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0154                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0374                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0549                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0903                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0976                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0620                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0017                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0222                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0536                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: INSPECT acc_0112                    flagged=0/10  suspects=0  steps_left=59
+    Step 22: INSPECT acc_0577                    flagged=0/10  suspects=0  steps_left=58
+    Step 23: INSPECT acc_0517                    flagged=0/10  suspects=0  steps_left=57
+    Step 24: INSPECT acc_0113                    flagged=0/10  suspects=0  steps_left=56
+    Step 25: INSPECT acc_0167                    flagged=0/10  suspects=0  steps_left=55
+    Step 26: INSPECT acc_0697                    flagged=0/10  suspects=0  steps_left=54
+    Step 27: INSPECT acc_0271                    flagged=0/10  suspects=0  steps_left=53
+    Step 28: INSPECT acc_0681                    flagged=0/10  suspects=0  steps_left=52
+    Step 29: INSPECT acc_0530                    flagged=0/10  suspects=0  steps_left=51
+    Step 30: INSPECT acc_0353                    flagged=0/10  suspects=0  steps_left=50
+    Step 31: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=49
+    Step 32: INSPECT acc_0777                    flagged=0/10  suspects=0  steps_left=48
+    Step 33: INSPECT acc_0265                    flagged=0/10  suspects=0  steps_left=47
+    Step 34: INSPECT acc_0788                    flagged=0/10  suspects=0  steps_left=46
+    Step 35: INSPECT acc_0033                    flagged=0/10  suspects=0  steps_left=45
+    Step 36: INSPECT acc_0187                    flagged=0/10  suspects=0  steps_left=44
+    Step 37: INSPECT acc_0445                    flagged=0/10  suspects=0  steps_left=43
+    Step 38: INSPECT acc_0846                    flagged=0/10  suspects=0  steps_left=42
+    Step 39: INSPECT acc_0659                    flagged=0/10  suspects=0  steps_left=41
+    Step 40: INSPECT acc_0768                    flagged=0/10  suspects=0  steps_left=40
+    Step 41: INSPECT acc_0677                    flagged=0/10  suspects=0  steps_left=39
+    Step 42: INSPECT acc_0539                    flagged=0/10  suspects=0  steps_left=38
+    Step 43: INSPECT acc_0742                    flagged=0/10  suspects=0  steps_left=37
+    Step 44: INSPECT acc_0503                    flagged=0/10  suspects=0  steps_left=36
+    Step 45: INSPECT acc_0876                    flagged=0/10  suspects=0  steps_left=35
+    Step 46: INSPECT acc_0639                    flagged=0/10  suspects=0  steps_left=34
+    Step 47: INSPECT acc_0494                    flagged=0/10  suspects=0  steps_left=33
+    Step 48: INSPECT acc_0898                    flagged=0/10  suspects=0  steps_left=32
+    Step 49: INSPECT acc_0553                    flagged=0/10  suspects=0  steps_left=31
+    Step 50: INSPECT acc_0588                    flagged=0/10  suspects=0  steps_left=30
+    Step 51: INSPECT acc_0194                    flagged=0/10  suspects=0  steps_left=29
+    Step 52: INSPECT acc_0810                    flagged=0/10  suspects=0  steps_left=28
+    Step 53: INSPECT acc_0355                    flagged=0/10  suspects=0  steps_left=27
+    Step 54: INSPECT acc_0363                    flagged=0/10  suspects=0  steps_left=26
+    Step 55: INSPECT acc_0221                    flagged=0/10  suspects=0  steps_left=25
+    Step 56: INSPECT acc_0580                    flagged=0/10  suspects=0  steps_left=24
+    Step 57: INSPECT acc_0534                    flagged=0/10  suspects=0  steps_left=23
+    Step 58: INSPECT acc_0778                    flagged=0/10  suspects=0  steps_left=22
+    Step 59: INSPECT acc_0998                    flagged=0/10  suspects=0  steps_left=21
+    Step 60: INSPECT acc_0233                    flagged=0/10  suspects=0  steps_left=20
+    Step 61: INSPECT acc_0052                    flagged=0/10  suspects=0  steps_left=19
+    Step 62: INSPECT acc_0813                    flagged=0/10  suspects=0  steps_left=18
+    Step 63: INSPECT acc_0035                    flagged=0/10  suspects=0  steps_left=17
+    Step 64: INSPECT acc_0667                    flagged=0/10  suspects=0  steps_left=16
+    Step 65: INSPECT acc_0019                    flagged=0/10  suspects=0  steps_left=15
+    Step 66: INSPECT acc_0959                    flagged=0/10  suspects=0  steps_left=14
+    Step 67: INSPECT acc_0212                    flagged=0/10  suspects=0  steps_left=13
+    Step 68: INSPECT acc_0776                    flagged=0/10  suspects=0  steps_left=12
+    Step 69: INSPECT acc_0049                    flagged=0/10  suspects=0  steps_left=11
+    Step 70: INSPECT acc_0434                    flagged=0/10  suspects=0  steps_left=10
+    Step 71: INSPECT acc_0827                    flagged=0/10  suspects=0  steps_left=9
+    Step 72: INSPECT acc_0583                    flagged=0/10  suspects=0  steps_left=8
+    Step 73: INSPECT acc_0065                    flagged=0/10  suspects=0  steps_left=7
+    Step 74: INSPECT acc_0107                    flagged=0/10  suspects=0  steps_left=6
+    Step 75: INSPECT acc_0761                    flagged=0/10  suspects=0  steps_left=5
+    Step 76: INSPECT acc_0995                    flagged=0/10  suspects=0  steps_left=4
+    Step 77: INSPECT acc_0157                    flagged=0/10  suspects=0  steps_left=3
+    Step 78: INSPECT acc_0936                    flagged=0/10  suspects=0  steps_left=2
+    Step 79: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=1
+    Step 80: INSPECT acc_0691                    flagged=0/10  suspects=0  steps_left=0
+    → Episode ended: [LOSS] TP=0 FP=0 FN=10 Recall=0.00 Precision=0.00 Episode reward=-9.80
+    ★ GRADER SCORE: 0.0000
+
+  --- LLM Agent: task=hard, seed=2, model=Bedrock/qwen.qwen3-next-80b-a3b ---
+    Step  1: INSPECT acc_0813                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0430                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0817                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0175                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0523                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0113                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0797                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0478                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0861                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0836                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0926                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0664                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0255                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0938                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0672                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: FLAG acc_0672                       flagged=1/10  suspects=6  steps_left=65
+    Step 17: INSPECT acc_0659                    flagged=1/10  suspects=6  steps_left=64
+    Step 18: FLAG acc_0659                       flagged=2/10  suspects=5  steps_left=64
+    Step 19: INSPECT acc_0290                    flagged=2/10  suspects=5  steps_left=63
+    Step 20: FLAG acc_0290                       flagged=3/10  suspects=5  steps_left=63
+    Step 21: INSPECT acc_0339                    flagged=3/10  suspects=5  steps_left=62
+    Step 22: FLAG acc_0339                       flagged=4/10  suspects=6  steps_left=62
+    Step 23: INSPECT acc_0544                    flagged=4/10  suspects=6  steps_left=61
+    Step 24: FLAG acc_0544                       flagged=5/10  suspects=5  steps_left=61
+    Step 25: INSPECT acc_0696                    flagged=5/10  suspects=5  steps_left=60
+    Step 26: FLAG acc_0696                       flagged=6/10  suspects=4  steps_left=60
+    Step 27: INSPECT acc_0541                    flagged=6/10  suspects=4  steps_left=59
+    Step 28: FLAG acc_0541                       flagged=7/10  suspects=3  steps_left=59
+    Step 29: INSPECT acc_0793                    flagged=7/10  suspects=3  steps_left=58
+    Step 30: FLAG acc_0793                       flagged=8/10  suspects=2  steps_left=58
+    Step 31: INSPECT acc_0214                    flagged=8/10  suspects=2  steps_left=57
+    Step 32: FLAG acc_0214                       flagged=9/10  suspects=1  steps_left=57
+    Step 33: INSPECT acc_0112                    flagged=9/10  suspects=1  steps_left=56
+    Step 34: FLAG acc_0112                       flagged=10/10  suspects=0  steps_left=56
+    Step 35: SUBMIT                              flagged=10/10  suspects=0  steps_left=56
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.76
+    ★ GRADER SCORE: 0.9700
+
+  hard: scores=['0.964', '0.000', '0.970']  mean=0.6446  var=0.207740
+
+============================================================
+EVALUATION COMPLETE
+============================================================
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26$

memory/reflections_easy.jsonl

@@ -1 +1 @@
-{"episode": 1, "reward": 18.88, "reflection": "Started by inspecting multiple low-profile accounts to identify early signals like comment_repeat_score and photo_reuse_score, which revealed coordinated spam patterns. Once a few clear gang members were flagged, their auto-become SUSPECT neighbors were rapidly confirmed and flagged, leveraging network propagation to efficiently uncover all 10 fake accounts without false positives. Always begin with targeted INSPECTs before FLAGging to validate signals and avoid premature assumptions."}
+{"episode": 1, "reward": 18.88, "reflection": "Starting with direct INSPECTs on low-numbered accounts revealed multiple gang members with high fake_risk_score, photo_reuse_score, and comment_repeat_score, confirming a coordinated pattern. Once the first few were FLAGGED, their auto-revealed neighbors automatically became high-priority SUSPECTs, allowing rapid expansion without extra steps\u2014this network-triggered propagation was key to 100% recall with zero false positives. Always begin with targeted INSPECTs to identify clear gang markers, then leverage FLAGging to unlock the full network efficiently."}

memory/reflections_hard.jsonl

@@ -1,3 +1,3 @@
-{"episode": 1, "reward": 17.84, "reflection": "We successfully identified the gang by first inspecting 10 low-suspicion accounts to establish baseline behavior, then flagged those showing high comment_repeat_score, photo_reuse_score, and bio_template_score \u2014 all clustering around shared_ip_count=9. The key was not jumping to flag early, but using initial inspections to confirm coordinated patterns before triggering network suspects, which automatically revealed the full gang without additional steps. Always verify multiple signals together before flagging \u2014 single indicators can be false positives."}
-{"episode": 2, "reward": 18.76, "reflection": "Starting with targeted INSPECTs on low-suspicion accounts revealed hidden patterns: multiple accounts shared identical bio templates and reused profile photos, exposing a coordinated gang. Once a single account with high fake_risk_score and shared_ip_count was confirmed, INVESTIGATE_NETWORK was used to expand to all 2-hop neighbors, uncovering the full network without false positives. Repeating this method\u2014inspecting first to validate signals before network expansion\u2014ensured precision and maximized recall."}
-{"episode": 3, "reward": 18.84, "reflection": "Started by inspecting multiple low-suspicion accounts to map the network, identifying key gang members through high comment_repeat_score, shared_ip_count, and bio_template_score. Once five clear gang members were confirmed, flagging them triggered automatic SUSPECT status on their neighbors, revealing the full cluster without unnecessary deep dives\u2014repeat this targeted inspection-to-flag pipeline to maximize coverage with minimal steps."}
+{"episode": 1, "reward": 17.84, "reflection": "The strategy succeeded by first INSPECTing 10 low-suspicion accounts to map the network\u2019s baseline behavior, confirming that comment_repeat_score and photo_reuse_score consistently spiked in gang members. Once five accounts showed matching red flags, FLAGging them triggered auto-suspect propagation, revealing the full gang without unnecessary INVESTIGATE_NETWORK steps\u2014this minimized steps while maximizing coverage. Future cases should always begin with targeted INSPECTs to validate signals before FLAGging, avoiding premature network expansion."}
+{"episode": 2, "reward": 18.76, "reflection": "By systematically INSPECTing high-risk accounts with elevated fake_risk_score and multiple red flags\u2014comment_repeat_score, shared_ip_count, and photo_reuse_score\u2014we uncovered a tightly coordinated gang operating under identical bio templates. Each confirmed fake account revealed neighbors with identical patterns, making network expansion unnecessary; we prioritized direct inspection over speculative INVESTIGATE_NETWORK, conserving steps while maximizing precision. Future cases should trust low-step inspection of high-signal accounts first, letting flagged neighbors naturally expose the rest."}
+{"episode": 3, "reward": 18.84, "reflection": "We successfully identified the gang by first inspecting 10 low-suspicion accounts to establish baseline behavior, then flagged those showing high comment_repeat_score, shared_ip_count, and photo_reuse_score \u2014 all consistent with coordinated fake accounts. The key was waiting to flag until multiple signals aligned, which prevented false positives and triggered auto-suspect expansion, revealing the full network without unnecessary steps. Always inspect first to calibrate, then flag only when signals converge."}

memory/reflections_medium.jsonl

@@ -1 +1 @@
-{"episode": 1, "reward": 18.86, "reflection": "The strategy succeeded by first INSPECTing multiple low-suspicion accounts to identify early signals like comment_repeat_score and photo_reuse_score, which revealed coordinated spam patterns. Once a few accounts with high fake_risk_score and shared_ip_count were flagged, the auto-triggered SUSPECT neighbors were immediately flagged without further inspection, efficiently expanding the takedown. Always start with targeted INSPECTs to map the network before FLAGging\u2014this minimizes steps and maximizes recall."}
+{"episode": 1, "reward": 18.86, "reflection": "Starting with targeted INSPECTs on low-activity accounts revealed multiple gang members with high comment_repeat_score, photo_reuse_score, and bio_template_score \u2014 all clustered under shared_ip_count > 5. Flagging the first five confirmed gang nodes triggered auto-suspect propagation, letting us identify the full network without unnecessary network expansion. Always begin with direct inspection of suspicious profiles before network moves \u2014 this minimizes steps and maximizes precision."}

memory/wins_easy.jsonl

@@ -8,3 +8,43 @@
 {"episode": 8, "won": true}
 {"episode": 9, "won": true}
 {"episode": 10, "won": true}
+{"episode": 11, "won": true}
+{"episode": 12, "won": true}
+{"episode": 13, "won": true}
+{"episode": 14, "won": true}
+{"episode": 15, "won": true}
+{"episode": 16, "won": true}
+{"episode": 17, "won": true}
+{"episode": 18, "won": true}
+{"episode": 19, "won": true}
+{"episode": 20, "won": true}
+{"episode": 21, "won": true}
+{"episode": 22, "won": true}
+{"episode": 23, "won": true}
+{"episode": 24, "won": true}
+{"episode": 25, "won": true}
+{"episode": 26, "won": true}
+{"episode": 27, "won": true}
+{"episode": 28, "won": true}
+{"episode": 29, "won": true}
+{"episode": 30, "won": true}
+{"episode": 31, "won": true}
+{"episode": 32, "won": true}
+{"episode": 33, "won": true}
+{"episode": 34, "won": true}
+{"episode": 35, "won": true}
+{"episode": 36, "won": true}
+{"episode": 37, "won": true}
+{"episode": 38, "won": true}
+{"episode": 39, "won": true}
+{"episode": 40, "won": true}
+{"episode": 41, "won": true}
+{"episode": 42, "won": true}
+{"episode": 43, "won": true}
+{"episode": 44, "won": true}
+{"episode": 45, "won": true}
+{"episode": 46, "won": true}
+{"episode": 47, "won": true}
+{"episode": 48, "won": true}
+{"episode": 49, "won": true}
+{"episode": 50, "won": true}

memory/wins_hard.jsonl

@@ -8,3 +8,28 @@
 {"episode": 8, "won": true}
 {"episode": 9, "won": true}
 {"episode": 10, "won": true}
+{"episode": 11, "won": true}
+{"episode": 12, "won": true}
+{"episode": 13, "won": true}
+{"episode": 14, "won": true}
+{"episode": 15, "won": true}
+{"episode": 16, "won": true}
+{"episode": 17, "won": true}
+{"episode": 18, "won": true}
+{"episode": 19, "won": true}
+{"episode": 20, "won": true}
+{"episode": 21, "won": true}
+{"episode": 22, "won": true}
+{"episode": 23, "won": true}
+{"episode": 24, "won": true}
+{"episode": 25, "won": true}
+{"episode": 26, "won": true}
+{"episode": 27, "won": true}
+{"episode": 28, "won": true}
+{"episode": 29, "won": true}
+{"episode": 30, "won": true}
+{"episode": 31, "won": true}
+{"episode": 32, "won": true}
+{"episode": 33, "won": true}
+{"episode": 34, "won": true}
+{"episode": 35, "won": true}

memory/wins_medium.jsonl

@@ -1,20 +1,44 @@
 {"episode": 1, "won": true}
-{"episode": 1, "won": true}
-{"episode": 2, "won": true}
 {"episode": 2, "won": true}
 {"episode": 3, "won": true}
-{"episode": 3, "won": true}
 {"episode": 4, "won": true}
-{"episode": 4, "won": true}
-{"episode": 5, "won": true}
 {"episode": 5, "won": true}
 {"episode": 6, "won": true}
-{"episode": 6, "won": true}
-{"episode": 7, "won": true}
 {"episode": 7, "won": true}
 {"episode": 8, "won": true}
-{"episode": 8, "won": true}
 {"episode": 9, "won": true}
-{"episode": 9, "won": true}
-{"episode": 10, "won": true}
 {"episode": 10, "won": true}
+{"episode": 11, "won": true}
+{"episode": 12, "won": true}
+{"episode": 13, "won": true}
+{"episode": 14, "won": true}
+{"episode": 15, "won": true}
+{"episode": 16, "won": true}
+{"episode": 17, "won": true}
+{"episode": 18, "won": true}
+{"episode": 19, "won": true}
+{"episode": 20, "won": true}
+{"episode": 21, "won": true}
+{"episode": 22, "won": true}
+{"episode": 23, "won": true}
+{"episode": 24, "won": true}
+{"episode": 25, "won": true}
+{"episode": 26, "won": true}
+{"episode": 27, "won": true}
+{"episode": 28, "won": true}
+{"episode": 29, "won": true}
+{"episode": 30, "won": true}
+{"episode": 31, "won": true}
+{"episode": 32, "won": true}
+{"episode": 33, "won": true}
+{"episode": 34, "won": true}
+{"episode": 35, "won": true}
+{"episode": 36, "won": true}
+{"episode": 37, "won": true}
+{"episode": 38, "won": true}
+{"episode": 39, "won": true}
+{"episode": 40, "won": true}
+{"episode": 41, "won": true}
+{"episode": 42, "won": true}
+{"episode": 43, "won": true}
+{"episode": 44, "won": true}

model-benchmark-logs/deepseek_judge_log.txt

@@ -0,0 +1,749 @@
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26$ python3 deepseek_test_judge_eval.py --url https:
+//pandago-graphstrike.hf.space --bedrock
+GraphStrike Judge Evaluation Simulator
+Target:  https://pandago-graphstrike.hf.space
+Backend: bedrock
+Model:   Bedrock/deepseek.v3.2
+Token:   set
+
+============================================================
+PHASE 0: Endpoint Verification
+============================================================
+  ✓ GET /health
+  ✓ GET /tasks
+  ✓ GET /metadata
+  ✓ GET /schema
+  ✓ GET /web
+  ✓ POST /reset
+  ✓ GET /state
+  ✓ POST /step
+  ✓ POST /step
+  ✓ GET /grader
+  ✓ POST /mcp
+  ✓ POST /baseline
+
+============================================================
+PHASE 1: Baseline Stability (3 runs)
+============================================================
+  Run 1: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 2: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 3: easy=0.9100  medium=0.9060  hard=0.9038
+  ✓ All 3 runs identical — baseline is deterministic
+
+============================================================
+PHASE 2: LLM Agent Evaluation (model=Bedrock/deepseek.v3.2)
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=29
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=28
+    Step  4: INSPECT acc_0001                    flagged=1/10  suspects=8  steps_left=27
+    Step  5: INSPECT acc_0014                    flagged=1/10  suspects=8  steps_left=26
+    Step  6: INSPECT acc_0012                    flagged=1/10  suspects=8  steps_left=25
+    Step  7: INSPECT acc_0000                    flagged=1/10  suspects=8  steps_left=24
+    Step  8: FLAG acc_0001                       flagged=2/10  suspects=8  steps_left=24
+    Step  9: INSPECT acc_0027                    flagged=2/10  suspects=8  steps_left=23
+    Step 10: FLAG acc_0027                       flagged=3/10  suspects=7  steps_left=23
+    Step 11: INSPECT acc_0047                    flagged=3/10  suspects=7  steps_left=22
+    Step 12: FLAG acc_0047                       flagged=4/10  suspects=6  steps_left=22
+    Step 13: INSPECT acc_0007                    flagged=4/10  suspects=6  steps_left=21
+    Step 14: FLAG acc_0014                       flagged=5/10  suspects=5  steps_left=21
+    Step 15: INSPECT acc_0028                    flagged=5/10  suspects=5  steps_left=20
+    Step 16: FLAG acc_0000                       flagged=6/10  suspects=4  steps_left=20
+    Step 17: FLAG acc_0007                       flagged=7/10  suspects=3  steps_left=20
+    Step 18: FLAG acc_0012                       flagged=8/10  suspects=2  steps_left=20
+    Step 19: FLAG acc_0036                       flagged=9/10  suspects=1  steps_left=20
+    Step 20: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=20
+    Step 21: SUBMIT                              flagged=10/10  suspects=0  steps_left=20
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.90
+    ★ GRADER SCORE: 0.9667
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=41
+    Step 11: INSPECT acc_0181                    flagged=2/10  suspects=8  steps_left=21
+    Step 12: INSPECT acc_0028                    flagged=3/10  suspects=7  steps_left=19
+    Step 13: FLAG acc_0047                       flagged=5/10  suspects=5  steps_left=19
+    Step 14: FLAG acc_0007                       flagged=7/10  suspects=3  steps_left=19
+    Step 15: FLAG acc_0012                       flagged=8/10  suspects=2  steps_left=19
+    Step 16: FLAG acc_0036                       flagged=9/10  suspects=1  steps_left=19
+    Step 17: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=19
+    Step 18: SUBMIT                              flagged=10/10  suspects=0  steps_left=18
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.88
+    ★ GRADER SCORE: 0.9600
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=76
+    Step  2: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=74
+    Step  3: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=70
+    Step  4: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=67
+    Step  5: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=64
+    Step  6: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=61
+    Step  7: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=59
+    Step  8: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=58
+    Step  9: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=55
+    Step 10: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=52
+    Step 11: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=49
+    Step 12: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=47
+    Step 13: FLAG acc_0439                       flagged=2/10  suspects=6  steps_left=45
+    Step 14: INSPECT acc_0389                    flagged=3/10  suspects=6  steps_left=44
+    Step 15: INSPECT acc_0160                    flagged=3/10  suspects=6  steps_left=41
+    Step 16: FLAG acc_0160                       flagged=4/10  suspects=6  steps_left=40
+    Step 17: INSPECT acc_0658                    flagged=6/10  suspects=4  steps_left=39
+    Step 18: INSPECT acc_0290                    flagged=10/10  suspects=0  steps_left=27
+    Step 19: SUBMIT                              flagged=10/10  suspects=0  steps_left=26
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=14.46
+    ★ GRADER SCORE: 0.9325
+
+  Summary: easy=0.9667  medium=0.9600  hard=0.9325
+
+============================================================
+PHASE 3: Score Variance (seeds=[0, 1, 2])
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0043                    flagged=10/10  suspects=0  steps_left=17
+    Step  2: SUBMIT                              flagged=10/10  suspects=0  steps_left=10
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.80
+    ★ GRADER SCORE: 0.9333
+
+  --- LLM Agent: task=easy, seed=1, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0034                    flagged=1/10  suspects=8  steps_left=28
+    Step  2: INSPECT acc_0047                    flagged=1/10  suspects=8  steps_left=26
+    Step  3: INSPECT acc_0002                    flagged=1/10  suspects=8  steps_left=23
+    Step  4: FLAG acc_0047                       flagged=2/10  suspects=8  steps_left=22
+    Step  5: INSPECT acc_0029                    flagged=4/10  suspects=6  steps_left=19
+    Step  6: FLAG acc_0048                       flagged=5/10  suspects=5  steps_left=18
+    Step  7: FLAG acc_0005                       flagged=6/10  suspects=4  steps_left=17
+    Step  8: FLAG acc_0046                       flagged=8/10  suspects=2  steps_left=17
+    Step  9: INSPECT acc_0014                    flagged=9/10  suspects=1  steps_left=16
+    Step 10: FLAG acc_0014                       flagged=10/10  suspects=0  steps_left=15
+    Step 11: SUBMIT                              flagged=10/10  suspects=0  steps_left=12
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.82
+    ★ GRADER SCORE: 0.9400
+
+  --- LLM Agent: task=easy, seed=2, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0040                    flagged=0/10  suspects=0  steps_left=17
+    Step  2: FLAG acc_0033                       flagged=1/10  suspects=9  steps_left=17
+    Step  3: INSPECT acc_0029                    flagged=6/10  suspects=4  steps_left=7
+    Step  4: FLAG acc_0015                       flagged=8/10  suspects=2  steps_left=7
+    Step  5: FLAG acc_0004                       flagged=10/10  suspects=0  steps_left=7
+    Step  6: SUBMIT                              flagged=0/10  suspects=0  steps_left=48
+    → Episode ended: [LOSS] TP=0 FP=0 FN=10 Recall=0.00 Precision=0.00 Episode reward=-2.02
+    ★ GRADER SCORE: 0.0000
+
+  easy: scores=['0.933', '0.940', '0.000']  mean=0.6244  var=0.194966
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=46
+    Step  2: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=49
+    Step  3: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=29
+    Step  4: INSPECT acc_0043                    flagged=1/10  suspects=9  steps_left=47
+    Step  5: INSPECT acc_0144                    flagged=1/10  suspects=9  steps_left=44
+    Step  6: INSPECT acc_0131                    flagged=3/10  suspects=7  steps_left=41
+    Step  7: INSPECT acc_0181                    flagged=5/10  suspects=5  steps_left=36
+    Step  8: INSPECT acc_0097                    flagged=6/10  suspects=4  steps_left=35
+    Step  9: INSPECT acc_0187                    flagged=7/10  suspects=3  steps_left=33
+    Step 10: FLAG acc_0187                       flagged=8/10  suspects=2  steps_left=32
+    Step 11: INSPECT acc_0058                    flagged=8/10  suspects=2  steps_left=30
+    Step 12: FLAG acc_0058                       flagged=9/10  suspects=1  steps_left=29
+    Step 13: FLAG acc_0093                       flagged=10/10  suspects=0  steps_left=27
+    Step 14: SUBMIT                              flagged=10/10  suspects=0  steps_left=26
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.76
+    ★ GRADER SCORE: 0.9520
+
+  --- LLM Agent: task=medium, seed=1, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0171                    flagged=0/10  suspects=0  steps_left=47
+    Step  2: INSPECT acc_0099                    flagged=0/10  suspects=0  steps_left=45
+    Step  3: INSPECT acc_0152                    flagged=0/10  suspects=0  steps_left=42
+    Step  4: INSPECT acc_0078                    flagged=0/10  suspects=0  steps_left=39
+    Step  5: INSPECT acc_0012                    flagged=0/10  suspects=0  steps_left=37
+    Step  6: FLAG acc_0012                       flagged=1/10  suspects=8  steps_left=36
+    Step  7: INSPECT acc_0174                    flagged=2/10  suspects=8  steps_left=34
+    Step  8: FLAG acc_0174                       flagged=3/10  suspects=7  steps_left=34
+    Step  9: INSPECT acc_0187                    flagged=10/10  suspects=0  steps_left=24
+    Step 10: SUBMIT                              flagged=10/10  suspects=0  steps_left=23
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.73
+    ★ GRADER SCORE: 0.9460
+
+  --- LLM Agent: task=medium, seed=2, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=47
+    Step  2: INSPECT acc_0107                    flagged=0/10  suspects=0  steps_left=45
+    Step  3: INSPECT acc_0175                    flagged=0/10  suspects=0  steps_left=43
+    Step  4: INSPECT acc_0030                    flagged=0/10  suspects=0  steps_left=40
+    Step  5: INSPECT acc_0054                    flagged=0/10  suspects=0  steps_left=38
+    Step  6: INSPECT acc_0199                    flagged=0/10  suspects=0  steps_left=35
+    Step  7: INSPECT acc_0166                    flagged=0/10  suspects=0  steps_left=32
+    Step  8: INSPECT acc_0082                    flagged=0/10  suspects=0  steps_left=15
+    Step  9: INSPECT acc_0016                    flagged=0/10  suspects=0  steps_left=28
+    Step 10: FLAG acc_0000                       flagged=10/10  suspects=0  steps_left=3
+    [RETRY] Network error: HTTP Error 400: Bad Request — retrying in 3s (1/3)
+    [RETRY] Network error: HTTP Error 400: Bad Request — retrying in 6s (2/3)
+Traceback (most recent call last):
+  File "/home/ubuntu/meta/meta-hack-26/deepseek_test_judge_eval.py", line 534, in <module>
+    test_variance(base, seeds=list(range(args.seeds)))
+  File "/home/ubuntu/meta/meta-hack-26/deepseek_test_judge_eval.py", line 458, in test_varia
+nce
+    score = test_llm_agent(base_url, task=task, seed=seed)
+            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/meta/meta-hack-26/deepseek_test_judge_eval.py", line 426, in test_llm_a
+gent
+    grader = http_get(f"{base_url}/grader")
+             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/home/ubuntu/meta/meta-hack-26/deepseek_test_judge_eval.py", line 116, in http_get
+    return _retry(_do)
+           ^^^^^^^^^^^
+  File "/home/ubuntu/meta/meta-hack-26/deepseek_test_judge_eval.py", line 85, in _retry
+    return fn()
+           ^^^^
+  File "/home/ubuntu/meta/meta-hack-26/deepseek_test_judge_eval.py", line 110, in _do
+    with urllib.request.urlopen(url, timeout=120) as resp:
+         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen
+    return opener.open(url, data, timeout)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/urllib/request.py", line 521, in open
+    response = meth(req, response)
+               ^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/urllib/request.py", line 630, in http_response
+    response = self.parent.error(
+               ^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/urllib/request.py", line 559, in error
+    return self._call_chain(*args)
+           ^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain
+    result = func(*args)
+             ^^^^^^^^^^^
+  File "/usr/lib/python3.12/urllib/request.py", line 639, in http_error_default
+    raise HTTPError(req.full_url, code, msg, hdrs, fp)
+urllib.error.HTTPError: HTTP Error 400: Bad Request
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26$ python3 deepseek_test_judge_eval.py --url https:
+//pandago-graphstrike.hf.space --bedrock
+GraphStrike Judge Evaluation Simulator
+Target:  https://pandago-graphstrike.hf.space
+Backend: bedrock
+Model:   Bedrock/deepseek.v3.2
+Token:   set
+
+============================================================
+PHASE 0: Endpoint Verification
+============================================================
+  ✓ GET /health
+  ✓ GET /tasks
+  ✓ GET /metadata
+  ✓ GET /schema
+  ✓ GET /web
+  ✓ POST /reset
+  ✓ GET /state
+  ✓ POST /step
+  ✓ POST /step
+  ✓ GET /grader
+  ✓ POST /mcp
+  ✓ POST /baseline
+
+============================================================
+PHASE 1: Baseline Stability (3 runs)
+============================================================
+  Run 1: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 2: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 3: easy=0.9100  medium=0.9060  hard=0.9038
+  ✓ All 3 runs identical — baseline is deterministic
+
+============================================================
+PHASE 2: LLM Agent Evaluation (model=Bedrock/deepseek.v3.2)
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=29
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=28
+    Step  4: INSPECT acc_0001                    flagged=1/10  suspects=8  steps_left=27
+    Step  5: INSPECT acc_0014                    flagged=1/10  suspects=8  steps_left=26
+    Step  6: INSPECT acc_0012                    flagged=1/10  suspects=8  steps_left=25
+    Step  7: INSPECT acc_0000                    flagged=1/10  suspects=8  steps_left=24
+    Step  8: FLAG acc_0001                       flagged=2/10  suspects=8  steps_left=24
+    Step  9: INSPECT acc_0027                    flagged=2/10  suspects=8  steps_left=23
+    Step 10: INSPECT acc_0047                    flagged=2/10  suspects=8  steps_left=22
+    Step 11: FLAG acc_0027                       flagged=3/10  suspects=7  steps_left=22
+    Step 12: INSPECT acc_0007                    flagged=3/10  suspects=7  steps_left=21
+    Step 13: FLAG acc_0047                       flagged=4/10  suspects=6  steps_left=21
+    Step 14: FLAG acc_0014                       flagged=1/10  suspects=1  steps_left=76
+    Step 15: INSPECT acc_0356                    flagged=1/10  suspects=1  steps_left=75
+    Step 16: INSPECT acc_0523                    flagged=1/10  suspects=1  steps_left=74
+    Step 17: INSPECT acc_0113                    flagged=1/10  suspects=1  steps_left=73
+    Step 18: INSPECT acc_0797                    flagged=1/10  suspects=1  steps_left=71
+    Step 19: INSPECT acc_0478                    flagged=1/10  suspects=1  steps_left=70
+    Step 20: INSPECT acc_0861                    flagged=1/10  suspects=1  steps_left=69
+    Step 21: INSPECT acc_0836                    flagged=1/10  suspects=1  steps_left=68
+    Step 22: INSPECT acc_0926                    flagged=1/10  suspects=1  steps_left=67
+    Step 23: INSPECT acc_0664                    flagged=1/10  suspects=1  steps_left=64
+    Step 24: INSPECT acc_0255                    flagged=1/10  suspects=1  steps_left=63
+    Step 25: INSPECT acc_0938                    flagged=1/10  suspects=1  steps_left=62
+    Step 26: INSPECT acc_0672                    flagged=1/10  suspects=1  steps_left=61
+    Step 27: FLAG acc_0672                       flagged=2/10  suspects=7  steps_left=61
+    Step 28: INSPECT acc_0659                    flagged=2/10  suspects=7  steps_left=60
+    Step 29: FLAG acc_0659                       flagged=3/10  suspects=6  steps_left=60
+    Step 30: INSPECT acc_0290                    flagged=3/10  suspects=6  steps_left=59
+    Step 31: INSPECT acc_0339                    flagged=3/10  suspects=6  steps_left=58
+    Step 32: INSPECT acc_0544                    flagged=3/10  suspects=6  steps_left=57
+    Step 33: INSPECT acc_0696                    flagged=3/10  suspects=6  steps_left=56
+    Step 34: INSPECT acc_0541                    flagged=3/10  suspects=6  steps_left=55
+    Step 35: FLAG acc_0290                       flagged=4/10  suspects=8  steps_left=55
+    Step 36: INSPECT acc_0793                    flagged=4/10  suspects=8  steps_left=54
+    Step 37: INSPECT acc_0214                    flagged=4/10  suspects=8  steps_left=53
+    Step 38: INSPECT acc_0112                    flagged=4/10  suspects=8  steps_left=52
+    Step 39: FLAG acc_0339                       flagged=5/10  suspects=7  steps_left=52
+    Step 40: FLAG acc_0112                       flagged=6/10  suspects=6  steps_left=52
+    Step 41: FLAG acc_0696                       flagged=7/10  suspects=5  steps_left=52
+    Step 42: FLAG acc_0544                       flagged=8/10  suspects=4  steps_left=52
+    Step 43: FLAG acc_0214                       flagged=9/10  suspects=3  steps_left=52
+    Step 44: FLAG acc_0793                       flagged=10/10  suspects=2  steps_left=52
+    Step 45: FLAG acc_0541                       flagged=11/10  suspects=1  steps_left=52
+    Step 46: SUBMIT                              flagged=11/10  suspects=1  steps_left=52
+    → Episode ended: [WIN] TP=10 FP=1 FN=0 Recall=1.00 Precision=0.91 Episode reward=17.22
+    ★ GRADER SCORE: 0.9514
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=40
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=40
+    Step 11: INSPECT acc_0181                    flagged=1/10  suspects=8  steps_left=38
+    Step 12: INSPECT acc_0022                    flagged=1/10  suspects=8  steps_left=37
+    Step 13: FLAG acc_0022                       flagged=2/10  suspects=8  steps_left=37
+    Step 14: INSPECT acc_0092                    flagged=2/10  suspects=8  steps_left=36
+    Step 15: INSPECT acc_0097                    flagged=2/10  suspects=8  steps_left=35
+    Step 16: FLAG acc_0097                       flagged=3/10  suspects=7  steps_left=35
+    Step 17: FLAG acc_0181                       flagged=4/10  suspects=6  steps_left=35
+    Step 18: INSPECT acc_0187                    flagged=4/10  suspects=6  steps_left=34
+    Step 19: INSPECT acc_0093                    flagged=4/10  suspects=6  steps_left=33
+    Step 20: FLAG acc_0093                       flagged=5/10  suspects=5  steps_left=33
+    Step 21: FLAG acc_0187                       flagged=6/10  suspects=4  steps_left=33
+    Step 22: INSPECT acc_0172                    flagged=6/10  suspects=4  steps_left=32
+    Step 23: FLAG acc_0172                       flagged=7/10  suspects=3  steps_left=32
+    Step 24: INSPECT acc_0058                    flagged=7/10  suspects=3  steps_left=31
+    Step 25: FLAG acc_0058                       flagged=8/10  suspects=2  steps_left=31
+    Step 26: INSPECT acc_0131                    flagged=8/10  suspects=2  steps_left=30
+    Step 27: FLAG acc_0131                       flagged=9/10  suspects=1  steps_left=30
+    Step 28: FLAG acc_0092                       flagged=10/10  suspects=0  steps_left=30
+    Step 29: SUBMIT                              flagged=10/10  suspects=0  steps_left=30
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.80
+    ★ GRADER SCORE: 0.9600
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0441                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0871                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0070                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0443                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0037                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0438                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: FLAG acc_0439                       flagged=1/10  suspects=5  steps_left=60
+    Step 22: INSPECT acc_0237                    flagged=1/10  suspects=5  steps_left=59
+    Step 23: INSPECT acc_0621                    flagged=1/10  suspects=5  steps_left=58
+    Step 24: FLAG acc_0621                       flagged=2/10  suspects=7  steps_left=58
+    Step 25: INSPECT acc_0389                    flagged=2/10  suspects=7  steps_left=57
+    Step 26: INSPECT acc_0160                    flagged=2/10  suspects=7  steps_left=56
+    Step 27: INSPECT acc_0549                    flagged=2/10  suspects=7  steps_left=55
+    Step 28: INSPECT acc_0658                    flagged=2/10  suspects=7  steps_left=54
+    Step 29: INSPECT acc_0290                    flagged=2/10  suspects=7  steps_left=53
+    Step 30: INSPECT acc_0124                    flagged=2/10  suspects=7  steps_left=52
+    Step 31: FLAG acc_0160                       flagged=3/10  suspects=7  steps_left=52
+    Step 32: INSPECT acc_0507                    flagged=3/10  suspects=7  steps_left=51
+    Step 33: FLAG acc_0237                       flagged=4/10  suspects=6  steps_left=51
+    Step 34: FLAG acc_0549                       flagged=5/10  suspects=5  steps_left=51
+    Step 35: FLAG acc_0290                       flagged=6/10  suspects=4  steps_left=51
+    Step 36: FLAG acc_0389                       flagged=8/10  suspects=10  steps_left=51
+    Step 37: INSPECT acc_0844                    flagged=9/10  suspects=9  steps_left=50
+    Step 38: INSPECT acc_0436                    flagged=9/10  suspects=9  steps_left=49
+    Step 39: INSPECT acc_0870                    flagged=9/10  suspects=9  steps_left=48
+    Step 40: FLAG acc_0507                       flagged=10/10  suspects=8  steps_left=48
+    Step 41: INSPECT acc_0142                    flagged=10/10  suspects=8  steps_left=47
+    Step 42: INSPECT acc_0833                    flagged=10/10  suspects=8  steps_left=46
+    Step 43: INSPECT acc_0271                    flagged=10/10  suspects=8  steps_left=45
+    Step 44: FLAG acc_0124                       flagged=11/10  suspects=7  steps_left=45
+    Step 45: SUBMIT                              flagged=11/10  suspects=7  steps_left=45
+    → Episode ended: [WIN] TP=10 FP=1 FN=0 Recall=1.00 Precision=0.91 Episode reward=16.15
+    ★ GRADER SCORE: 0.9426
+
+  Summary: easy=0.9514  medium=0.9600  hard=0.9426
+
+============================================================
+PHASE 3: Score Variance (seeds=[0, 1, 2])
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=29
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=28
+    Step  4: INSPECT acc_0001                    flagged=1/10  suspects=8  steps_left=27
+    Step  5: INSPECT acc_0014                    flagged=1/10  suspects=8  steps_left=26
+    Step  6: INSPECT acc_0012                    flagged=1/10  suspects=8  steps_left=25
+    Step  7: INSPECT acc_0000                    flagged=1/10  suspects=8  steps_left=24
+    Step  8: FLAG acc_0001                       flagged=2/10  suspects=8  steps_left=24
+    Step  9: INSPECT acc_0027                    flagged=2/10  suspects=8  steps_left=23
+    Step 10: FLAG acc_0027                       flagged=3/10  suspects=7  steps_left=23
+    Step 11: INSPECT acc_0047                    flagged=3/10  suspects=7  steps_left=22
+    Step 12: FLAG acc_0047                       flagged=4/10  suspects=6  steps_left=22
+    Step 13: INSPECT acc_0007                    flagged=4/10  suspects=6  steps_left=21
+    Step 14: INSPECT acc_0028                    flagged=4/10  suspects=6  steps_left=20
+    Step 15: FLAG acc_0014                       flagged=5/10  suspects=5  steps_left=20
+    Step 16: FLAG acc_0007                       flagged=6/10  suspects=4  steps_left=20
+    Step 17: FLAG acc_0000                       flagged=7/10  suspects=3  steps_left=20
+    Step 18: FLAG acc_0012                       flagged=8/10  suspects=2  steps_left=20
+    Step 19: FLAG acc_0036                       flagged=9/10  suspects=1  steps_left=20
+    Step 20: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=20
+    Step 21: SUBMIT                              flagged=10/10  suspects=0  steps_left=20
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.90
+    ★ GRADER SCORE: 0.9667
+
+  --- LLM Agent: task=easy, seed=1, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0034                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: INSPECT acc_0003                    flagged=0/10  suspects=0  steps_left=28
+    Step  3: INSPECT acc_0049                    flagged=0/10  suspects=0  steps_left=27
+    Step  4: INSPECT acc_0006                    flagged=0/10  suspects=0  steps_left=26
+    Step  5: INSPECT acc_0047                    flagged=0/10  suspects=0  steps_left=25
+    Step  6: FLAG acc_0047                       flagged=1/10  suspects=9  steps_left=25
+    Step  7: INSPECT acc_0009                    flagged=1/10  suspects=9  steps_left=24
+    Step  8: INSPECT acc_0046                    flagged=1/10  suspects=9  steps_left=23
+    Step  9: FLAG acc_0009                       flagged=2/10  suspects=8  steps_left=23
+    Step 10: FLAG acc_0046                       flagged=3/10  suspects=7  steps_left=23
+    Step 11: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=22
+    Step 12: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=22
+    Step 13: INSPECT acc_0021                    flagged=4/10  suspects=6  steps_left=21
+    Step 14: FLAG acc_0021                       flagged=5/10  suspects=5  steps_left=21
+    Step 15: INSPECT acc_0002                    flagged=5/10  suspects=5  steps_left=20
+    Step 16: FLAG acc_0002                       flagged=6/10  suspects=4  steps_left=20
+    Step 17: INSPECT acc_0048                    flagged=6/10  suspects=4  steps_left=19
+    Step 18: INSPECT acc_0029                    flagged=6/10  suspects=4  steps_left=18
+    Step 19: FLAG acc_0029                       flagged=7/10  suspects=3  steps_left=18
+    Step 20: INSPECT acc_0015                    flagged=7/10  suspects=3  steps_left=17
+    Step 21: FLAG acc_0015                       flagged=8/10  suspects=2  steps_left=17
+    Step 22: FLAG acc_0048                       flagged=9/10  suspects=1  steps_left=17
+    Step 23: INSPECT acc_0005                    flagged=9/10  suspects=1  steps_left=16
+    Step 24: FLAG acc_0005                       flagged=10/10  suspects=0  steps_left=16
+    Step 25: SUBMIT                              flagged=10/10  suspects=0  steps_left=16
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.86
+    ★ GRADER SCORE: 0.9533
+
+  --- LLM Agent: task=easy, seed=2, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0040                    flagged=0/10  suspects=0  steps_left=30
+    ★ GRADER SCORE: 0.0000
+
+  easy: scores=['0.967', '0.953', '0.000']  mean=0.6400  var=0.204830
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=41
+    Step 11: INSPECT acc_0181                    flagged=1/10  suspects=8  steps_left=40
+    Step 12: INSPECT acc_0022                    flagged=1/10  suspects=8  steps_left=39
+    Step 13: FLAG acc_0022                       flagged=2/10  suspects=8  steps_left=39
+    Step 14: INSPECT acc_0092                    flagged=2/10  suspects=8  steps_left=38
+    Step 15: INSPECT acc_0097                    flagged=2/10  suspects=8  steps_left=37
+    Step 16: FLAG acc_0181                       flagged=3/10  suspects=7  steps_left=37
+    Step 17: INSPECT acc_0187                    flagged=3/10  suspects=7  steps_left=36
+    Step 18: FLAG acc_0092                       flagged=4/10  suspects=6  steps_left=36
+    Step 19: INSPECT acc_0093                    flagged=4/10  suspects=6  steps_left=35
+    Step 20: FLAG acc_0093                       flagged=5/10  suspects=5  steps_left=35
+    Step 21: FLAG acc_0097                       flagged=6/10  suspects=4  steps_left=35
+    Step 22: INSPECT acc_0172                    flagged=6/10  suspects=4  steps_left=34
+    Step 23: FLAG acc_0187                       flagged=7/10  suspects=3  steps_left=34
+    Step 24: INSPECT acc_0058                    flagged=7/10  suspects=3  steps_left=33
+    Step 25: FLAG acc_0058                       flagged=8/10  suspects=2  steps_left=33
+    Step 26: FLAG acc_0172                       flagged=9/10  suspects=1  steps_left=33
+    Step 27: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=32
+    Step 28: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=32
+    Step 29: SUBMIT                              flagged=10/10  suspects=0  steps_left=32
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.82
+    ★ GRADER SCORE: 0.9640
+
+  --- LLM Agent: task=medium, seed=1, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0171                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0099                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0152                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0092                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0078                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0112                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0012                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: FLAG acc_0012                       flagged=1/10  suspects=8  steps_left=43
+    Step  9: INSPECT acc_0033                    flagged=1/10  suspects=8  steps_left=42
+    Step 10: FLAG acc_0033                       flagged=2/10  suspects=8  steps_left=42
+    Step 11: INSPECT acc_0174                    flagged=2/10  suspects=8  steps_left=41
+    Step 12: INSPECT acc_0187                    flagged=2/10  suspects=8  steps_left=40
+    Step 13: FLAG acc_0187                       flagged=3/10  suspects=7  steps_left=40
+    Step 14: INSPECT acc_0079                    flagged=3/10  suspects=7  steps_left=39
+    Step 15: INSPECT acc_0032                    flagged=3/10  suspects=7  steps_left=38
+    Step 16: INSPECT acc_0023                    flagged=3/10  suspects=7  steps_left=37
+    Step 17: INSPECT acc_0146                    flagged=3/10  suspects=7  steps_left=36
+    Step 18: FLAG acc_0174                       flagged=4/10  suspects=6  steps_left=36
+    Step 19: INSPECT acc_0019                    flagged=4/10  suspects=6  steps_left=35
+    Step 20: FLAG acc_0023                       flagged=5/10  suspects=5  steps_left=35
+    Step 21: INSPECT acc_0131                    flagged=5/10  suspects=5  steps_left=34
+    Step 22: FLAG acc_0079                       flagged=6/10  suspects=4  steps_left=34
+    Step 23: FLAG acc_0019                       flagged=7/10  suspects=3  steps_left=34
+    Step 24: FLAG acc_0146                       flagged=8/10  suspects=2  steps_left=34
+    Step 25: FLAG acc_0131                       flagged=9/10  suspects=1  steps_left=34
+    Step 26: FLAG acc_0032                       flagged=10/10  suspects=0  steps_left=34
+    Step 27: SUBMIT                              flagged=10/10  suspects=0  steps_left=34
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.84
+    ★ GRADER SCORE: 0.9680
+
+  --- LLM Agent: task=medium, seed=2, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0107                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0175                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0030                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0041                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0054                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0199                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0181                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0166                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: INSPECT acc_0098                    flagged=0/10  suspects=0  steps_left=40
+    Step 11: INSPECT acc_0121                    flagged=0/10  suspects=0  steps_left=39
+    Step 12: INSPECT acc_0053                    flagged=0/10  suspects=0  steps_left=38
+    Step 13: INSPECT acc_0103                    flagged=0/10  suspects=0  steps_left=37
+    Step 14: INSPECT acc_0000                    flagged=0/10  suspects=0  steps_left=36
+    Step 15: INSPECT acc_0168                    flagged=0/10  suspects=0  steps_left=35
+    Step 16: INSPECT acc_0040                    flagged=0/10  suspects=0  steps_left=34
+    Step 17: INSPECT acc_0149                    flagged=0/10  suspects=0  steps_left=33
+    Step 18: INSPECT acc_0064                    flagged=0/10  suspects=0  steps_left=32
+    Step 19: INSPECT acc_0016                    flagged=0/10  suspects=0  steps_left=31
+    Step 20: INSPECT acc_0105                    flagged=0/10  suspects=0  steps_left=30
+    Step 21: INSPECT acc_0035                    flagged=0/10  suspects=0  steps_left=29
+    Step 22: FLAG acc_0035                       flagged=1/10  suspects=9  steps_left=29
+    Step 23: INSPECT acc_0020                    flagged=1/10  suspects=9  steps_left=28
+    Step 24: INSPECT acc_0036                    flagged=1/10  suspects=9  steps_left=27
+    Step 25: FLAG acc_0036                       flagged=2/10  suspects=8  steps_left=27
+    Step 26: INSPECT acc_0050                    flagged=2/10  suspects=8  steps_left=26
+    Step 27: FLAG acc_0020                       flagged=3/10  suspects=7  steps_left=26
+    Step 28: INSPECT acc_0051                    flagged=3/10  suspects=7  steps_left=25
+    Step 29: INSPECT acc_0085                    flagged=3/10  suspects=7  steps_left=24
+    Step 30: FLAG acc_0050                       flagged=4/10  suspects=6  steps_left=24
+    Step 31: FLAG acc_0085                       flagged=5/10  suspects=5  steps_left=24
+    Step 32: INSPECT acc_0177                    flagged=5/10  suspects=5  steps_left=23
+    Step 33: INSPECT acc_0170                    flagged=5/10  suspects=5  steps_left=22
+    Step 34: FLAG acc_0170                       flagged=6/10  suspects=4  steps_left=22
+    [LLM RETRY] An error occurred (ValidationException) when calling the Converse operation:
+ The model returned the following errors: {"error":{"code":"validation_error","message":"Int
+ernal server error","param":null,"type":"invalid_request_error"}} — retrying in 3s
+    Step 35: INSPECT acc_0055                    flagged=6/10  suspects=4  steps_left=21
+    Step 36: FLAG acc_0177                       flagged=7/10  suspects=3  steps_left=21
+    Step 37: INSPECT acc_0094                    flagged=7/10  suspects=3  steps_left=20
+    Step 38: FLAG acc_0094                       flagged=8/10  suspects=2  steps_left=20
+    Step 39: FLAG acc_0055                       flagged=9/10  suspects=1  steps_left=20
+    Step 40: FLAG acc_0051                       flagged=10/10  suspects=0  steps_left=20
+    Step 41: SUBMIT                              flagged=10/10  suspects=0  steps_left=20
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.70
+    ★ GRADER SCORE: 0.9400
+
+  medium: scores=['0.964', '0.968', '0.940']  mean=0.9573  var=0.000153
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0441                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0871                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0070                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0443                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0037                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0438                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: FLAG acc_0439                       flagged=1/10  suspects=5  steps_left=60
+    Step 22: INSPECT acc_0237                    flagged=1/10  suspects=5  steps_left=59
+    Step 23: FLAG acc_0237                       flagged=2/10  suspects=6  steps_left=59
+    Step 24: INSPECT acc_0621                    flagged=2/10  suspects=6  steps_left=58
+    Step 25: FLAG acc_0621                       flagged=3/10  suspects=6  steps_left=58
+    Step 26: INSPECT acc_0389                    flagged=3/10  suspects=6  steps_left=57
+    Step 27: INSPECT acc_0160                    flagged=3/10  suspects=6  steps_left=56
+    Step 28: INSPECT acc_0549                    flagged=3/10  suspects=6  steps_left=55
+    Step 29: INSPECT acc_0658                    flagged=3/10  suspects=6  steps_left=54
+    Step 30: FLAG acc_0160                       flagged=4/10  suspects=6  steps_left=54
+    Step 31: INSPECT acc_0290                    flagged=4/10  suspects=6  steps_left=53
+    Step 32: INSPECT acc_0124                    flagged=4/10  suspects=6  steps_left=52
+    Step 33: INSPECT acc_0507                    flagged=4/10  suspects=6  steps_left=51
+    Step 34: FLAG acc_0549                       flagged=5/10  suspects=5  steps_left=51
+    Step 35: FLAG acc_0290                       flagged=6/10  suspects=4  steps_left=51
+    Step 36: FLAG acc_0389                       flagged=7/10  suspects=3  steps_left=51
+    Step 37: FLAG acc_0658                       flagged=8/10  suspects=2  steps_left=51
+    Step 38: FLAG acc_0507                       flagged=9/10  suspects=1  steps_left=51
+    Step 39: FLAG acc_0124                       flagged=10/10  suspects=0  steps_left=51
+    Step 40: SUBMIT                              flagged=10/10  suspects=0  steps_left=51
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.71
+    ★ GRADER SCORE: 0.9637
+
+  --- LLM Agent: task=hard, seed=1, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0014                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0835                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0855                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0930                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0336                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0929                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0076                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0543                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0590                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0401                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0322                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0154                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0374                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0549                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0903                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0976                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0620                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0017                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0222                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0536                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: INSPECT acc_0112                    flagged=0/10  suspects=0  steps_left=59
+    Step 22: INSPECT acc_0577                    flagged=0/10  suspects=0  steps_left=58
+    Step 23: INSPECT acc_0517                    flagged=0/10  suspects=0  steps_left=57
+    Step 24: INSPECT acc_0113                    flagged=0/10  suspects=0  steps_left=56
+    Step 25: INSPECT acc_0167                    flagged=0/10  suspects=0  steps_left=55
+    Step 26: INSPECT acc_0697                    flagged=0/10  suspects=0  steps_left=54
+    Step 27: INSPECT acc_0271                    flagged=0/10  suspects=0  steps_left=53
+    Step 28: INSPECT acc_0681                    flagged=0/10  suspects=0  steps_left=52
+    Step 29: INSPECT acc_0530                    flagged=0/10  suspects=0  steps_left=51
+    Step 30: INSPECT acc_0353                    flagged=0/10  suspects=0  steps_left=50
+    Step 31: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=49
+    Step 32: INSPECT acc_0777                    flagged=0/10  suspects=0  steps_left=48
+    Step 33: INSPECT acc_0265                    flagged=0/10  suspects=0  steps_left=47
+    Step 34: INSPECT acc_0788                    flagged=0/10  suspects=0  steps_left=46
+    Step 35: INSPECT acc_0033                    flagged=0/10  suspects=0  steps_left=45
+    Step 36: INSPECT acc_0187                    flagged=0/10  suspects=0  steps_left=44
+    Step 37: INSPECT acc_0445                    flagged=0/10  suspects=0  steps_left=43
+    Step 38: INSPECT acc_0846                    flagged=0/10  suspects=0  steps_left=42
+    Step 39: INSPECT acc_0659                    flagged=0/10  suspects=0  steps_left=41
+    Step 40: INSPECT acc_0768                    flagged=0/10  suspects=0  steps_left=40
+    Step 41: INSPECT acc_0677                    flagged=0/10  suspects=0  steps_left=39
+    Step 42: INSPECT acc_0539                    flagged=0/10  suspects=0  steps_left=38
+    Step 43: INSPECT acc_0742                    flagged=0/10  suspects=0  steps_left=37
+    Step 44: INSPECT acc_0503                    flagged=0/10  suspects=0  steps_left=36
+    Step 45: INSPECT acc_0876                    flagged=0/10  suspects=0  steps_left=35
+    Step 46: INSPECT acc_0639                    flagged=0/10  suspects=0  steps_left=34
+    Step 47: INSPECT acc_0494                    flagged=0/10  suspects=0  steps_left=33
+    Step 48: INSPECT acc_0898                    flagged=0/10  suspects=0  steps_left=32
+    Step 49: INSPECT acc_0553                    flagged=0/10  suspects=0  steps_left=31
+    Step 50: INSPECT acc_0588                    flagged=0/10  suspects=0  steps_left=30
+    Step 51: INSPECT acc_0194                    flagged=0/10  suspects=0  steps_left=29
+    Step 52: INSPECT acc_0810                    flagged=0/10  suspects=0  steps_left=28
+    Step 53: INSPECT acc_0355                    flagged=0/10  suspects=0  steps_left=27
+    Step 54: INSPECT acc_0363                    flagged=0/10  suspects=0  steps_left=26
+    Step 55: INSPECT acc_0221                    flagged=0/10  suspects=0  steps_left=25
+    Step 56: INSPECT acc_0580                    flagged=0/10  suspects=0  steps_left=24
+    Step 57: INSPECT acc_0534                    flagged=0/10  suspects=0  steps_left=23
+    Step 58: INSPECT acc_0778                    flagged=0/10  suspects=0  steps_left=22
+    Step 59: INSPECT acc_0998                    flagged=0/10  suspects=0  steps_left=21
+    Step 60: INSPECT acc_0233                    flagged=0/10  suspects=0  steps_left=20
+    Step 61: INSPECT acc_0052                    flagged=0/10  suspects=0  steps_left=19
+    Step 62: INSPECT acc_0813                    flagged=0/10  suspects=0  steps_left=18
+    Step 63: INSPECT acc_0035                    flagged=0/10  suspects=0  steps_left=17
+    Step 64: INSPECT acc_0667                    flagged=0/10  suspects=0  steps_left=16
+    Step 65: INSPECT acc_0019                    flagged=0/10  suspects=0  steps_left=15
+    Step 66: INSPECT acc_0959                    flagged=0/10  suspects=0  steps_left=14
+    Step 67: INSPECT acc_0212                    flagged=0/10  suspects=0  steps_left=13
+    Step 68: INSPECT acc_0776                    flagged=0/10  suspects=0  steps_left=12
+    Step 69: INSPECT acc_0049                    flagged=0/10  suspects=0  steps_left=11
+    Step 70: INSPECT acc_0434                    flagged=0/10  suspects=0  steps_left=10
+    Step 71: INSPECT acc_0827                    flagged=0/10  suspects=0  steps_left=9
+    Step 72: INSPECT acc_0583                    flagged=0/10  suspects=0  steps_left=8
+    Step 73: INSPECT acc_0065                    flagged=0/10  suspects=0  steps_left=7
+    Step 74: INSPECT acc_0107                    flagged=0/10  suspects=0  steps_left=6
+    Step 75: INSPECT acc_0107                    flagged=0/10  suspects=0  steps_left=5
+    Step 76: INSPECT acc_0761                    flagged=0/10  suspects=0  steps_left=4
+    Step 77: INSPECT acc_0995                    flagged=0/10  suspects=0  steps_left=3
+    Step 78: INSPECT acc_0157                    flagged=0/10  suspects=0  steps_left=2
+    Step 79: INSPECT acc_0936                    flagged=0/10  suspects=0  steps_left=1
+    Step 80: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=0
+    → Episode ended: [LOSS] TP=0 FP=0 FN=10 Recall=0.00 Precision=0.00 Episode reward=-9.80
+    ★ GRADER SCORE: 0.0000
+
+  --- LLM Agent: task=hard, seed=2, model=Bedrock/deepseek.v3.2 ---
+    Step  1: INSPECT acc_0813                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0430                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0817                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0175                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0523                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0113                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0797                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0478                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0861                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0836                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0926                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0664                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0255                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0938                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0672                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: FLAG acc_0672                       flagged=1/10  suspects=6  steps_left=65
+    Step 17: INSPECT acc_0659                    flagged=1/10  suspects=6  steps_left=64
+    Step 18: INSPECT acc_0290                    flagged=1/10  suspects=6  steps_left=63
+    Step 19: FLAG acc_0290                       flagged=2/10  suspects=6  steps_left=63
+    Step 20: INSPECT acc_0339                    flagged=2/10  suspects=6  steps_left=62
+    Step 21: FLAG acc_0659                       flagged=3/10  suspects=7  steps_left=62
+    Step 22: INSPECT acc_0544                    flagged=3/10  suspects=7  steps_left=61
+    Step 23: FLAG acc_0339                       flagged=4/10  suspects=6  steps_left=61
+    Step 24: INSPECT acc_0696                    flagged=4/10  suspects=6  steps_left=60
+    Step 25: FLAG acc_0544                       flagged=5/10  suspects=5  steps_left=60
+    Step 26: INSPECT acc_0541                    flagged=5/10  suspects=5  steps_left=59
+    Step 27: FLAG acc_0696                       flagged=6/10  suspects=4  steps_left=59
+    Step 28: FLAG acc_0541                       flagged=7/10  suspects=3  steps_left=59
+    Step 29: INSPECT acc_0793                    flagged=7/10  suspects=3  steps_left=58
+    Step 30: FLAG acc_0793                       flagged=8/10  suspects=2  steps_left=58
+    Step 31: INSPECT acc_0214                    flagged=8/10  suspects=2  steps_left=57
+    Step 32: INSPECT acc_0112                    flagged=8/10  suspects=2  steps_left=56
+    Step 33: FLAG acc_0112                       flagged=9/10  suspects=1  steps_left=56
+    Step 34: FLAG acc_0214                       flagged=10/10  suspects=0  steps_left=56
+    Step 35: SUBMIT                              flagged=10/10  suspects=0  steps_left=56
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.76
+    ★ GRADER SCORE: 0.9700
+
+  hard: scores=['0.964', '0.000', '0.970']  mean=0.6446  var=0.207740
+
+============================================================
+EVALUATION COMPLETE
+============================================================
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26$

model-benchmark-logs/meta_judge_log.txt

@@ -0,0 +1,826 @@
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26/eval-models$ python3 llama_test_judge_eval.py --url https://pandago-graphstrike.hf.space --bedrock
+GraphStrike Judge Evaluation Simulator
+Target:  https://pandago-graphstrike.hf.space
+Backend: bedrock
+Model:   Bedrock/us.meta.llama4-scout-17b-instruct-v1:0
+Token:   set
+
+============================================================
+PHASE 0: Endpoint Verification
+============================================================
+  ✓ GET /health
+  ✓ GET /tasks
+  ✓ GET /metadata
+  ✓ GET /schema
+  ✓ GET /web
+  ✓ POST /reset
+  ✓ GET /state
+  ✓ POST /step
+  ✓ POST /step
+  ✓ GET /grader
+  ✓ POST /mcp
+  ✓ POST /baseline
+
+============================================================
+PHASE 1: Baseline Stability (3 runs)
+============================================================
+  Run 1: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 2: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 3: easy=0.9100  medium=0.9060  hard=0.9038
+  ✓ All 3 runs identical — baseline is deterministic
+
+============================================================
+PHASE 2: LLM Agent Evaluation (model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0)
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=29
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=28
+    Step  4: FLAG acc_0036                       flagged=2/10  suspects=8  steps_left=28
+    Step  5: INSPECT acc_0001                    flagged=2/10  suspects=8  steps_left=27
+    Step  6: FLAG acc_0001                       flagged=3/10  suspects=7  steps_left=27
+    Step  7: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=26
+    Step  8: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=26
+    Step  9: INSPECT acc_0012                    flagged=4/10  suspects=6  steps_left=25
+    Step 10: FLAG acc_0012                       flagged=5/10  suspects=5  steps_left=25
+    Step 11: INSPECT acc_0000                    flagged=5/10  suspects=5  steps_left=24
+    Step 12: FLAG acc_0000                       flagged=6/10  suspects=4  steps_left=24
+    Step 13: INSPECT acc_0027                    flagged=6/10  suspects=4  steps_left=23
+    Step 14: FLAG acc_0027                       flagged=7/10  suspects=3  steps_left=23
+    Step 15: INSPECT acc_0047                    flagged=7/10  suspects=3  steps_left=22
+    Step 16: FLAG acc_0047                       flagged=8/10  suspects=2  steps_left=22
+    Step 17: INSPECT acc_0007                    flagged=8/10  suspects=2  steps_left=21
+    Step 18: FLAG acc_0007                       flagged=9/10  suspects=1  steps_left=21
+    Step 19: INSPECT acc_0028                    flagged=9/10  suspects=1  steps_left=20
+    Step 20: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=20
+    Step 21: INSPECT acc_0035                    flagged=10/10  suspects=0  steps_left=19
+    Step 22: INSPECT acc_0022                    flagged=10/10  suspects=0  steps_left=18
+    Step 23: INSPECT acc_0041                    flagged=10/10  suspects=0  steps_left=17
+    Step 24: INSPECT acc_0029                    flagged=10/10  suspects=0  steps_left=16
+    Step 25: INSPECT acc_0018                    flagged=10/10  suspects=0  steps_left=15
+    Step 26: INSPECT acc_0019                    flagged=10/10  suspects=0  steps_left=14
+    Step 27: INSPECT acc_0041                    flagged=10/10  suspects=0  steps_left=13
+    Step 28: INSPECT acc_0040                    flagged=10/10  suspects=0  steps_left=12
+    Step 29: INSPECT acc_0026                    flagged=10/10  suspects=0  steps_left=11
+    Step 30: INSPECT acc_0021                    flagged=10/10  suspects=0  steps_left=10
+    Step 31: INSPECT acc_0009                    flagged=10/10  suspects=0  steps_left=9
+    Step 32: INSPECT acc_0033                    flagged=10/10  suspects=0  steps_left=8
+    Step 33: INSPECT acc_0044                    flagged=10/10  suspects=0  steps_left=7
+    Step 34: SUBMIT                              flagged=10/10  suspects=0  steps_left=7
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.77
+    ★ GRADER SCORE: 0.9233
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=41
+    Step 11: INSPECT acc_0181                    flagged=1/10  suspects=8  steps_left=40
+    Step 12: FLAG acc_0181                       flagged=2/10  suspects=8  steps_left=40
+    Step 13: INSPECT acc_0022                    flagged=2/10  suspects=8  steps_left=39
+    Step 14: FLAG acc_0022                       flagged=3/10  suspects=7  steps_left=39
+    Step 15: INSPECT acc_0092                    flagged=3/10  suspects=7  steps_left=38
+    Step 16: FLAG acc_0092                       flagged=4/10  suspects=6  steps_left=38
+    Step 17: INSPECT acc_0097                    flagged=4/10  suspects=6  steps_left=37
+    Step 18: FLAG acc_0097                       flagged=5/10  suspects=5  steps_left=37
+    Step 19: INSPECT acc_0187                    flagged=5/10  suspects=5  steps_left=36
+    Step 20: FLAG acc_0187                       flagged=6/10  suspects=4  steps_left=36
+    Step 21: INSPECT acc_0093                    flagged=6/10  suspects=4  steps_left=35
+    Step 22: FLAG acc_0093                       flagged=7/10  suspects=3  steps_left=35
+    Step 23: INSPECT acc_0172                    flagged=7/10  suspects=3  steps_left=34
+    Step 24: FLAG acc_0172                       flagged=8/10  suspects=2  steps_left=34
+    Step 25: INSPECT acc_0058                    flagged=8/10  suspects=2  steps_left=33
+    Step 26: FLAG acc_0058                       flagged=9/10  suspects=1  steps_left=33
+    Step 27: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=32
+    Step 28: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=32
+    Step 29: INSPECT acc_0148                    flagged=10/10  suspects=0  steps_left=31
+    Step 30: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=30
+    Step 31: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=29
+    Step 32: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=28
+    Step 33: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=27
+    Step 34: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=26
+    Step 35: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=25
+    Step 36: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=24
+    Step 37: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=23
+    Step 38: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=22
+    Step 39: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=21
+    Step 40: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=20
+    Step 41: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=19
+    Step 42: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=18
+    Step 43: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=17
+    Step 44: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=16
+    Step 45: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=15
+    Step 46: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=14
+    Step 47: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=13
+    Step 48: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=12
+    Step 49: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=11
+    Step 50: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=10
+    Step 51: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=9
+    Step 52: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=8
+    Step 53: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=7
+    Step 54: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=6
+    Step 55: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=5
+    Step 56: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=4
+    Step 57: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=3
+    Step 58: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=2
+    Step 59: SUBMIT                              flagged=10/10  suspects=0  steps_left=2
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.52
+    ★ GRADER SCORE: 0.9040
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0441                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0871                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0070                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0443                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0037                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0438                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: FLAG acc_0439                       flagged=1/10  suspects=5  steps_left=60
+    Step 22: INSPECT acc_0237                    flagged=1/10  suspects=5  steps_left=59
+    Step 23: FLAG acc_0237                       flagged=2/10  suspects=6  steps_left=59
+    Step 24: INSPECT acc_0621                    flagged=2/10  suspects=6  steps_left=58
+    Step 25: FLAG acc_0621                       flagged=3/10  suspects=6  steps_left=58
+    Step 26: INSPECT acc_0389                    flagged=3/10  suspects=6  steps_left=57
+    Step 27: FLAG acc_0389                       flagged=4/10  suspects=6  steps_left=57
+    Step 28: INSPECT acc_0160                    flagged=4/10  suspects=6  steps_left=56
+    Step 29: FLAG acc_0160                       flagged=5/10  suspects=5  steps_left=56
+    Step 30: INSPECT acc_0549                    flagged=5/10  suspects=5  steps_left=55
+    Step 31: FLAG acc_0549                       flagged=6/10  suspects=4  steps_left=55
+    Step 32: INSPECT acc_0658                    flagged=6/10  suspects=4  steps_left=54
+    Step 33: FLAG acc_0658                       flagged=7/10  suspects=3  steps_left=54
+    Step 34: INSPECT acc_0290                    flagged=7/10  suspects=3  steps_left=53
+    Step 35: FLAG acc_0290                       flagged=8/10  suspects=2  steps_left=53
+    Step 36: INSPECT acc_0124                    flagged=8/10  suspects=2  steps_left=52
+    Step 37: FLAG acc_0124                       flagged=9/10  suspects=1  steps_left=52
+    Step 38: INSPECT acc_0507                    flagged=9/10  suspects=1  steps_left=51
+    Step 39: FLAG acc_0507                       flagged=10/10  suspects=0  steps_left=51
+    Step 40: INSPECT acc_0086                    flagged=10/10  suspects=0  steps_left=50
+    Step 41: INSPECT acc_0497                    flagged=10/10  suspects=0  steps_left=49
+    Step 42: INSPECT acc_0610                    flagged=10/10  suspects=0  steps_left=48
+    Step 43: INSPECT acc_0579                    flagged=10/10  suspects=0  steps_left=47
+    Step 44: INSPECT acc_0573                    flagged=10/10  suspects=0  steps_left=46
+    Step 45: INSPECT acc_0479                    flagged=10/10  suspects=0  steps_left=45
+    Step 46: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=44
+    Step 47: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=43
+    Step 48: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=42
+    Step 49: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=41
+    Step 50: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=40
+    Step 51: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=39
+    Step 52: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=38
+    Step 53: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=37
+    Step 54: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=36
+    Step 55: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=35
+    Step 56: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=34
+    Step 57: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=33
+    Step 58: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=32
+    Step 59: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=31
+    Step 60: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=30
+    Step 61: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=29
+    Step 62: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=28
+    Step 63: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=27
+    Step 64: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=26
+    Step 65: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=25
+    Step 66: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=24
+    Step 67: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=23
+    Step 68: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=22
+    Step 69: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=21
+    Step 70: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=20
+    Step 71: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=19
+    Step 72: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=18
+    Step 73: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=17
+    Step 74: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=16
+    Step 75: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=15
+    Step 76: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=14
+    Step 77: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=13
+    Step 78: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=12
+    Step 79: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=11
+    Step 80: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=10
+    Step 81: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=9
+    Step 82: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=8
+    Step 83: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=7
+    Step 84: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=6
+    Step 85: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=5
+    Step 86: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=4
+    Step 87: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=3
+    Step 88: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=2
+    Step 89: SUBMIT                              flagged=10/10  suspects=0  steps_left=2
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=13.22
+    ★ GRADER SCORE: 0.9025
+
+  Summary: easy=0.9233  medium=0.9040  hard=0.9025
+
+============================================================
+PHASE 3: Score Variance (seeds=[0, 1, 2])
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=29
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=28
+    Step  4: FLAG acc_0036                       flagged=2/10  suspects=8  steps_left=28
+    Step  5: INSPECT acc_0001                    flagged=2/10  suspects=8  steps_left=27
+    Step  6: FLAG acc_0001                       flagged=3/10  suspects=7  steps_left=27
+    Step  7: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=26
+    Step  8: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=26
+    Step  9: INSPECT acc_0012                    flagged=4/10  suspects=6  steps_left=25
+    Step 10: FLAG acc_0012                       flagged=5/10  suspects=5  steps_left=25
+    Step 11: INSPECT acc_0000                    flagged=5/10  suspects=5  steps_left=24
+    Step 12: FLAG acc_0000                       flagged=6/10  suspects=4  steps_left=24
+    Step 13: INSPECT acc_0027                    flagged=6/10  suspects=4  steps_left=23
+    Step 14: FLAG acc_0027                       flagged=7/10  suspects=3  steps_left=23
+    Step 15: INSPECT acc_0047                    flagged=7/10  suspects=3  steps_left=22
+    Step 16: FLAG acc_0047                       flagged=8/10  suspects=2  steps_left=22
+    Step 17: INSPECT acc_0007                    flagged=8/10  suspects=2  steps_left=21
+    Step 18: FLAG acc_0007                       flagged=9/10  suspects=1  steps_left=21
+    Step 19: INSPECT acc_0028                    flagged=9/10  suspects=1  steps_left=20
+    Step 20: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=20
+    Step 21: INSPECT acc_0035                    flagged=10/10  suspects=0  steps_left=19
+    Step 22: INSPECT acc_0022                    flagged=10/10  suspects=0  steps_left=18
+    Step 23: INSPECT acc_0041                    flagged=10/10  suspects=0  steps_left=17
+    Step 24: INSPECT acc_0029                    flagged=10/10  suspects=0  steps_left=16
+    Step 25: INSPECT acc_0018                    flagged=10/10  suspects=0  steps_left=15
+    Step 26: INSPECT acc_0019                    flagged=10/10  suspects=0  steps_left=14
+    Step 27: INSPECT acc_0041                    flagged=10/10  suspects=0  steps_left=13
+    Step 28: INSPECT acc_0040                    flagged=10/10  suspects=0  steps_left=12
+    Step 29: INSPECT acc_0026                    flagged=10/10  suspects=0  steps_left=11
+    Step 30: INSPECT acc_0021                    flagged=10/10  suspects=0  steps_left=10
+    Step 31: INSPECT acc_0009                    flagged=10/10  suspects=0  steps_left=9
+    Step 32: INSPECT acc_0033                    flagged=10/10  suspects=0  steps_left=8
+    Step 33: INSPECT acc_0044                    flagged=10/10  suspects=0  steps_left=7
+    Step 34: SUBMIT                              flagged=10/10  suspects=0  steps_left=7
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.77
+    ★ GRADER SCORE: 0.9233
+
+  --- LLM Agent: task=easy, seed=1, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0034                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: INSPECT acc_0003                    flagged=0/10  suspects=0  steps_left=28
+    Step  3: INSPECT acc_0049                    flagged=0/10  suspects=0  steps_left=27
+    Step  4: INSPECT acc_0006                    flagged=0/10  suspects=0  steps_left=26
+    Step  5: INSPECT acc_0047                    flagged=0/10  suspects=0  steps_left=25
+    Step  6: FLAG acc_0047                       flagged=1/10  suspects=9  steps_left=25
+    Step  7: INSPECT acc_0009                    flagged=1/10  suspects=9  steps_left=24
+    Step  8: FLAG acc_0009                       flagged=2/10  suspects=8  steps_left=24
+    Step  9: INSPECT acc_0046                    flagged=2/10  suspects=8  steps_left=23
+    Step 10: FLAG acc_0046                       flagged=3/10  suspects=7  steps_left=23
+    Step 11: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=22
+    Step 12: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=22
+    Step 13: INSPECT acc_0021                    flagged=4/10  suspects=6  steps_left=21
+    Step 14: FLAG acc_0021                       flagged=5/10  suspects=5  steps_left=21
+    Step 15: INSPECT acc_0002                    flagged=5/10  suspects=5  steps_left=20
+    Step 16: FLAG acc_0002                       flagged=6/10  suspects=4  steps_left=20
+    Step 17: INSPECT acc_0048                    flagged=6/10  suspects=4  steps_left=19
+    Step 18: FLAG acc_0048                       flagged=7/10  suspects=3  steps_left=19
+    Step 19: INSPECT acc_0029                    flagged=7/10  suspects=3  steps_left=18
+    Step 20: FLAG acc_0029                       flagged=8/10  suspects=2  steps_left=18
+    Step 21: INSPECT acc_0015                    flagged=8/10  suspects=2  steps_left=17
+    Step 22: FLAG acc_0015                       flagged=9/10  suspects=1  steps_left=17
+    Step 23: INSPECT acc_0005                    flagged=9/10  suspects=1  steps_left=16
+    Step 24: FLAG acc_0005                       flagged=10/10  suspects=0  steps_left=16
+    Step 25: INSPECT acc_0036                    flagged=10/10  suspects=0  steps_left=15
+    Step 26: INSPECT acc_0027                    flagged=10/10  suspects=0  steps_left=14
+    Step 27: INSPECT acc_0043                    flagged=10/10  suspects=0  steps_left=13
+    Step 28: INSPECT acc_0044                    flagged=10/10  suspects=0  steps_left=12
+    Step 29: INSPECT acc_0038                    flagged=10/10  suspects=0  steps_left=11
+    Step 30: INSPECT acc_0039                    flagged=10/10  suspects=0  steps_left=10
+    Step 31: INSPECT acc_0028                    flagged=10/10  suspects=0  steps_left=9
+    Step 32: INSPECT acc_0022                    flagged=10/10  suspects=0  steps_left=8
+    Step 33: INSPECT acc_0025                    flagged=10/10  suspects=0  steps_left=7
+    Step 34: INSPECT acc_0031                    flagged=10/10  suspects=0  steps_left=6
+    Step 35: INSPECT acc_0007                    flagged=10/10  suspects=0  steps_left=5
+    Step 36: INSPECT acc_0026                    flagged=10/10  suspects=0  steps_left=4
+    Step 37: SUBMIT                              flagged=10/10  suspects=0  steps_left=4
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.74
+    ★ GRADER SCORE: 0.9133
+
+  --- LLM Agent: task=easy, seed=2, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0040                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: INSPECT acc_0017                    flagged=0/10  suspects=0  steps_left=28
+    Step  3: INSPECT acc_0025                    flagged=0/10  suspects=0  steps_left=27
+    Step  4: INSPECT acc_0026                    flagged=0/10  suspects=0  steps_left=26
+    Step  5: INSPECT acc_0038                    flagged=0/10  suspects=0  steps_left=25
+    Step  6: INSPECT acc_0029                    flagged=0/10  suspects=0  steps_left=24
+    Step  7: FLAG acc_0029                       flagged=1/10  suspects=9  steps_left=24
+    Step  8: INSPECT acc_0006                    flagged=1/10  suspects=9  steps_left=23
+    Step  9: FLAG acc_0006                       flagged=2/10  suspects=8  steps_left=23
+    Step 10: INSPECT acc_0033                    flagged=2/10  suspects=8  steps_left=22
+    Step 11: FLAG acc_0033                       flagged=3/10  suspects=7  steps_left=22
+    Step 12: INSPECT acc_0015                    flagged=3/10  suspects=7  steps_left=21
+    Step 13: FLAG acc_0015                       flagged=4/10  suspects=6  steps_left=21
+    Step 14: INSPECT acc_0022                    flagged=4/10  suspects=6  steps_left=20
+    Step 15: FLAG acc_0022                       flagged=5/10  suspects=5  steps_left=20
+    Step 16: INSPECT acc_0009                    flagged=5/10  suspects=5  steps_left=19
+    Step 17: FLAG acc_0009                       flagged=6/10  suspects=4  steps_left=19
+    Step 18: INSPECT acc_0004                    flagged=6/10  suspects=4  steps_left=18
+    Step 19: FLAG acc_0004                       flagged=7/10  suspects=3  steps_left=18
+    Step 20: INSPECT acc_0024                    flagged=7/10  suspects=3  steps_left=17
+    Step 21: FLAG acc_0024                       flagged=8/10  suspects=2  steps_left=17
+    Step 22: INSPECT acc_0049                    flagged=8/10  suspects=2  steps_left=16
+    Step 23: FLAG acc_0049                       flagged=9/10  suspects=1  steps_left=16
+    Step 24: INSPECT acc_0035                    flagged=9/10  suspects=1  steps_left=15
+    Step 25: FLAG acc_0035                       flagged=10/10  suspects=0  steps_left=15
+    Step 26: INSPECT acc_0044                    flagged=10/10  suspects=0  steps_left=14
+    Step 27: INSPECT acc_0016                    flagged=10/10  suspects=0  steps_left=13
+    Step 28: INSPECT acc_0043                    flagged=10/10  suspects=0  steps_left=12
+    Step 29: INSPECT acc_0003                    flagged=10/10  suspects=0  steps_left=11
+    Step 30: INSPECT acc_0028                    flagged=10/10  suspects=0  steps_left=10
+    Step 31: INSPECT acc_0027                    flagged=10/10  suspects=0  steps_left=9
+    Step 32: INSPECT acc_0023                    flagged=10/10  suspects=0  steps_left=8
+    Step 33: INSPECT acc_0041                    flagged=10/10  suspects=0  steps_left=7
+    Step 34: INSPECT acc_0045                    flagged=10/10  suspects=0  steps_left=6
+    Step 35: INSPECT acc_0039                    flagged=10/10  suspects=0  steps_left=5
+    Step 36: INSPECT acc_0048                    flagged=10/10  suspects=0  steps_left=4
+    Step 37: INSPECT acc_0046                    flagged=10/10  suspects=0  steps_left=3
+    Step 38: SUBMIT                              flagged=10/10  suspects=0  steps_left=3
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.73
+    ★ GRADER SCORE: 0.9100
+
+  easy: scores=['0.923', '0.913', '0.910']  mean=0.9155  var=0.000032
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=41
+    Step 11: INSPECT acc_0181                    flagged=1/10  suspects=8  steps_left=40
+    Step 12: FLAG acc_0181                       flagged=2/10  suspects=8  steps_left=40
+    Step 13: INSPECT acc_0022                    flagged=2/10  suspects=8  steps_left=39
+    Step 14: FLAG acc_0022                       flagged=3/10  suspects=7  steps_left=39
+    Step 15: INSPECT acc_0092                    flagged=3/10  suspects=7  steps_left=38
+    Step 16: FLAG acc_0092                       flagged=4/10  suspects=6  steps_left=38
+    Step 17: INSPECT acc_0097                    flagged=4/10  suspects=6  steps_left=37
+    Step 18: FLAG acc_0097                       flagged=5/10  suspects=5  steps_left=37
+    Step 19: INSPECT acc_0187                    flagged=5/10  suspects=5  steps_left=36
+    Step 20: FLAG acc_0187                       flagged=6/10  suspects=4  steps_left=36
+    Step 21: INSPECT acc_0093                    flagged=6/10  suspects=4  steps_left=35
+    Step 22: FLAG acc_0093                       flagged=7/10  suspects=3  steps_left=35
+    Step 23: INSPECT acc_0172                    flagged=7/10  suspects=3  steps_left=34
+    Step 24: FLAG acc_0172                       flagged=8/10  suspects=2  steps_left=34
+    Step 25: INSPECT acc_0058                    flagged=8/10  suspects=2  steps_left=33
+    Step 26: FLAG acc_0058                       flagged=9/10  suspects=1  steps_left=33
+    Step 27: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=32
+    Step 28: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=32
+    Step 29: INSPECT acc_0148                    flagged=10/10  suspects=0  steps_left=31
+    Step 30: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=30
+    Step 31: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=29
+    Step 32: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=28
+    Step 33: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=27
+    Step 34: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=26
+    Step 35: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=25
+    Step 36: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=24
+    Step 37: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=23
+    Step 38: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=22
+    Step 39: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=21
+    Step 40: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=20
+    Step 41: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=19
+    Step 42: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=18
+    Step 43: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=17
+    Step 44: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=16
+    Step 45: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=15
+    Step 46: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=14
+    Step 47: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=13
+    Step 48: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=12
+    Step 49: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=11
+    Step 50: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=10
+    Step 51: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=9
+    Step 52: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=8
+    Step 53: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=7
+    Step 54: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=6
+    Step 55: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=5
+    Step 56: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=4
+    Step 57: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=3
+    Step 58: INSPECT acc_0078                    flagged=10/10  suspects=0  steps_left=2
+    Step 59: INSPECT acc_0179                    flagged=10/10  suspects=0  steps_left=1
+    Step 60: SUBMIT                              flagged=10/10  suspects=0  steps_left=1
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.51
+    ★ GRADER SCORE: 0.9020
+
+  --- LLM Agent: task=medium, seed=1, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0171                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0099                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0152                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0092                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0078                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0112                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0012                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: FLAG acc_0012                       flagged=1/10  suspects=8  steps_left=43
+    Step  9: INSPECT acc_0033                    flagged=1/10  suspects=8  steps_left=42
+    Step 10: FLAG acc_0033                       flagged=2/10  suspects=8  steps_left=42
+    Step 11: INSPECT acc_0174                    flagged=2/10  suspects=8  steps_left=41
+    Step 12: FLAG acc_0174                       flagged=3/10  suspects=7  steps_left=41
+    Step 13: INSPECT acc_0187                    flagged=3/10  suspects=7  steps_left=40
+    Step 14: FLAG acc_0187                       flagged=4/10  suspects=6  steps_left=40
+    Step 15: INSPECT acc_0079                    flagged=4/10  suspects=6  steps_left=39
+    Step 16: FLAG acc_0079                       flagged=5/10  suspects=5  steps_left=39
+    Step 17: INSPECT acc_0032                    flagged=5/10  suspects=5  steps_left=38
+    Step 18: FLAG acc_0032                       flagged=6/10  suspects=4  steps_left=38
+    Step 19: INSPECT acc_0023                    flagged=6/10  suspects=4  steps_left=37
+    Step 20: FLAG acc_0023                       flagged=7/10  suspects=3  steps_left=37
+    Step 21: INSPECT acc_0146                    flagged=7/10  suspects=3  steps_left=36
+    Step 22: FLAG acc_0146                       flagged=8/10  suspects=2  steps_left=36
+    Step 23: INSPECT acc_0019                    flagged=8/10  suspects=2  steps_left=35
+    Step 24: FLAG acc_0019                       flagged=9/10  suspects=1  steps_left=35
+    Step 25: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=34
+    Step 26: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=34
+    Step 27: INSPECT acc_0168                    flagged=10/10  suspects=0  steps_left=33
+    Step 28: INSPECT acc_0198                    flagged=10/10  suspects=0  steps_left=32
+    Step 29: INSPECT acc_0186                    flagged=10/10  suspects=0  steps_left=31
+    Step 30: INSPECT acc_0099                    flagged=10/10  suspects=0  steps_left=30
+    Step 31: INSPECT acc_0084                    flagged=10/10  suspects=0  steps_left=29
+    Step 32: INSPECT acc_0117                    flagged=10/10  suspects=0  steps_left=28
+    Step 33: INSPECT acc_0192                    flagged=10/10  suspects=0  steps_left=27
+    Step 34: INSPECT acc_0025                    flagged=10/10  suspects=0  steps_left=26
+    Step 35: INSPECT acc_0176                    flagged=10/10  suspects=0  steps_left=25
+    Step 36: INSPECT acc_0185                    flagged=10/10  suspects=0  steps_left=24
+    Step 37: INSPECT acc_0027                    flagged=10/10  suspects=0  steps_left=23
+    Step 38: INSPECT acc_0199                    flagged=10/10  suspects=0  steps_left=22
+    Step 39: INSPECT acc_0135                    flagged=10/10  suspects=0  steps_left=21
+    Step 40: INSPECT acc_0082                    flagged=10/10  suspects=0  steps_left=20
+    Step 41: INSPECT acc_0002                    flagged=10/10  suspects=0  steps_left=19
+    Step 42: INSPECT acc_0161                    flagged=10/10  suspects=0  steps_left=18
+    Step 43: INSPECT acc_0067                    flagged=10/10  suspects=0  steps_left=17
+    Step 44: INSPECT acc_0062                    flagged=10/10  suspects=0  steps_left=16
+    Step 45: INSPECT acc_0034                    flagged=10/10  suspects=0  steps_left=15
+    Step 46: INSPECT acc_0010                    flagged=10/10  suspects=0  steps_left=14
+    Step 47: INSPECT acc_0173                    flagged=10/10  suspects=0  steps_left=13
+    Step 48: INSPECT acc_0081                    flagged=10/10  suspects=0  steps_left=12
+    Step 49: INSPECT acc_0132                    flagged=10/10  suspects=0  steps_left=11
+    Step 50: INSPECT acc_0094                    flagged=10/10  suspects=0  steps_left=10
+    Step 51: INSPECT acc_0089                    flagged=10/10  suspects=0  steps_left=9
+    Step 52: INSPECT acc_0046                    flagged=10/10  suspects=0  steps_left=8
+    Step 53: INSPECT acc_0116                    flagged=10/10  suspects=0  steps_left=7
+    Step 54: INSPECT acc_0121                    flagged=10/10  suspects=0  steps_left=6
+    Step 55: INSPECT acc_0156                    flagged=10/10  suspects=0  steps_left=5
+    Step 56: INSPECT acc_0141                    flagged=10/10  suspects=0  steps_left=4
+    Step 57: INSPECT acc_0188                    flagged=10/10  suspects=0  steps_left=3
+    Step 58: SUBMIT                              flagged=10/10  suspects=0  steps_left=3
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.53
+    ★ GRADER SCORE: 0.9060
+
+  --- LLM Agent: task=medium, seed=2, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0107                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0175                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0030                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0041                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0054                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0199                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0181                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0166                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: INSPECT acc_0098                    flagged=0/10  suspects=0  steps_left=40
+    Step 11: INSPECT acc_0121                    flagged=0/10  suspects=0  steps_left=39
+    Step 12: INSPECT acc_0053                    flagged=0/10  suspects=0  steps_left=38
+    Step 13: INSPECT acc_0103                    flagged=0/10  suspects=0  steps_left=37
+    Step 14: INSPECT acc_0168                    flagged=0/10  suspects=0  steps_left=36
+    Step 15: INSPECT acc_0000                    flagged=0/10  suspects=0  steps_left=35
+    Step 16: INSPECT acc_0040                    flagged=0/10  suspects=0  steps_left=34
+    Step 17: INSPECT acc_0149                    flagged=0/10  suspects=0  steps_left=33
+    Step 18: INSPECT acc_0064                    flagged=0/10  suspects=0  steps_left=32
+    Step 19: INSPECT acc_0016                    flagged=0/10  suspects=0  steps_left=31
+    Step 20: INSPECT acc_0105                    flagged=0/10  suspects=0  steps_left=30
+    Step 21: INSPECT acc_0035                    flagged=0/10  suspects=0  steps_left=29
+    Step 22: FLAG acc_0035                       flagged=1/10  suspects=9  steps_left=29
+    Step 23: INSPECT acc_0020                    flagged=1/10  suspects=9  steps_left=28
+    Step 24: FLAG acc_0020                       flagged=2/10  suspects=8  steps_left=28
+    Step 25: INSPECT acc_0036                    flagged=2/10  suspects=8  steps_left=27
+    Step 26: FLAG acc_0036                       flagged=3/10  suspects=7  steps_left=27
+    Step 27: INSPECT acc_0050                    flagged=3/10  suspects=7  steps_left=26
+    Step 28: FLAG acc_0050                       flagged=4/10  suspects=6  steps_left=26
+    Step 29: INSPECT acc_0051                    flagged=4/10  suspects=6  steps_left=25
+    Step 30: FLAG acc_0051                       flagged=5/10  suspects=5  steps_left=25
+    Step 31: INSPECT acc_0085                    flagged=5/10  suspects=5  steps_left=24
+    Step 32: FLAG acc_0085                       flagged=6/10  suspects=4  steps_left=24
+    Step 33: INSPECT acc_0177                    flagged=6/10  suspects=4  steps_left=23
+    Step 34: FLAG acc_0177                       flagged=7/10  suspects=3  steps_left=23
+    Step 35: INSPECT acc_0170                    flagged=7/10  suspects=3  steps_left=22
+    Step 36: FLAG acc_0170                       flagged=8/10  suspects=2  steps_left=22
+    Step 37: INSPECT acc_0055                    flagged=8/10  suspects=2  steps_left=21
+    Step 38: FLAG acc_0055                       flagged=9/10  suspects=1  steps_left=21
+    Step 39: INSPECT acc_0094                    flagged=9/10  suspects=1  steps_left=20
+    Step 40: FLAG acc_0094                       flagged=10/10  suspects=0  steps_left=20
+    Step 41: INSPECT acc_0185                    flagged=10/10  suspects=0  steps_left=19
+    Step 42: INSPECT acc_0175                    flagged=10/10  suspects=0  steps_left=18
+    Step 43: INSPECT acc_0192                    flagged=10/10  suspects=0  steps_left=17
+    Step 44: INSPECT acc_0013                    flagged=10/10  suspects=0  steps_left=16
+    Step 45: INSPECT acc_0099                    flagged=10/10  suspects=0  steps_left=15
+    Step 46: INSPECT acc_0075                    flagged=10/10  suspects=0  steps_left=14
+    Step 47: INSPECT acc_0058                    flagged=10/10  suspects=0  steps_left=13
+    Step 48: INSPECT acc_0072                    flagged=10/10  suspects=0  steps_left=12
+    Step 49: INSPECT acc_0070                    flagged=10/10  suspects=0  steps_left=11
+    Step 50: INSPECT acc_0071                    flagged=10/10  suspects=0  steps_left=10
+    Step 51: INSPECT acc_0175                    flagged=10/10  suspects=0  steps_left=9
+    Step 52: INSPECT acc_0181                    flagged=10/10  suspects=0  steps_left=8
+    Step 53: INSPECT acc_0175                    flagged=10/10  suspects=0  steps_left=7
+    Step 54: INSPECT acc_0181                    flagged=10/10  suspects=0  steps_left=6
+    Step 55: INSPECT acc_0175                    flagged=10/10  suspects=0  steps_left=5
+    Step 56: INSPECT acc_0181                    flagged=10/10  suspects=0  steps_left=4
+    Step 57: INSPECT acc_0175                    flagged=10/10  suspects=0  steps_left=3
+    Step 58: INSPECT acc_0181                    flagged=10/10  suspects=0  steps_left=2
+    Step 59: INSPECT acc_0175                    flagged=10/10  suspects=0  steps_left=1
+    Step 60: SUBMIT                              flagged=10/10  suspects=0  steps_left=1
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.51
+    ★ GRADER SCORE: 0.9020
+
+  medium: scores=['0.902', '0.906', '0.902']  mean=0.9033  var=0.000004
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0441                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0871                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0070                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0443                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0037                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0438                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: FLAG acc_0439                       flagged=1/10  suspects=5  steps_left=60
+    Step 22: INSPECT acc_0237                    flagged=1/10  suspects=5  steps_left=59
+    Step 23: FLAG acc_0237                       flagged=2/10  suspects=6  steps_left=59
+    Step 24: INSPECT acc_0621                    flagged=2/10  suspects=6  steps_left=58
+    Step 25: FLAG acc_0621                       flagged=3/10  suspects=6  steps_left=58
+    Step 26: INSPECT acc_0389                    flagged=3/10  suspects=6  steps_left=57
+    Step 27: FLAG acc_0389                       flagged=4/10  suspects=6  steps_left=57
+    Step 28: INSPECT acc_0160                    flagged=4/10  suspects=6  steps_left=56
+    Step 29: FLAG acc_0160                       flagged=5/10  suspects=5  steps_left=56
+    Step 30: INSPECT acc_0549                    flagged=5/10  suspects=5  steps_left=55
+    Step 31: FLAG acc_0549                       flagged=6/10  suspects=4  steps_left=55
+    Step 32: INSPECT acc_0658                    flagged=6/10  suspects=4  steps_left=54
+    Step 33: FLAG acc_0658                       flagged=7/10  suspects=3  steps_left=54
+    Step 34: INSPECT acc_0290                    flagged=7/10  suspects=3  steps_left=53
+    Step 35: FLAG acc_0290                       flagged=8/10  suspects=2  steps_left=53
+    Step 36: INSPECT acc_0124                    flagged=8/10  suspects=2  steps_left=52
+    Step 37: INSPECT acc_0507                    flagged=8/10  suspects=2  steps_left=51
+    Step 38: INSPECT acc_0124                    flagged=8/10  suspects=2  steps_left=50
+    Step 39: FLAG acc_0507                       flagged=9/10  suspects=1  steps_left=50
+    Step 40: FLAG acc_0124                       flagged=10/10  suspects=0  steps_left=50
+    Step 41: INSPECT acc_0086                    flagged=10/10  suspects=0  steps_left=49
+    Step 42: INSPECT acc_0497                    flagged=10/10  suspects=0  steps_left=48
+    Step 43: INSPECT acc_0610                    flagged=10/10  suspects=0  steps_left=47
+    Step 44: INSPECT acc_0579                    flagged=10/10  suspects=0  steps_left=46
+    Step 45: INSPECT acc_0573                    flagged=10/10  suspects=0  steps_left=45
+    Step 46: INSPECT acc_0479                    flagged=10/10  suspects=0  steps_left=44
+    Step 47: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=43
+    Step 48: INSPECT acc_0438                    flagged=10/10  suspects=0  steps_left=42
+    Step 49: INSPECT acc_0287                    flagged=10/10  suspects=0  steps_left=41
+    Step 50: INSPECT acc_0960                    flagged=10/10  suspects=0  steps_left=40
+    Step 51: INSPECT acc_0455                    flagged=10/10  suspects=0  steps_left=39
+    Step 52: INSPECT acc_0248                    flagged=10/10  suspects=0  steps_left=38
+    Step 53: INSPECT acc_0964                    flagged=10/10  suspects=0  steps_left=37
+    Step 54: INSPECT acc_0480                    flagged=10/10  suspects=0  steps_left=36
+    Step 55: INSPECT acc_0154                    flagged=10/10  suspects=0  steps_left=35
+    Step 56: INSPECT acc_0368                    flagged=10/10  suspects=0  steps_left=34
+    Step 57: INSPECT acc_0426                    flagged=10/10  suspects=0  steps_left=33
+    Step 58: INSPECT acc_0810                    flagged=10/10  suspects=0  steps_left=32
+    Step 59: INSPECT acc_0040                    flagged=10/10  suspects=0  steps_left=31
+    Step 60: INSPECT acc_0538                    flagged=10/10  suspects=0  steps_left=30
+    Step 61: INSPECT acc_0940                    flagged=10/10  suspects=0  steps_left=29
+    Step 62: INSPECT acc_0668                    flagged=10/10  suspects=0  steps_left=28
+    Step 63: INSPECT acc_0721                    flagged=10/10  suspects=0  steps_left=27
+    Step 64: INSPECT acc_0787                    flagged=10/10  suspects=0  steps_left=26
+    Step 65: INSPECT acc_0639                    flagged=10/10  suspects=0  steps_left=25
+    Step 66: INSPECT acc_0700                    flagged=10/10  suspects=0  steps_left=24
+    Step 67: INSPECT acc_0353                    flagged=10/10  suspects=0  steps_left=23
+    Step 68: INSPECT acc_0620                    flagged=10/10  suspects=0  steps_left=22
+    Step 69: INSPECT acc_0499                    flagged=10/10  suspects=0  steps_left=21
+    Step 70: INSPECT acc_0207                    flagged=10/10  suspects=0  steps_left=20
+    Step 71: INSPECT acc_0011                    flagged=10/10  suspects=0  steps_left=19
+    Step 72: INSPECT acc_0524                    flagged=10/10  suspects=0  steps_left=18
+    Step 73: INSPECT acc_0553                    flagged=10/10  suspects=0  steps_left=17
+    Step 74: INSPECT acc_0948                    flagged=10/10  suspects=0  steps_left=16
+    Step 75: INSPECT acc_0333                    flagged=10/10  suspects=0  steps_left=15
+    Step 76: INSPECT acc_0574                    flagged=10/10  suspects=0  steps_left=14
+    Step 77: INSPECT acc_0258                    flagged=10/10  suspects=0  steps_left=13
+    Step 78: INSPECT acc_0742                    flagged=10/10  suspects=0  steps_left=12
+    Step 79: INSPECT acc_0354                    flagged=10/10  suspects=0  steps_left=11
+    Step 80: INSPECT acc_0022                    flagged=10/10  suspects=0  steps_left=10
+    Step 81: INSPECT acc_0232                    flagged=10/10  suspects=0  steps_left=9
+    Step 82: INSPECT acc_0123                    flagged=10/10  suspects=0  steps_left=8
+    Step 83: INSPECT acc_0844                    flagged=10/10  suspects=0  steps_left=7
+    Step 84: INSPECT acc_0757                    flagged=10/10  suspects=0  steps_left=6
+    Step 85: INSPECT acc_0653                    flagged=10/10  suspects=0  steps_left=5
+    Step 86: INSPECT acc_0119                    flagged=10/10  suspects=0  steps_left=4
+    Step 87: INSPECT acc_0514                    flagged=10/10  suspects=0  steps_left=3
+    Step 88: SUBMIT                              flagged=10/10  suspects=0  steps_left=3
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=13.23
+    ★ GRADER SCORE: 0.9038
+
+  --- LLM Agent: task=hard, seed=1, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0014                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0835                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0855                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0930                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0336                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0929                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0076                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0543                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0590                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0401                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0322                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0154                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0374                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0549                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0903                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0976                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0620                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0017                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0222                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0536                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: INSPECT acc_0112                    flagged=0/10  suspects=0  steps_left=59
+    Step 22: INSPECT acc_0577                    flagged=0/10  suspects=0  steps_left=58
+    Step 23: INSPECT acc_0517                    flagged=0/10  suspects=0  steps_left=57
+    Step 24: INSPECT acc_0113                    flagged=0/10  suspects=0  steps_left=56
+    Step 25: INSPECT acc_0167                    flagged=0/10  suspects=0  steps_left=55
+    Step 26: INSPECT acc_0697                    flagged=0/10  suspects=0  steps_left=54
+    Step 27: INSPECT acc_0271                    flagged=0/10  suspects=0  steps_left=53
+    Step 28: INSPECT acc_0681                    flagged=0/10  suspects=0  steps_left=52
+    Step 29: INSPECT acc_0530                    flagged=0/10  suspects=0  steps_left=51
+    Step 30: INSPECT acc_0353                    flagged=0/10  suspects=0  steps_left=50
+    Step 31: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=49
+    Step 32: INSPECT acc_0777                    flagged=0/10  suspects=0  steps_left=48
+    Step 33: INSPECT acc_0265                    flagged=0/10  suspects=0  steps_left=47
+    Step 34: INSPECT acc_0788                    flagged=0/10  suspects=0  steps_left=46
+    Step 35: INSPECT acc_0033                    flagged=0/10  suspects=0  steps_left=45
+    Step 36: INSPECT acc_0187                    flagged=0/10  suspects=0  steps_left=44
+    Step 37: INSPECT acc_0445                    flagged=0/10  suspects=0  steps_left=43
+    Step 38: INSPECT acc_0846                    flagged=0/10  suspects=0  steps_left=42
+    Step 39: INSPECT acc_0659                    flagged=0/10  suspects=0  steps_left=41
+    Step 40: INSPECT acc_0677                    flagged=0/10  suspects=0  steps_left=40
+    Step 41: INSPECT acc_0768                    flagged=0/10  suspects=0  steps_left=39
+    Step 42: INSPECT acc_0539                    flagged=0/10  suspects=0  steps_left=38
+    Step 43: INSPECT acc_0742                    flagged=0/10  suspects=0  steps_left=37
+    Step 44: INSPECT acc_0503                    flagged=0/10  suspects=0  steps_left=36
+    Step 45: INSPECT acc_0876                    flagged=0/10  suspects=0  steps_left=35
+    Step 46: INSPECT acc_0639                    flagged=0/10  suspects=0  steps_left=34
+    Step 47: INSPECT acc_0494                    flagged=0/10  suspects=0  steps_left=33
+    Step 48: INSPECT acc_0898                    flagged=0/10  suspects=0  steps_left=32
+    Step 49: INSPECT acc_0553                    flagged=0/10  suspects=0  steps_left=31
+    Step 50: INSPECT acc_0588                    flagged=0/10  suspects=0  steps_left=30
+    Step 51: INSPECT acc_0194                    flagged=0/10  suspects=0  steps_left=29
+    Step 52: INSPECT acc_0810                    flagged=0/10  suspects=0  steps_left=28
+    Step 53: INSPECT acc_0355                    flagged=0/10  suspects=0  steps_left=27
+    Step 54: INSPECT acc_0363                    flagged=0/10  suspects=0  steps_left=26
+    Step 55: INSPECT acc_0221                    flagged=0/10  suspects=0  steps_left=25
+    Step 56: INSPECT acc_0580                    flagged=0/10  suspects=0  steps_left=24
+    Step 57: INSPECT acc_0534                    flagged=0/10  suspects=0  steps_left=23
+    Step 58: INSPECT acc_0778                    flagged=0/10  suspects=0  steps_left=22
+    Step 59: INSPECT acc_0998                    flagged=0/10  suspects=0  steps_left=21
+    Step 60: INSPECT acc_0233                    flagged=0/10  suspects=0  steps_left=20
+    Step 61: INSPECT acc_0052                    flagged=0/10  suspects=0  steps_left=19
+    Step 62: INSPECT acc_0813                    flagged=0/10  suspects=0  steps_left=18
+    Step 63: INSPECT acc_0035                    flagged=0/10  suspects=0  steps_left=17
+    Step 64: INSPECT acc_0667                    flagged=0/10  suspects=0  steps_left=16
+    Step 65: INSPECT acc_0019                    flagged=0/10  suspects=0  steps_left=15
+    Step 66: INSPECT acc_0959                    flagged=0/10  suspects=0  steps_left=14
+    Step 67: INSPECT acc_0212                    flagged=0/10  suspects=0  steps_left=13
+    Step 68: INSPECT acc_0776                    flagged=0/10  suspects=0  steps_left=12
+    Step 69: INSPECT acc_0049                    flagged=0/10  suspects=0  steps_left=11
+    Step 70: INSPECT acc_0434                    flagged=0/10  suspects=0  steps_left=10
+    Step 71: INSPECT acc_0827                    flagged=0/10  suspects=0  steps_left=9
+    Step 72: INSPECT acc_0583                    flagged=0/10  suspects=0  steps_left=8
+    Step 73: INSPECT acc_0065                    flagged=0/10  suspects=0  steps_left=7
+    Step 74: INSPECT acc_0107                    flagged=0/10  suspects=0  steps_left=6
+    Step 75: INSPECT acc_0761                    flagged=0/10  suspects=0  steps_left=5
+    Step 76: INSPECT acc_0995                    flagged=0/10  suspects=0  steps_left=4
+    Step 77: INSPECT acc_0157                    flagged=0/10  suspects=0  steps_left=3
+    Step 78: INSPECT acc_0936                    flagged=0/10  suspects=0  steps_left=2
+    Step 79: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=1
+    Step 80: INSPECT acc_0691                    flagged=0/10  suspects=0  steps_left=0
+    → Episode ended: [LOSS] TP=0 FP=0 FN=10 Recall=0.00 Precision=0.00 Episode reward=-9.80
+    ★ GRADER SCORE: 0.0000
+
+  --- LLM Agent: task=hard, seed=2, model=Bedrock/us.meta.llama4-scout-17b-instruct-v1:0 ---
+    Step  1: INSPECT acc_0813                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0430                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0817                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0175                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0523                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0113                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0797                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0478                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0861                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0836                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0926                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0664                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0255                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0938                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0672                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: FLAG acc_0672                       flagged=1/10  suspects=6  steps_left=65
+    Step 17: INSPECT acc_0659                    flagged=1/10  suspects=6  steps_left=64
+    Step 18: FLAG acc_0659                       flagged=2/10  suspects=5  steps_left=64
+    Step 19: INSPECT acc_0290                    flagged=2/10  suspects=5  steps_left=63
+    Step 20: FLAG acc_0290                       flagged=3/10  suspects=5  steps_left=63
+    Step 21: INSPECT acc_0339                    flagged=3/10  suspects=5  steps_left=62
+    Step 22: FLAG acc_0339                       flagged=4/10  suspects=6  steps_left=62
+    Step 23: INSPECT acc_0544                    flagged=4/10  suspects=6  steps_left=61
+    Step 24: FLAG acc_0544                       flagged=5/10  suspects=5  steps_left=61
+    Step 25: INSPECT acc_0696                    flagged=5/10  suspects=5  steps_left=60
+    Step 26: FLAG acc_0696                       flagged=6/10  suspects=4  steps_left=60
+    Step 27: INSPECT acc_0541                    flagged=6/10  suspects=4  steps_left=59
+    Step 28: FLAG acc_0541                       flagged=7/10  suspects=3  steps_left=59
+    Step 29: INSPECT acc_0793                    flagged=7/10  suspects=3  steps_left=58
+    Step 30: FLAG acc_0793                       flagged=8/10  suspects=2  steps_left=58
+    Step 31: INSPECT acc_0214                    flagged=8/10  suspects=2  steps_left=57
+    Step 32: FLAG acc_0214                       flagged=9/10  suspects=1  steps_left=57
+    Step 33: INSPECT acc_0112                    flagged=9/10  suspects=1  steps_left=56
+    Step 34: FLAG acc_0112                       flagged=10/10  suspects=0  steps_left=56
+    Step 35: INSPECT acc_0348                    flagged=10/10  suspects=0  steps_left=55
+    Step 36: INSPECT acc_0721                    flagged=10/10  suspects=0  steps_left=54
+    Step 37: INSPECT acc_0321                    flagged=10/10  suspects=0  steps_left=53
+    Step 38: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=52
+    Step 39: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=51
+    Step 40: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=50
+    Step 41: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=49
+    Step 42: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=48
+    Step 43: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=47
+    Step 44: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=46
+    Step 45: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=45
+    Step 46: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=44
+    Step 47: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=43
+    Step 48: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=42
+    Step 49: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=41
+    Step 50: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=40
+    Step 51: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=39
+    Step 52: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=38
+    Step 53: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=37
+    Step 54: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=36
+    Step 55: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=35
+    Step 56: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=34
+    Step 57: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=33
+    Step 58: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=32
+    Step 59: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=31
+    Step 60: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=30
+    Step 61: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=29
+    Step 62: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=28
+    Step 63: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=27
+    Step 64: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=26
+    Step 65: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=25
+    Step 66: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=24
+    Step 67: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=23
+    Step 68: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=22
+    Step 69: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=21
+    Step 70: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=20
+    Step 71: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=19
+    Step 72: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=18
+    Step 73: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=17
+    Step 74: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=16
+    Step 75: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=15
+    Step 76: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=14
+    Step 77: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=13
+    Step 78: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=12
+    Step 79: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=11
+    Step 80: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=10
+    Step 81: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=9
+    Step 82: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=8
+    Step 83: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=7
+    Step 84: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=6
+    Step 85: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=5
+    Step 86: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=4
+    Step 87: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=3
+    Step 88: INSPECT acc_0349                    flagged=10/10  suspects=0  steps_left=2
+    Step 89: INSPECT acc_0523                    flagged=10/10  suspects=0  steps_left=1
+    Step 90: SUBMIT                              flagged=10/10  suspects=0  steps_left=1
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=13.21
+    ★ GRADER SCORE: 0.9012
+
+  hard: scores=['0.904', '0.000', '0.901']  mean=0.6017  var=0.181003
+
+============================================================
+EVALUATION COMPLETE
+============================================================
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26/eval-models$

model-benchmark-logs/mistral_judge_log.txt

@@ -0,0 +1,410 @@
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26$ python3 mistral_test_judge_eval.py --url https://pandago-graphstrike.hf.space --bedrock
+GraphStrike Judge Evaluation Simulator
+Target:  https://pandago-graphstrike.hf.space
+Backend: bedrock
+Model:   Bedrock/mistral.ministral-3-8b-instruct
+Token:   set
+
+============================================================
+PHASE 0: Endpoint Verification
+============================================================
+  ✓ GET /health
+  ✓ GET /tasks
+  ✓ GET /metadata
+  ✓ GET /schema
+  ✓ GET /web
+  ✓ POST /reset
+  ✓ GET /state
+  ✓ POST /step
+  ✓ POST /step
+  ✓ GET /grader
+  ✓ POST /mcp
+  ✓ POST /baseline
+
+============================================================
+PHASE 1: Baseline Stability (3 runs)
+============================================================
+  Run 1: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 2: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 3: easy=0.9100  medium=0.9060  hard=0.9038
+  ✓ All 3 runs identical — baseline is deterministic
+
+============================================================
+PHASE 2: LLM Agent Evaluation (model=Bedrock/mistral.ministral-3-8b-instruct)
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=29
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=28
+    Step  4: FLAG acc_0036                       flagged=2/10  suspects=8  steps_left=28
+    Step  5: INSPECT acc_0001                    flagged=2/10  suspects=8  steps_left=27
+    Step  6: FLAG acc_0001                       flagged=3/10  suspects=7  steps_left=27
+    Step  7: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=26
+    Step  8: INSPECT acc_0012                    flagged=3/10  suspects=7  steps_left=25
+    Step  9: INSPECT acc_0000                    flagged=3/10  suspects=7  steps_left=24
+    Step 10: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=24
+    Step 11: FLAG acc_0000                       flagged=5/10  suspects=5  steps_left=24
+    Step 12: INSPECT acc_0027                    flagged=5/10  suspects=5  steps_left=23
+    Step 13: FLAG acc_0012                       flagged=6/10  suspects=4  steps_left=23
+    Step 14: FLAG acc_0027                       flagged=7/10  suspects=3  steps_left=23
+    Step 15: INSPECT acc_0047                    flagged=7/10  suspects=3  steps_left=22
+    Step 16: FLAG acc_0047                       flagged=8/10  suspects=2  steps_left=22
+    Step 17: INSPECT acc_0007                    flagged=8/10  suspects=2  steps_left=21
+    Step 18: FLAG acc_0007                       flagged=9/10  suspects=1  steps_left=21
+    Step 19: INSPECT acc_0028                    flagged=9/10  suspects=1  steps_left=20
+    Step 20: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=20
+    Step 21: SUBMIT                              flagged=10/10  suspects=0  steps_left=20
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.90
+    ★ GRADER SCORE: 0.9667
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=41
+    Step 11: INSPECT acc_0181                    flagged=1/10  suspects=8  steps_left=40
+    Step 12: FLAG acc_0181                       flagged=2/10  suspects=8  steps_left=40
+    Step 13: INSPECT acc_0022                    flagged=2/10  suspects=8  steps_left=39
+    Step 14: FLAG acc_0022                       flagged=3/10  suspects=7  steps_left=39
+    Step 15: INSPECT acc_0092                    flagged=3/10  suspects=7  steps_left=38
+    Step 16: FLAG acc_0092                       flagged=4/10  suspects=6  steps_left=38
+    Step 17: INSPECT acc_0097                    flagged=4/10  suspects=6  steps_left=37
+    Step 18: FLAG acc_0097                       flagged=5/10  suspects=5  steps_left=37
+    Step 19: INSPECT acc_0187                    flagged=5/10  suspects=5  steps_left=36
+    Step 20: FLAG acc_0187                       flagged=6/10  suspects=4  steps_left=36
+    Step 21: INSPECT acc_0093                    flagged=6/10  suspects=4  steps_left=35
+    Step 22: FLAG acc_0093                       flagged=7/10  suspects=3  steps_left=35
+    Step 23: INSPECT acc_0172                    flagged=7/10  suspects=3  steps_left=34
+    Step 24: FLAG acc_0172                       flagged=8/10  suspects=2  steps_left=34
+    Step 25: INSPECT acc_0058                    flagged=8/10  suspects=2  steps_left=33
+    Step 26: FLAG acc_0058                       flagged=9/10  suspects=1  steps_left=33
+    Step 27: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=32
+    Step 28: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=32
+    Step 29: SUBMIT                              flagged=10/10  suspects=0  steps_left=32
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.82
+    ★ GRADER SCORE: 0.9640
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0441                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0871                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0070                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0443                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0037                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0438                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: FLAG acc_0439                       flagged=1/10  suspects=5  steps_left=60
+    Step 22: INSPECT acc_0237                    flagged=1/10  suspects=5  steps_left=59
+    Step 23: FLAG acc_0237                       flagged=2/10  suspects=6  steps_left=59
+    Step 24: INSPECT acc_0621                    flagged=2/10  suspects=6  steps_left=58
+    Step 25: FLAG acc_0621                       flagged=3/10  suspects=6  steps_left=58
+    Step 26: INSPECT acc_0389                    flagged=3/10  suspects=6  steps_left=57
+    Step 27: INSPECT acc_0160                    flagged=3/10  suspects=6  steps_left=56
+    Step 28: FLAG acc_0160                       flagged=4/10  suspects=6  steps_left=56
+    Step 29: INSPECT acc_0549                    flagged=4/10  suspects=6  steps_left=55
+    Step 30: FLAG acc_0549                       flagged=5/10  suspects=5  steps_left=55
+    Step 31: INSPECT acc_0658                    flagged=5/10  suspects=5  steps_left=54
+    Step 32: FLAG acc_0658                       flagged=6/10  suspects=4  steps_left=54
+    Step 33: INSPECT acc_0290                    flagged=6/10  suspects=4  steps_left=53
+    Step 34: FLAG acc_0389                       flagged=7/10  suspects=3  steps_left=53
+    Step 35: FLAG acc_0290                       flagged=8/10  suspects=2  steps_left=53
+    Step 36: INSPECT acc_0124                    flagged=8/10  suspects=2  steps_left=52
+    Step 37: FLAG acc_0124                       flagged=9/10  suspects=1  steps_left=52
+    Step 38: INSPECT acc_0507                    flagged=9/10  suspects=1  steps_left=51
+    Step 39: FLAG acc_0507                       flagged=10/10  suspects=0  steps_left=51
+    Step 40: SUBMIT                              flagged=10/10  suspects=0  steps_left=51
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.71
+    ★ GRADER SCORE: 0.9637
+
+  Summary: easy=0.9667  medium=0.9640  hard=0.9637
+
+============================================================
+PHASE 3: Score Variance (seeds=[0, 1, 2])
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=29
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=28
+    Step  4: FLAG acc_0036                       flagged=2/10  suspects=8  steps_left=28
+    Step  5: INSPECT acc_0001                    flagged=2/10  suspects=8  steps_left=27
+    Step  6: FLAG acc_0001                       flagged=3/10  suspects=7  steps_left=27
+    Step  7: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=26
+
+    Step  8: INSPECT acc_0012                    flagged=3/10  suspects=7  steps_left=25
+    Step  9: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=25
+    Step 10: INSPECT acc_0000                    flagged=4/10  suspects=6  steps_left=24
+    Step 11: FLAG acc_0012                       flagged=5/10  suspects=5  steps_left=24
+    Step 12: INSPECT acc_0027                    flagged=5/10  suspects=5  steps_left=23
+    Step 13: FLAG acc_0000                       flagged=6/10  suspects=4  steps_left=23
+    Step 14: INSPECT acc_0047                    flagged=6/10  suspects=4  steps_left=22
+    Step 15: FLAG acc_0027                       flagged=7/10  suspects=3  steps_left=22
+    Step 16: FLAG acc_0047                       flagged=8/10  suspects=2  steps_left=22
+    Step 17: INSPECT acc_0007                    flagged=8/10  suspects=2  steps_left=21
+    Step 18: FLAG acc_0007                       flagged=9/10  suspects=1  steps_left=21
+    Step 19: INSPECT acc_0028                    flagged=9/10  suspects=1  steps_left=20
+    Step 20: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=20
+    Step 21: SUBMIT                              flagged=10/10  suspects=0  steps_left=20
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.90
+    ★ GRADER SCORE: 0.9667
+
+  --- LLM Agent: task=easy, seed=1, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0034                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: INSPECT acc_0003                    flagged=0/10  suspects=0  steps_left=28
+    Step  3: INSPECT acc_0049                    flagged=0/10  suspects=0  steps_left=27
+    Step  4: INSPECT acc_0006                    flagged=0/10  suspects=0  steps_left=26
+    Step  5: INSPECT acc_0047                    flagged=0/10  suspects=0  steps_left=25
+    Step  6: FLAG acc_0047                       flagged=1/10  suspects=9  steps_left=25
+    Step  7: INSPECT acc_0009                    flagged=1/10  suspects=9  steps_left=24
+    Step  8: FLAG acc_0009                       flagged=2/10  suspects=8  steps_left=24
+    Step  9: INSPECT acc_0046                    flagged=2/10  suspects=8  steps_left=23
+    Step 10: FLAG acc_0046                       flagged=3/10  suspects=7  steps_left=23
+    Step 11: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=22
+    Step 12: INSPECT acc_0021                    flagged=3/10  suspects=7  steps_left=21
+    Step 13: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=21
+    Step 14: INSPECT acc_0002                    flagged=4/10  suspects=6  steps_left=20
+    Step 15: FLAG acc_0021                       flagged=5/10  suspects=5  steps_left=20
+    Step 16: INSPECT acc_0048                    flagged=5/10  suspects=5  steps_left=19
+    Step 17: INSPECT acc_0029                    flagged=5/10  suspects=5  steps_left=18
+    Step 18: FLAG acc_0029                       flagged=6/10  suspects=4  steps_left=18
+    Step 19: FLAG acc_0048                       flagged=7/10  suspects=3  steps_left=18
+    Step 20: FLAG acc_0002                       flagged=8/10  suspects=2  steps_left=18
+    Step 21: INSPECT acc_0015                    flagged=8/10  suspects=2  steps_left=17
+    Step 22: FLAG acc_0015                       flagged=9/10  suspects=1  steps_left=17
+    Step 23: INSPECT acc_0005                    flagged=9/10  suspects=1  steps_left=16
+    Step 24: FLAG acc_0005                       flagged=10/10  suspects=0  steps_left=16
+    Step 25: SUBMIT                              flagged=10/10  suspects=0  steps_left=16
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.86
+    ★ GRADER SCORE: 0.9533
+
+  --- LLM Agent: task=easy, seed=2, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0040                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: INSPECT acc_0017                    flagged=0/10  suspects=0  steps_left=28
+    Step  3: INSPECT acc_0025                    flagged=0/10  suspects=0  steps_left=27
+    Step  4: INSPECT acc_0026                    flagged=0/10  suspects=0  steps_left=26
+    Step  5: INSPECT acc_0038                    flagged=0/10  suspects=0  steps_left=25
+    Step  6: FLAG acc_0029                       flagged=1/10  suspects=9  steps_left=25
+    Step  7: INSPECT acc_0006                    flagged=1/10  suspects=9  steps_left=24
+    Step  8: FLAG acc_0006                       flagged=2/10  suspects=8  steps_left=24
+    Step  9: INSPECT acc_0033                    flagged=2/10  suspects=8  steps_left=23
+    Step 10: FLAG acc_0033                       flagged=3/10  suspects=7  steps_left=23
+    Step 11: INSPECT acc_0015                    flagged=3/10  suspects=7  steps_left=22
+    Step 12: FLAG acc_0015                       flagged=4/10  suspects=6  steps_left=22
+    Step 13: INSPECT acc_0022                    flagged=4/10  suspects=6  steps_left=21
+    Step 14: INSPECT acc_0009                    flagged=4/10  suspects=6  steps_left=20
+    Step 15: FLAG acc_0022                       flagged=5/10  suspects=5  steps_left=20
+    Step 16: INSPECT acc_0004                    flagged=5/10  suspects=5  steps_left=19
+    Step 17: FLAG acc_0009                       flagged=6/10  suspects=4  steps_left=19
+    Step 18: FLAG acc_0004                       flagged=7/10  suspects=3  steps_left=19
+    Step 19: INSPECT acc_0024                    flagged=7/10  suspects=3  steps_left=18
+    Step 20: FLAG acc_0024                       flagged=8/10  suspects=2  steps_left=18
+    Step 21: INSPECT acc_0049                    flagged=8/10  suspects=2  steps_left=17
+    Step 22: FLAG acc_0049                       flagged=9/10  suspects=1  steps_left=17
+    Step 23: INSPECT acc_0035                    flagged=9/10  suspects=1  steps_left=16
+    Step 24: FLAG acc_0035                       flagged=10/10  suspects=0  steps_left=16
+    Step 25: SUBMIT                              flagged=10/10  suspects=0  steps_left=16
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.86
+    ★ GRADER SCORE: 0.9533
+
+  easy: scores=['0.967', '0.953', '0.953']  mean=0.9578  var=0.000040
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=41
+    Step 11: INSPECT acc_0181                    flagged=1/10  suspects=8  steps_left=40
+    Step 12: FLAG acc_0181                       flagged=2/10  suspects=8  steps_left=40
+    Step 13: INSPECT acc_0022                    flagged=2/10  suspects=8  steps_left=39
+    Step 14: FLAG acc_0022                       flagged=3/10  suspects=7  steps_left=39
+    Step 15: INSPECT acc_0092                    flagged=3/10  suspects=7  steps_left=38
+    Step 16: FLAG acc_0092                       flagged=4/10  suspects=6  steps_left=38
+    Step 17: INSPECT acc_0097                    flagged=4/10  suspects=6  steps_left=37
+    Step 18: FLAG acc_0097                       flagged=5/10  suspects=5  steps_left=37
+    Step 19: INSPECT acc_0187                    flagged=5/10  suspects=5  steps_left=36
+    Step 20: FLAG acc_0187                       flagged=6/10  suspects=4  steps_left=36
+    Step 21: INSPECT acc_0093                    flagged=6/10  suspects=4  steps_left=35
+    Step 22: FLAG acc_0093                       flagged=7/10  suspects=3  steps_left=35
+    Step 23: INSPECT acc_0172                    flagged=7/10  suspects=3  steps_left=34
+    Step 24: FLAG acc_0172                       flagged=8/10  suspects=2  steps_left=34
+    Step 25: INSPECT acc_0058                    flagged=8/10  suspects=2  steps_left=33
+    Step 26: FLAG acc_0058                       flagged=9/10  suspects=1  steps_left=33
+    Step 27: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=32
+    Step 28: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=32
+    Step 29: SUBMIT                              flagged=10/10  suspects=0  steps_left=32
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.82
+    ★ GRADER SCORE: 0.9640
+
+  --- LLM Agent: task=medium, seed=1, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0171                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0099                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0152                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0092                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0078                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0112                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0012                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: FLAG acc_0012                       flagged=1/10  suspects=8  steps_left=43
+    Step  9: INSPECT acc_0033                    flagged=1/10  suspects=8  steps_left=42
+    Step 10: FLAG acc_0033                       flagged=2/10  suspects=8  steps_left=42
+    Step 11: INSPECT acc_0174                    flagged=2/10  suspects=8  steps_left=41
+    Step 12: FLAG acc_0174                       flagged=3/10  suspects=7  steps_left=41
+    Step 13: INSPECT acc_0187                    flagged=3/10  suspects=7  steps_left=40
+    Step 14: FLAG acc_0187                       flagged=4/10  suspects=6  steps_left=40
+    Step 15: INSPECT acc_0079                    flagged=4/10  suspects=6  steps_left=39
+    Step 16: FLAG acc_0032                       flagged=5/10  suspects=5  steps_left=39
+    Step 17: INSPECT acc_0023                    flagged=5/10  suspects=5  steps_left=38
+    Step 18: FLAG acc_0023                       flagged=6/10  suspects=4  steps_left=38
+    Step 19: INSPECT acc_0146                    flagged=6/10  suspects=4  steps_left=37
+    Step 20: FLAG acc_0079                       flagged=7/10  suspects=3  steps_left=37
+    Step 21: INSPECT acc_0019                    flagged=7/10  suspects=3  steps_left=36
+    Step 22: FLAG acc_0146                       flagged=8/10  suspects=2  steps_left=36
+    Step 23: FLAG acc_0019                       flagged=9/10  suspects=1  steps_left=36
+    Step 24: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=35
+    Step 25: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=35
+    Step 26: SUBMIT                              flagged=10/10  suspects=0  steps_left=35
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.85
+    ★ GRADER SCORE: 0.9700
+
+  --- LLM Agent: task=medium, seed=2, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0107                    flagged=4/10  suspects=6  steps_left=20
+    Step  3: FLAG acc_0014                       flagged=10/10  suspects=0  steps_left=20
+    Step  4: SUBMIT                              flagged=0/10  suspects=0  steps_left=42
+    → Episode ended: [LOSS] TP=0 FP=0 FN=10 Recall=0.00 Precision=0.00 Episode reward=-2.08
+    ★ GRADER SCORE: 0.0000
+
+  medium: scores=['0.964', '0.970', '0.000']  mean=0.6447  var=0.207804
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=31
+    Step  3: INSPECT acc_0105                    flagged=0/10  suspects=0  steps_left=67
+    Step  4: INSPECT acc_0374                    flagged=0/10  suspects=0  steps_left=57
+    Step  5: INSPECT acc_0577                    flagged=10/10  suspects=0  steps_left=0
+    ★ GRADER SCORE: 0.9000
+
+  --- LLM Agent: task=hard, seed=1, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0014                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0835                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0855                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0930                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0336                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0929                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0076                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0543                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0590                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0401                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0322                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0154                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0374                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0549                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0903                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0976                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0620                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0017                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0222                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0536                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: INSPECT acc_0112                    flagged=0/10  suspects=0  steps_left=59
+    Step 22: INSPECT acc_0577                    flagged=0/10  suspects=0  steps_left=58
+    Step 23: INSPECT acc_0517                    flagged=0/10  suspects=0  steps_left=57
+    Step 24: INSPECT acc_0113                    flagged=0/10  suspects=0  steps_left=56
+    Step 25: INSPECT acc_0167                    flagged=0/10  suspects=0  steps_left=55
+    Step 26: INSPECT acc_0697                    flagged=0/10  suspects=0  steps_left=54
+    Step 27: INSPECT acc_0271                    flagged=0/10  suspects=0  steps_left=53
+    Step 28: INSPECT acc_0681                    flagged=0/10  suspects=0  steps_left=52
+    Step 29: INSPECT acc_0530                    flagged=0/10  suspects=0  steps_left=51
+    Step 30: INSPECT acc_0353                    flagged=0/10  suspects=0  steps_left=50
+    Step 31: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=49
+    Step 32: INSPECT acc_0777                    flagged=0/10  suspects=0  steps_left=48
+    Step 33: INSPECT acc_0265                    flagged=0/10  suspects=0  steps_left=47
+    Step 34: INSPECT acc_0788                    flagged=0/10  suspects=0  steps_left=46
+    Step 35: INSPECT acc_0033                    flagged=0/10  suspects=0  steps_left=45
+    Step 36: INSPECT acc_0187                    flagged=0/10  suspects=0  steps_left=44
+    Step 37: INSPECT acc_0445                    flagged=0/10  suspects=0  steps_left=43
+    Step 38: INSPECT acc_0846                    flagged=0/10  suspects=0  steps_left=42
+    Step 39: INSPECT acc_0659                    flagged=0/10  suspects=0  steps_left=41
+    Step 40: INSPECT acc_0768                    flagged=0/10  suspects=0  steps_left=40
+    Step 41: INSPECT acc_0677                    flagged=0/10  suspects=0  steps_left=39
+    Step 42: INSPECT acc_0539                    flagged=0/10  suspects=0  steps_left=38
+    Step 43: INSPECT acc_0742                    flagged=0/10  suspects=0  steps_left=37
+    Step 44: INSPECT acc_0503                    flagged=0/10  suspects=0  steps_left=36
+    Step 45: INSPECT acc_0876                    flagged=0/10  suspects=0  steps_left=35
+    Step 46: INSPECT acc_0639                    flagged=0/10  suspects=0  steps_left=34
+    Step 47: INSPECT acc_0494                    flagged=0/10  suspects=0  steps_left=33
+    Step 48: INSPECT acc_0898                    flagged=0/10  suspects=0  steps_left=32
+    Step 49: INSPECT acc_0553                    flagged=0/10  suspects=0  steps_left=31
+    Step 50: INSPECT acc_0588                    flagged=0/10  suspects=0  steps_left=30
+    Step 51: INSPECT acc_0194                    flagged=0/10  suspects=0  steps_left=29
+    Step 52: INSPECT acc_0810                    flagged=0/10  suspects=0  steps_left=28
+    Step 53: INSPECT acc_0355                    flagged=0/10  suspects=0  steps_left=27
+    Step 54: INSPECT acc_0363                    flagged=0/10  suspects=0  steps_left=26
+    Step 55: INSPECT acc_0221                    flagged=0/10  suspects=0  steps_left=25
+    Step 56: INSPECT acc_0580                    flagged=0/10  suspects=0  steps_left=24
+    Step 57: INSPECT acc_0534                    flagged=0/10  suspects=0  steps_left=23
+    Step 58: INSPECT acc_0778                    flagged=0/10  suspects=0  steps_left=22
+    Step 59: INSPECT acc_0998                    flagged=0/10  suspects=0  steps_left=21
+    Step 60: INSPECT acc_0233                    flagged=0/10  suspects=0  steps_left=20
+    Step 61: INSPECT acc_0052                    flagged=0/10  suspects=0  steps_left=19
+    Step 62: INSPECT acc_0813                    flagged=4/10  suspects=6  steps_left=21
+    Step 63: FLAG acc_0014                       flagged=5/10  suspects=5  steps_left=21
+    Step 64: FLAG acc_0028                       flagged=6/10  suspects=4  steps_left=21
+    Step 65: FLAG acc_0000                       flagged=7/10  suspects=3  steps_left=21
+    Step 66: FLAG acc_0012                       flagged=8/10  suspects=2  steps_left=21
+    Step 67: FLAG acc_0007                       flagged=9/10  suspects=1  steps_left=21
+    Step 68: FLAG acc_0036                       flagged=10/10  suspects=0  steps_left=21
+    Step 69: SUBMIT                              flagged=10/10  suspects=0  steps_left=21
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.91
+    ★ GRADER SCORE: 0.9700
+
+  --- LLM Agent: task=hard, seed=2, model=Bedrock/mistral.ministral-3-8b-instruct ---
+    Step  1: INSPECT acc_0813                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0430                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0817                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0175                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0523                    flagged=1/10  suspects=1  steps_left=72
+    Step  6: INSPECT acc_0797                    flagged=1/10  suspects=1  steps_left=66
+    Step  7: INSPECT acc_0664                    flagged=1/10  suspects=1  steps_left=65
+    Step  8: INSPECT acc_0255                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=40
+    Step 11: INSPECT acc_0181                    flagged=1/10  suspects=8  steps_left=39
+    Step 12: FLAG acc_0181                       flagged=7/10  suspects=11  steps_left=51
+    Step 13: FLAG acc_0389                       flagged=8/10  suspects=10  steps_left=51
+    Step 14: FLAG acc_0658                       flagged=9/10  suspects=9  steps_left=51
+    Step 15: FLAG acc_0507                       flagged=10/10  suspects=0  steps_left=16
+    Step 16: SUBMIT                              flagged=0/10  suspects=0  steps_left=30
+    → Episode ended: [LOSS] TP=0 FP=0 FN=10 Recall=0.00 Precision=0.00 Episode reward=-2.00
+    ★ GRADER SCORE: 0.0000
+
+  hard: scores=['0.900', '0.970', '0.000']  mean=0.6233  var=0.195089
+
+============================================================
+EVALUATION COMPLETE
+============================================================
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26$
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26$

model-benchmark-logs/nvidia_judge_log.txt

@@ -0,0 +1,545 @@
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26$ python3 nvidia_test_judge_eval.py --url https://
+pandago-graphstrike.hf.space --bedrock
+GraphStrike Judge Evaluation Simulator
+Target:  https://pandago-graphstrike.hf.space
+Backend: bedrock
+Model:   Bedrock/nvidia.nemotron-super-3-120b
+Token:   set
+
+============================================================
+PHASE 0: Endpoint Verification
+============================================================
+  ✓ GET /health
+  ✓ GET /tasks
+  ✓ GET /metadata
+  ✓ GET /schema
+  ✓ GET /web
+  ✓ POST /reset
+  ✓ GET /state
+  ✓ POST /step
+  ✓ POST /step
+  ✓ GET /grader
+  ✓ POST /mcp
+  ✓ POST /baseline
+
+============================================================
+PHASE 1: Baseline Stability (3 runs)
+============================================================
+  Run 1: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 2: easy=0.9100  medium=0.9060  hard=0.9038
+  Run 3: easy=0.9100  medium=0.9060  hard=0.9038
+  ✓ All 3 runs identical — baseline is deterministic
+
+============================================================
+PHASE 2: LLM Agent Evaluation (model=Bedrock/nvidia.nemotron-super-3-120b)
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=27
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=27
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=25
+    Step  4: FLAG acc_0036                       flagged=2/10  suspects=8  steps_left=25
+    Step  5: INSPECT acc_0047                    flagged=2/10  suspects=8  steps_left=24
+    Step  6: FLAG acc_0047                       flagged=3/10  suspects=7  steps_left=23
+    Step  7: INSPECT acc_0007                    flagged=3/10  suspects=7  steps_left=22
+    Step  8: FLAG acc_0007                       flagged=4/10  suspects=6  steps_left=21
+    Step  9: INSPECT acc_0001                    flagged=4/10  suspects=6  steps_left=20
+    Step 10: FLAG acc_0001                       flagged=5/10  suspects=5  steps_left=19
+    Step 11: INSPECT acc_0014                    flagged=5/10  suspects=5  steps_left=18
+    Step 12: FLAG acc_0014                       flagged=6/10  suspects=4  steps_left=17
+    Step 13: INSPECT acc_0012                    flagged=6/10  suspects=4  steps_left=16
+    Step 14: FLAG acc_0012                       flagged=7/10  suspects=3  steps_left=15
+    Step 15: INSPECT acc_0000                    flagged=7/10  suspects=3  steps_left=14
+    Step 16: FLAG acc_0000                       flagged=8/10  suspects=2  steps_left=13
+    Step 17: INSPECT acc_0027                    flagged=8/10  suspects=2  steps_left=12
+    Step 18: FLAG acc_0027                       flagged=9/10  suspects=1  steps_left=11
+    Step 19: INSPECT acc_0028                    flagged=9/10  suspects=1  steps_left=10
+    Step 20: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=9
+    Step 21: SUBMIT                              flagged=10/10  suspects=0  steps_left=9
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.79
+    ★ GRADER SCORE: 0.9300
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=48
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=45
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=43
+    Step  4: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=41
+    Step  5: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=39
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=37
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=35
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=33
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=31
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=9  steps_left=31
+    Step 11: INSPECT acc_0131                    flagged=1/10  suspects=9  steps_left=29
+    Step 12: FLAG acc_0131                       flagged=2/10  suspects=8  steps_left=29
+    Step 13: INSPECT acc_0181                    flagged=2/10  suspects=8  steps_left=27
+    Step 14: FLAG acc_0181                       flagged=3/10  suspects=7  steps_left=26
+    Step 15: FLAG acc_0022                       flagged=4/10  suspects=6  steps_left=26
+    Step 16: INSPECT acc_0092                    flagged=4/10  suspects=6  steps_left=24
+    Step 17: FLAG acc_0092                       flagged=5/10  suspects=5  steps_left=24
+    Step 18: INSPECT acc_0097                    flagged=5/10  suspects=5  steps_left=22
+    Step 19: FLAG acc_0097                       flagged=6/10  suspects=4  steps_left=22
+    Step 20: INSPECT acc_0187                    flagged=6/10  suspects=4  steps_left=20
+    Step 21: FLAG acc_0187                       flagged=7/10  suspects=3  steps_left=20
+    Step 22: INSPECT acc_0093                    flagged=7/10  suspects=3  steps_left=19
+    Step 23: FLAG acc_0093                       flagged=8/10  suspects=2  steps_left=18
+    Step 24: INSPECT acc_0172                    flagged=8/10  suspects=2  steps_left=17
+    Step 25: FLAG acc_0172                       flagged=9/10  suspects=1  steps_left=16
+    Step 26: INSPECT acc_0058                    flagged=0/10  suspects=0  steps_left=79
+    Step 27: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=78
+    Step 28: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=77
+    Step 29: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=75
+    Step 30: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=73
+    Step 31: INSPECT acc_0441                    flagged=0/10  suspects=0  steps_left=70
+    Step 32: INSPECT acc_0871                    flagged=0/10  suspects=0  steps_left=68
+    Step 33: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=67
+    Step 34: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=65
+    Step 35: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=63
+    Step 36: INSPECT acc_0070                    flagged=0/10  suspects=0  steps_left=61
+    Step 37: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=58
+    Step 38: INSPECT acc_0443                    flagged=0/10  suspects=0  steps_left=56
+    Step 39: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=54
+    Step 40: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=53
+    Step 41: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=52
+    Step 42: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=50
+    Step 43: INSPECT acc_0037                    flagged=0/10  suspects=0  steps_left=47
+    Step 44: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=46
+    Step 45: INSPECT acc_0438                    flagged=0/10  suspects=0  steps_left=44
+    Step 46: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=42
+    Step 47: FLAG acc_0439                       flagged=1/10  suspects=5  steps_left=41
+    Step 48: INSPECT acc_0237                    flagged=1/10  suspects=5  steps_left=40
+    Step 49: FLAG acc_0237                       flagged=2/10  suspects=6  steps_left=40
+    Step 50: INSPECT acc_0621                    flagged=2/10  suspects=6  steps_left=39
+    Step 51: FLAG acc_0621                       flagged=3/10  suspects=6  steps_left=39
+    Step 52: INSPECT acc_0389                    flagged=3/10  suspects=6  steps_left=38
+    Step 53: FLAG acc_0389                       flagged=4/10  suspects=6  steps_left=38
+    Step 54: INSPECT acc_0160                    flagged=4/10  suspects=6  steps_left=37
+    Step 55: FLAG acc_0160                       flagged=5/10  suspects=5  steps_left=37
+    Step 56: INSPECT acc_0549                    flagged=5/10  suspects=5  steps_left=36
+    Step 57: FLAG acc_0549                       flagged=6/10  suspects=4  steps_left=36
+    Step 58: INSPECT acc_0658                    flagged=6/10  suspects=4  steps_left=35
+    Step 59: FLAG acc_0658                       flagged=7/10  suspects=3  steps_left=35
+    Step 60: INSPECT acc_0290                    flagged=7/10  suspects=3  steps_left=34
+    Step 61: FLAG acc_0290                       flagged=8/10  suspects=2  steps_left=34
+    Step 62: INSPECT acc_0124                    flagged=8/10  suspects=2  steps_left=33
+    Step 63: FLAG acc_0507                       flagged=9/10  suspects=1  steps_left=33
+    Step 64: FLAG acc_0124                       flagged=10/10  suspects=0  steps_left=33
+    Step 65: SUBMIT                              flagged=10/10  suspects=0  steps_left=33
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=14.53
+    ★ GRADER SCORE: 0.9413
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0441                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0871                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0070                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0443                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0037                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0438                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: FLAG acc_0439                       flagged=1/10  suspects=5  steps_left=60
+    Step 22: INSPECT acc_0237                    flagged=1/10  suspects=5  steps_left=59
+    Step 23: FLAG acc_0237                       flagged=2/10  suspects=6  steps_left=59
+    Step 24: INSPECT acc_0621                    flagged=2/10  suspects=6  steps_left=58
+    Step 25: FLAG acc_0621                       flagged=3/10  suspects=6  steps_left=58
+    Step 26: INSPECT acc_0389                    flagged=3/10  suspects=6  steps_left=57
+    Step 27: FLAG acc_0389                       flagged=4/10  suspects=6  steps_left=57
+    Step 28: INSPECT acc_0160                    flagged=4/10  suspects=6  steps_left=56
+    Step 29: FLAG acc_0160                       flagged=5/10  suspects=5  steps_left=56
+    Step 30: INSPECT acc_0549                    flagged=5/10  suspects=5  steps_left=55
+    Step 31: FLAG acc_0549                       flagged=6/10  suspects=4  steps_left=55
+    Step 32: INSPECT acc_0658                    flagged=6/10  suspects=4  steps_left=54
+    Step 33: FLAG acc_0658                       flagged=7/10  suspects=3  steps_left=54
+    Step 34: INSPECT acc_0290                    flagged=7/10  suspects=3  steps_left=53
+    Step 35: FLAG acc_0290                       flagged=8/10  suspects=2  steps_left=53
+    Step 36: INSPECT acc_0124                    flagged=8/10  suspects=2  steps_left=52
+    Step 37: FLAG acc_0124                       flagged=9/10  suspects=1  steps_left=52
+    Step 38: INSPECT acc_0507                    flagged=9/10  suspects=1  steps_left=51
+    Step 39: FLAG acc_0507                       flagged=10/10  suspects=0  steps_left=51
+    Step 40: SUBMIT                              flagged=10/10  suspects=0  steps_left=51
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.71
+    ★ GRADER SCORE: 0.9637
+
+  Summary: easy=0.9300  medium=0.9413  hard=0.9637
+
+============================================================
+PHASE 3: Score Variance (seeds=[0, 1, 2])
+============================================================
+
+  --- LLM Agent: task=easy, seed=0, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0043                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: FLAG acc_0043                       flagged=1/10  suspects=8  steps_left=29
+    Step  3: INSPECT acc_0036                    flagged=1/10  suspects=8  steps_left=28
+    Step  4: FLAG acc_0036                       flagged=2/10  suspects=8  steps_left=28
+    Step  5: INSPECT acc_0001                    flagged=2/10  suspects=8  steps_left=27
+    Step  6: FLAG acc_0001                       flagged=3/10  suspects=7  steps_left=27
+    Step  7: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=26
+    Step  8: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=26
+    Step  9: INSPECT acc_0012                    flagged=4/10  suspects=6  steps_left=25
+    Step 10: FLAG acc_0012                       flagged=5/10  suspects=5  steps_left=25
+    Step 11: INSPECT acc_0000                    flagged=5/10  suspects=5  steps_left=24
+    Step 12: FLAG acc_0000                       flagged=6/10  suspects=4  steps_left=24
+    Step 13: INSPECT acc_0027                    flagged=6/10  suspects=4  steps_left=23
+    Step 14: FLAG acc_0027                       flagged=7/10  suspects=3  steps_left=23
+    Step 15: INSPECT acc_0047                    flagged=7/10  suspects=3  steps_left=22
+    Step 16: FLAG acc_0047                       flagged=8/10  suspects=2  steps_left=22
+    Step 17: INSPECT acc_0007                    flagged=8/10  suspects=2  steps_left=21
+    Step 18: FLAG acc_0007                       flagged=9/10  suspects=1  steps_left=21
+    Step 19: INSPECT acc_0028                    flagged=9/10  suspects=1  steps_left=20
+    Step 20: FLAG acc_0028                       flagged=10/10  suspects=0  steps_left=20
+    Step 21: SUBMIT                              flagged=10/10  suspects=0  steps_left=20
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.90
+    ★ GRADER SCORE: 0.9667
+
+  --- LLM Agent: task=easy, seed=1, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0034                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: INSPECT acc_0003                    flagged=0/10  suspects=0  steps_left=28
+    Step  3: INSPECT acc_0049                    flagged=0/10  suspects=0  steps_left=27
+    Step  4: INSPECT acc_0006                    flagged=0/10  suspects=0  steps_left=26
+    Step  5: INSPECT acc_0047                    flagged=0/10  suspects=0  steps_left=25
+    Step  6: FLAG acc_0047                       flagged=1/10  suspects=9  steps_left=25
+    Step  7: INSPECT acc_0009                    flagged=1/10  suspects=9  steps_left=24
+    Step  8: FLAG acc_0009                       flagged=2/10  suspects=8  steps_left=24
+    Step  9: INSPECT acc_0046                    flagged=2/10  suspects=8  steps_left=23
+    Step 10: FLAG acc_0046                       flagged=3/10  suspects=7  steps_left=23
+    Step 11: INSPECT acc_0014                    flagged=3/10  suspects=7  steps_left=22
+    Step 12: FLAG acc_0014                       flagged=4/10  suspects=6  steps_left=22
+    Step 13: INSPECT acc_0021                    flagged=4/10  suspects=6  steps_left=21
+    Step 14: FLAG acc_0021                       flagged=5/10  suspects=5  steps_left=21
+    Step 15: INSPECT acc_0002                    flagged=5/10  suspects=5  steps_left=20
+    Step 16: FLAG acc_0002                       flagged=6/10  suspects=4  steps_left=20
+    Step 17: INSPECT acc_0048                    flagged=6/10  suspects=4  steps_left=19
+    Step 18: FLAG acc_0048                       flagged=7/10  suspects=3  steps_left=19
+    Step 19: INSPECT acc_0029                    flagged=7/10  suspects=3  steps_left=18
+    Step 20: FLAG acc_0029                       flagged=8/10  suspects=2  steps_left=18
+    Step 21: INSPECT acc_0015                    flagged=8/10  suspects=2  steps_left=17
+    Step 22: FLAG acc_0015                       flagged=9/10  suspects=1  steps_left=17
+    Step 23: INSPECT acc_0005                    flagged=9/10  suspects=1  steps_left=16
+    Step 24: FLAG acc_0005                       flagged=10/10  suspects=0  steps_left=16
+    Step 25: SUBMIT                              flagged=10/10  suspects=0  steps_left=16
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.86
+    ★ GRADER SCORE: 0.9533
+
+  --- LLM Agent: task=easy, seed=2, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0040                    flagged=0/10  suspects=0  steps_left=29
+    Step  2: INSPECT acc_0017                    flagged=0/10  suspects=0  steps_left=28
+    Step  3: INSPECT acc_0025                    flagged=0/10  suspects=0  steps_left=27
+    Step  4: INSPECT acc_0026                    flagged=0/10  suspects=0  steps_left=26
+    Step  5: INSPECT acc_0038                    flagged=0/10  suspects=0  steps_left=25
+    Step  6: INSPECT acc_0029                    flagged=0/10  suspects=0  steps_left=24
+    Step  7: FLAG acc_0029                       flagged=1/10  suspects=9  steps_left=24
+    Step  8: INSPECT acc_0006                    flagged=1/10  suspects=9  steps_left=23
+    Step  9: FLAG acc_0006                       flagged=2/10  suspects=8  steps_left=23
+    Step 10: INSPECT acc_0033                    flagged=2/10  suspects=8  steps_left=22
+    Step 11: FLAG acc_0033                       flagged=3/10  suspects=7  steps_left=22
+    Step 12: INSPECT acc_0015                    flagged=3/10  suspects=7  steps_left=21
+    Step 13: FLAG acc_0015                       flagged=4/10  suspects=6  steps_left=21
+    Step 14: INSPECT acc_0022                    flagged=4/10  suspects=6  steps_left=20
+    Step 15: FLAG acc_0022                       flagged=5/10  suspects=5  steps_left=20
+    Step 16: INSPECT acc_0009                    flagged=5/10  suspects=5  steps_left=19
+    Step 17: FLAG acc_0009                       flagged=6/10  suspects=4  steps_left=19
+    Step 18: INSPECT acc_0004                    flagged=6/10  suspects=4  steps_left=18
+    Step 19: FLAG acc_0004                       flagged=7/10  suspects=3  steps_left=18
+    Step 20: INSPECT acc_0024                    flagged=7/10  suspects=3  steps_left=17
+    Step 21: FLAG acc_0024                       flagged=8/10  suspects=2  steps_left=17
+    Step 22: INSPECT acc_0049                    flagged=8/10  suspects=2  steps_left=16
+    Step 23: FLAG acc_0049                       flagged=9/10  suspects=1  steps_left=16
+    Step 24: INSPECT acc_0035                    flagged=9/10  suspects=1  steps_left=15
+    Step 25: FLAG acc_0035                       flagged=10/10  suspects=0  steps_left=15
+    Step 26: SUBMIT                              flagged=10/10  suspects=0  steps_left=15
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.85
+    ★ GRADER SCORE: 0.9500
+
+  easy: scores=['0.967', '0.953', '0.950']  mean=0.9567  var=0.000052
+
+  --- LLM Agent: task=medium, seed=0, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0008                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0074                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0179                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0096                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0177                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0122                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0174                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0144                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: FLAG acc_0144                       flagged=1/10  suspects=8  steps_left=41
+    Step 11: INSPECT acc_0181                    flagged=1/10  suspects=8  steps_left=40
+    Step 12: FLAG acc_0181                       flagged=2/10  suspects=8  steps_left=40
+    Step 13: INSPECT acc_0022                    flagged=2/10  suspects=8  steps_left=39
+    Step 14: FLAG acc_0022                       flagged=3/10  suspects=7  steps_left=39
+    Step 15: INSPECT acc_0092                    flagged=3/10  suspects=7  steps_left=38
+    Step 16: FLAG acc_0092                       flagged=4/10  suspects=6  steps_left=38
+    Step 17: INSPECT acc_0097                    flagged=4/10  suspects=6  steps_left=37
+    Step 18: FLAG acc_0097                       flagged=5/10  suspects=5  steps_left=37
+    Step 19: INSPECT acc_0187                    flagged=5/10  suspects=5  steps_left=36
+    Step 20: FLAG acc_0187                       flagged=6/10  suspects=4  steps_left=36
+    Step 21: INSPECT acc_0093                    flagged=6/10  suspects=4  steps_left=35
+    Step 22: FLAG acc_0093                       flagged=7/10  suspects=3  steps_left=34
+    Step 23: INSPECT acc_0172                    flagged=7/10  suspects=3  steps_left=33
+    Step 24: FLAG acc_0172                       flagged=8/10  suspects=2  steps_left=33
+    Step 25: INSPECT acc_0058                    flagged=8/10  suspects=2  steps_left=31
+    Step 26: FLAG acc_0058                       flagged=9/10  suspects=1  steps_left=31
+    Step 27: INSPECT acc_0131                    flagged=9/10  suspects=1  steps_left=29
+    Step 28: FLAG acc_0131                       flagged=10/10  suspects=0  steps_left=29
+    Step 29: SUBMIT                              flagged=10/10  suspects=0  steps_left=29
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.79
+    ★ GRADER SCORE: 0.9580
+
+  --- LLM Agent: task=medium, seed=1, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0171                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0099                    flagged=1/10  suspects=3  steps_left=48
+    Step  3: INSPECT acc_0012                    flagged=1/10  suspects=3  steps_left=46
+    Step  4: FLAG acc_0012                       flagged=2/10  suspects=7  steps_left=46
+    Step  5: INSPECT acc_0033                    flagged=2/10  suspects=7  steps_left=45
+    Step  6: FLAG acc_0033                       flagged=3/10  suspects=7  steps_left=44
+    Step  7: INSPECT acc_0174                    flagged=3/10  suspects=7  steps_left=43
+    Step  8: FLAG acc_0174                       flagged=4/10  suspects=6  steps_left=43
+    Step  9: INSPECT acc_0187                    flagged=4/10  suspects=6  steps_left=42
+    Step 10: FLAG acc_0187                       flagged=5/10  suspects=5  steps_left=42
+    Step 11: INSPECT acc_0079                    flagged=5/10  suspects=5  steps_left=41
+    Step 12: FLAG acc_0079                       flagged=6/10  suspects=4  steps_left=41
+    Step 13: INSPECT acc_0019                    flagged=6/10  suspects=4  steps_left=40
+    Step 14: FLAG acc_0019                       flagged=7/10  suspects=3  steps_left=40
+    Step 15: INSPECT acc_0023                    flagged=7/10  suspects=3  steps_left=39
+    Step 16: FLAG acc_0023                       flagged=8/10  suspects=2  steps_left=39
+    Step 17: INSPECT acc_0032                    flagged=8/10  suspects=2  steps_left=38
+    Step 18: FLAG acc_0032                       flagged=9/10  suspects=1  steps_left=38
+    Step 19: INSPECT acc_0146                    flagged=9/10  suspects=1  steps_left=37
+    Step 20: FLAG acc_0146                       flagged=10/10  suspects=0  steps_left=37
+    Step 21: SUBMIT                              flagged=10/10  suspects=0  steps_left=37
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=18.87
+    ★ GRADER SCORE: 0.9740
+
+  --- LLM Agent: task=medium, seed=2, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0189                    flagged=0/10  suspects=0  steps_left=49
+    Step  2: INSPECT acc_0107                    flagged=0/10  suspects=0  steps_left=48
+    Step  3: INSPECT acc_0175                    flagged=0/10  suspects=0  steps_left=47
+    Step  4: INSPECT acc_0030                    flagged=0/10  suspects=0  steps_left=46
+    Step  5: INSPECT acc_0041                    flagged=0/10  suspects=0  steps_left=45
+    Step  6: INSPECT acc_0054                    flagged=0/10  suspects=0  steps_left=44
+    Step  7: INSPECT acc_0199                    flagged=0/10  suspects=0  steps_left=43
+    Step  8: INSPECT acc_0181                    flagged=0/10  suspects=0  steps_left=42
+    Step  9: INSPECT acc_0166                    flagged=0/10  suspects=0  steps_left=41
+    Step 10: INSPECT acc_0098                    flagged=0/10  suspects=0  steps_left=40
+    Step 11: INSPECT acc_0121                    flagged=0/10  suspects=0  steps_left=39
+    Step 12: INSPECT acc_0053                    flagged=0/10  suspects=0  steps_left=38
+    Step 13: INSPECT acc_0103                    flagged=0/10  suspects=0  steps_left=37
+    Step 14: INSPECT acc_0000                    flagged=0/10  suspects=0  steps_left=36
+    Step 15: INSPECT acc_0168                    flagged=0/10  suspects=0  steps_left=35
+    Step 16: INSPECT acc_0040                    flagged=0/10  suspects=0  steps_left=34
+    Step 17: INSPECT acc_0149                    flagged=0/10  suspects=0  steps_left=33
+    Step 18: INSPECT acc_0064                    flagged=0/10  suspects=0  steps_left=32
+    Step 19: INSPECT acc_0016                    flagged=0/10  suspects=0  steps_left=31
+    Step 20: INSPECT acc_0105                    flagged=0/10  suspects=0  steps_left=30
+    Step 21: INSPECT acc_0035                    flagged=0/10  suspects=0  steps_left=29
+    Step 22: FLAG acc_0035                       flagged=1/10  suspects=9  steps_left=29
+    Step 23: INSPECT acc_0020                    flagged=1/10  suspects=9  steps_left=28
+    Step 24: FLAG acc_0020                       flagged=2/10  suspects=8  steps_left=28
+    Step 25: INSPECT acc_0036                    flagged=2/10  suspects=8  steps_left=27
+    Step 26: FLAG acc_0036                       flagged=3/10  suspects=7  steps_left=27
+    Step 27: INSPECT acc_0050                    flagged=3/10  suspects=7  steps_left=26
+    Step 28: FLAG acc_0050                       flagged=4/10  suspects=6  steps_left=26
+    Step 29: INSPECT acc_0051                    flagged=4/10  suspects=6  steps_left=25
+    Step 30: FLAG acc_0051                       flagged=5/10  suspects=5  steps_left=25
+    Step 31: INSPECT acc_0085                    flagged=5/10  suspects=5  steps_left=24
+    Step 32: FLAG acc_0085                       flagged=6/10  suspects=4  steps_left=24
+    Step 33: INSPECT acc_0177                    flagged=6/10  suspects=4  steps_left=23
+    Step 34: FLAG acc_0177                       flagged=7/10  suspects=3  steps_left=23
+    Step 35: INSPECT acc_0170                    flagged=7/10  suspects=3  steps_left=22
+    Step 36: FLAG acc_0170                       flagged=8/10  suspects=2  steps_left=22
+    Step 37: INSPECT acc_0055                    flagged=8/10  suspects=2  steps_left=21
+    Step 38: FLAG acc_0055                       flagged=9/10  suspects=1  steps_left=21
+    Step 39: INSPECT acc_0094                    flagged=9/10  suspects=1  steps_left=20
+    Step 40: FLAG acc_0094                       flagged=10/10  suspects=0  steps_left=20
+    Step 41: SUBMIT                              flagged=10/10  suspects=0  steps_left=20
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.70
+    ★ GRADER SCORE: 0.9400
+
+  medium: scores=['0.958', '0.974', '0.940']  mean=0.9573  var=0.000193
+
+  --- LLM Agent: task=hard, seed=0, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0704                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0289                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0826                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0927                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0441                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0871                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0880                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0939                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0070                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0915                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0443                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0970                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0028                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0792                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0579                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0037                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0295                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0438                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0439                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: FLAG acc_0439                       flagged=1/10  suspects=5  steps_left=60
+    Step 22: INSPECT acc_0237                    flagged=1/10  suspects=5  steps_left=59
+    Step 23: FLAG acc_0237                       flagged=2/10  suspects=6  steps_left=59
+    Step 24: INSPECT acc_0621                    flagged=2/10  suspects=6  steps_left=58
+    Step 25: FLAG acc_0621                       flagged=3/10  suspects=6  steps_left=58
+    Step 26: INSPECT acc_0389                    flagged=3/10  suspects=6  steps_left=57
+    Step 27: INSPECT acc_0160                    flagged=3/10  suspects=6  steps_left=56
+    Step 28: FLAG acc_0389                       flagged=4/10  suspects=6  steps_left=56
+    Step 29: FLAG acc_0160                       flagged=5/10  suspects=5  steps_left=56
+    Step 30: INSPECT acc_0549                    flagged=5/10  suspects=5  steps_left=55
+    Step 31: FLAG acc_0549                       flagged=6/10  suspects=4  steps_left=55
+    Step 32: INSPECT acc_0658                    flagged=6/10  suspects=4  steps_left=54
+    Step 33: FLAG acc_0658                       flagged=7/10  suspects=3  steps_left=54
+    Step 34: INSPECT acc_0290                    flagged=7/10  suspects=3  steps_left=53
+    Step 35: FLAG acc_0290                       flagged=8/10  suspects=2  steps_left=53
+    Step 36: INSPECT acc_0124                    flagged=8/10  suspects=2  steps_left=52
+    Step 37: FLAG acc_0124                       flagged=9/10  suspects=1  steps_left=52
+    Step 38: INSPECT acc_0507                    flagged=9/10  suspects=1  steps_left=51
+    Step 39: FLAG acc_0507                       flagged=10/10  suspects=0  steps_left=51
+    Step 40: SUBMIT                              flagged=10/10  suspects=0  steps_left=51
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.71
+    ★ GRADER SCORE: 0.9637
+
+  --- LLM Agent: task=hard, seed=1, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0014                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0835                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0855                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0930                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0336                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0929                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0076                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0543                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0590                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0401                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0322                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0154                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0374                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0549                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0903                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: INSPECT acc_0976                    flagged=0/10  suspects=0  steps_left=64
+    Step 17: INSPECT acc_0620                    flagged=0/10  suspects=0  steps_left=63
+    Step 18: INSPECT acc_0017                    flagged=0/10  suspects=0  steps_left=62
+    Step 19: INSPECT acc_0222                    flagged=0/10  suspects=0  steps_left=61
+    Step 20: INSPECT acc_0536                    flagged=0/10  suspects=0  steps_left=60
+    Step 21: INSPECT acc_0112                    flagged=0/10  suspects=0  steps_left=59
+    Step 22: INSPECT acc_0577                    flagged=0/10  suspects=0  steps_left=58
+    Step 23: INSPECT acc_0517                    flagged=0/10  suspects=0  steps_left=57
+    Step 24: INSPECT acc_0113                    flagged=0/10  suspects=0  steps_left=56
+    Step 25: INSPECT acc_0167                    flagged=0/10  suspects=0  steps_left=55
+    Step 26: INSPECT acc_0697                    flagged=0/10  suspects=0  steps_left=54
+    Step 27: INSPECT acc_0271                    flagged=0/10  suspects=0  steps_left=53
+    Step 28: INSPECT acc_0681                    flagged=0/10  suspects=0  steps_left=52
+    Step 29: INSPECT acc_0530                    flagged=0/10  suspects=0  steps_left=51
+    Step 30: INSPECT acc_0353                    flagged=0/10  suspects=0  steps_left=50
+    Step 31: INSPECT acc_0652                    flagged=0/10  suspects=0  steps_left=49
+    Step 32: INSPECT acc_0777                    flagged=0/10  suspects=0  steps_left=48
+    Step 33: INSPECT acc_0265                    flagged=0/10  suspects=0  steps_left=47
+    Step 34: INSPECT acc_0788                    flagged=0/10  suspects=0  steps_left=46
+    Step 35: INSPECT acc_0033                    flagged=0/10  suspects=0  steps_left=45
+    Step 36: INSPECT acc_0187                    flagged=0/10  suspects=0  steps_left=44
+    Step 37: INSPECT acc_0445                    flagged=0/10  suspects=0  steps_left=43
+    Step 38: INSPECT acc_0846                    flagged=0/10  suspects=0  steps_left=42
+    Step 39: INSPECT acc_0659                    flagged=0/10  suspects=0  steps_left=41
+    Step 40: INSPECT acc_0768                    flagged=0/10  suspects=0  steps_left=40
+    Step 41: INSPECT acc_0677                    flagged=0/10  suspects=0  steps_left=39
+    Step 42: INSPECT acc_0539                    flagged=0/10  suspects=0  steps_left=38
+    Step 43: INSPECT acc_0742                    flagged=1/10  suspects=10  steps_left=37
+    Step 44: INSPECT acc_0876                    flagged=1/10  suspects=10  steps_left=35
+    Step 45: INSPECT acc_0884                    flagged=1/10  suspects=10  steps_left=33
+    Step 46: INSPECT acc_0245                    flagged=1/10  suspects=10  steps_left=32
+    Step 47: INSPECT acc_0223                    flagged=1/10  suspects=10  steps_left=31
+    Step 48: INSPECT acc_0647                    flagged=1/10  suspects=10  steps_left=30
+    Step 49: INSPECT acc_0994                    flagged=1/10  suspects=10  steps_left=29
+    Step 50: INSPECT acc_0811                    flagged=1/10  suspects=10  steps_left=28
+    Step 51: INSPECT acc_0794                    flagged=1/10  suspects=10  steps_left=27
+    Step 52: INSPECT acc_0582                    flagged=1/10  suspects=10  steps_left=26
+    Step 53: INSPECT acc_0503                    flagged=1/10  suspects=10  steps_left=25
+    Step 54: FLAG acc_0174                       flagged=1/10  suspects=10  steps_left=25
+    Step 55: INSPECT acc_0639                    flagged=1/10  suspects=10  steps_left=24
+    Step 56: INSPECT acc_0494                    flagged=1/10  suspects=10  steps_left=23
+    Step 57: INSPECT acc_0898                    flagged=1/10  suspects=10  steps_left=22
+    Step 58: INSPECT acc_0553                    flagged=1/10  suspects=10  steps_left=21
+    Step 59: INSPECT acc_0588                    flagged=1/10  suspects=10  steps_left=20
+    Step 60: INSPECT acc_0194                    flagged=1/10  suspects=10  steps_left=19
+    Step 61: INSPECT acc_0810                    flagged=1/10  suspects=10  steps_left=18
+    Step 62: INSPECT acc_0355                    flagged=1/10  suspects=10  steps_left=17
+    Step 63: INSPECT acc_0363                    flagged=1/10  suspects=10  steps_left=16
+    Step 64: INSPECT acc_0221                    flagged=1/10  suspects=10  steps_left=15
+    Step 65: INSPECT acc_0580                    flagged=1/10  suspects=10  steps_left=14
+    Step 66: INSPECT acc_0534                    flagged=1/10  suspects=10  steps_left=13
+    Step 67: INSPECT acc_0778                    flagged=1/10  suspects=10  steps_left=12
+    Step 68: INSPECT acc_0998                    flagged=1/10  suspects=10  steps_left=11
+    Step 69: INSPECT acc_0233                    flagged=1/10  suspects=10  steps_left=10
+    Step 70: INSPECT acc_0052                    flagged=1/10  suspects=10  steps_left=9
+    Step 71: INSPECT acc_0813                    flagged=1/10  suspects=10  steps_left=8
+    Step 72: INSPECT acc_0035                    flagged=1/10  suspects=10  steps_left=7
+    Step 73: INSPECT acc_0667                    flagged=1/10  suspects=10  steps_left=6
+    Step 74: INSPECT acc_0019                    flagged=1/10  suspects=10  steps_left=5
+    Step 75: INSPECT acc_0959                    flagged=1/10  suspects=10  steps_left=4
+    Step 76: INSPECT acc_0212                    flagged=1/10  suspects=10  steps_left=3
+    Step 77: INSPECT acc_0776                    flagged=1/10  suspects=10  steps_left=2
+    Step 78: INSPECT acc_0049                    flagged=1/10  suspects=10  steps_left=1
+    Step 79: FLAG acc_0174                       flagged=1/10  suspects=10  steps_left=1
+    Step 80: FLAG acc_0434                       flagged=2/10  suspects=18  steps_left=1
+    Step 81: INSPECT acc_0369                    flagged=2/10  suspects=18  steps_left=0
+    → Episode ended: [LOSS] TP=0 FP=2 FN=10 Recall=0.00 Precision=0.00 Episode reward=-10.80
+    ★ GRADER SCORE: 0.0000
+
+  --- LLM Agent: task=hard, seed=2, model=Bedrock/nvidia.nemotron-super-3-120b ---
+    Step  1: INSPECT acc_0813                    flagged=0/10  suspects=0  steps_left=79
+    Step  2: INSPECT acc_0430                    flagged=0/10  suspects=0  steps_left=78
+    Step  3: INSPECT acc_0817                    flagged=0/10  suspects=0  steps_left=77
+    Step  4: INSPECT acc_0175                    flagged=0/10  suspects=0  steps_left=76
+    Step  5: INSPECT acc_0523                    flagged=0/10  suspects=0  steps_left=75
+    Step  6: INSPECT acc_0113                    flagged=0/10  suspects=0  steps_left=74
+    Step  7: INSPECT acc_0797                    flagged=0/10  suspects=0  steps_left=73
+    Step  8: INSPECT acc_0478                    flagged=0/10  suspects=0  steps_left=72
+    Step  9: INSPECT acc_0861                    flagged=0/10  suspects=0  steps_left=71
+    Step 10: INSPECT acc_0836                    flagged=0/10  suspects=0  steps_left=70
+    Step 11: INSPECT acc_0926                    flagged=0/10  suspects=0  steps_left=69
+    Step 12: INSPECT acc_0664                    flagged=0/10  suspects=0  steps_left=68
+    Step 13: INSPECT acc_0255                    flagged=0/10  suspects=0  steps_left=67
+    Step 14: INSPECT acc_0938                    flagged=0/10  suspects=0  steps_left=66
+    Step 15: INSPECT acc_0672                    flagged=0/10  suspects=0  steps_left=65
+    Step 16: FLAG acc_0672                       flagged=1/10  suspects=6  steps_left=65
+    Step 17: INSPECT acc_0659                    flagged=1/10  suspects=6  steps_left=64
+    Step 18: FLAG acc_0659                       flagged=2/10  suspects=5  steps_left=64
+    Step 19: INSPECT acc_0290                    flagged=2/10  suspects=5  steps_left=63
+    Step 20: FLAG acc_0290                       flagged=3/10  suspects=5  steps_left=63
+    Step 21: INSPECT acc_0339                    flagged=3/10  suspects=5  steps_left=62
+    Step 22: FLAG acc_0339                       flagged=4/10  suspects=6  steps_left=62
+    Step 23: INSPECT acc_0544                    flagged=4/10  suspects=6  steps_left=61
+    Step 24: FLAG acc_0544                       flagged=5/10  suspects=5  steps_left=61
+    Step 25: INSPECT acc_0696                    flagged=5/10  suspects=5  steps_left=60
+    Step 26: FLAG acc_0696                       flagged=6/10  suspects=4  steps_left=60
+    Step 27: INSPECT acc_0541                    flagged=6/10  suspects=4  steps_left=59
+    Step 28: FLAG acc_0541                       flagged=7/10  suspects=3  steps_left=59
+    Step 29: INSPECT acc_0793                    flagged=7/10  suspects=3  steps_left=58
+    Step 30: FLAG acc_0793                       flagged=8/10  suspects=2  steps_left=58
+    Step 31: INSPECT acc_0214                    flagged=8/10  suspects=2  steps_left=57
+    Step 32: FLAG acc_0214                       flagged=9/10  suspects=1  steps_left=57
+    Step 33: INSPECT acc_0112                    flagged=9/10  suspects=1  steps_left=56
+    Step 34: FLAG acc_0112                       flagged=10/10  suspects=0  steps_left=56
+    Step 35: SUBMIT                              flagged=10/10  suspects=0  steps_left=56
+    → Episode ended: [WIN] TP=10 FP=0 FN=0 Recall=1.00 Precision=1.00 Episode reward=17.76
+    ★ GRADER SCORE: 0.9700
+
+  hard: scores=['0.964', '0.000', '0.970']  mean=0.6446  var=0.207740
+
+============================================================
+EVALUATION COMPLETE
+============================================================
+ubuntu@ip-172-31-33-59:~/meta/meta-hack-26$

server/app.py

@@ -11,8 +11,9 @@ sys.path.insert(0, str(Path(__file__).parent))
 sys.path.insert(0, str(Path(__file__).parent.parent))
 
 from fastapi import FastAPI, HTTPException
-from fastapi.responses import HTMLResponse
+from fastapi.responses import HTMLResponse, RedirectResponse
 from fastapi.middleware.cors import CORSMiddleware
+from fastapi.staticfiles import StaticFiles
 from pydantic import BaseModel
 from typing import Any, Dict, Optional
 
@@ -20,7 +21,7 @@ from models import FakeGangAction, FakeGangObservation, FakeGangState, ActionTyp
 from environment import FakeGangEnvironment
 
 # ---------------------------------------------------------------------------
-# App
+# App + environment
 # ---------------------------------------------------------------------------
 
 app = FastAPI(
@@ -28,19 +29,19 @@ app = FastAPI(
     description="RL environment for detecting coordinated fake account rings in social networks.",
     version="1.0.0",
 )
+app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
 
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=["*"], allow_methods=["*"], allow_headers=["*"],
-)
+# Serve images at /assets/* and /images/* (used by the Gradio README tab)
+_PROJECT_ROOT = Path(__file__).parent.parent
+_ASSETS_DIR = _PROJECT_ROOT / "assets"
+_IMAGES_DIR = _PROJECT_ROOT / "images"
+if _ASSETS_DIR.exists():
+    app.mount("/assets", StaticFiles(directory=str(_ASSETS_DIR)), name="assets")
+if _IMAGES_DIR.exists():
+    app.mount("/images", StaticFiles(directory=str(_IMAGES_DIR)), name="images")
 
 _env = FakeGangEnvironment()
 
-
-# ---------------------------------------------------------------------------
-# Schemas
-# ---------------------------------------------------------------------------
-
 class ResetRequest(BaseModel):
     task: str = "easy"
     seed: Optional[int] = None
@@ -52,7 +53,6 @@ class StepResponse(BaseModel):
     reward: Optional[float]
     message: str
 
-
 # ---------------------------------------------------------------------------
 # OpenEnv API endpoints
 # ---------------------------------------------------------------------------
@@ -100,9 +100,8 @@ def grader():
 @app.get("/metadata")
 def metadata():
     return {
-        "name": "graphstrike",
+        "name": "graphstrike", "version": "1.0.0", "author": "Pandago",
         "description": "RL environment for detecting coordinated fake account rings in social networks.",
-        "version": "1.0.0", "author": "Pandago",
         "tags": ["social-network", "fraud-detection", "graph", "rl"],
     }
 
@@ -137,124 +136,662 @@ def baseline():
     return {"scores": scores, "agent": "rule_based"}
 
 
-# HF Spaces probes GET /web to detect if a web UI exists.
-# Must return 200 BEFORE Gradio mount (Gradio's catch-all would shadow it).
-@app.get("/web", response_class=HTMLResponse)
-def web_view():
-    return """<!DOCTYPE html>
-<html><head><meta http-equiv="refresh" content="0;url=/"><title>GraphStrike</title></head>
-<body><p>Loading <a href="/">GraphStrike</a>...</p></body></html>"""
+# HF Spaces probes /web — redirect to root (must be on FastAPI before Gradio mount)
+@app.get("/web", response_class=RedirectResponse)
+def web_redirect():
+    return RedirectResponse(url="/")
 
 
 # ---------------------------------------------------------------------------
-# Gradio web interface — mounted at /
+# Gradio UI
 # ---------------------------------------------------------------------------
 
+import pandas as pd
+
+# ── Benchmark data ───────────────────────────────────────────────────────────
+
+BENCH_SEED0 = [
+    # [Model, Params, Easy, Medium, Hard, Mean]  — sorted by Mean desc
+    ["Llama 4 Scout 17B",  "17B",  0.960, 0.979, 0.976, 0.972],
+    ["Ministral 3 8B",     "8B",   0.967, 0.964, 0.964, 0.965],
+    ["DeepSeek V3.2",      "685B", 0.967, 0.960, 0.933, 0.953],
+    ["Nemotron Super 3",   "49B",  0.930, 0.941, 0.964, 0.945],
+    ["Rule-Based Baseline","—",    0.910, 0.906, 0.904, 0.907],
+    ["Gemma 3 12B",        "12B",  0.900, 0.908, 0.908, 0.905],
+]
+
+BENCH_VARIANCE = [
+    # [Model, Easy mean, Easy var, Med mean, Med var, Hard mean, Hard var]
+    ["Llama 4 Scout 17B", 0.960, 0.000007, 0.979, 0.000001, 0.976, 0.000063],
+    ["Nemotron Super 3",  0.957, 0.000,    0.957, 0.000,    0.645, 0.208],
+    ["Ministral 3 8B",    0.958, 0.000,    0.645, 0.208,    0.623, 0.195],
+    ["DeepSeek V3.2",     0.640, 0.205,    0.957, 0.000,    0.645, 0.208],
+    ["Gemma 3 12B",       0.912, 0.000,    0.917, 0.000,    0.603, 0.182],
+]
+
+PROFILE_HEADERS = ["Account", "Status", "Risk", "Node", "Beh", "Graph", "Hub", "Photo", "Bio", "IP", "F.Nbrs"]
+
+# Long-format DataFrame for BarPlot
+_bench_long_rows = []
+for _r in BENCH_SEED0:
+    _bench_long_rows += [
+        {"Model": _r[0], "Task": "Easy",   "Score": _r[2]},
+        {"Model": _r[0], "Task": "Medium", "Score": _r[3]},
+        {"Model": _r[0], "Task": "Hard",   "Score": _r[4]},
+    ]
+BENCH_LONG_DF = pd.DataFrame(_bench_long_rows)
+
+
+# ── HTML table builders ──────────────────────────────────────────────────────
+
+def _score_color(s: float) -> str:
+    if s >= 0.960: return "#22c55e"
+    if s >= 0.930: return "#86efac"
+    if s >= 0.910: return "#facc15"
+    return "#f97316"
+
+def _var_color(v: float) -> str:
+    if v < 0.001:  return "#22c55e"
+    if v < 0.05:   return "#facc15"
+    return "#f87171"
+
+_TH = "padding:11px 16px;font-weight:600;white-space:nowrap;"
+_TD = "padding:10px 16px;white-space:nowrap;"
+_TABLE_WRAP = (
+    "overflow-x:auto;border-radius:10px;border:1px solid #1e3a5f;"
+    "font-family:'IBM Plex Mono',monospace;font-size:13.5px;"
+)
+_THEAD_BG = "background:#0c2340;"
+
+def _leaderboard_html() -> str:
+    header = (
+        f"<thead><tr style='{_THEAD_BG}'>"
+        f"<th style='{_TH}color:#64748b;'>#</th>"
+        f"<th style='{_TH}color:#e2e8f0;text-align:left;'>Model</th>"
+        f"<th style='{_TH}color:#94a3b8;text-align:center;'>Params</th>"
+        f"<th style='{_TH}color:#4ade80;text-align:center;'>Easy</th>"
+        f"<th style='{_TH}color:#facc15;text-align:center;'>Medium</th>"
+        f"<th style='{_TH}color:#f87171;text-align:center;'>Hard</th>"
+        f"<th style='{_TH}color:#c084fc;text-align:center;'>Mean</th>"
+        f"</tr></thead>"
+    )
+    rows = ""
+    for i, r in enumerate(BENCH_SEED0):
+        bg  = "#162032" if i % 2 == 0 else "#0f172a"
+        is_base = r[0] == "Rule-Based Baseline"
+        name_cell = (
+            f"{r[0]} <span style='color:#64748b;font-size:11px;'>(baseline)</span>"
+            if is_base else r[0]
+        )
+        name_color = "#94a3b8" if is_base else "#e2e8f0"
+        rows += (
+            f"<tr style='background:{bg};'>"
+            f"<td style='{_TD}color:#475569;text-align:center;'>{i+1}</td>"
+            f"<td style='{_TD}color:{name_color};'>{name_cell}</td>"
+            f"<td style='{_TD}color:#64748b;text-align:center;'>{r[1]}</td>"
+            + "".join(
+                f"<td style='{_TD}color:{_score_color(r[j])};font-weight:700;"
+                f"text-align:center;'>{r[j]:.3f}</td>"
+                for j in (2, 3, 4)
+            )
+            + f"<td style='{_TD}color:{_score_color(r[5])};font-weight:800;"
+              f"font-size:14px;text-align:center;'>{r[5]:.3f}</td>"
+            f"</tr>"
+        )
+    return f"<div style='{_TABLE_WRAP}'><table style='width:100%;border-collapse:collapse;'>{header}<tbody>{rows}</tbody></table></div>"
+
+
+def _variance_html() -> str:
+    header = (
+        f"<thead><tr style='{_THEAD_BG}'>"
+        f"<th style='{_TH}color:#e2e8f0;text-align:left;'>Model</th>"
+        f"<th style='{_TH}color:#4ade80;text-align:center;'>Easy — mean / var</th>"
+        f"<th style='{_TH}color:#facc15;text-align:center;'>Medium — mean / var</th>"
+        f"<th style='{_TH}color:#f87171;text-align:center;'>Hard — mean / var</th>"
+        f"</tr></thead>"
+    )
+    rows = ""
+    for i, r in enumerate(BENCH_VARIANCE):
+        bg = "#162032" if i % 2 == 0 else "#0f172a"
+        def cell(mean, var):
+            return (
+                f"<td style='{_TD}text-align:center;'>"
+                f"<span style='color:#e2e8f0;font-weight:600;'>{mean:.3f}</span>"
+                f" <span style='color:{_var_color(var)};font-size:11px;'>/ {var:.1e}</span>"
+                f"</td>"
+            )
+        rows += (
+            f"<tr style='background:{bg};'>"
+            f"<td style='{_TD}color:#e2e8f0;font-weight:500;'>{r[0]}</td>"
+            + cell(r[1], r[2]) + cell(r[3], r[4]) + cell(r[5], r[6])
+            + "</tr>"
+        )
+    return f"<div style='{_TABLE_WRAP};margin-top:20px;'><table style='width:100%;border-collapse:collapse;'>{header}<tbody>{rows}</tbody></table></div>"
+
+
+def _baseline_html() -> str:
+    rows_data = [
+        ("Easy",   0.9100, "100%", "#4ade80"),
+        ("Medium", 0.9060, "84%",  "#facc15"),
+        ("Hard",   0.9038, "52%",  "#f87171"),
+    ]
+    header = (
+        f"<thead><tr style='{_THEAD_BG}'>"
+        f"<th style='{_TH}color:#e2e8f0;'>Task</th>"
+        f"<th style='{_TH}color:#e2e8f0;text-align:center;'>Score (seed=0)</th>"
+        f"<th style='{_TH}color:#e2e8f0;text-align:center;'>Win Rate (50 seeds)</th>"
+        f"</tr></thead>"
+    )
+    rows = ""
+    for i, (task, score, wr, col) in enumerate(rows_data):
+        bg = "#162032" if i % 2 == 0 else "#0f172a"
+        rows += (
+            f"<tr style='background:{bg};'>"
+            f"<td style='{_TD}color:{col};font-weight:600;'>{task}</td>"
+            f"<td style='{_TD}color:#e2e8f0;font-weight:700;text-align:center;'>{score:.4f}</td>"
+            f"<td style='{_TD}color:{col};font-weight:600;text-align:center;'>{wr}</td>"
+            f"</tr>"
+        )
+    return f"<div style='{_TABLE_WRAP};margin-top:4px;'><table style='width:100%;border-collapse:collapse;'>{header}<tbody>{rows}</tbody></table></div>"
+
+
 try:
     import gradio as gr
 
+    # ── Observation / profile helpers ─────────────────────────────────────────
+
     def _fmt_obs(d: dict) -> str:
         lines = []
-        lines.append(f"**Task:** {d.get('task','?')}  |  **Done:** {d.get('done',False)}  |  **Steps remaining:** {d.get('steps_remaining','?')}")
+        task  = d.get('task', '?').upper()
+        done  = d.get('done', False)
+        steps = d.get('steps_remaining', '?')
+        state_label = "Done" if done else "In Progress"
+        lines.append(f"### Task: **{task}**  |  Steps remaining: **{steps}**  |  {state_label}")
         if d.get('reward') is not None:
-            lines.append(f"**Reward:** {d['reward']:.2f}")
+            lines.append(f"**Final Reward:** `{d['reward']:.2f}`")
         fl = d.get('flagged_ids', [])
-        lines.append(f"**Flagged ({len(fl)}/10):** {fl}")
-        su = d.get('suspect_ids', [])
-        lines.append(f"**Suspects ({len(su)}):** {su}")
+        lines.append(f"**Flagged ({len(fl)}/10):** " + (" ".join(f"`{f}`" for f in fl) if fl else "*none*"))
+        su  = d.get('suspect_ids', [])
+        ins = set(d.get('inspected_ids', []))
+        uninspected_sus = [s for s in su if s not in ins]
+        if uninspected_sus:
+            lines.append(f"**Suspects — uninspected ({len(uninspected_sus)}):** " + " ".join(f"`{s}`" for s in uninspected_sus))
         lines.append(f"**Visible:** {len(d.get('visible_account_ids',[]))} IDs  |  **Inspected:** {len(d.get('inspected_ids',[]))} accounts")
         if d.get('evasion_triggered'):
-            lines.append(f"**Evasion events:** {d.get('evasion_count',0)}")
-        lines.append(f"**Message:** {d.get('message','')}")
+            lines.append(f"**Evasion events fired:** {d.get('evasion_count', 0)}")
+        lines.append(f"\n> {d.get('message', '')}")
         return "\n\n".join(lines)
 
-    def _fmt_profiles(d: dict) -> str:
+    def _profile_rows(d: dict) -> list:
         accs = d.get("visible_accounts", [])
         if not accs:
-            return "No accounts inspected yet. Use **INSPECT** to reveal profiles."
-        rows = ["| Account | Status | Risk | Node | Beh | Graph | Hub | Photo | Bio | F.Nbrs |",
-                "|---------|--------|------|------|-----|-------|-----|-------|-----|--------|"]
-        for a in sorted(accs, key=lambda x: x.get("fake_risk_score",0), reverse=True)[:25]:
-            rows.append(f"| {a.get('account_id','')} | {a.get('status','?')} | {a.get('fake_risk_score',0):.3f} "
-                        f"| {a.get('node_risk',0):.2f} | {a.get('behavior_risk',0):.2f} | {a.get('graph_risk',0):.2f} "
-                        f"| {a.get('hub_legitimacy_score',0):.2f} | {a.get('photo_reuse_score',0):.2f} "
-                        f"| {a.get('bio_template_score',0):.2f} | {a.get('flagged_neighbor_count',0)} |")
-        return "\n".join(rows)
+            return []
+        STATUS_MAP = {
+            "confirmed_fake": "confirmed_fake [flagged]",
+            "suspect":        "suspect",
+            "normal":         "normal",
+        }
+        rows = []
+        for a in sorted(accs, key=lambda x: x.get("fake_risk_score", 0), reverse=True)[:40]:
+            rows.append([
+                a.get("account_id", ""),
+                STATUS_MAP.get(a.get("status", ""), a.get("status", "")),
+                round(a.get("fake_risk_score", 0), 3),
+                round(a.get("node_risk", 0), 3),
+                round(a.get("behavior_risk", 0), 3),
+                round(a.get("graph_risk", 0), 3),
+                round(a.get("hub_legitimacy_score", 0), 3),
+                round(a.get("photo_reuse_score", 0), 3),
+                round(a.get("bio_template_score", 0), 3),
+                a.get("shared_ip_count", 0),
+                a.get("flagged_neighbor_count", 0),
+            ])
+        return rows
+
+    def _fmt_visible_ids(d: dict) -> str:
+        ins      = set(d.get('inspected_ids', []))
+        suspects = set(d.get('suspect_ids', []))
+        flagged  = set(d.get('flagged_ids', []))
+        visible  = d.get('visible_account_ids', [])
+        if not visible:
+            return "*No visible accounts yet.*"
+        parts = []
+        for vid in visible:
+            if vid in flagged:
+                parts.append(f"**[F]** `{vid}`")
+            elif vid in suspects and vid not in ins:
+                parts.append(f"**[S]** `{vid}`")
+            elif vid in ins:
+                parts.append(f"`{vid}`")
+            else:
+                parts.append(f"`{vid}`")
+        return "  ".join(parts)
+
+    # ── Playground callbacks ──────────────────────────────────────────────────
 
     def gr_reset(task, seed):
         try:
             obs = _env.reset(task=task, seed=int(seed))
-            d = obs.model_dump()
-            return _fmt_obs(d), _fmt_profiles(d), json.dumps(d, indent=2, default=str)
+            d   = obs.model_dump()
+            return _fmt_obs(d), _profile_rows(d), _fmt_visible_ids(d), json.dumps(d, indent=2, default=str)
         except Exception as e:
-            return f"**Error:** {e}", "", "{}"
+            return f"**Error:** {e}", [], "", "{}"
 
     def gr_step(action_type, account_id):
         try:
-            acc = account_id.strip() if action_type != "submit" else None
+            acc    = account_id.strip() if action_type != "submit" else None
             action = FakeGangAction(action_type=ActionType(action_type), account_id=acc)
-            obs = _env.step(action)
-            d = obs.model_dump()
-            return _fmt_obs(d), _fmt_profiles(d), json.dumps(d, indent=2, default=str)
+            obs    = _env.step(action)
+            d      = obs.model_dump()
+            return _fmt_obs(d), _profile_rows(d), _fmt_visible_ids(d), json.dumps(d, indent=2, default=str)
         except Exception as e:
-            return f"**Error:** {e}", "", "{}"
+            return f"**Error:** {e}", [], "", "{}"
 
     def gr_grader():
         if not _env._done:
-            return "Episode not complete. Call SUBMIT first."
-        return json.dumps({"score": _env._last_grader_score, "task": _env._task, "episode_id": _env._episode_id}, indent=2)
+            return "Episode not complete — call SUBMIT first."
+        return (
+            f"**Score:** `{_env._last_grader_score:.4f}`  |  "
+            f"**Task:** {_env._task}  |  "
+            f"**Episode:** `{_env._episode_id}`"
+        )
 
     def gr_baseline():
         sys.path.insert(0, str(Path(__file__).parent.parent))
         from inference import run_rule_based_episode
         scores = {t: run_rule_based_episode(_env, task=t, seed=0) for t in ["easy", "medium", "hard"]}
-        return json.dumps({"scores": scores, "agent": "rule_based"}, indent=2)
+        mean = sum(scores.values()) / 3
+        return (
+            f"**Baseline (rule-based, seed=0)**\n\n"
+            f"Easy: `{scores['easy']:.4f}`  |  Medium: `{scores['medium']:.4f}`  |  "
+            f"Hard: `{scores['hard']:.4f}`  |  Mean: `{mean:.4f}`"
+        )
+
+    # ── Build Gradio UI ───────────────────────────────────────────────────────
+
+    # ── README content (rendered as styled HTML) ─────────────────────────────
+
+    _README_HTML = """
+<style>
+.gs-readme { font-family: 'Inter', system-ui, sans-serif; color: #cbd5e1; line-height: 1.7; max-width: 960px; margin: 0 auto; padding: 8px 4px 32px; }
+.gs-readme h2 { color: #e2e8f0; font-size: 1.12em; font-weight: 700; border-bottom: 1px solid #1e3a5f; padding-bottom: 8px; margin: 32px 0 14px; letter-spacing: -0.2px; }
+.gs-readme h3 { color: #7dd3fc; font-size: 0.97em; font-weight: 600; margin: 20px 0 8px; }
+.gs-readme p  { margin: 0 0 10px; font-size: 0.92em; }
+.gs-readme code { background: #0c2340; color: #7dd3fc; padding: 2px 7px; border-radius: 4px; font-family: 'IBM Plex Mono', monospace; font-size: 0.84em; }
+.gs-readme pre { background: #0a1628; border: 1px solid #1e3a5f; border-radius: 8px; padding: 14px 18px; overflow-x: auto; margin: 10px 0 16px; }
+.gs-readme pre code { background: none; padding: 0; color: #93c5fd; font-size: 0.82em; }
+.gs-table { width: 100%; border-collapse: collapse; margin: 10px 0 18px; font-size: 0.86em; }
+.gs-table th { background: #0c2340; color: #94a3b8; font-weight: 600; padding: 9px 14px; text-align: left; border-bottom: 1px solid #1e3a5f; }
+.gs-table td { padding: 8px 14px; border-bottom: 1px solid #0f1e30; color: #cbd5e1; }
+.gs-table tr:nth-child(even) td { background: #060e1a; }
+.gs-badge { display:inline-block; padding: 2px 9px; border-radius: 4px; font-size: 0.78em; font-weight: 700; }
+.gs-badge-easy   { background:#052e16; color:#4ade80; border:1px solid #166534; }
+.gs-badge-medium { background:#2d1f00; color:#facc15; border:1px solid #92400e; }
+.gs-badge-hard   { background:#2d0a0a; color:#f87171; border:1px solid #7f1d1d; }
+.gs-card { background: #0a1628; border: 1px solid #1e3a5f; border-radius: 10px; padding: 16px 20px; margin: 10px 0; }
+.gs-card h3 { margin-top: 0; }
+.gs-formula { background: #050d18; border-left: 3px solid #3b82f6; padding: 12px 18px; border-radius: 0 8px 8px 0; margin: 12px 0; font-family: 'IBM Plex Mono', monospace; font-size: 0.83em; color: #93c5fd; white-space: pre; overflow-x: auto; }
+.gs-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; margin: 14px 0; }
+.gs-stat { background: #0a1628; border: 1px solid #1e3a5f; border-radius: 8px; padding: 14px 16px; text-align: center; }
+.gs-stat-val { font-size: 1.7em; font-weight: 800; color: #38bdf8; font-family: 'IBM Plex Mono', monospace; display: block; }
+.gs-stat-lbl { font-size: 0.77em; color: #64748b; margin-top: 4px; display: block; }
+.gs-img { width: 100%; border-radius: 10px; border: 1px solid #1e3a5f; margin: 14px 0; display: block; background: #0a1628; }
+.gs-img-pair { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; margin: 14px 0; }
+.gs-img-caption { font-size: 0.78em; color: #475569; text-align: center; margin-top: -8px; margin-bottom: 12px; font-style: italic; }
+.gs-divider { border: none; border-top: 1px solid #0f1e30; margin: 28px 0; }
+</style>
+
+<div class="gs-readme">
+
+<!-- OVERVIEW -->
+<div class="gs-card" style="border-color:#2563eb;margin-bottom:20px;border-width:1px 1px 1px 3px;">
+  <h3 style="color:#7dd3fc;font-size:1.05em;">What is GraphStrike?</h3>
+  <p>An <strong style="color:#e2e8f0;">OpenEnv-compatible</strong> reinforcement learning environment where an LLM agent
+  must identify all 10 members of a coordinated fake account ring hidden inside a synthetic social network.
+  The agent learns via <strong>Reflexion</strong> and a <strong>dynamic hybrid rule/LLM policy</strong> — no gradient
+  updates, no fine-tuning required.</p>
+  <p style="margin:0;">Submitted to the <strong style="color:#e2e8f0;">OpenEnv Hackathon × SCALER School of Technology</strong>.
+  Judges deploy this container, run their own LLM agent against it, and score on task quality, environment design,
+  code quality, creativity, and domain quality.</p>
+</div>
+
+<!-- KEY STATS -->
+<div class="gs-grid">
+  <div class="gs-stat"><span class="gs-stat-val">10</span><span class="gs-stat-lbl">Gang members to find per episode</span></div>
+  <div class="gs-stat"><span class="gs-stat-val">3</span><span class="gs-stat-lbl">Difficulty tiers (easy / medium / hard)</span></div>
+  <div class="gs-stat"><span class="gs-stat-val">150</span><span class="gs-stat-lbl">Pre-generated episodes (50 per task)</span></div>
+  <div class="gs-stat"><span class="gs-stat-val">24</span><span class="gs-stat-lbl">Automated validator checks</span></div>
+</div>
+
+<!-- SYSTEM ARCHITECTURE -->
+<h2>System Architecture</h2>
+<img src="/assets/sys arch.png" class="gs-img" alt="System Architecture" onerror="this.style.display='none'">
+<p class="gs-img-caption">End-to-end pipeline: episode generation → environment server → hybrid agent → reflexion memory</p>
+
+<!-- DIFFICULTY -->
+<h2>Task Difficulty Tiers</h2>
+<table class="gs-table">
+  <tr><th>Task</th><th>Network Size</th><th>Gang</th><th>Decoys</th><th>Max Steps</th><th>Win Condition</th><th>Baseline Score</th></tr>
+  <tr><td><span class="gs-badge gs-badge-easy">Easy</span></td><td>50 accounts</td><td>10</td><td>0</td><td>30</td><td>Recall ≥ 0.8, Precision ≥ 0.7</td><td>0.910</td></tr>
+  <tr><td><span class="gs-badge gs-badge-medium">Medium</span></td><td>200 accounts</td><td>10</td><td>20</td><td>50</td><td>Recall ≥ 0.8, Precision ≥ 0.7</td><td>0.906</td></tr>
+  <tr><td><span class="gs-badge gs-badge-hard">Hard</span></td><td>1000 accounts</td><td>10</td><td>50</td><td>80</td><td>Recall ≥ 0.9, Precision ≥ 0.8</td><td>0.904</td></tr>
+</table>
+<p style="font-size:0.84em;color:#64748b;margin-top:-8px;">Hard mode fires 4 evasion events (steps 15, 30, 45, 60) that drop intra-gang follow edges mid-investigation, destroying graph signals.</p>
+
+<hr class="gs-divider">
+
+<!-- DETECTION SIGNALS -->
+<h2>Detection Signal Hierarchy</h2>
+<img src="/assets/gs.png" class="gs-img" alt="Signal Hierarchy" onerror="this.style.display='none'">
+<p class="gs-img-caption">Node signals (offline) → Behavioral signals (temporal/device) → Graph signals (live at INSPECT) → False-positive control via hub legitimacy</p>
+
+<h3>Node Signals (pre-computed offline)</h3>
+<table class="gs-table">
+  <tr><th>Feature</th><th>Fake Range</th><th>Real Range</th><th>What it measures</th></tr>
+  <tr><td><code>photo_reuse_score</code></td><td>0.30 – 0.95</td><td>0.00 – 0.15</td><td>Stolen celebrity photos via pHash fingerprint matching</td></tr>
+  <tr><td><code>bio_template_score</code></td><td>0.20 – 0.90</td><td>0.00 – 0.12</td><td>Cosine similarity to known fake bio templates</td></tr>
+  <tr><td><code>comment_repeat_score</code></td><td>0.60 – 0.90</td><td>0.00 – 0.08</td><td>Fraction of copy-pasted spam comments across accounts</td></tr>
+</table>
+
+<h3>Behavioral Signals (temporal + device)</h3>
+<table class="gs-table">
+  <tr><th>Feature</th><th>Fake Pattern</th></tr>
+  <tr><td><code>avg_post_hour</code></td><td>All 10 gang members post within ±0.5h of each other (coordinated scheduling)</td></tr>
+  <tr><td><code>account_age_days</code></td><td>Created same week — base_age ± 7 days</td></tr>
+  <tr><td><code>shared_ip_count</code></td><td>= 9 for all gang members (one IP subnet per episode, unique seed)</td></tr>
+</table>
+
+<h3>Graph Signals (computed live at INSPECT)</h3>
+<table class="gs-table">
+  <tr><th>Feature</th><th>Fake Pattern</th></tr>
+  <tr><td><code>mutual_follow_rate</code></td><td>0.6 – 0.9 (dense intra-gang mutual follows)</td></tr>
+  <tr><td><code>flagged_neighbor_count</code></td><td>Grows as investigation proceeds — strongest late-game signal</td></tr>
+  <tr><td><code>avg_neighbor_photo_reuse</code></td><td>High when cluster shares stolen content</td></tr>
+</table>
+
+<hr class="gs-divider">
+
+<!-- EPISODE FLOW -->
+<h2>Episode Lifecycle &amp; Action Mechanics</h2>
+<img src="/assets/episode.png" class="gs-img" alt="Episode Flow" onerror="this.style.display='none'">
+<p class="gs-img-caption">Episode flow: reset → inspect/flag/investigate loop → dual SUSPECT cascade → submit → grader score</p>
+
+<h3>Action Space</h3>
+<table class="gs-table">
+  <tr><th>Action</th><th>Step Cost</th><th>Effect</th></tr>
+  <tr><td><code>INSPECT acc_XXXX</code></td><td>1 step</td><td>Reveals full AccountProfile + follow list; adds 1-hop neighbors to visible set</td></tr>
+  <tr><td><code>INVESTIGATE_NETWORK acc_XXXX</code></td><td>2 steps</td><td>Bidirectional 2-hop expansion (outgoing + incoming edges); re-cascades SUSPECT</td></tr>
+  <tr><td><code>FLAG acc_XXXX</code></td><td>FREE</td><td>Marks as fake; triggers dual SUSPECT cascade (follow-graph + IP cluster)</td></tr>
+  <tr><td><code>UNFLAG acc_XXXX</code></td><td>FREE</td><td>Removes flag; clears CONFIRMED_FAKE status</td></tr>
+  <tr><td><code>SUBMIT</code></td><td>FREE</td><td>Ends episode; triggers grader scoring</td></tr>
+</table>
+
+<h3>Dual SUSPECT Cascade (triggered by FLAG)</h3>
+<div style="display:grid;grid-template-columns:1fr 1fr;gap:12px;margin:10px 0;">
+  <div class="gs-card">
+    <h3 style="color:#4ade80;margin-top:0;">Cascade 1 — Follow-Graph</h3>
+    <p style="margin:0;font-size:0.88em;">Every account the flagged member <em>follows</em> (<code>_live_edges</code>) becomes SUSPECT if visible and NORMAL. Gang follow density is 0.70+ so this is high-precision.</p>
+  </div>
+  <div class="gs-card">
+    <h3 style="color:#facc15;margin-top:0;">Cascade 2 — IP Cluster</h3>
+    <p style="margin:0;font-size:0.88em;">Every visible account sharing the same <code>ip_cluster_id</code> becomes SUSPECT. Gang shares <code>ip_gang_&lt;seed&gt;</code>; real accounts have unique IPs. <strong>Zero false positives.</strong></p>
+  </div>
+</div>
+
+<hr class="gs-divider">
+
+<!-- RISK SCORING -->
+<h2>Risk Scoring Mathematics</h2>
+<img src="/images/big.png" class="gs-img" alt="Risk Scoring Overview" onerror="this.style.display='none'">
+<p class="gs-img-caption">All scoring functions are stateless and deterministic — called inside _build_profile() at every INSPECT</p>
+
+<div class="gs-img-pair">
+  <div>
+    <img src="/assets/formulas-1.png" class="gs-img" alt="Risk Formulas Part 1" onerror="this.style.display='none'">
+    <p class="gs-img-caption">Node risk, Behavior risk, Graph risk components</p>
+  </div>
+  <div>
+    <img src="/assets/formulas-2.png" class="gs-img" alt="Risk Formulas Part 2" onerror="this.style.display='none'">
+    <p class="gs-img-caption">Hub legitimacy, Composite fake_risk_score formula</p>
+  </div>
+</div>
+
+<div class="gs-formula">fake_risk = clip(
+  0.30 × node_risk        ← content signals (photo reuse, bio templates)
++ 0.25 × behavior_risk   ← temporal + age clustering
++ 0.45 × graph_risk      ← structural coordination (highest weight — hardest to fake)
+− 0.25 × hub_legitimacy, ← subtractive: celebrities score ≈ 0 before clip
+0.0, 1.0)</div>
+
+<h3>Grader Score Formula</h3>
+<div class="gs-formula">recall    = tp / 10
+precision = tp / max(tp + fp, 1)
+efficiency = max(0, (max_steps − steps_used) / max_steps)
+
+if recall ≥ 0.8 and precision ≥ 0.7:
+    score = 0.55 + 0.20×recall + 0.15×precision + 0.10×efficiency
+else:
+    score = 0.30×recall + 0.10×precision
+
+# Maximum possible: 1.00  |  Win threshold: ~0.815</div>
+
+<hr class="gs-divider">
+
+<!-- REFLEXION -->
+<h2>Reflexion Learning</h2>
+<img src="/assets/reflexion.png" class="gs-img" alt="Reflexion Learning Loop" onerror="this.style.display='none'">
+<p class="gs-img-caption">Post-episode lessons injected into every future prompt — learning without weight updates</p>
+
+<p>The LLM (Qwen3-80B via AWS Bedrock) cannot be fine-tuned — it is a black-box API.
+Instead, a separate Qwen3 call generates a 2–3 sentence lesson after each episode.
+The best winning trajectory is stored as a few-shot example injected into all future prompts.</p>
+
+<pre><code>Episode N:
+  LLM acts using: system_prompt + reflections[last 4] + best_trajectory
+  Episode ends → WIN or LOSS
+  LOSS → generate_reflection(action_log, outcome) → lesson stored
+  WIN  → save trajectory if better reward + generate_success_reflection
+
+Episode N+1:
+  last 4 reflections + best win trajectory injected into prompt
+  → LLM has learned from its past without any weight updates</code></pre>
+
+<hr class="gs-divider">
+
+<!-- HYBRID POLICY -->
+<h2>Hybrid Policy — The Novel Contribution</h2>
+<img src="/assets/hybrid.png" class="gs-img" alt="Hybrid Policy Architecture" onerror="this.style.display='none'">
+<p class="gs-img-caption">Dynamic alpha-weighted blend: rules dominate early, LLM earns trust through wins and reflections</p>
+
+<p>A <strong>dynamic α-weighted blend</strong> of a deterministic rule engine and the LLM. α represents trust in the LLM —
+starts at 0.20 (rules dominate), climbs as the LLM wins consistently and accumulates reflections, capped per task
+to prevent the LLM from overriding correct high-confidence rule decisions.</p>
+
+<div class="gs-formula">reflection_factor = min(1.0, n_reflections / 4.0)
+raw   = 0.20 + reflection_factor × (0.80 × recent_win_rate + 0.12)
+alpha = clamp(raw, 0.20, task_cap)
+
+Per-task caps:  easy → 0.50  |  medium → 0.70  |  hard → 0.85</div>
+
+<img src="/images/plot.png" class="gs-img" alt="Alpha progression over training" onerror="this.style.display='none'">
+<p class="gs-img-caption">Alpha progression: rule-dominated early training → LLM earns authority through wins</p>
+
+<h3>Rule Confidence Levels</h3>
+<table class="gs-table">
+  <tr><th>Situation</th><th>Rule Action</th><th>Confidence</th></tr>
+  <tr><td>Steps remaining = 0</td><td>SUBMIT</td><td>1.00</td></tr>
+  <tr><td>Uninspected SUSPECT accounts exist</td><td>INSPECT suspects[0]</td><td>0.95</td></tr>
+  <tr><td><code>fake_risk ≥ 0.85</code></td><td>FLAG that account</td><td>0.95</td></tr>
+  <tr><td><code>fake_risk</code> in [threshold, 0.85)</td><td>FLAG that account</td><td>0.70 – 0.94</td></tr>
+  <tr><td>10 flags placed</td><td>SUBMIT</td><td>0.85</td></tr>
+  <tr><td>Steps remaining ≤ 3</td><td>SUBMIT</td><td>0.90</td></tr>
+  <tr><td>Uninspected accounts available</td><td>INSPECT top candidate</td><td>0.30</td></tr>
+</table>
+<p style="font-size:0.85em;color:#64748b;">When <code>rule_confidence ≥ alpha</code> the rule engine overrides. At easy cap (0.50), the LLM controls only exploratory INSPECT decisions. At hard cap (0.85), the LLM controls most decisions except forced submits and suspect cascade.</p>
+
+</div>
+"""
+
+    _HEADER_HTML = """
+<style>
+.gr-dataframe th { background:#0c2340!important;color:#94a3b8!important;font-weight:700!important;font-size:12px!important;padding:10px 12px!important;border-bottom:1px solid #1e3a5f!important; }
+.gr-dataframe td { font-size:12.5px!important;padding:8px 12px!important; }
+</style>
+<div style="background:linear-gradient(135deg,#050d1a 0%,#0b1f3a 50%,#060f1e 100%);
+            padding:24px 32px 20px;border-radius:12px;
+            border:1px solid #1e3a5f;margin-bottom:2px;
+            box-shadow:0 4px 24px rgba(0,0,0,0.5);">
+  <div style="display:flex;align-items:center;gap:16px;margin-bottom:8px;">
+    <div>
+      <h1 style="color:#e2e8f0;margin:0;font-size:1.9em;font-weight:800;letter-spacing:-0.5px;
+                  font-family:'Inter',system-ui,sans-serif;">GraphStrike</h1>
+      <p style="color:#475569;margin:3px 0 0;font-size:0.88em;letter-spacing:0.3px;font-family:'IBM Plex Mono',monospace;">
+        COORDINATED FAKE ACCOUNT RING DETECTION &mdash; OPENENV RL ENVIRONMENT
+      </p>
+    </div>
+  </div>
+  <div style="display:flex;gap:10px;flex-wrap:wrap;margin-top:12px;">
+    <span style="background:#052e16;color:#4ade80;padding:3px 10px;border-radius:20px;font-size:0.78em;font-weight:600;border:1px solid #166534;">OpenEnv Hackathon</span>
+    <span style="background:#0c1a2e;color:#7dd3fc;padding:3px 10px;border-radius:20px;font-size:0.78em;font-weight:600;border:1px solid #1e40af;">Reinforcement Learning</span>
+    <span style="background:#1c0533;color:#c084fc;padding:3px 10px;border-radius:20px;font-size:0.78em;font-weight:600;border:1px solid #6b21a8;">Hybrid Policy</span>
+    <span style="background:#2d1f00;color:#fbbf24;padding:3px 10px;border-radius:20px;font-size:0.78em;font-weight:600;border:1px solid #92400e;">Reflexion Learning</span>
+    <span style="background:#1a0505;color:#f87171;padding:3px 10px;border-radius:20px;font-size:0.78em;font-weight:600;border:1px solid #7f1d1d;">Fraud Detection</span>
+  </div>
+</div>"""
+
+    _FOOTER_HTML = """
+<div style="text-align:center;padding:24px 0 8px;color:#1e3a5f;font-size:12px;
+            border-top:1px solid #0f1e30;margin-top:28px;font-family:'IBM Plex Mono',monospace;">
+  GraphStrike &mdash; OpenEnv Hackathon &times; SCALER School of Technology &nbsp;|&nbsp;
+  <a href="/docs" style="color:#334155;text-decoration:none;">API Docs</a>
+</div>"""
 
     with gr.Blocks(title="GraphStrike") as demo:
-        gr.Markdown(
-            "# GraphStrike\n"
-            "### Coordinated Fake Account Ring Detection — OpenEnv RL Environment\n\n"
-            "Detect all 10 members of a coordinated fake account ring hidden in a social network.\n"
-            "Use **INSPECT** to reveal profiles, **FLAG** to mark fakes, **SUBMIT** to end.\n\n"
-            "`/reset` `/step` `/state` `/grader` `/baseline` `/tasks` `/health` — [Swagger](/docs)"
-        )
-        with gr.Row():
-            with gr.Column():
-                gr.Markdown("#### 1. Start Episode")
-                task_dd = gr.Dropdown(["easy","medium","hard"], value="easy", label="Task")
-                seed_in = gr.Number(value=0, label="Seed", precision=0)
-                reset_btn = gr.Button("Reset Episode", variant="primary", size="lg")
-            with gr.Column():
-                gr.Markdown("#### 2. Take Actions")
-                action_dd = gr.Dropdown(["inspect","investigate_network","flag","unflag","submit"], value="inspect", label="Action Type")
-                acc_in = gr.Textbox(label="Account ID", placeholder="e.g. acc_0012")
-                step_btn = gr.Button("Step", variant="primary", size="lg")
-
-        obs_md = gr.Markdown(value="*Click 'Reset Episode' to begin.*")
-        with gr.Accordion("Account Profiles (sorted by risk)", open=True):
-            prof_md = gr.Markdown(value="")
-        with gr.Row():
-            grader_btn = gr.Button("Get Grader Score")
-            baseline_btn = gr.Button("Run Baseline (all 3 tasks)")
-        result_box = gr.Textbox(label="Result", lines=5, interactive=False)
-        with gr.Accordion("Raw JSON", open=False):
-            raw_json = gr.Textbox(label="Raw JSON", lines=15, interactive=False)
-
-        reset_btn.click(gr_reset, [task_dd, seed_in], [obs_md, prof_md, raw_json])
-        step_btn.click(gr_step, [action_dd, acc_in], [obs_md, prof_md, raw_json])
-        grader_btn.click(gr_grader, [], result_box)
-        baseline_btn.click(gr_baseline, [], result_box)
+
+        gr.HTML(_HEADER_HTML)
+
+        with gr.Tabs():
+
+            # ══════════════ TAB 1: README ══════════════
+            with gr.Tab("Overview"):
+                gr.HTML(_README_HTML)
+
+            # ══════════════ TAB 2: PLAYGROUND ══════════════
+            with gr.Tab("Playground"):
+                with gr.Row():
+                    with gr.Column(scale=1, min_width=220):
+                        gr.Markdown("**1 — Episode**")
+                        task_dd   = gr.Dropdown(["easy","medium","hard"], value="easy", label="Task")
+                        seed_in   = gr.Number(value=0, label="Seed", precision=0)
+                        reset_btn = gr.Button("Reset", variant="primary")
+
+                    with gr.Column(scale=1, min_width=220):
+                        gr.Markdown("**2 — Action**")
+                        action_dd = gr.Dropdown(
+                            ["inspect","investigate_network","flag","unflag","submit"],
+                            value="inspect", label="Action")
+                        acc_in   = gr.Textbox(label="Account ID", placeholder="acc_0012")
+                        step_btn = gr.Button("Step", variant="primary")
+
+                    with gr.Column(scale=1, min_width=180):
+                        gr.Markdown("**3 — Score**")
+                        gr.Markdown("<br>", container=False)
+                        grader_btn   = gr.Button("Grader Score",   size="sm")
+                        baseline_btn = gr.Button("Baseline Agent", size="sm")
+                        gr.Button("API Docs (Swagger)", size="sm", link="/docs", link_target="_blank")
+
+                obs_md = gr.Markdown(value="*Reset an episode to begin.*")
+
+                gr.Markdown("**Account Profiles** — sorted by fake risk score (highest first)")
+                prof_table = gr.Dataframe(
+                    headers=PROFILE_HEADERS,
+                    datatype=["str","str","number","number","number","number",
+                               "number","number","number","number","number"],
+                    value=[],
+                    interactive=False,
+                    wrap=False,
+                    column_widths=["110px","160px","70px","70px","70px",
+                                   "70px","70px","70px","70px","55px","70px"],
+                )
+
+                result_md = gr.Markdown(value="")
+
+                with gr.Accordion("All Visible IDs", open=False):
+                    vis_md = gr.Markdown(value="")
+                with gr.Accordion("Raw JSON", open=False):
+                    raw_json = gr.Textbox(lines=20, interactive=False)
+
+                reset_btn.click(gr_reset,      [task_dd, seed_in],  [obs_md, prof_table, vis_md, raw_json])
+                step_btn.click( gr_step,       [action_dd, acc_in], [obs_md, prof_table, vis_md, raw_json])
+                grader_btn.click(gr_grader,    [],                   result_md)
+                baseline_btn.click(gr_baseline,[],                   result_md)
+
+            # ══════════════ TAB 2: BENCHMARKS ══════════════
+            with gr.Tab("Benchmarks"):
+                gr.Markdown(
+                    "### LLM Agent Evaluation — GraphStrike Environment\n"
+                    "Agents evaluated with identical system prompts and structured inference. "
+                    "Grader score range: **0.0 – 1.0** (win threshold ≥ 0.815). "
+                    "Score colours: "
+                    "<span style='color:#22c55e'>■</span> ≥0.960 &nbsp; "
+                    "<span style='color:#86efac'>■</span> ≥0.930 &nbsp; "
+                    "<span style='color:#facc15'>■</span> ≥0.910 &nbsp; "
+                    "<span style='color:#f97316'>■</span> below",
+                    sanitize_html=False,
+                )
+
+                gr.Markdown("#### Leaderboard — Single Seed (seed=0)")
+                gr.HTML(_leaderboard_html())
+
+                gr.Markdown("#### Score Distribution by Task")
+                gr.BarPlot(
+                    value=BENCH_LONG_DF,
+                    x="Model", y="Score", color="Task",
+                    title="Agent Scores by Task (seed=0)",
+                    color_map={"Easy": "#4ade80", "Medium": "#facc15", "Hard": "#f87171"},
+                    y_lim=[0.50, 1.0],
+                    x_label_angle=-25,
+                    height=340,
+                )
+
+                gr.Markdown(
+                    "#### Stability — 3-Seed Variance Check (seeds 0, 1, 2)\n"
+                    "Variance colour: "
+                    "<span style='color:#22c55e'>■</span> stable (&lt;0.001) &nbsp; "
+                    "<span style='color:#facc15'>■</span> moderate &nbsp; "
+                    "<span style='color:#f87171'>■</span> high",
+                    sanitize_html=False,
+                )
+                gr.HTML(_variance_html())
+
+                gr.Markdown("#### Rule-Based Baseline (no LLM, deterministic)")
+                gr.HTML(_baseline_html())
+
+                gr.Markdown(
+                    "#### Key Observations\n"
+                    "- Hard task is the real differentiator — evasion events destroy graph signals "
+                    "mid-investigation, requiring adaptive reasoning beyond memorised patterns.\n"
+                    "- Llama 4 Scout 17B achieves the lowest variance on hard (6e-5), "
+                    "outperforming models with 40× more parameters.\n"
+                    "- The rule-based baseline is competitive at mean 0.907, confirming "
+                    "the environment's signal quality. LLM value is in evasion adaptation.\n"
+                    "- All frontier models exceed 0.93 on easy/medium — cascade mechanics "
+                    "are learnable from the structured observation format."
+                )
+
+        gr.HTML(_FOOTER_HTML)
 
     app = gr.mount_gradio_app(app, demo, path="/")
     print("[GraphStrike] Gradio UI mounted at /", flush=True)
 
 except Exception as exc:
+    import traceback
     print(f"[GraphStrike] Gradio unavailable: {exc}", flush=True)
+    traceback.print_exc()
 
     @app.get("/", response_class=HTMLResponse)
     def root_fallback():
-        return "<html><body><h1>GraphStrike</h1><p>API-only mode. <a href='/docs'>Swagger</a></p></body></html>"
-
+        return "<html><body><h1>GraphStrike</h1><p>API mode. <a href='/docs'>Swagger</a></p></body></html>"
 
 # ---------------------------------------------------------------------------
 # Entry point

server/environment.py

@@ -527,11 +527,53 @@ class FakeGangEnvironment(_OpenEnvBase):
             suspicious_mutual_ratio=suspicious_mutual_ratio,
         )
 
+    def _build_hint(self) -> str:
+        """Generate actionable hints for the agent based on current state."""
+        hints = []
+
+        # Hint 1: Uninspected suspects (highest priority)
+        suspect_ids = [
+            sid for sid in self._visible_ids
+            if sid not in self._flagged
+            and self._account_statuses.get(sid, "normal") == "suspect"
+        ]
+        uninspected_suspects = [s for s in suspect_ids if s not in self._inspected]
+        if uninspected_suspects:
+            hints.append(f"HINT: {len(uninspected_suspects)} SUSPECT accounts need inspection — INSPECT {uninspected_suspects[0]} next (auto-elevated by cascade, likely gang member).")
+
+        # Hint 2: Unflagged accounts with strong fake signals
+        unflagged_fakes = []
+        for acc_id in self._inspected:
+            if acc_id in self._flagged:
+                continue
+            p = self._profiled.get(acc_id)
+            if not p:
+                continue
+            if (p.shared_ip_count >= 5
+                or (p.photo_reuse_score >= 0.50 and p.bio_template_score >= 0.40
+                    and p.hub_legitimacy_score < 0.70)):
+                unflagged_fakes.append(acc_id)
+        if unflagged_fakes and not uninspected_suspects:
+            hints.append(f"HINT: FLAG {unflagged_fakes[0]} — strong fake signals detected (photo_reuse/bio_template/shared_ip). FLAG is FREE (costs 0 steps).")
+
+        # Hint 3: Submit reminder
+        steps_left = max(0, self._max_steps - self._step_count)
+        if len(self._flagged) >= 10:
+            hints.append("HINT: You have 10 flags — SUBMIT now to end the episode and get scored.")
+        elif steps_left <= 3 and not self._done:
+            hints.append(f"HINT: Only {steps_left} steps left — consider SUBMIT to lock in your score.")
+
+        return " ".join(hints)
+
     def _make_observation(
         self,
         message: str = "",
         terminal_reward: Optional[float] = None,
     ) -> FakeGangObservation:
+        # Append hints to message for agent guidance
+        hint = self._build_hint() if not self._done else ""
+        full_message = f"{message} {hint}".strip() if hint else message
+
         return FakeGangObservation(
             done=self._done,
             reward=terminal_reward,
@@ -549,7 +591,7 @@ class FakeGangEnvironment(_OpenEnvBase):
             evasion_triggered=self._evasion_triggered,
             evasion_count=self._evasion_count,
             task=self._task,
-            message=message,
+            message=full_message,
             suspect_ids=[
                 sid for sid in self._visible_ids
                 if sid not in self._flagged

server/generator.py

@@ -261,9 +261,19 @@ def generate_episode(task: str, seed: int) -> Dict[str, Any]:
 
     _build_edges(rng, accounts, gang_ids, cfg["intra_gang_density"])
 
-    # Choose starting visible accounts (mix of real + maybe 1 gang member)
+    # Choose starting visible accounts.
+    # Guarantee exactly 1 gang member is included so the cascade CAN start
+    # regardless of seed. The agent still has to identify WHICH account is fake
+    # (requires inspecting profiles) — so difficulty is preserved.
+    # Without this, ~31% of easy episodes and ~82% of hard episodes start with
+    # zero gang members visible, making score variance seed-luck rather than
+    # agent skill.
     starting_count = cfg["starting_visible"]
-    starting_visible = rng.sample(all_ids, starting_count)
+    forced_gang = rng.sample(gang_ids, 1)          # exactly 1 gang member
+    rest_pool = [i for i in all_ids if i not in forced_gang]
+    additional = rng.sample(rest_pool, starting_count - 1)
+    starting_visible = forced_gang + additional
+    rng.shuffle(starting_visible)                   # don't reveal which is fake
 
     return {
         "episode_id": str(uuid.uuid4()),