HACKATHON_PITCH.md

# 🚀 Codebase Navigation & Repair — OpenEnv

## 🧨 The Problem
AI coding agents fail silently and unpredictably. 
Worse, **no one knows WHY they fail.** 
Current benchmarks just give a final Pass/Fail grade. Did the agent read the wrong files? Hallucinate a fix? Ignore the tests entirely? There is no way to know. 

## 💡 The Solution
**We track and evaluate every step of the agent’s reasoning and actions.**
Codebase Navigation & Repair is a system that makes AI coding agents reliable in real-world scenarios. We don't just grade the final output; we grade the *entire journey*.

---

## 🛠️ What It Is
It is a fully OpenEnv-compliant, production-ready testing environment for AI software engineers. 

You drop an AI agent into an unfamiliar Python repository with a hidden bug. The agent cannot cheat by seeing all files at once. It must explore the codebase step-by-step, find the issue, write the fix, and run actual tests to prove it works—exactly like a human engineer.

---

## 🌎 Why It Matters
For developers building autonomous agents (like Devin, Copilot, or Cursor), **reliability** is the biggest unsolved problem. Our system provides a high-fidelity diagnostic layer so researchers can find the exact weak spots in their models and fix them.

---

## 🎬 Demo Walkthrough: A Realistic Bug Scenario

Imagine an e-commerce agent tasked with fixing an order processing failure.

**BEFORE:** ❌ `test_process_valid_order` is failing. 

1. **Step 1:** Agent reads `tests/test_orders.py` to understand the expected behavior.
2. **Step 2:** Agent reads `src/order_processor.py` and spots the bug: a missing `datetime` import causing the script to crash.
3. **Step 3:** Agent writes the fix to `src/order_processor.py`.
4. **Step 4:** Agent runs `pytest`.
5. **Step 5:** Agent submits the fixed codebase.

**AFTER:** ✅ All tests pass. 

Our system records this perfect execution. But if an agent *fails*, our **Process-Based Evaluation** engine flags exactly what went wrong: e.g., *"Agent wasted 14 steps reading irrelevant files and submitted without testing."*

---

## 🏗️ How It Works (Simplified)
1. **The Server:** A FastAPI engine loads a Python repository with a verifiable bug.
2. **The Agent:** An AI model (we provide a Hugging Face Inference agent) requests the current state and explores the repo tree.
3. **The Loop:** The agent interacts via structured actions (`read_file`, `write_file`, `run_tests`).
4. **The Evaluation:** Every action is logged, timed, and scored against our 6-axis Reliability Grader.
5. **The UI:** A beautiful Gradio interface lets you watch the AI operate in real-time or explore its trajectory post-flight.

---

## 🥇 Why It’s Better (Our USP)

We test **Process and Reliability, not just Correctness**. 

- **Flight Data Recorder:** Full trajectory replay. Debug the AI's thought process step-by-step.
- **Dynamic Fault Injection:** Real code is messy. We inject misleading comments and red herring files to see if the AI gets distracted.
- **Proactive Security:** We scan the AI's output for dangerous patterns (like `os.system`) to prevent destructive actions.
- **Context Efficiency:** We penalize agents that waste API tokens by reading identical files over and over.

---

## ⏱️ Why Now?
The rise of autonomous agents is here. But enterprise adoption is stalled because these agents are unpredictable. Moving from "cool toy" to "reliable teammate" requires rigorous, process-level evaluation. Our system directly solves the reliability bottleneck.

---

## 🤝 Hackathon Alignment
We built this explicitly for the Meta OpenEnv hackathon:
- **100% OpenEnv Compliant:** Implements standard `/reset`, `/step`, and `/state` APIs.
- **Live & Deployed:** Running live on Hugging Face Spaces with a Gradio frontend.
- **Inference Ready:** Built-in agent using Hugging Face inference (`run_agent.py`).
- **Sandboxed:** Secure, dockerized test execution.

---

## 🚀 Why This Wins
This is not infrastructure; it is a **diagnostic product for the AI era**. 
It features immense technical depth (sandboxed execution, multi-dimensional scoring, fault injection), massive real-world relevance, and a polished user experience. It doesn't just test AI agents—it shows us how to make them better.