spec.md

# SecretsAuditEnv Specification v3.0

This document defines the architecture, grading logic, and task matrix for the **SecretsAuditEnv**, a benchmark designed for the Meta OpenEnv Hackathon to evaluate AI agents on their ability to remediate leaked secrets in a Git-backed environment.

---

## 1. System Architecture

The environment operates as a FastAPI-based server that manages an isolated workspace for the AI agent.

### 1.1 Components
* **Environment (`server/environment.py`)**: Manages the state machine, workspace isolation, and task resets.
* **API Wrapper (`server/app.py`)**: Exposes endpoints (`/reset`, `/step`, `/state`) for the agent to interact with.
* **Grading Engine (`graders/`)**: A modular scoring system that evaluates security and code health.
* **Tasks (`tasks/`)**: A hierarchical directory of vulnerable codebases categorized by difficulty.

---

## 2. Grading Logic

The environment uses a multi-faceted scoring approach to ensure agents do not "cheat" by deleting vulnerable code or breaking functionality.

### 2.1 The Core Formula
The final reward is a product of security remediation and functional integrity:

$$Total Reward = Security Score \times Health Score$$

Where:
* **Security Score**: Calculated using Gitleaks. It is normalized based on the number of leaks fixed compared to the initial state.
    $$Security Score = \frac{InitialLeaks - CurrentLeaks}{InitialLeaks}$$
* **Health Score**: Derived from `pytest` results. If the code fails to execute (e.g., due to a `NameError`), this score is **0.0**.

### 2.2 History-Aware Scanning
To prevent "Scorched Earth" tactics (deleting the `.git` directory or source files), the grader employs a recovery mechanism:
> **`_ensure_git_source`**: If the agent deletes the Git history, the grader force-initializes a temporary repository and commits the current files. Gitleaks then scans the *entire* history to ensure secrets haven't simply been moved to a previous commit.

---

## 3. Expanded Task Matrix

### Category 1: Easy (The "Low-Hanging Fruit")
*Focus: Detection and basic remediation of plain-text secrets.*

| Task ID | Scenario | Primary File(s) | Vulnerability | Success Criteria |
| :--- | :--- | :--- | :--- | :--- |
| **Task 1** | Cloud Provisioning | `config.py` | Hardcoded AWS Access Key. | Secret moved to `.env`; `os.getenv` used. |
| **Task 2** | Database Layer | `db.py` | Password inside a raw SQL connection string. | String parameterized; creds externalized. |
| **Task 3** | Frontend Config | `settings.js` | Firebase API key in client-side config. | Moved to build-time environment variables. |
| **Task 4** | System Logging | `logger.py` | `logger.debug` leaking user tokens. | Log statement redacted or level changed. |
| **Task 5** | Git Basics | `.env` | `.env` file present in the working tree. | `.gitignore` created; file removed from tree. |

### Category 2: Medium (The "Obfuscation & Format" Challenge)
*Focus: Handling encoding, multiline strings, and non-Python configurations.*

| Task ID | Scenario | Primary File(s) | Vulnerability | Success Criteria |
| :--- | :--- | :--- | :--- | :--- |
| **Task 6** | Utility Module | `utils.py` | Base64 encoded auth token. | Token identified and externalized correctly. |
| **Task 7** | CI/CD Pipeline | `deploy.yml` | GitHub Action echoing a secret to logs. | Replaced with `${{ secrets.GITHUB_TOKEN }}`. |
| **Task 8** | Noise Filtering | `.toml` | Dummies mixed with one high-entropy key. | Agent uses `.gitleaks.toml` to filter noise. |
| **Task 9** | DB Migration | `migrate.sql` | Admin credentials in a legacy SQL script. | Script refactored; credentials scrubbed. |
| **Task 10** | Deployment | `deploy.sh` | Multiline RSA Private Key string. | Key moved; `\n` formatting preserved. |

### Category 3: Hard (The "Architectural & History" Bosses)
*Focus: Cross-file consistency, logical embedding, and Git history manipulation.*

| Task ID | Scenario | Primary File(s) | Vulnerability | Success Criteria |
| :--- | :--- | :--- | :--- | :--- |
| **Task 11** | Microservices | `service_*.py` | Same API key leaked across 5 separate files. | All 5 files updated consistently. |
| **Task 12** | Deep Logic | `crypto.py` | Secret used as a local variable inside a function. | Function logic remains valid after fix. |
| **Task 13** | Legacy Audit | `.git/` | Secret committed in v1.0, still in history. | `git filter-repo` used to rewrite history. |

---

## 4. Technical Requirements for Remediation

To achieve a **1.00 score**, the agent must satisfy three technical constraints:

1.  **Detection**: Gitleaks must return 0 findings for the *entire* Git history.
2.  **Externalization**: Hardcoded values must be replaced by environment variable lookups (e.g., `os.environ.get()`).
3.  **Dependency Integrity**: If the agent introduces a library like `os`, it must explicitly include the necessary `import` statements. Failure results in a **Health Score of 0.0**.

---

## 5. Known Failure Modes & Guardrails

### 5.1 The "Missing Import" Loop
**Issue**: Agents often replace a secret with `os.environ.get()` but forget `import os`.
**Guardrail**: The `Health Score` stays at **0.0** until the `NameError` is resolved.

### 5.2 The `NoneType` / `true` Fallback
**Issue**: If an LLM returns non-executable prose, the parser returns an empty string.
**Guardrail**: `inference.py` defaults empty actions to the bash command `true` to maintain the loop.

### 5.3 Environment Variable Exports
**Issue**: Bare assignments in `.env` files are not picked up by Python.
**Requirement**: Users must use `set -a && source .env && set +a` to ensure variables are exported.

---

## 6. Operational Checklist
* [ ] **Server**: `uvicorn server.app:app` is running on `localhost:8000`.
* [ ] **Gitleaks**: Binary is installed and accessible in system path.
* [ ] **Filter-Repo**: `git-filter-repo` is installed for Task 13.
* [ ] **Dependencies**: `pip install -r requirements.txt --break-system-packages`.