| # SecretsAuditEnv Specification v3.0 |
|
|
| This document defines the architecture, grading logic, and task matrix for the **SecretsAuditEnv**, a benchmark designed for the Meta OpenEnv Hackathon to evaluate AI agents on their ability to remediate leaked secrets in a Git-backed environment. |
|
|
| --- |
|
|
| ## 1. System Architecture |
|
|
| The environment operates as a FastAPI-based server that manages an isolated workspace for the AI agent. |
|
|
| ### 1.1 Components |
| * **Environment (`server/environment.py`)**: Manages the state machine, workspace isolation, and task resets. |
| * **API Wrapper (`server/app.py`)**: Exposes endpoints (`/reset`, `/step`, `/state`) for the agent to interact with. |
| * **Grading Engine (`graders/`)**: A modular scoring system that evaluates security and code health. |
| * **Tasks (`tasks/`)**: A hierarchical directory of vulnerable codebases categorized by difficulty. |
|
|
| --- |
|
|
| ## 2. Grading Logic |
|
|
| The environment uses a multi-faceted scoring approach to ensure agents do not "cheat" by deleting vulnerable code or breaking functionality. |
|
|
| ### 2.1 The Core Formula |
| The final reward is a product of security remediation and functional integrity: |
|
|
| $$Total Reward = Security Score \times Health Score$$ |
|
|
| Where: |
| * **Security Score**: Calculated using Gitleaks. It is normalized based on the number of leaks fixed compared to the initial state. |
| $$Security Score = \frac{InitialLeaks - CurrentLeaks}{InitialLeaks}$$ |
| * **Health Score**: Derived from `pytest` results. If the code fails to execute (e.g., due to a `NameError`), this score is **0.0**. |
| |
| ### 2.2 History-Aware Scanning |
| To prevent "Scorched Earth" tactics (deleting the `.git` directory or source files), the grader employs a recovery mechanism: |
| > **`_ensure_git_source`**: If the agent deletes the Git history, the grader force-initializes a temporary repository and commits the current files. Gitleaks then scans the *entire* history to ensure secrets haven't simply been moved to a previous commit. |
| |
| --- |
| |
| ## 3. Expanded Task Matrix |
| |
| ### Category 1: Easy (The "Low-Hanging Fruit") |
| *Focus: Detection and basic remediation of plain-text secrets.* |
| |
| | Task ID | Scenario | Primary File(s) | Vulnerability | Success Criteria | |
| | :--- | :--- | :--- | :--- | :--- | |
| | **Task 1** | Cloud Provisioning | `config.py` | Hardcoded AWS Access Key. | Secret moved to `.env`; `os.getenv` used. | |
| | **Task 2** | Database Layer | `db.py` | Password inside a raw SQL connection string. | String parameterized; creds externalized. | |
| | **Task 3** | Frontend Config | `settings.js` | Firebase API key in client-side config. | Moved to build-time environment variables. | |
| | **Task 4** | System Logging | `logger.py` | `logger.debug` leaking user tokens. | Log statement redacted or level changed. | |
| | **Task 5** | Git Basics | `.env` | `.env` file present in the working tree. | `.gitignore` created; file removed from tree. | |
| |
| ### Category 2: Medium (The "Obfuscation & Format" Challenge) |
| *Focus: Handling encoding, multiline strings, and non-Python configurations.* |
| |
| | Task ID | Scenario | Primary File(s) | Vulnerability | Success Criteria | |
| | :--- | :--- | :--- | :--- | :--- | |
| | **Task 6** | Utility Module | `utils.py` | Base64 encoded auth token. | Token identified and externalized correctly. | |
| | **Task 7** | CI/CD Pipeline | `deploy.yml` | GitHub Action echoing a secret to logs. | Replaced with `${{ secrets.GITHUB_TOKEN }}`. | |
| | **Task 8** | Noise Filtering | `.toml` | Dummies mixed with one high-entropy key. | Agent uses `.gitleaks.toml` to filter noise. | |
| | **Task 9** | DB Migration | `migrate.sql` | Admin credentials in a legacy SQL script. | Script refactored; credentials scrubbed. | |
| | **Task 10** | Deployment | `deploy.sh` | Multiline RSA Private Key string. | Key moved; `\n` formatting preserved. | |
|
|
| ### Category 3: Hard (The "Architectural & History" Bosses) |
| *Focus: Cross-file consistency, logical embedding, and Git history manipulation.* |
|
|
| | Task ID | Scenario | Primary File(s) | Vulnerability | Success Criteria | |
| | :--- | :--- | :--- | :--- | :--- | |
| | **Task 11** | Microservices | `service_*.py` | Same API key leaked across 5 separate files. | All 5 files updated consistently. | |
| | **Task 12** | Deep Logic | `crypto.py` | Secret used as a local variable inside a function. | Function logic remains valid after fix. | |
| | **Task 13** | Legacy Audit | `.git/` | Secret committed in v1.0, still in history. | `git filter-repo` used to rewrite history. | |
|
|
| --- |
|
|
| ## 4. Technical Requirements for Remediation |
|
|
| To achieve a **1.00 score**, the agent must satisfy three technical constraints: |
|
|
| 1. **Detection**: Gitleaks must return 0 findings for the *entire* Git history. |
| 2. **Externalization**: Hardcoded values must be replaced by environment variable lookups (e.g., `os.environ.get()`). |
| 3. **Dependency Integrity**: If the agent introduces a library like `os`, it must explicitly include the necessary `import` statements. Failure results in a **Health Score of 0.0**. |
|
|
| --- |
|
|
| ## 5. Known Failure Modes & Guardrails |
|
|
| ### 5.1 The "Missing Import" Loop |
| **Issue**: Agents often replace a secret with `os.environ.get()` but forget `import os`. |
| **Guardrail**: The `Health Score` stays at **0.0** until the `NameError` is resolved. |
|
|
| ### 5.2 The `NoneType` / `true` Fallback |
| **Issue**: If an LLM returns non-executable prose, the parser returns an empty string. |
| **Guardrail**: `inference.py` defaults empty actions to the bash command `true` to maintain the loop. |
|
|
| ### 5.3 Environment Variable Exports |
| **Issue**: Bare assignments in `.env` files are not picked up by Python. |
| **Requirement**: Users must use `set -a && source .env && set +a` to ensure variables are exported. |
|
|
| --- |
|
|
| ## 6. Operational Checklist |
| * [ ] **Server**: `uvicorn server.app:app` is running on `localhost:8000`. |
| * [ ] **Gitleaks**: Binary is installed and accessible in system path. |
| * [ ] **Filter-Repo**: `git-filter-repo` is installed for Task 13. |
| * [ ] **Dependencies**: `pip install -r requirements.txt --break-system-packages`. |
|
|
