spec.md

SecretsAuditEnv Specification v3.0

This document defines the architecture, grading logic, and task matrix for the SecretsAuditEnv, a benchmark designed for the Meta OpenEnv Hackathon to evaluate AI agents on their ability to remediate leaked secrets in a Git-backed environment.

---

1. System Architecture

The environment operates as a FastAPI-based server that manages an isolated workspace for the AI agent.

1.1 Components

• Environment (server/environment.py): Manages the state machine, workspace isolation, and task resets.
• API Wrapper (server/app.py): Exposes endpoints (/reset, /step, /state) for the agent to interact with.
• Grading Engine (graders/): A modular scoring system that evaluates security and code health.
• Tasks (tasks/): A hierarchical directory of vulnerable codebases categorized by difficulty.

---

2. Grading Logic

The environment uses a multi-faceted scoring approach to ensure agents do not "cheat" by deleting vulnerable code or breaking functionality.

2.1 The Core Formula

The final reward is a product of security remediation and functional integrity:

$$TotalReward=SecurityScore\times HealthScoreTotal\; Reward\; =\; Security\; Score\; \backslash times\; Health\; Score$$

Where:

• Security Score: Calculated using Gitleaks. It is normalized based on the number of leaks fixed compared to the initial state. $$Security Score = \frac{InitialLeaks - CurrentLeaks}{InitialLeaks}$$
• Health Score: Derived from pytest results. If the code fails to execute (e.g., due to a NameError), this score is 0.0.

2.2 History-Aware Scanning

To prevent "Scorched Earth" tactics (deleting the .git directory or source files), the grader employs a recovery mechanism:

_ensure_git_source: If the agent deletes the Git history, the grader force-initializes a temporary repository and commits the current files. Gitleaks then scans the entire history to ensure secrets haven't simply been moved to a previous commit.

---

3. Expanded Task Matrix

Category 1: Easy (The "Low-Hanging Fruit")

Focus: Detection and basic remediation of plain-text secrets.

Task ID	Scenario	Primary File(s)	Vulnerability	Success Criteria
Task 1	Cloud Provisioning	config.py	Hardcoded AWS Access Key.	Secret moved to .env; os.getenv used.
Task 2	Database Layer	db.py	Password inside a raw SQL connection string.	String parameterized; creds externalized.
Task 3	Frontend Config	settings.js	Firebase API key in client-side config.	Moved to build-time environment variables.
Task 4	System Logging	logger.py	logger.debug leaking user tokens.	Log statement redacted or level changed.
Task 5	Git Basics	.env	.env file present in the working tree.	.gitignore created; file removed from tree.

Category 2: Medium (The "Obfuscation & Format" Challenge)

Focus: Handling encoding, multiline strings, and non-Python configurations.

Task ID	Scenario	Primary File(s)	Vulnerability	Success Criteria
Task 6	Utility Module	utils.py	Base64 encoded auth token.	Token identified and externalized correctly.
Task 7	CI/CD Pipeline	deploy.yml	GitHub Action echoing a secret to logs.	Replaced with ${{ secrets.GITHUB_TOKEN }}.
Task 8	Noise Filtering	.toml	Dummies mixed with one high-entropy key.	Agent uses .gitleaks.toml to filter noise.
Task 9	DB Migration	migrate.sql	Admin credentials in a legacy SQL script.	Script refactored; credentials scrubbed.
Task 10	Deployment	deploy.sh	Multiline RSA Private Key string.	Key moved; \n formatting preserved.

Category 3: Hard (The "Architectural & History" Bosses)

Focus: Cross-file consistency, logical embedding, and Git history manipulation.

Task ID	Scenario	Primary File(s)	Vulnerability	Success Criteria
Task 11	Microservices	service_*.py	Same API key leaked across 5 separate files.	All 5 files updated consistently.
Task 12	Deep Logic	crypto.py	Secret used as a local variable inside a function.	Function logic remains valid after fix.
Task 13	Legacy Audit	.git/	Secret committed in v1.0, still in history.	git filter-repo used to rewrite history.

---

4. Technical Requirements for Remediation

To achieve a 1.00 score, the agent must satisfy three technical constraints:

1. Detection: Gitleaks must return 0 findings for the entire Git history.
2. Externalization: Hardcoded values must be replaced by environment variable lookups (e.g., os.environ.get()).
3. Dependency Integrity: If the agent introduces a library like os, it must explicitly include the necessary import statements. Failure results in a Health Score of 0.0.

---

5. Known Failure Modes & Guardrails

5.1 The "Missing Import" Loop

Issue: Agents often replace a secret with os.environ.get() but forget import os. Guardrail: The Health Score stays at 0.0 until the NameError is resolved.

5.2 The NoneType / true Fallback

Issue: If an LLM returns non-executable prose, the parser returns an empty string. Guardrail: inference.py defaults empty actions to the bash command true to maintain the loop.

5.3 Environment Variable Exports

Issue: Bare assignments in .env files are not picked up by Python. Requirement: Users must use set -a && source .env && set +a to ensure variables are exported.

---

6. Operational Checklist

• Server: uvicorn server.app:app is running on localhost:8000.
• Gitleaks: Binary is installed and accessible in system path.
• Filter-Repo: git-filter-repo is installed for Task 13.
• Dependencies: pip install -r requirements.txt --break-system-packages.