Hugging Face's logo

dexter-2k
/
openenv-secaudit

Model card Files
xet
 Community
openenv-secaudit / README.md
dexter-2k's picture
dexter-2k
Upload folder using huggingface_hub
ee929ba verified
preview code
|
Raw
History Blame Contribute Delete
14.9 kB

SecretsAuditEnv

A reinforcement-learning benchmark that drops an AI agent into a git-backed codebase seeded with realistic secret leaks — hardcoded API keys, base64-encoded tokens, credentials buried in git history — and grades how quickly and safely the agent remediates every one. The environment runs as a stateless FastAPI server; the agent interacts over HTTP with bash commands and structured inspection actions, receiving a composite reward after every step that captures security progress, code health, and time efficiency.

Why This Exists

Production secret leaks remain a top-5 cause of cloud breaches. Developers commit API keys, push .env files, or leave credentials in migration scripts — then scramble to rotate and rewrite history. Existing linting tools flag secrets but don't fix them; LLM agents can, but there's no standardized benchmark to measure how well. SecretsAuditEnv fills that gap with a 13-task curriculum spanning trivial single-file fixes to multi-service cascading leaks with git-history rewriting, complete with a deterministic grading pipeline and an anti-gaming reward function.

Task Curriculum

All 13 tasks are defined in tasks/ with full metadata in each task.json. Secret types are detected by graders/security.py using regex patterns for AWS keys, GitHub tokens, Firebase keys, connection strings, private keys, SQL passwords, assignment secrets, and base64-encoded variants.

ID Difficulty Title Description Scan Mode Visibility Tiers Conflict Map
1 Easy Cloud Provisioning Hardcoded AWS Access Key in config.py dir surface, surface, shallow —
2 Easy Database Layer Password embedded in a raw SQL connection string dir surface, surface, shallow —
3 Easy Frontend Config Firebase API key exposed in a client-side config file dir surface, surface, shallow —
4 Easy System Logging Debug logging leaks a user token dir surface, surface, shallow —
5 Easy Git Basics A tracked .env file leaks credentials in the working tree dir surface, surface, shallow —
6 Medium Utility Module A base64-encoded auth token hides in utils.py dir surface, shallow, deep —
7 Medium CI/CD Pipeline A deployment workflow prints a secret directly to logs dir surface, shallow, deep —
8 Medium Noise Filtering High-entropy dummy values are mixed with one real secret in TOML dir surface, shallow, deep —
9 Medium DB Migration A legacy migration embeds administrator credentials dir surface, shallow, deep —
10 Medium Deployment A multiline RSA private key is embedded in a shell script dir surface, shallow, deep —
11 Hard Microservices The same API key is duplicated across five services dir surface, deep, cascading ✓
12 Hard Deep Logic A secret is embedded as a local variable inside a function dir surface, deep, cascading ✓
13 Hard Legacy Audit A secret was committed in v1.0 and still exists in Git history git surface, deep, cascading ✓

Scan modes: dir scans the working directory only. git scans the full git commit history (all revisions), so agents must use git filter-repo to clean history — deleting files won't work.

Secret Visibility Tiers

Secrets are not all visible at episode start. The environment implements a 4-tier progressive disclosure system defined per-secret in task.json:

Tier When Visible Typical Use
SURFACE Immediately on /reset Obvious hardcoded keys in source files
SHALLOW After agent calls inspect_file <path> Secrets that require reading the file to notice
DEEP After inspect_git_history or inspect_encoded Secrets in git commits or base64-encoded blobs
CASCADING After a specified trigger secret is fixed Secrets that only become relevant after another is remediated

Hard tasks (11–13) include conflict maps that encode dependencies: fixing secret s1 may reveal s3, while s2 may block s3 until resolved.

Reward Formula

Defined in graders/reward.py. Computed fresh after every /step:

detection_score = (initial_leaks - current_leaks) / initial_leaks
base            = 0.4 × detection_score + 0.6 × detection_score     # (remediation = detection for now)
health_score    = pytest_passed / pytest_total                       # 0.0 if errors > 0 or total == 0
efficiency      = 0.15 × max(0, 1 - steps_taken / step_budget)      # decays linearly per step

if base > 0:
    total_reward = min(1.0, base × health_score + efficiency)
else:
    total_reward = 0.0                                               # no free points for doing nothing

Key properties:

  • Reward is 0.0 until the agent actually fixes something — read-only commands like cat or ls cannot earn reward
  • Health gate: breaking the test suite (e.g., deleting imports) multiplies reward toward zero
  • Efficiency bonus (0.0–0.15): rewards agents that solve in fewer steps. Decays to 0 at step budget
  • Step budgets: Easy = 10, Medium = 20, Hard = 30

Action Space

Actions are sent as the action field in POST /step. Two categories:

Structured Actions (intercepted before bash)

Action Format Effect
inspect_file inspect_file <path> Marks file as inspected → unlocks SHALLOW secrets for that path
inspect_git_history inspect_git_history [path] Scans git history → unlocks all DEEP secrets requiring git inspection
inspect_encoded inspect_encoded <path> [line] Decodes base64 blobs → unlocks DEEP encoded secrets for that path

Bash Commands (executed in workspace via /usr/bin/bash -lc)

Any string that doesn't match a structured prefix is executed as a bash command in the task workspace. Common patterns:

  • cat config.py — read file contents
  • sed -i 's/AKIA.../os.getenv("AWS_KEY")/' config.py — redact a secret
  • git filter-repo --replace-text <(echo 'ghp_xxx==>REDACTED') --force — clean git history
  • gitleaks detect --no-git --source . — run leak scanner

Commands time out after 90 seconds. Exit code, stdout, and stderr are returned in the observation.

Observation Keys

Every /step and /reset response returns a session object with these fields (also listed in openenv.yaml):

Key Type Description
visible_secrets list[dict] Secrets the agent can currently see (filtered by visibility tier)
hidden_count_hint int Number of secrets not yet visible — tells agent more exist
ranked_actions list[dict] Top-5 heuristic action suggestions sorted by priority (0.0–1.0)
top_blocker string One-sentence description of the highest-priority next action
step_budget int Total step budget for this difficulty tier
steps_taken int Steps consumed so far
steps_remaining int step_budget - steps_taken
efficiency_bonus float Current efficiency bonus value (decays each step)
conflict_map dict Dependency graph between visible secrets (reveals/blocks relationships)
security_score float Fraction of initial leaks fixed (0.0–1.0)
health_score float Fraction of pytest tests passing (0.0–1.0)
reward float Composite reward after this step
observation string Combined stdout/stderr from last command + health/security messages
last_result dict Raw action, exit_code, stdout, stderr, timed_out from last command

Ranked Actions

Generated by server/observation.py using 5 heuristics:

  1. Visible unfixed secrets → priority 0.92 (fix these first)
  2. Uninspected high-risk files → priority based on filename suspicion score
  3. Git history not yet scanned → priority 0.75 (medium/hard only)
  4. Hidden secrets remaining → priority 0.68 (suggest encoded inspection)
  5. All visible fixed → priority 0.85 (run gitleaks validation)

Quick Start

Local Development 

# Install dependencies
pip install -r requirements.txt

# Generate task workspaces
python tools/generate_tasks.py

# Start the server
uvicorn server.app:app --host 0.0.0.0 --port 7860

# Open the debug UI
open http://localhost:7860/web

# Run the agent (requires LLM API)
export API_BASE_URL="https://openrouter.ai/api/v1"
export HF_TOKEN="your-api-key"
export MODEL_NAME="nvidia/nemotron-3-super-120b-a12b:free"
export ENV_URL="http://localhost:7860"
python inference.py --task-id 1

Docker 

docker build -t secretsauditenv .
docker run -p 7860:7860 secretsauditenv
# Server auto-generates tasks and starts on port 7860

The Dockerfile installs Python 3.11-slim with bash, git, git-filter-repo, and all pip dependencies. The entrypoint runs tools/generate_tasks.py then starts uvicorn.

inference.py — Agent Loop

The baseline agent (inference.py) implements a single-turn ReAct loop over any OpenAI-compatible API:

  1. Reset — POST /reset with the target task ID
  2. Prompt — Builds a structured prompt from the current session state (reward, leaks, health, last command output, recent actions)
  3. Call LLM — Sends prompt to the model, extracts a bash command from the response
  4. Normalize — Strips markdown fences, extracts from XML tool_call formats, filters prose prefixes, enforces atomic (no && or ; chaining)
  5. Step — POST /step with the action, reads updated state
  6. Repeat until reward >= 1.0 or MAX_STEPS (40) reached

Environment variables:

Variable Required Description
API_BASE_URL Yes OpenAI-compatible API endpoint (e.g., https://openrouter.ai/api/v1)
HF_TOKEN Yes API key for the model provider
MODEL_NAME Yes Model identifier (e.g., nvidia/nemotron-3-super-120b-a12b:free)
ENV_URL No Server URL, defaults to http://localhost:7860
TASK_ID No Default task, overridden by --task-id CLI arg

Anti-loop features:

  • Detects 3 consecutive identical actions with identical rewards → injects a CRITICAL WARNING into the prompt and forces a different command
  • Filters natural language preambles (e.g., "Let me check...", "Looking at...") before sending to bash

Grading Pipeline

Security Grader (graders/security.py)

Custom regex-based scanner (no external tools required). Detects:

  • AWS Access Keys (AKIA...)
  • GitHub tokens (ghp_...)
  • Firebase API keys (AIza...)
  • Service tokens (tok_live_..., sk_test_...)
  • Private keys (PEM format)
  • SQL connection strings (postgres://user:pass@host/db)
  • SQL passwords (PASSWORD 'value')
  • High-entropy assignment secrets (Shannon entropy ≥ 3.2)
  • Base64-encoded variants of all the above

Supports .gitleaks.toml allowlists. For scan_mode: git, scans all commits via git rev-list --all.

Anti-gaming: if an agent deletes .git, the grader recovers by creating a fresh snapshot and scanning that — the secret still gets found.

Health Grader (graders/health.py)

Runs pytest -q --junitxml in the task workspace and parses the JUnit XML report. Score = passed / total. Returns 0.0 if any errors or if pytest times out (60s default).

Web Debug UI

Available at GET /web. A single-page dark-mode dashboard served from server/web_ui.html that lets you:

  • Start any of the 13 tasks with one click
  • Send structured actions or raw bash commands
  • View real-time metrics (reward, leaks, hidden count, steps, efficiency, health)
  • See ranked action suggestions and conflict maps
  • Browse visible secrets and full observation text
  • Review action history

No external dependencies — pure HTML/CSS/JS.

Validation 

# Run the environment test suite (34 tests)
python -m pytest tests/ -v

# Run the submission validator (checks connection, Docker build, spec compliance, reward integrity)
bash validate-submission.sh

The validator checks:

  1. Connection — /reset returns HTTP 200
  2. Docker portability — no absolute host paths leak into the image
  3. Spec compliance — spec.md exists, all 13 task directories present, openenv validate passes
  4. Reward integrity — smoke test runs inference.py for 2 steps and verifies [START]/[STEP]/[END] log format

Project Structure 

.
├── server/
│   ├── app.py              # FastAPI routes (/reset, /step, /state, /tasks, /web)
│   ├── environment.py      # Core environment logic, visibility tiers, action parsing
│   ├── observation.py      # Ranked action heuristics and top_blocker computation
│   └── web_ui.html         # Debug dashboard (served at /web)
├── graders/
│   ├── security.py         # Regex-based secret scanner with git history support
│   ├── health.py           # Pytest-based health grader with JUnit parsing
│   ├── reward.py           # Composite reward: security × health + efficiency
│   ├── gitleaks_eval.py    # Spec-aligned wrapper with git integrity reporting
│   └── health_eval.py      # Spec-aligned wrapper with failure messaging
├── tasks/
│   ├── easy/task_01..05/   # 5 easy tasks (single-file, surface+shallow secrets)
│   ├── medium/task_06..10/ # 5 medium tasks (multi-format, surface+shallow+deep)
│   └── hard/task_11..13/   # 3 hard tasks (cascading, conflict maps, git history)
├── tests/
│   ├── test_hidden_leaks.py    # 11 tests: visibility tier logic
│   ├── test_ranked_actions.py  # 12 tests: heuristic action suggestions
│   └── test_reward.py         # 11 tests: reward formula and efficiency bonus
├── tools/
│   ├── generate_tasks.py   # Generates all 13 task workspaces with seeded secrets
│   ├── check_space_ready.py
│   ├── prepare_hf_space_bundle.sh
│   └── prepare_github_upload_bundle.sh
├── inference.py            # Baseline LLM agent loop (OpenAI-compatible)
├── openenv.yaml            # OpenEnv spec declaration
├── Dockerfile              # Python 3.11-slim + git + git-filter-repo
├── hf_space_entrypoint.sh  # Docker entrypoint: generate tasks → start uvicorn
├── validate-submission.sh  # 4-check submission validator
├── requirements.txt        # Pip dependencies
├── pyproject.toml          # Package metadata
└── spec.md                 # Environment specification document

License

MIT