Spaces:
Sleeping
Sleeping
File size: 5,011 Bytes
d83fb27 115612d d73d6cb 115612d 0c5f81b 115612d d57d02f 115612d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 | ---
title: OpenEnv-CloudSOC
emoji: 🔒
colorFrom: blue
colorTo: indigo
sdk: docker
app_file: inference.py
pinned: false
---
# OpenEnv-CloudSOC Benchmark
A production-ready benchmark environment for evaluating LLM agents on cloud security incident response tasks.
## Architecture
**Zero-DB Design**: Entire cloud state managed via in-memory Python dictionaries and Pydantic models. No external databases required.
**Hardware Target**: 2 vCPU / 8 GB RAM Docker container
## Features (12+ Core Mechanics)
| # | Mechanic | Description |
|---|----------|-------------|
| 1 | Deceptive Environment | Logs contain noise, red herrings, and real attack indicators |
| 2 | Partial Observability | Query costs: basic=-0.01, deep=-0.05 |
| 3 | Strict Preconditions | Must snapshot before isolate, detach before rotate |
| 4 | Adversarial Traps | Terminating compromised instances destroys evidence (-1.0) |
| 5 | Gradient Rewards | +0.02 per new flag discovered |
| 6 | Memory Pressure | Sliding context window (last 6 turns) |
| 7 | Tool Abstraction | Strict JSON schema with Pydantic validation |
| 8 | Rich Scoring | Breakdown by IR phase (investigation/containment/eradication/recovery) |
| 9 | Deterministic Seeds | 100% reproducible benchmarks |
| 10 | CoT Prompting | Required thought/tool/args format with dynamic hints |
| 11 | Multi-Task Campaign | Easy→Medium→Hard with shared state |
| 12 | Timeline Reconstruction | Jaccard similarity + order preservation scoring |
### Additional Enhancements
- **24 Tools**: Expanded tool set including aws.config, aws.ssm, aws.lambda, aws.sts
- **MITRE ATT&CK Mapping**: Scenarios tagged with technique IDs
- **Adaptive Temperature**: Retry logic with increasing temperature for diversity
- **Robust JSON Parsing**: Multi-strategy parser with recovery for malformed responses
- **State Serialization**: Full state export for debugging and checkpointing
- **Progress Hints**: Dynamic hints based on agent progress
## Installation
```bash
pip install -r requirements.txt
pip install fastapi uvicorn
```
**For detailed setup instructions, see [SETUP.md](SETUP.md)**
## Usage
### Single Task
```bash
python inference.py --task easy --seed 42
```
### Full Campaign
```bash
python inference.py --task campaign --seed 42
```
### Verbose Mode (for debugging)
```bash
python inference.py --task easy --verbose
```
### Environment Variables
```bash
export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4.1-mini"
export HF_TOKEN="your_token_here"
```
## Files
| File | Size | Description |
|------|------|-------------|
| `cloud_soc_env.py` | ~65KB | Gymnasium environment with state engine and grader |
| `inference.py` | ~18KB | LLM evaluation loop with hackathon-compliant output |
| `openenv.yaml` | ~11KB | Benchmark metadata specification |
| `requirements.txt` | - | Dependencies (gymnasium, pydantic, openai) |
| `Dockerfile` | - | Docker build for 2 vCPU/8GB constraint |
## Hugging Face Spaces Deployment
To deploy this benchmark to Hugging Face Spaces:
1. **Fork or clone** this repository to your GitHub account
2. **Create a new Space** at https://huggingface.co/new-space
- Space name: `openenv-cloudsoc`
- SDK: **Docker**
- Repository: Link to your GitHub fork
3. **Add HF_TOKEN Secret**:
- Go to Space Settings → Secrets
- Add `HF_TOKEN` with your Hugging Face API token (get one at https://huggingface.co/settings/tokens)
4. **Space will build and deploy automatically**
Once deployed, the Space will run inference on startup. Monitor the logs in the Space UI.
## Scenarios
### Easy: Leaky S3 Bucket Discovery
- **Steps**: 15 max
- **Flags**: s3_public_identified, credentials_found, public_access_blocked
- **MITRE**: T1530 (Data from Cloud Storage Object)
### Medium: Credential Compromise Response
- **Steps**: 25 max
- **Flags**: leaked_creds_identified, compromised_role_found, role_detached, credentials_rotated
- **MITRE**: T1078, T1552 (Valid Accounts, Credentials in Files)
### Hard: Full Incident Response - Ransomware
- **Steps**: 40 max
- **Flags**: ransomware_detected, backdoor_identified, forensic_snapshot_taken, instance_isolated, backdoor_removed, all_creds_rotated, systems_verified
- **MITRE**: T1486, T1098, T1078 (Data Encrypted, Account Manipulation)
## Output Format
```
[START] task=easy env=cloudsoc model=gpt-4.1-mini
[STEP] step=1 action=aws.soc.get_alerts({}) reward=0.00 done=false error=null
[STEP] step=2 action=aws.s3.get_bucket_policy({"bucket_name":"company-backup-2024"}) reward=0.02 done=false error=null
...
[END] success=true steps=8 rewards=0.00,0.02,0.02,0.10,0.05,0.10,0.05,0.25
```
## Scoring
Final score is a weighted average across IR phases:
```python
score = (
investigation * phase_weights["investigation"] +
containment * phase_weights["containment"] +
eradication * phase_weights["eradication"] +
recovery * phase_weights["recovery"]
)
```
Timeline accuracy uses Jaccard similarity with order preservation bonus.
## License
MIT
|