sandbox/Local_Testing_Guide.md

Local Test Runbook (Windows + PowerShell)

This guide is intentionally explicit so you can run a full sandbox validation without guessing any steps.

---

0) Prerequisites

Install and verify:

• Docker Desktop (running)
• Python 3.11+
• Git
• curl.exe (bundled with Windows 10+; do NOT rely on the curl alias which on Windows PowerShell 5.x is mapped to Invoke-WebRequest and does not accept POSIX-style flags like -X POST -d ...)

PowerShell checks:

docker --version
docker compose version
python --version
git --version
curl.exe --version

If Docker is not running, start Docker Desktop first.

---

1) Open PowerShell in repo root

cd F:\Coding\OpenEnv

Confirm:

ls

You should see sandbox, server, env, evaluate.py.

---

2) Build and start sandbox cluster

docker compose -f "sandbox/docker-compose.yml" up --build -d

Check containers:

docker compose -f "sandbox/docker-compose.yml" ps

---

3) Validate service health endpoints

curl.exe http://localhost:5001/health
curl.exe http://localhost:5002/health
curl.exe http://localhost:5003/health
curl.exe http://localhost:5004/health
curl.exe http://localhost:5005/health
curl.exe http://localhost:6660/health
curl.exe http://localhost:6660/chaos/status

Expected: JSON responses for all endpoints.

---

4) Create Python env and install project deps

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -U pip
pip install -e ".[dev]"

---

5) Start OpenEnv API in sandbox mode

Use a new PowerShell window for the API server:

cd F:\Coding\OpenEnv
.\.venv\Scripts\Activate.ps1
$env:OPENENV_SANDBOX="true"
$env:OPENENV_SANDBOX_CLUSTER_URL="http://localhost"
$env:OPENENV_SANDBOX_CHAOS_URL="http://localhost:6660"
python -m server.app

Keep this running.

---

6) API smoke test in another PowerShell window

cd F:\Coding\OpenEnv
.\.venv\Scripts\Activate.ps1
python sandbox\smoke_test.py --base-url http://localhost --api-url http://localhost:7860

Expected final line: Sandbox smoke test passed.

---

7) Manual step-by-step API checks

7.1 Reset incident

$bodyObj = @{
  mode = "incident"
  difficulty = "easy"
  seed = 0
}
$body = $bodyObj | ConvertTo-Json -Compress
$body

Invoke-RestMethod -Method Post -Uri "http://localhost:7860/reset" `
  -Headers @{ "X-Session-ID" = "demo-session" } `
  -ContentType "application/json" `
  -Body $body

7.1b Reset incident with drill mode

$bodyObj = @{
  mode = "incident"
  difficulty = "hard"
  seed = 0
  drill_mode = $true
  drill_seed = 7
}
$body = $bodyObj | ConvertTo-Json -Compress
$body

Invoke-RestMethod -Method Post -Uri "http://localhost:7860/reset" `
  -Headers @{ "X-Session-ID" = "drill-session" } `
  -ContentType "application/json" `
  -Body $body

7.2 Check monitoring

$bodyObj = @{
  action = @{
    action_type = "check_monitoring"
  }
}
$body = $bodyObj | ConvertTo-Json -Compress
$body

Invoke-RestMethod -Method Post -Uri "http://localhost:7860/step" `
  -Headers @{ "X-Session-ID" = "demo-session" } `
  -ContentType "application/json" `
  -Body $body

7.3 Fetch logs

$bodyObj = @{
  action = @{
    action_type = "fetch_logs"
    service_name = "auth"
    time_range = "last_5m"
  }
}
$body = $bodyObj | ConvertTo-Json -Compress
$body

Invoke-RestMethod -Method Post -Uri "http://localhost:7860/step" `
  -Headers @{ "X-Session-ID" = "demo-session" } `
  -ContentType "application/json" `
  -Body $body

7.4 Apply + verify fix

$applyBodyObj = @{
  action = @{
    action_type = "apply_fix"
    service_name = "auth"
    fix_type = "restart_service"
  }
}
$applyBody = $applyBodyObj | ConvertTo-Json -Compress
$applyBody

Invoke-RestMethod -Method Post -Uri "http://localhost:7860/step" `
  -Headers @{ "X-Session-ID" = "demo-session" } `
  -ContentType "application/json" `
  -Body $applyBody

$verifyBodyObj = @{
  action = @{
    action_type = "verify_fix"
    service_name = "auth"
  }
}
$verifyBody = $verifyBodyObj | ConvertTo-Json -Compress
$verifyBody

Invoke-RestMethod -Method Post -Uri "http://localhost:7860/step" `
  -Headers @{ "X-Session-ID" = "demo-session" } `
  -ContentType "application/json" `
  -Body $verifyBody

7.5 Confirm sandbox payload is attached

In responses, check:

• info.sandbox
• observation.tool_results.sandbox_live

---

8) Sandbox evaluation runs

Use two profiles:

• Smoke check (--episodes-per-difficulty 1) for quick verification.
• Final reporting (--episodes-per-difficulty 5 or 10) for stable metrics.

8.1 Smoke check (fast)

cd F:\Coding\OpenEnv
.\.venv\Scripts\Activate.ps1
python evaluate.py --policy compare --episodes-per-difficulty 1 --sandbox --output-dir artifacts/eval_sandbox

8.2 Final sandbox report (recommended)

python evaluate.py --policy compare --compare-trained-policy trained_checkpoint --checkpoint-dir artifacts/train/trained_adapter --checkpoint-base-model Qwen/Qwen2.5-3B-Instruct --episodes-per-difficulty 5 --sandbox --output-dir artifacts/eval_sandbox

For publication-quality numbers, use --episodes-per-difficulty 10.

trained_checkpoint requires a completed training run that produced artifacts/train/trained_adapter.

8.3 Final transfer benchmark (sim + sandbox in one run)

python evaluate.py --policy compare --compare-trained-policy trained_checkpoint --checkpoint-dir artifacts/train/trained_adapter --checkpoint-base-model Qwen/Qwen2.5-3B-Instruct --episodes-per-difficulty 5 --transfer-report --output-dir artifacts/eval_transfer

8.4 Final transfer benchmark with drill mode

python evaluate.py --policy compare --compare-trained-policy trained_checkpoint --checkpoint-dir artifacts/train/trained_adapter --checkpoint-base-model Qwen/Qwen2.5-3B-Instruct --episodes-per-difficulty 5 --transfer-report --sandbox-drill-mode --sandbox-drill-seed 7 --output-dir artifacts/eval_transfer_drill

8.5 Mentor-friendly plots (easy/medium/hard stage curves)

Run compare with plotting enabled (use 7 for your requested 6-7 stages):

python evaluate.py --policy compare --compare-trained-policy trained_checkpoint --checkpoint-dir artifacts/train/trained_adapter --checkpoint-base-model Qwen/Qwen2.5-3B-Instruct --episodes-per-difficulty 7 --plot --sandbox --output-dir artifacts/eval_sandbox

Generated files:

• artifacts/eval_sandbox/reward_curve_easy.png
• artifacts/eval_sandbox/reward_curve_medium.png
• artifacts/eval_sandbox/reward_curve_hard.png

---

9) Run tests

python -m pytest tests/test_sandbox_env.py -q
python -m pytest tests/test_server_api.py -q

---

10) Shutdown + cleanup

Stop server with Ctrl+C in API window.

Then:

docker compose -f "sandbox/docker-compose.yml" down

Optional full cleanup (remove images/volumes):

docker compose -f "sandbox/docker-compose.yml" down -v --rmi local

---

Troubleshooting

• connection refused on ports 500x/6660: cluster not running -> run compose up.
• sandbox info missing in /step: ensure OPENENV_SANDBOX=true in server window.
• Docker not available: start Docker Desktop and retry.
• Health is down right after reset: expected if failure injected; use apply_fix then verify_fix.
• PowerShell request parsing issues: use Invoke-RestMethod examples in section 7 (recommended over curl/curl.exe on Windows).
• Port already in use (5001-5005, 6660, 7860): stop conflicting processes or change the host port mapping in sandbox/docker-compose.yml.