docs/analysis_and_next_steps.md

OpenSecOpsEnv — Full Audit, MDP Definition & Next Steps

---

1 · Full MDP Definition

This is the formal Markov Decision Process that your environment implements.

State Space S

The hidden state (never given to the agent directly) is:

Variable	Type	Description
true_root_cause	str	"infra_failure" or "cyber_attack"
subtype	str	"memory_leak", "ddos", "data_exfiltration"
affected_services	list[str]	Services that are actually compromised
attack_progress	float ∈ [0,1]	How far the attack/failure has progressed
noise_level	float ∈ [0,1]	Fraction of log/alert signals that are misleading
metrics[svc]	dict	Per-service {cpu, memory, latency, error_rate}
isolated_services	set[str]	Services the agent has isolated
blocked_ips	set[str]	IPs the agent has blocked
step_count	int	Current episode step

The observation (what the agent actually sees) is a stochastic projection of the hidden state:

O = f(S) + noise
  = {alerts, metrics_snapshot, partial_logs, topology, last_action_result, time_step}

---

Action Space A

Discrete (9 actions), parameterized:

Action	Parameters	Class
query_logs	{service}	Investigation
inspect_metrics	{service?}	Investigation
run_security_scan	{target}	Investigation
restart_service	{service}	Mitigation
scale_service	{service, replicas}	Mitigation
block_ip	{ip}	Mitigation
rollback_deployment	{service, version}	Mitigation
isolate_service	{service}	Mitigation
submit_diagnosis	{label}	Terminal

---

Transition Function T(s, a) → s'

State evolves deterministically (seeded RNG) after each action:

1. _drift_metrics() — Brownian drift + scenario-specific escalation:

   • Memory leak: memory += 1.5/step, latency += 10/step per affected svc
   • DDoS: cpu += attack_progress × 5, latency += attack_progress × 100
   • Exfiltration: cpu += attack_progress × 1.5, memory += attack_progress × 0.8

2. _evolve_attack() — attack_progress += 0.08/step (×0.5 if an affected service is isolated)

3. Mitigation side-effects (when correct action taken):

   • restart_service → resets memory to 35%, latency to 80ms
   • block_ip → reduces affected svc cpu/latency, attack_progress −0.2
   • isolate_service → attack_progress −0.35
   • scale_service → cpu −20%, latency −200ms

---

Reward Function R(s, a) → ℝ

Dense per-step reward ∈ [−1.0, +1.0]:

Investigation on affected service     → +0.2
inspect_metrics({}) (global)          → +0.2   (always useful)
run_security_scan on affected + cyber → +0.3
Correct mitigation                    → +0.5
Correct submit_diagnosis              → +1.0

Investigation on NON-affected svc     → −0.05
Ineffective mitigation                → −0.1
Wrong service (not in metrics)        → −0.2
Harmful action (wrong ip/isolate)     → −0.5
Wrong submit_diagnosis                → −1.0

---

Terminal Condition

Episode ends when:

• Agent calls submit_diagnosis (always terminates, reward ±1.0), OR
• step_count >= max_steps (30 / 40 / 50 per task)

---

Grader G(episode) → [0,1]

score = 0.5 × diagnosis_correct
      + 0.3 × action_efficiency
      + 0.2 × investigation_quality

diagnosis_correct  = 1.0 (exact) | 0.5 (category) | 0.0
action_efficiency  = 0.7 × mitigation_recall + 0.3 × step_bonus
  mitigation_recall = |done ∩ correct| / |correct|
  step_bonus        = min(1, ideal_steps / step_count)   ideal = max(n_correct×3, 5)
investigation_quality = |investigated_affected| / |affected|

---

2 · Gap Analysis vs Competition Rubric

Real-world utility (30%) — Current: ~25/30 → Target: 28/30

Issue	Impact	Fix?
Only 3 scenario types (memory_leak, DDoS, exfil)	Medium	Add 4th task type
False-alert signal in HARD task is labeled "FALSE POSITIVE" in the alert text itself	Medium	Remove the self-labeling hint
gateway not treated as a valid investigation target (penalized)	Low-Med	See §3-B

---

Task & grader quality (25%) — Current: ~20/25 → Target: 24/25

Issue	Impact	Fix?
run_security_scan counted as investigation not mitigation — but HARD task lists it in correct_mitigations. Results in the agent never getting credit for scans even when done.	Critical	§3-A
Hard task: affected_services = ["db","auth"], but agent gets no IQ credit for scanning — only for query_logs/inspect_metrics. run_security_scan:db should count toward investigation_quality	High	§3-A
grader step_bonus uses ideal_steps = max(n_correct×3, 5) — for easy task with 1 mitigation this gives ideal=5 but optimal is 4 steps. Slightly penalizes perfect play	Low	§3-C

---

Environment design (20%) — Current: ~17/20 → Target: 19/20

Issue	Impact	Fix?
inspect_metrics({}) gives same +0.2 as targeted inspect. No penalty for already-seen services	Low	Cosmetic
_generate_logs() called twice per step (in _build_observation and in _act_query_logs) — same pool, different random shuffle	Low	§3-D
No cooldown on repeat actions — agent can query_logs:auth 20× for +4.0 reward with no diminishing returns	Medium	§3-E

---

Code quality & spec compliance (15%) — Current: ~13/15 → Target: 15/15

Issue	Impact	Fix?
score in log_end() formatted as :.2f — spec says 2 decimal places, this is fine	None	—
openenv.yaml lists run_security_scan under correct_mitigations for HARD task, but grader never counts it as mitigation	Minor	§3-A
No LICENSE file despite README saying MIT	Low	§3-F
Docker HEALTHCHECK hits /health — ✓ good	—	—
inference.py falls back to heuristic when no LLM key — ✓ good	—	—

---

Creativity & novelty (10%) — Current: ~8/10 → Target: 9/10

Issue	Impact	Fix?
The cache false-alert says "FALSE POSITIVE" in the message text — removes the discovery challenge	Medium	§3-B
Adding a 4th task would add novelty	Medium	§3-G (optional)

---

3 · Prioritized Fixes (High → Low)

A · [CRITICAL] Fix run_security_scan in HARD grader — grader.py + env.py

Problem: run_security_scan:db is listed in correct_mitigations for the hard task (openenv.yaml line 143-144), but INVESTIGATION_ACTIONS in env.py means it's never appended to state.mitigation_actions — only to state.investigation_actions. The grader only counts mitigation_actions toward mitigation_recall. So an agent doing the right scans gets 0 mitigation credit.

Fix in grader.py: Count security scans listed in correct_mitigations as valid — check both mitigation_actions and investigation_actions whose keys appear in correct_mitigations.

Also fix in grader.py: Let run_security_scan:X count toward investigation_quality for service X (already works because investigation_actions includes them, but the set intersection logic uses parts[1] which gives the target correctly — this part is actually fine).

# In grade(), replace mitigation recall calculation:
# Check BOTH lists for items that appear in correct_mitigations
all_taken_actions = set(mitigation_actions) | set(investigation_actions)
n_correct = len(all_taken_actions & set(correct_mitigations))

---

B · [HIGH] Remove self-labeling hint from HARD task alert — task_definitions.py

Problem: The false-alert message says "(FALSE POSITIVE)" in the text. This tells the agent exactly that it's a false alarm, defeating the point of the noise.

Fix in task_definitions.py: Change:

# Before
"message": "Cache memory 82% – possible memory leak (FALSE POSITIVE)"
# After  
"message": "Cache memory 82% – triggering high-memory threshold policy"

---

C · [MEDIUM] Fix step_bonus for easy task — grader.py

Problem: ideal_steps = max(n_correct * 3, 5) gives 5 for easy task (1 correct mitigation × 3 = 3, max with 5 = 5). But the optimal trajectory is 4 steps (inspect, query, restart, submit). An agent doing it perfectly in 4 steps gets step_bonus = 5/4 = 1.25 → clamped to 1.0 — still OK. For medium: 3 × 3 = 9, optimal is 7 — fine. This is actually acceptable, just add a note.

---

D · [MEDIUM] Prevent log double-generation and add repeat-action diminishing returns — env.py

Problem: When query_logs is called, _act_query_logs calls _generate_logs() and then _build_observation also calls _generate_logs() again. Two different shuffles, slightly incoherent. Fix by building the observation first with the same log pool.

Also: An agent can harvest +0.2 per step by repeatedly calling query_logs:auth on the affected service. Add a per-service "already investigated" diminishing return.

Fix in env.py: Maintain a _investigated: dict[str, int] counter. First time: full reward. Second time same service: +0.05. Third+ time: 0.

# In __init__:
self._investigated: dict[str, int] = {}

# In _act_query_logs / _act_inspect_metrics:
count = self._investigated.get(svc, 0)
self._investigated[svc] = count + 1
if svc in self._hidden.affected_services:
    reward = 0.2 if count == 0 else (0.05 if count == 1 else 0.0)

---

E · [LOW] Add LICENSE file — root

Create a LICENSE file with the MIT license text. Required for open-source credibility and referenced by README.

---

F · [LOW / OPTIONAL] Add a 4th task — task_definitions.py + openenv.yaml

A medium_hard task such as a bad deployment rollback scenario would:

• Fill the gap between medium (0.73) and hard (0.795) — scores are surprisingly close
• Add a misconfiguration:bad_config or infra_failure:service_crash scenario using rollback_deployment action (currently never the correct mitigation in any task!)

This is optional but would increase creativity score and make rollback_deployment useful.

---

4 · Files to Edit (Summary Table)

File	Change	Priority
opensecops_env/grader.py	Fix mitigation_recall to include investigation_actions ∩ correct_mitigations	🔴 Critical
opensecops_env/tasks/task_definitions.py	Remove "(FALSE POSITIVE)" from hard task alert text	🟠 High
opensecops_env/env.py	Add diminishing returns for repeat investigation on same service	🟡 Medium
opensecops_env/env.py	Fix double log generation (minor)	🟡 Medium
openenv.yaml	Clarify that run_security_scan mitigations in HARD task are also graded as investigation	🟡 Medium
LICENSE	Add MIT license text	🟢 Low
README.md	Update baseline scores to actual measured values (1.00, 0.73, 0.795)	🟢 Low
opensecops_env/tasks/task_definitions.py	(Optional) Add 4th task	🔵 Optional

---

5 · Should You Fine-tune?

No, you do not need to fine-tune any model. Your environment is the submission, not the model.

The competition evaluates your environment quality — judges will run their own models against it. The inference script produces a baseline score that shows the environment is solvable.

What you should optimize instead:

Concern	What to do
Baseline score too low	Improve the system prompt in inference.py — more explicit strategy hints
Baseline score too high	Make tasks harder (more noise, more red herrings)
Scores not deterministic	Ensure _rng is seeded before EVERY _generate_logs call
Runtime > 20min	MAX_STEPS = 20 already set — keep it; 3 tasks × 20 steps × ~1s/LLM call ≈ 1 min

System prompt improvements to inference.py that would raise the baseline score:

1. Add explicit instruction: "When investigating, always check ALL services in the topology, not just the most obvious one"
2. Add: "The gateway / frontend service is rarely the root cause; look at the backend services it calls"
3. Add: "You MUST call run_security_scan on suspicious services before submitting if you suspect a cyber attack"

---

6 · Deployment Checklist (Before Submitting to HF Spaces)

• Run docker build -t opensecops-env:latest . locally ✓ (user is doing this now)
• Run docker run -p 8000:8000 opensecops-env:latest and verify /health returns 200
• curl -X POST http://localhost:8000/reset -d '{}' -H 'Content-Type: application/json' returns observation
• Set HF_TOKEN and run python inference.py — verify 3× [START]/[STEP]/[END] blocks print
• Push to HF Spaces with correct README.md sdk: docker header
• Set Space secrets: API_BASE_URL, MODEL_NAME, HF_TOKEN
• Ping https://your-space.hf.space/reset with POST — verify 200

HF Spaces README.md header (required at top of HF repo README):

---
title: OpenSecOpsEnv
emoji: 🔐
colorFrom: blue
colorTo: green
sdk: docker
app_port: 8000
tags:
  - openenv
  - reinforcement-learning
  - secops
  - agent-evaluation
---