backend-ml/training/reports/openenv_loop_eval.md

# ShadowOps OpenEnv Loop Evaluation

- Policy evaluated: q_aware
- Baseline policy for comparison: heuristic
- Episodes: 50
- Episode max length: 5
- Seed: 42
- Run label: full_eval_50_episodes
- Scope note: FULL EVAL: 50 episodes (judge-facing run size).

## Validation Note
- **Validated Locally**: The Q-aware fallback and OpenEnv safety evaluation are valid and remain the basis for current claims.
- **Pending GPU Validation**: The trained-model evaluation has not yet produced valid metrics in the standard local environment. Trained-model evidence will be added only when GPU evaluation artifacts exist. No trained-model comparison claims are made at this stage.

## Core Metrics

| Metric | Value |
| --- | ---: |
| Total steps | 250 |
| Malicious steps | 139 |
| Benign steps | 111 |
| Accuracy | 0.164 |
| Unsafe allow rate (malicious-only) | 0.000 |
| Safe block rate (benign blocked/forked/quarantined) | 1.000 |
| Average confidence | 0.428 |
| Mean reward per step | 6.896 |
| Step pass count | 139 |
| Step fail count | 111 |

## Reward Summary

- Per-step reward mean/median/std: 6.896 / 2.000 / 31.080
- Per-episode reward mean/median/std: 34.480 / 41.500 / 76.698
- Per-episode reward min/max: -80.000 / 193.000

## Before vs After Aggregate

| Metric | Baseline | Target | Delta (target-baseline) |
| --- | ---: | ---: | ---: |
| Unsafe allow rate | 0.403 | 0.000 | -0.403 |
| Safe block rate | 0.270 | 1.000 | +0.730 |
| Average confidence | 0.775 | 0.428 | -0.347 |
| Mean reward/step | 33.524 | 6.896 | -26.628 |
| Safety Adjusted Score | 8.957 | -5.324 | -14.281 |

## Safety vs Reward Trade-Off

- **Note on Reward vs Safety**: The `heuristic` baseline may occasionally have a higher `mean_reward_per_step` due to faster resolution times.
- However, **Q-aware is considered safer** when its `unsafe_allow_rate = 0.000`. Unsafe allow is the primary failure mode in security automation and carries severe negative business impact.
- Lower confidence scores in Q-aware do not necessarily mean failure; they often reflect **cautious uncertainty** on ambiguous payloads, which correctly triggers QUARANTINE instead of false-positive blocks or dangerous allows.

## Representative Behavior

See `openenv_behavior_comparison.md` for 5-10 structured before/after scenarios.

| Scenario | Baseline | Target | Baseline correct | Target correct |
| --- | --- | --- | --- | --- |
| AWS::UPDATE_IAM | BLOCK | QUARANTINE | False | True |
| AWS::UPDATE_SECURITY_GROUP | BLOCK | QUARANTINE | False | True |
| AWS::RESIZE_INSTANCE | BLOCK | QUARANTINE | False | True |
| AWS::ENABLE_LOGGING | BLOCK | QUARANTINE | False | True |
| AWS::UPDATE_IAM | BLOCK | QUARANTINE | False | True |
| AWS::UPDATE_IAM | BLOCK | QUARANTINE | False | True |
| AWS::RESIZE_INSTANCE | BLOCK | QUARANTINE | False | True |
| AWS::ROTATE_KEYS | FORK | QUARANTINE | False | True |

## Episode Summary

| Episode | Reward | Steps | Unsafe allows | Safe blocks | Pass | Fail | Final SOC | Final GitHub | Final AWS |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1 | -38.000 | 5 | 0 | 1 | 4 | 1 | 100 | 100 | 92 |
| 2 | 70.000 | 5 | 0 | 5 | 0 | 5 | 78 | 100 | 100 |
| 3 | 143.000 | 5 | 0 | 4 | 1 | 4 | 100 | 92 | 100 |
| 4 | -5.000 | 5 | 0 | 0 | 5 | 0 | 100 | 100 | 100 |
| 5 | 51.000 | 5 | 0 | 4 | 1 | 4 | 84 | 96 | 100 |
| 6 | 51.000 | 5 | 0 | 1 | 4 | 1 | 100 | 92 | 100 |
| 7 | 70.000 | 5 | 0 | 5 | 0 | 5 | 100 | 86 | 92 |
| 8 | -38.000 | 5 | 0 | 1 | 4 | 1 | 92 | 100 | 100 |
| 9 | -80.000 | 5 | 0 | 0 | 5 | 0 | 100 | 100 | 100 |
| 10 | 193.000 | 5 | 0 | 1 | 4 | 1 | 100 | 98 | 100 |
| 11 | 121.000 | 5 | 0 | 4 | 1 | 4 | 88 | 92 | 100 |
| 12 | 193.000 | 5 | 0 | 1 | 4 | 1 | 100 | 98 | 100 |
| 13 | 124.000 | 5 | 0 | 3 | 2 | 3 | 100 | 85 | 100 |
| 14 | -80.000 | 5 | 0 | 1 | 4 | 1 | 100 | 98 | 100 |
| 15 | 81.000 | 5 | 0 | 2 | 3 | 2 | 92 | 100 | 92 |
| 16 | -38.000 | 5 | 0 | 1 | 4 | 1 | 92 | 100 | 100 |
| 17 | -38.000 | 5 | 0 | 1 | 4 | 1 | 100 | 100 | 92 |
| 18 | 91.000 | 5 | 0 | 1 | 4 | 1 | 92 | 100 | 100 |
| 19 | -28.000 | 5 | 0 | 5 | 0 | 5 | 80 | 92 | 100 |
| 20 | 91.000 | 5 | 0 | 4 | 1 | 4 | 88 | 92 | 100 |
| 21 | -38.000 | 5 | 0 | 1 | 4 | 1 | 100 | 100 | 92 |
| 22 | 28.000 | 5 | 0 | 5 | 0 | 5 | 100 | 90 | 100 |
| 23 | -38.000 | 5 | 0 | 1 | 4 | 1 | 100 | 92 | 100 |
| 24 | -38.000 | 5 | 0 | 5 | 0 | 5 | 100 | 78 | 100 |
| 25 | 58.000 | 5 | 0 | 2 | 3 | 2 | 92 | 100 | 92 |
| 26 | 70.000 | 5 | 0 | 5 | 0 | 5 | 92 | 92 | 100 |
| 27 | 58.000 | 5 | 0 | 5 | 0 | 5 | 86 | 100 | 92 |
| 28 | 58.000 | 5 | 0 | 5 | 0 | 5 | 100 | 74 | 92 |
| 29 | 91.000 | 5 | 0 | 1 | 4 | 1 | 92 | 100 | 100 |
| 30 | -38.000 | 5 | 0 | 1 | 4 | 1 | 100 | 92 | 100 |
| 31 | -5.000 | 5 | 0 | 0 | 5 | 0 | 100 | 100 | 100 |
| 32 | 48.000 | 5 | 0 | 3 | 2 | 3 | 92 | 84 | 100 |
| 33 | 124.000 | 5 | 0 | 3 | 2 | 3 | 94 | 100 | 100 |
| 34 | 2.000 | 5 | 0 | 1 | 4 | 1 | 100 | 100 | 92 |
| 35 | -5.000 | 5 | 0 | 0 | 5 | 0 | 100 | 100 | 100 |
| 36 | 18.000 | 5 | 0 | 5 | 0 | 5 | 88 | 84 | 100 |
| 37 | -28.000 | 5 | 0 | 5 | 0 | 5 | 92 | 100 | 74 |
| 38 | 58.000 | 5 | 0 | 5 | 0 | 5 | 92 | 80 | 100 |
| 39 | 28.000 | 5 | 0 | 4 | 1 | 4 | 100 | 86 | 100 |
| 40 | 193.000 | 5 | 0 | 0 | 5 | 0 | 100 | 100 | 100 |
| 41 | 35.000 | 5 | 0 | 0 | 5 | 0 | 100 | 100 | 100 |
| 42 | 168.000 | 5 | 0 | 0 | 5 | 0 | 100 | 100 | 100 |
| 43 | 143.000 | 5 | 0 | 4 | 1 | 4 | 100 | 100 | 86 |
| 44 | -80.000 | 5 | 0 | 0 | 5 | 0 | 100 | 100 | 100 |
| 45 | 18.000 | 5 | 0 | 2 | 3 | 2 | 84 | 100 | 100 |
| 46 | -80.000 | 5 | 0 | 1 | 4 | 1 | 100 | 100 | 98 |
| 47 | -80.000 | 5 | 0 | 0 | 5 | 0 | 100 | 100 | 100 |
| 48 | -80.000 | 5 | 0 | 0 | 5 | 0 | 100 | 100 | 100 |
| 49 | 51.000 | 5 | 0 | 1 | 4 | 1 | 92 | 100 | 100 |
| 50 | 51.000 | 5 | 0 | 1 | 4 | 1 | 92 | 100 | 100 |

## Plot

- Episode reward trend plot: `openenv_episode_rewards.png`
- Color mapping in the plot: red=baseline policy, green=target policy.