Divyansh Agrawal commited on
Commit
fedca64
·
1 Parent(s): c2b19b3

updated grader

Browse files
Files changed (1) hide show
  1. openenv.yaml +6 -6
openenv.yaml CHANGED
@@ -13,7 +13,7 @@ tasks:
13
  type: llm
14
  prompt_template: |
15
  You are grading a Predictive Scaling task (Task-1) in a microservice SRE simulation.
16
- Score the agent's performance from 0.0 (worst) to 1.0 (best) based on the episode trace.
17
 
18
  Scoring criteria:
19
  - Traffic increases linearly; the agent should scale up BEFORE queues overflow (boot delay = 5 ticks).
@@ -22,7 +22,7 @@ tasks:
22
  - Reward proactive SCALE_UP actions that precede traffic spikes by 5+ steps.
23
  - Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
24
 
25
- Return a single float between 0.0 and 1.0.
26
 
27
  - id: task-2
28
  difficulty: medium
@@ -31,7 +31,7 @@ tasks:
31
  type: llm
32
  prompt_template: |
33
  You are grading a Fault Tolerance task (Task-2) in a microservice SRE simulation.
34
- Score the agent's performance from 0.0 (worst) to 1.0 (best) based on the episode trace.
35
 
36
  Scoring criteria:
37
  - A node fails randomly; the agent must detect and respond quickly (within a few steps).
@@ -41,7 +41,7 @@ tasks:
41
  - Penalize heavily for node-0 (VIP, weight=4) experiencing high latency post-failure.
42
  - Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
43
 
44
- Return a single float between 0.0 and 1.0.
45
 
46
  - id: task-3
47
  difficulty: hard
@@ -50,7 +50,7 @@ tasks:
50
  type: llm
51
  prompt_template: |
52
  You are grading a Stability Under Surge task (Task-3) in a microservice SRE simulation.
53
- Score the agent's performance from 0.0 (worst) to 1.0 (best) based on the episode trace.
54
 
55
  Scoring criteria:
56
  - A major traffic surge hits non-critical nodes; node-0 (Payment Gateway VIP) must be protected.
@@ -60,4 +60,4 @@ tasks:
60
  - REROUTE_TRAFFIC decays 50% per tick; reward persistent re-application to redirect load.
61
  - Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
62
 
63
- Return a single float between 0.0 and 1.0.
 
13
  type: llm
14
  prompt_template: |
15
  You are grading a Predictive Scaling task (Task-1) in a microservice SRE simulation.
16
+ Score the agent's performance strictly between 0 and 1 (exclusive; do not return exactly 0 or 1) based on the episode trace.
17
 
18
  Scoring criteria:
19
  - Traffic increases linearly; the agent should scale up BEFORE queues overflow (boot delay = 5 ticks).
 
22
  - Reward proactive SCALE_UP actions that precede traffic spikes by 5+ steps.
23
  - Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
24
 
25
+ Return a single float strictly between 0 and 1 (exclusive; do not return exactly 0 or 1).
26
 
27
  - id: task-2
28
  difficulty: medium
 
31
  type: llm
32
  prompt_template: |
33
  You are grading a Fault Tolerance task (Task-2) in a microservice SRE simulation.
34
+ Score the agent's performance strictly between 0 and 1 (exclusive; do not return exactly 0 or 1) based on the episode trace.
35
 
36
  Scoring criteria:
37
  - A node fails randomly; the agent must detect and respond quickly (within a few steps).
 
41
  - Penalize heavily for node-0 (VIP, weight=4) experiencing high latency post-failure.
42
  - Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
43
 
44
+ Return a single float strictly between 0 and 1 (exclusive; do not return exactly 0 or 1).
45
 
46
  - id: task-3
47
  difficulty: hard
 
50
  type: llm
51
  prompt_template: |
52
  You are grading a Stability Under Surge task (Task-3) in a microservice SRE simulation.
53
+ Score the agent's performance strictly between 0 and 1 (exclusive; do not return exactly 0 or 1) based on the episode trace.
54
 
55
  Scoring criteria:
56
  - A major traffic surge hits non-critical nodes; node-0 (Payment Gateway VIP) must be protected.
 
60
  - REROUTE_TRAFFIC decays 50% per tick; reward persistent re-application to redirect load.
61
  - Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
62
 
63
+ Return a single float strictly between 0 and 1 (exclusive; do not return exactly 0 or 1).