Divyansh Agrawal commited on
Commit ·
fedca64
1
Parent(s): c2b19b3
updated grader
Browse files- openenv.yaml +6 -6
openenv.yaml
CHANGED
|
@@ -13,7 +13,7 @@ tasks:
|
|
| 13 |
type: llm
|
| 14 |
prompt_template: |
|
| 15 |
You are grading a Predictive Scaling task (Task-1) in a microservice SRE simulation.
|
| 16 |
-
Score the agent's performance
|
| 17 |
|
| 18 |
Scoring criteria:
|
| 19 |
- Traffic increases linearly; the agent should scale up BEFORE queues overflow (boot delay = 5 ticks).
|
|
@@ -22,7 +22,7 @@ tasks:
|
|
| 22 |
- Reward proactive SCALE_UP actions that precede traffic spikes by 5+ steps.
|
| 23 |
- Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
|
| 24 |
|
| 25 |
-
Return a single float between 0
|
| 26 |
|
| 27 |
- id: task-2
|
| 28 |
difficulty: medium
|
|
@@ -31,7 +31,7 @@ tasks:
|
|
| 31 |
type: llm
|
| 32 |
prompt_template: |
|
| 33 |
You are grading a Fault Tolerance task (Task-2) in a microservice SRE simulation.
|
| 34 |
-
Score the agent's performance
|
| 35 |
|
| 36 |
Scoring criteria:
|
| 37 |
- A node fails randomly; the agent must detect and respond quickly (within a few steps).
|
|
@@ -41,7 +41,7 @@ tasks:
|
|
| 41 |
- Penalize heavily for node-0 (VIP, weight=4) experiencing high latency post-failure.
|
| 42 |
- Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
|
| 43 |
|
| 44 |
-
Return a single float between 0
|
| 45 |
|
| 46 |
- id: task-3
|
| 47 |
difficulty: hard
|
|
@@ -50,7 +50,7 @@ tasks:
|
|
| 50 |
type: llm
|
| 51 |
prompt_template: |
|
| 52 |
You are grading a Stability Under Surge task (Task-3) in a microservice SRE simulation.
|
| 53 |
-
Score the agent's performance
|
| 54 |
|
| 55 |
Scoring criteria:
|
| 56 |
- A major traffic surge hits non-critical nodes; node-0 (Payment Gateway VIP) must be protected.
|
|
@@ -60,4 +60,4 @@ tasks:
|
|
| 60 |
- REROUTE_TRAFFIC decays 50% per tick; reward persistent re-application to redirect load.
|
| 61 |
- Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
|
| 62 |
|
| 63 |
-
Return a single float between 0
|
|
|
|
| 13 |
type: llm
|
| 14 |
prompt_template: |
|
| 15 |
You are grading a Predictive Scaling task (Task-1) in a microservice SRE simulation.
|
| 16 |
+
Score the agent's performance strictly between 0 and 1 (exclusive; do not return exactly 0 or 1) based on the episode trace.
|
| 17 |
|
| 18 |
Scoring criteria:
|
| 19 |
- Traffic increases linearly; the agent should scale up BEFORE queues overflow (boot delay = 5 ticks).
|
|
|
|
| 22 |
- Reward proactive SCALE_UP actions that precede traffic spikes by 5+ steps.
|
| 23 |
- Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
|
| 24 |
|
| 25 |
+
Return a single float strictly between 0 and 1 (exclusive; do not return exactly 0 or 1).
|
| 26 |
|
| 27 |
- id: task-2
|
| 28 |
difficulty: medium
|
|
|
|
| 31 |
type: llm
|
| 32 |
prompt_template: |
|
| 33 |
You are grading a Fault Tolerance task (Task-2) in a microservice SRE simulation.
|
| 34 |
+
Score the agent's performance strictly between 0 and 1 (exclusive; do not return exactly 0 or 1) based on the episode trace.
|
| 35 |
|
| 36 |
Scoring criteria:
|
| 37 |
- A node fails randomly; the agent must detect and respond quickly (within a few steps).
|
|
|
|
| 41 |
- Penalize heavily for node-0 (VIP, weight=4) experiencing high latency post-failure.
|
| 42 |
- Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
|
| 43 |
|
| 44 |
+
Return a single float strictly between 0 and 1 (exclusive; do not return exactly 0 or 1).
|
| 45 |
|
| 46 |
- id: task-3
|
| 47 |
difficulty: hard
|
|
|
|
| 50 |
type: llm
|
| 51 |
prompt_template: |
|
| 52 |
You are grading a Stability Under Surge task (Task-3) in a microservice SRE simulation.
|
| 53 |
+
Score the agent's performance strictly between 0 and 1 (exclusive; do not return exactly 0 or 1) based on the episode trace.
|
| 54 |
|
| 55 |
Scoring criteria:
|
| 56 |
- A major traffic surge hits non-critical nodes; node-0 (Payment Gateway VIP) must be protected.
|
|
|
|
| 60 |
- REROUTE_TRAFFIC decays 50% per tick; reward persistent re-application to redirect load.
|
| 61 |
- Ignore sensor noise readings (0 or -1 values) when evaluating decisions.
|
| 62 |
|
| 63 |
+
Return a single float strictly between 0 and 1 (exclusive; do not return exactly 0 or 1).
|