Spaces:
Sleeping
Sleeping
I766446 commited on
Commit ·
71b519d
1
Parent(s): 1d3b3a7
simplify tasks for validator
Browse files- openenv.yaml +0 -30
openenv.yaml
CHANGED
|
@@ -142,43 +142,13 @@ reward:
|
|
| 142 |
# ---------------------------------------------------------------------------
|
| 143 |
tasks:
|
| 144 |
- id: easy
|
| 145 |
-
difficulty: easy
|
| 146 |
-
description: >
|
| 147 |
-
Clear-cut claims with overwhelming contradicting evidence from authoritative
|
| 148 |
-
sources (WHO, CDC, NASA, official government). Decision should be
|
| 149 |
-
straightforward for a competent agent.
|
| 150 |
-
dataset: tasks/easy/scenarios.json
|
| 151 |
-
n_scenarios: 5
|
| 152 |
grader: tasks.easy.grader.grade
|
| 153 |
-
weights: { decision: 0.50, evidence: 0.25, reasoning: 0.15, efficiency: 0.10 }
|
| 154 |
-
ideal_max_steps: 8
|
| 155 |
-
ground_truth_distribution: { allow: 1, flag: 1, remove: 3 }
|
| 156 |
|
| 157 |
- id: medium
|
| 158 |
-
difficulty: medium
|
| 159 |
-
description: >
|
| 160 |
-
Partially true or misleading content. Mixed evidence with some supporting
|
| 161 |
-
and some contradicting sources. Requires careful claim-by-claim verification
|
| 162 |
-
and evidence weighing.
|
| 163 |
-
dataset: tasks/medium/scenarios.json
|
| 164 |
-
n_scenarios: 5
|
| 165 |
grader: tasks.medium.grader.grade
|
| 166 |
-
weights: { decision: 0.35, evidence: 0.35, reasoning: 0.20, efficiency: 0.10 }
|
| 167 |
-
ideal_max_steps: 12
|
| 168 |
-
ground_truth_distribution: { allow: 1, flag: 4, remove: 0 }
|
| 169 |
|
| 170 |
- id: hard
|
| 171 |
-
difficulty: hard
|
| 172 |
-
description: >
|
| 173 |
-
Conflicting evidence from credible sources; genuinely contested ground truth.
|
| 174 |
-
Requires nuanced multi-claim reasoning, evidence reliability weighting, and
|
| 175 |
-
tolerance for ambiguity. All four scoring dimensions count equally.
|
| 176 |
-
dataset: tasks/hard/scenarios.json
|
| 177 |
-
n_scenarios: 5
|
| 178 |
grader: tasks.hard.grader.grade
|
| 179 |
-
weights: { decision: 0.30, evidence: 0.30, reasoning: 0.30, efficiency: 0.10 }
|
| 180 |
-
ideal_max_steps: 18
|
| 181 |
-
ground_truth_distribution: { allow: 0, flag: 5, remove: 0 }
|
| 182 |
|
| 183 |
# ---------------------------------------------------------------------------
|
| 184 |
# Grader
|
|
|
|
| 142 |
# ---------------------------------------------------------------------------
|
| 143 |
tasks:
|
| 144 |
- id: easy
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
grader: tasks.easy.grader.grade
|
|
|
|
|
|
|
|
|
|
| 146 |
|
| 147 |
- id: medium
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 148 |
grader: tasks.medium.grader.grade
|
|
|
|
|
|
|
|
|
|
| 149 |
|
| 150 |
- id: hard
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
grader: tasks.hard.grader.grade
|
|
|
|
|
|
|
|
|
|
| 152 |
|
| 153 |
# ---------------------------------------------------------------------------
|
| 154 |
# Grader
|