upload sim_eval_phase_b/eval_summary.md
Browse files
sim_eval_phase_b/eval_summary.md
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Sim eval — ep-0950_sr-0.527 (Phase B confirm (50 ep/task, 3 seeds))
|
| 2 |
+
|
| 3 |
+
- **phase**: `Phase B confirm (50 ep/task, 3 seeds)`
|
| 4 |
+
- **mean_success_rate**: `0.506` ± `0.007571877794400371`
|
| 5 |
+
- **n_test**: 500
|
| 6 |
+
- **n_test_per_task**: 50
|
| 7 |
+
- **test_start_seed**: 1000
|
| 8 |
+
- **num_exp**: 3
|
| 9 |
+
|
| 10 |
+
## Per-task SR
|
| 11 |
+
|
| 12 |
+
| Task | SR |
|
| 13 |
+
|------|-----|
|
| 14 |
+
| `KITCHEN_SCENE4_put_the_black_bowl_in_the_bottom_drawer_of_th` | 94.7% |
|
| 15 |
+
| `KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it` | 88.7% |
|
| 16 |
+
| `KITCHEN_SCENE6_put_the_yellow_and_white_mug_in_the_microwave` | 74.0% |
|
| 17 |
+
| `STUDY_SCENE1_pick_up_the_book_and_place_it_in_the_back_compa` | 66.0% |
|
| 18 |
+
| `LIVING_ROOM_SCENE2_put_both_the_cream_cheese_box_and_the_but` | 56.7% |
|
| 19 |
+
| `LIVING_ROOM_SCENE5_put_the_white_mug_on_the_left_plate_and_p` | 48.0% |
|
| 20 |
+
| `KITCHEN_SCENE8_put_both_moka_pots_on_the_stove` | 43.3% |
|
| 21 |
+
| `LIVING_ROOM_SCENE6_put_the_white_mug_on_the_plate_and_put_th` | 34.0% |
|
| 22 |
+
| `LIVING_ROOM_SCENE2_put_both_the_alphabet_soup_and_the_tomato` | 0.7% |
|
| 23 |
+
| `LIVING_ROOM_SCENE1_put_both_the_alphabet_soup_and_the_cream_` | 0.0% |
|