SFT eval on 22-task held-out split — fill in leaderboard

Job 69ed97e5d2c8bd8662bcf2ad ran eval_lora.py over both adapters on
the eval split (L40S, ~30 min). Parsed the raw stdout into per-adapter
results.json files matching the format the dashboard already expects;
both rows now populate from disk instead of rendering '—'.

Numbers: 4K SFT 0.006 avg / 0% success; 8K SFT 0.011 avg / 0% success.
~6-11x lift over the vanilla student baseline (0.001) but every episode
still bottoms at the env's 0.005 reward floor — the SFT'd model produces
parseable code that doesn't mutate files in ways the grader rewards.
The next step is GRPO continuation against the env's reward signal.

Also adds a parallel 'Eval artifacts' table under the SFT training
artifacts table on both README and the /dashboard, linking the eval
job page + raw_logs.txt for provenance.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

README.md

@@ -67,8 +67,18 @@ logged on HF Jobs L40S, ~$0.50–0.80 each. The 8K run is online at
 | **MiniMaxAI/MiniMax-M2.1** (frontier baseline) | 0.390 | 41% | 0.293 | 0.445 | 0.485 |
 | **moonshotai/Kimi-K2.5** (teacher) | 0.481 | 52% | 0.370 | 0.472 | 0.673 |
 | **Qwen/Qwen2.5-Coder-3B-Instruct** (student baseline) | **0.001** | 0% | 0.001 | 0.001 | 0.001 |
-| **Qwen3-Coder-3B + LoRA SFT (4K)** *(eval pending)* | *coming* | *coming* | – | – | – |
-| **Qwen3-Coder-3B + LoRA SFT (8K)** *(eval pending)* | *coming* | *coming* | – | – | – |
+| **Qwen2.5-Coder-3B + LoRA SFT (4K)** | 0.006 | 0% | 0.007 | 0.005 | 0.005 |
+| **Qwen2.5-Coder-3B + LoRA SFT (8K)** | 0.011 | 0% | 0.018 | 0.005 | 0.005 |
+
+> **Reading the SFT rows.** Both adapters lift the vanilla baseline ~6–11×
+> on the eval set, but every episode still bottoms out at the env's reward
+> floor (0.005) — the model produces *parseable* code but it doesn't mutate
+> the source file in ways the grader rewards. The SFT loss is well-
+> converged (0.19 on the training distribution), so the gap is a
+> generalization-from-Kimi-trajectories problem, not an under-training one.
+> The *next* step — GRPO continuation directly against the env's reward
+> signal — is what's expected to close this. See [`train_grpo.py`](train_grpo.py)
+> and the rollout-format note in [`edits.md`](edits.md) Phase 13.
 
 Reproduce any row:
 
@@ -122,6 +132,17 @@ Re-parse any HF Job's stdout into clean metrics + a loss curve PNG with
 — takes a `--job-id` and emits `training_metrics.jsonl`, `summary.json`,
 and `sft_loss_curve.png`. Both runs above were generated this way.
 
+**Eval artifacts — both SFT adapters scored against the 22-task held-out split:**
+
+| Run | Eval results.json | Raw stdout log | HF Job page |
+|---|---|---|---|
+| 4K context | [results.json](runs/sft_eval_v2/bpHigh_qwen3b-office-sft-kimi/results.json) | [raw_logs.txt](runs/sft_eval_v2/raw_logs.txt) | [Job 69ed97e5…2ad](https://huggingface.co/jobs/bpHigh/69ed97e5d2c8bd8662bcf2ad) |
+| 8K context | [results.json](runs/sft_eval_v2/bpHigh_qwen3b-office-sft-kimi-long/results.json) | [raw_logs.txt](runs/sft_eval_v2/raw_logs.txt) | [Job 69ed97e5…2ad](https://huggingface.co/jobs/bpHigh/69ed97e5d2c8bd8662bcf2ad) |
+
+Both adapters were evaluated in a single HF Jobs run (L40S, ~30 min, ~$1)
+via [`eval_lora.py --adapters A,B`](eval_lora.py) — the base model loads
+once and each adapter is detached/reattached without reloading.
+
 ---
 
 ## Task inventory (119 total)

runs/sft_eval_v2/bpHigh_qwen3b-office-sft-kimi-long/results.json

@@ -0,0 +1,245 @@
+{
+  "model": "bpHigh/qwen3b-office-sft-kimi-long",
+  "split": "eval",
+  "family": "all",
+  "n_tasks": 22,
+  "avg_score": 0.0107,
+  "success_rate": 0.0,
+  "total_elapsed_s": 5257.8,
+  "by_family": {
+    "docx": {
+      "n": 4,
+      "avg": 0.005
+    },
+    "pptx": {
+      "n": 8,
+      "avg": 0.005
+    },
+    "xlsx": {
+      "n": 10,
+      "avg": 0.0175
+    }
+  },
+  "results": [
+    {
+      "task_id": "osworld_0a0faba3",
+      "family": "docx",
+      "primary_tag": "check_tabstops",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 136.3
+    },
+    {
+      "task_id": "osworld_0b17a146",
+      "family": "docx",
+      "primary_tag": "compare_docx_files",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 433.3
+    },
+    {
+      "task_id": "osworld_66399b0d",
+      "family": "docx",
+      "primary_tag": "compare_docx_tables",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 163.6
+    },
+    {
+      "task_id": "osworld_0810415c",
+      "family": "docx",
+      "primary_tag": "compare_line_spacing",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 126.1
+    },
+    {
+      "task_id": "pptarena_case_60_fix_text_placement",
+      "family": "pptx",
+      "primary_tag": "Alignment, Distribution & Z-order",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 439.5
+    },
+    {
+      "task_id": "pptarena_case_7_update_quarter_two_data_b",
+      "family": "pptx",
+      "primary_tag": "Charts",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 221.5
+    },
+    {
+      "task_id": "pptarena_case_32_arrange_image_and_text",
+      "family": "pptx",
+      "primary_tag": "Images & Pictures",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 261.6
+    },
+    {
+      "task_id": "pptarena_case_36_add_speaker_notes",
+      "family": "pptx",
+      "primary_tag": "Slide/Section Management & Footers",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 241.1
+    },
+    {
+      "task_id": "pptarena_case_49_normalize_thousand_separators",
+      "family": "pptx",
+      "primary_tag": "Tables",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 271.8
+    },
+    {
+      "task_id": "pptarena_case_35_structural_fix",
+      "family": "pptx",
+      "primary_tag": "Text & Typography",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 510.1
+    },
+    {
+      "task_id": "pptarena_case_40_hindu_center_titles",
+      "family": "pptx",
+      "primary_tag": "Text & Typography",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 244.0
+    },
+    {
+      "task_id": "pptarena_case_26_match_slide_colors_to_theme",
+      "family": "pptx",
+      "primary_tag": "Theme & Background",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 501.9
+    },
+    {
+      "task_id": "finch_10",
+      "family": "xlsx",
+      "primary_tag": "Calculation",
+      "split": "eval",
+      "score": 0.03,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 200.3
+    },
+    {
+      "task_id": "finch_35",
+      "family": "xlsx",
+      "primary_tag": "Calculation",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 155.6
+    },
+    {
+      "task_id": "finch_38",
+      "family": "xlsx",
+      "primary_tag": "Calculation",
+      "split": "eval",
+      "score": 0.03,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 154.6
+    },
+    {
+      "task_id": "finch_112",
+      "family": "xlsx",
+      "primary_tag": "Cross-sheet/file Retrieval",
+      "split": "eval",
+      "score": 0.03,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 116.6
+    },
+    {
+      "task_id": "finch_154",
+      "family": "xlsx",
+      "primary_tag": "Data Entry / Import",
+      "split": "eval",
+      "score": 0.03,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 156.3
+    },
+    {
+      "task_id": "finch_14",
+      "family": "xlsx",
+      "primary_tag": "Financial Modeling",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 163.4
+    },
+    {
+      "task_id": "finch_168",
+      "family": "xlsx",
+      "primary_tag": "Structuring / Formatting",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 238.0
+    },
+    {
+      "task_id": "finch_59",
+      "family": "xlsx",
+      "primary_tag": "Structuring / Formatting",
+      "split": "eval",
+      "score": 0.03,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 111.0
+    },
+    {
+      "task_id": "finch_122",
+      "family": "xlsx",
+      "primary_tag": "Summary / Visualization",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 226.1
+    },
+    {
+      "task_id": "finch_158",
+      "family": "xlsx",
+      "primary_tag": "Validation / Review",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 185.1
+    }
+  ]
+}

runs/sft_eval_v2/bpHigh_qwen3b-office-sft-kimi/results.json

@@ -0,0 +1,245 @@
+{
+  "model": "bpHigh/qwen3b-office-sft-kimi",
+  "split": "eval",
+  "family": "all",
+  "n_tasks": 22,
+  "avg_score": 0.0061,
+  "success_rate": 0.0,
+  "total_elapsed_s": 5901.1,
+  "by_family": {
+    "docx": {
+      "n": 4,
+      "avg": 0.005
+    },
+    "pptx": {
+      "n": 8,
+      "avg": 0.005
+    },
+    "xlsx": {
+      "n": 10,
+      "avg": 0.0075
+    }
+  },
+  "results": [
+    {
+      "task_id": "osworld_0a0faba3",
+      "family": "docx",
+      "primary_tag": "check_tabstops",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 315.9
+    },
+    {
+      "task_id": "osworld_0b17a146",
+      "family": "docx",
+      "primary_tag": "compare_docx_files",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 202.0
+    },
+    {
+      "task_id": "osworld_66399b0d",
+      "family": "docx",
+      "primary_tag": "compare_docx_tables",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 167.1
+    },
+    {
+      "task_id": "osworld_0810415c",
+      "family": "docx",
+      "primary_tag": "compare_line_spacing",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 212.0
+    },
+    {
+      "task_id": "pptarena_case_60_fix_text_placement",
+      "family": "pptx",
+      "primary_tag": "Alignment, Distribution & Z-order",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 264.9
+    },
+    {
+      "task_id": "pptarena_case_7_update_quarter_two_data_b",
+      "family": "pptx",
+      "primary_tag": "Charts",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 296.3
+    },
+    {
+      "task_id": "pptarena_case_32_arrange_image_and_text",
+      "family": "pptx",
+      "primary_tag": "Images & Pictures",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 343.5
+    },
+    {
+      "task_id": "pptarena_case_36_add_speaker_notes",
+      "family": "pptx",
+      "primary_tag": "Slide/Section Management & Footers",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 221.6
+    },
+    {
+      "task_id": "pptarena_case_49_normalize_thousand_separators",
+      "family": "pptx",
+      "primary_tag": "Tables",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 228.5
+    },
+    {
+      "task_id": "pptarena_case_35_structural_fix",
+      "family": "pptx",
+      "primary_tag": "Text & Typography",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 327.4
+    },
+    {
+      "task_id": "pptarena_case_40_hindu_center_titles",
+      "family": "pptx",
+      "primary_tag": "Text & Typography",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 388.8
+    },
+    {
+      "task_id": "pptarena_case_26_match_slide_colors_to_theme",
+      "family": "pptx",
+      "primary_tag": "Theme & Background",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 531.2
+    },
+    {
+      "task_id": "finch_10",
+      "family": "xlsx",
+      "primary_tag": "Calculation",
+      "split": "eval",
+      "score": 0.03,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 234.0
+    },
+    {
+      "task_id": "finch_35",
+      "family": "xlsx",
+      "primary_tag": "Calculation",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 271.9
+    },
+    {
+      "task_id": "finch_38",
+      "family": "xlsx",
+      "primary_tag": "Calculation",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 203.0
+    },
+    {
+      "task_id": "finch_112",
+      "family": "xlsx",
+      "primary_tag": "Cross-sheet/file Retrieval",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 268.3
+    },
+    {
+      "task_id": "finch_154",
+      "family": "xlsx",
+      "primary_tag": "Data Entry / Import",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 159.2
+    },
+    {
+      "task_id": "finch_14",
+      "family": "xlsx",
+      "primary_tag": "Financial Modeling",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 199.8
+    },
+    {
+      "task_id": "finch_168",
+      "family": "xlsx",
+      "primary_tag": "Structuring / Formatting",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 224.1
+    },
+    {
+      "task_id": "finch_59",
+      "family": "xlsx",
+      "primary_tag": "Structuring / Formatting",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 227.5
+    },
+    {
+      "task_id": "finch_122",
+      "family": "xlsx",
+      "primary_tag": "Summary / Visualization",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 327.6
+    },
+    {
+      "task_id": "finch_158",
+      "family": "xlsx",
+      "primary_tag": "Validation / Review",
+      "split": "eval",
+      "score": 0.005,
+      "success": false,
+      "steps": 15,
+      "elapsed_s": 286.5
+    }
+  ]
+}

runs/sft_eval_v2/cross_summary.json

@@ -0,0 +1,38 @@
+{
+  "bpHigh_qwen3b-office-sft-kimi": {
+    "avg_score": 0.0061,
+    "success_rate": 0.0,
+    "by_family": {
+      "docx": {
+        "n": 4,
+        "avg": 0.005
+      },
+      "pptx": {
+        "n": 8,
+        "avg": 0.005
+      },
+      "xlsx": {
+        "n": 10,
+        "avg": 0.0075
+      }
+    }
+  },
+  "bpHigh_qwen3b-office-sft-kimi-long": {
+    "avg_score": 0.0107,
+    "success_rate": 0.0,
+    "by_family": {
+      "docx": {
+        "n": 4,
+        "avg": 0.005
+      },
+      "pptx": {
+        "n": 8,
+        "avg": 0.005
+      },
+      "xlsx": {
+        "n": 10,
+        "avg": 0.0175
+      }
+    }
+  }
+}

server/app.py

@@ -696,6 +696,22 @@ def build_dashboard() -> gr.Blocks:
             "[`data_pipeline/analyze_sft_logs.py`](https://github.com/bp-high/openenv_financial_task_env/blob/main/data_pipeline/analyze_sft_logs.py) "
             "(works on any HF Job ID, not just these two)."
         )
+        gr.Markdown(
+            "**Eval artifacts** — both adapters scored on the 22-task held-out split:\n\n"
+            "| Run | Eval results.json | Raw stdout log | HF Job page |\n"
+            "|---|---|---|---|\n"
+            "| 4K context | "
+            "[results.json](https://raw.githubusercontent.com/bp-high/openenv_financial_task_env/main/runs/sft_eval_v2/bpHigh_qwen3b-office-sft-kimi/results.json) | "
+            "[raw_logs.txt](https://raw.githubusercontent.com/bp-high/openenv_financial_task_env/main/runs/sft_eval_v2/raw_logs.txt) | "
+            "[Job 69ed97e5...2ad](https://huggingface.co/jobs/bpHigh/69ed97e5d2c8bd8662bcf2ad) |\n"
+            "| 8K context | "
+            "[results.json](https://raw.githubusercontent.com/bp-high/openenv_financial_task_env/main/runs/sft_eval_v2/bpHigh_qwen3b-office-sft-kimi-long/results.json) | "
+            "[raw_logs.txt](https://raw.githubusercontent.com/bp-high/openenv_financial_task_env/main/runs/sft_eval_v2/raw_logs.txt) | "
+            "[Job 69ed97e5...2ad](https://huggingface.co/jobs/bpHigh/69ed97e5d2c8bd8662bcf2ad) |\n\n"
+            "Both adapters were evaluated in a single HF Jobs run (L40S, ~30 min, ~$1) "
+            "via [`eval_lora.py`](https://github.com/bp-high/openenv_financial_task_env/blob/main/eval_lora.py) "
+            "with `--adapters A,B` so the base model loads once."
+        )
 
         # ---- Plot ----
         gr.Markdown("## 4K vs 8K context length ablation")