# Benchmark Integration Progress

## Status: Batches 1-5 Complete

## Batch Plan

### Batch 1 (Highest Priority -- Easy HF, High Influence)
| Benchmark | Slug | Status | HF Dataset | View Type |
|-----------|------|--------|------------|-----------|
| MBPP+ | `mbppplus` | Done | `evalplus/mbppplus` | Simple |
| ClassEval | `classeval` | Done | `FudanSELab/ClassEval` | Simple |
| LiveCodeBench | `livecodebench` | Done | `livecodebench/code_generation_lite` | Simple |
| DebugBench | `debugbench` | Done | `Rtian/DebugBench` | Before/After |
| HumanEval-X | `humanevalx` | Done | `THUDM/humaneval-x` | Multi-language |

**Refactoring done:** Multi-language syntax highlighting via `get_lexer_by_name()`. Before/after code diff view. Multi-language tab view.

### Batch 2
| Benchmark | Slug | Status | HF Dataset | View Type |
|-----------|------|--------|------------|-----------|
| SWE-bench Lite | `swebenchlite` | Done | `princeton-nlp/SWE-bench_Lite` | Diff |
| CodeContests | `codecontests` | Done | `deepmind/code_contests` | Multi-solution |
| APPS | `apps` | Done | `codeparrot/apps` | Multi-solution / Simple |
| CanItEdit | `canitedit` | Done | `nuprl/CanItEdit` | Before/After |
| MBPP | `mbpp` | Done | `google-research-datasets/mbpp` | Simple |

**New views:** Unified diff view for SWE-bench patches. Multi-solution view extended to show language labels for CodeContests.

### Batch 3
| Benchmark | Slug | Status | HF Dataset | View Type |
|-----------|------|--------|------------|-----------|
| SAFIM | `safim` | Done | `gonglinyuan/safim` | Fill-in-the-Middle |
| BigVul | `bigvul` | Done | `bstee615/bigvul` | Vulnerability |
| DiverseVul | `diversevul` | Done | `claudios/DiverseVul` | Vulnerability |
| PrimeVul | `primevul` | Done | `starsofchance/PrimeVul` | Vulnerability |
| CodeEditorBench | `codeeditorbench` | Done | `m-a-p/CodeEditorBench` | Before/After |

**New views:** Fill-in-the-Middle view showing code with [HOLE] marker and ground truth. Vulnerability view with CWE badges and vulnerable/patched code comparison.

### Batch 4
| Benchmark | Slug | Status | HF Dataset | View Type |
|-----------|------|--------|------------|-----------|
| SWE-bench Verified | `swebenchverified` | Done | `princeton-nlp/SWE-bench_Verified` | Diff |
| CodeSearchNet | `codesearchnet` | Done | `code-search-net/code_search_net` | Simple |
| Devign | `devign` | Done | `google/code_x_glue_cc_defect_detection` | Vulnerability |

### Dropped from original plan
| Benchmark | Reason |
|-----------|--------|
| DS-1000 | Complex library-specific format, limited visualization value |
| RepoBench | Repo-level context too complex for per-problem viewing |
| MultiPL-E | 22 languages but same problems as HumanEval/MBPP already covered |
| McEval | Very large (40 languages), complex format |
| xCodeEval | Very large (25M rows), 7 tasks, too complex |
| CrossVul | Similar to DiverseVul/BigVul, diminishing returns |

### Batch 5
| Benchmark | Slug | Status | HF Dataset | View Type |
|-----------|------|--------|------------|-----------|
| BigCodeBench | `bigcodebench` | Done | `bigcode/bigcodebench` | Simple |
| HumanEvalPack | `humanevalpack` | Done | `bigcode/humanevalpack` | Multi-language + Before/After |
| CodeXGLUE Refinement | `codexgluerefinement` | Done | `google/code_x_glue_cc_code_refinement` | Before/After |
| SWE-bench | `swebenchfull` | Done | `princeton-nlp/SWE-bench` | Diff |
| CommitBench | `commitbench` | Done | `Maxscha/commitbench` | Diff |
| EffiBench | `effibench` | Done | `DONG19/EffiBench` | Simple |

**New views:** Multi-language view with canonical/buggy code toggle (HumanEvalPack). CommitBench reuses diff view. CodeXGLUE Refinement uses before/after Java view.

### Deferred (GitHub-only or complex infrastructure)
CoderEval, NaturalCodeBench, DevEval, RunBugRun, Defects4J, ConDefects, FixEval, TransCoder, AVATAR, TypeEvalPy, VJBench, SVEN, PyTER

## Architecture Decisions

### Multi-language Support
- `highlight_code()` in `app.py` accepts `language` parameter (default: `"python"`)
- Uses `get_lexer_by_name()` from Pygments for automatic lexer selection
- Adapters pass language when calling `_highlight_code(code, language=...)`

### View Types Implemented
1. **BigOBench view** -- multiple solutions with complexity badges
2. **Simple view** -- code + inputs/outputs + test suite (HumanEval+, MBPP+, MBPP, ClassEval, LiveCodeBench, APPS, CodeSearchNet)
3. **CRUXEval view** -- given/predict task selector
4. **DREval view** -- full interactive view with coverage, arrows, ground truth
5. **Before/After view** -- side-by-side buggy/fixed code (DebugBench, CanItEdit, CodeEditorBench)
6. **Multi-language view** -- same problem in multiple languages (HumanEval-X, HumanEvalPack)
7. **Diff view** -- unified diff patch visualization (SWE-bench Lite, SWE-bench Verified, SWE-bench, CommitBench)
8. **Fill-in-the-Middle view** -- prefix + [HOLE] + suffix (SAFIM)
9. **Vulnerability view** -- vulnerable/patched code + CWE labels (BigVul, DiverseVul, PrimeVul, Devign)

### Batch 6 — Long Code Arena (6 project-level tasks)
| Benchmark | Slug | Status | HF Dataset | View Type |
|-----------|------|--------|------------|-----------|
| LCA Library-Based Code Gen | `lca-libcodegen` | Done | `JetBrains-Research/lca-library-based-code-generation` | Simple |
| LCA Project-Level Completion | `lca-codecompletion` | Done | `JetBrains-Research/lca-project-level-code-completion` | Simple |
| LCA Bug Localization | `lca-buglocalization` | Done | `JetBrains-Research/lca-bug-localization` | Diff |
| LCA Commit Message Gen | `lca-commitmsg` | Done | `JetBrains-Research/lca-commit-message-generation` | Diff |
| LCA CI Builds Repair | `lca-cirepair` | Done | `JetBrains-Research/lca-ci-builds-repair` | Diff |
| LCA Module Summarization | `lca-modulesumm` | Done | `JetBrains-Research/lca-module-summarization` | Simple |

**New adapter module:** `adapters/long_code_arena.py` — all 6 Long Code Arena project-level tasks.

### Batch 7 — dpaia & Additional Benchmarks (7 datasets)
| Benchmark | Slug | Status | Source | View Type |
|-----------|------|--------|--------|-----------|
| DPAIA EE-Dataset | `dpaia-ee` | Done | `github.com/dpaia/ee-dataset` (JSON) | Diff (SWE-bench style) |
| Multi-SWE-bench | `multiswebench` | Done | `ByteDance-Seed/Multi-SWE-bench` (JSONL) | Diff |
| SWE-bench Multilingual | `swebenchmultilingual` | Done | `SWE-bench/SWE-bench_Multilingual` | Diff |
| CrossCodeEval | `crosscodeeval` | Done | `Vincentvmt/CrossCodeEval` (JSONL) | Fill-in-the-Middle |
| McEval | `mceval` | Done | `Multilingual-Multimodal-NLP/McEval` | Simple |
| MultiPL-E | `multiple` | Done | `nuprl/MultiPL-E` | Multi-language |
| Defects4J | `defects4j` | Done | `rufimelo/defects4j` | Before/After |

### Dropped from Batch 7
| Benchmark | Reason |
|-----------|--------|
| RepoBench | HF repo has only a deprecated loading script (`repobench-p.py`), no actual data files |

**New adapter module:** `adapters/additional.py` — dpaia EE-Dataset, Multi-SWE-bench, SWE-bench Multilingual, CrossCodeEval, McEval, MultiPL-E, Defects4J.

**Sources:**
- Long Code Arena: https://huggingface.co/collections/JetBrains-Research/long-code-arena (OpenReview: aQoUjxlgNE)
- DPAIA EE-Dataset: https://github.com/dpaia/ee-dataset (Java/Spring SWE-bench-style)
- Multi-SWE-bench: ByteDance multilingual SWE-bench (7 languages, 1632 problems across 40 repos)
- SWE-bench Multilingual: Official SWE-bench multilingual extension (42 repos)
- CrossCodeEval: Cross-file code completion (4 languages, Amazon, 9928 problems)
- McEval: Massively multilingual code evaluation (40 languages)
- MultiPL-E: Multi-language HumanEval/MBPP translation (9 languages loaded)
- Defects4J: Classic Java bug-fix benchmark (467 bugs)
- Arxiv survey reference: https://arxiv.org/abs/2505.08903

## Total Datasets: 41
Base (4): REval, CRUXEval, HumanEval+, BigOBench
Batch 1 (5): MBPP+, ClassEval, LiveCodeBench, DebugBench, HumanEval-X
Batch 2 (5): SWE-bench Lite, CodeContests, APPS, CanItEdit, MBPP
Batch 3 (5): SAFIM, BigVul, DiverseVul, PrimeVul, CodeEditorBench
Batch 4 (3): SWE-bench Verified, CodeSearchNet, Devign
Batch 5 (6): BigCodeBench, HumanEvalPack, CodeXGLUE Refinement, SWE-bench, CommitBench, EffiBench
Batch 6 (6): LCA Library-Based Code Gen, LCA Project-Level Completion, LCA Bug Localization, LCA Commit Message Gen, LCA CI Builds Repair, LCA Module Summarization
Batch 7 (7): DPAIA EE-Dataset, Multi-SWE-bench, SWE-bench Multilingual, CrossCodeEval, McEval, MultiPL-E, Defects4J

## Changelog

- 2026-03-03: Initial benchmark analysis and prioritization complete
- 2026-03-03: Batch 1 complete (MBPP+, ClassEval, LiveCodeBench, DebugBench, HumanEval-X)
- 2026-03-03: Batch 2 complete (SWE-bench Lite, CodeContests, APPS, CanItEdit, MBPP)
- 2026-03-03: Batch 3 complete (SAFIM, BigVul, DiverseVul, PrimeVul, CodeEditorBench)
- 2026-03-03: Batch 4 complete (SWE-bench Verified, CodeSearchNet, Devign)
- 2026-03-03: Fixed APPS loading (refs/convert/parquet), PrimeVul (direct JSONL), CodeEditorBench (per-task JSONL)
- 2026-03-03: All 22 datasets verified loading successfully
- 2026-03-04: Refactored adapters into submodules (adapters/code_generation.py, code_editing.py, code_reasoning.py, vulnerability.py)
- 2026-03-04: Extracted CSS and JS into static/ directory (static/problem.css, static/problem.js)
- 2026-03-04: Added sampling for large datasets (cap at 1000 with seed=42)
- 2026-03-04: Enhanced FIM view (merged code with ground truth highlighting)
- 2026-03-04: Enhanced Before/After view (diff highlighting)
- 2026-03-04: Enhanced SWE-bench diff view (full file with diff chunks)
- 2026-03-04: Batch 5 complete (BigCodeBench, HumanEvalPack, CodeXGLUE Refinement, SWE-bench, CommitBench, EffiBench)
- 2026-03-04: All 28 datasets verified loading successfully
- 2026-03-04: Batch 6 complete (Long Code Arena — 6 project-level tasks)
- 2026-03-04: Batch 7 complete (dpaia EE-Dataset, Multi-SWE-bench, SWE-bench Multilingual, CrossCodeEval, McEval, MultiPL-E, Defects4J)
- 2026-03-04: Dropped RepoBench (HF repo has only deprecated loading script, no data files)
- 2026-03-04: Fixed Multi-SWE-bench (load per-repo JSONL files directly instead of `load_dataset`)
- 2026-03-04: Fixed CrossCodeEval (load per-language JSONL files directly, inconsistent columns across files)
- 2026-03-04: Fixed Defects4J (split="train" not "test", fields: bug_id/func_before/func_after)
- 2026-03-04: All 41 datasets verified loading successfully