| # Confirmatory GitLab Runner dataset |
|
|
| ## Freeze and separation |
|
|
| The confirmatory set was frozen on 2026-07-18 before running E01-E05. It |
| contains 15 test-backed GitLab Runner fixes from March-May 2026. None of their |
| gold commits overlap the five E00 development fixes. E00 remains excluded from |
| confirmatory estimates. |
|
|
| Every task stores a visible issue statement, exact parent and gold commits, |
| hidden source and test patches, changed source files, Tree-sitter-derived gold |
| symbols, validation commands, and provenance. The generator is |
| `scripts/build_confirmatory_tasks.py`; the frozen artifacts are under |
| `tasks/manifests` and `tasks/patches`. |
|
|
| | Task | Gold commit | Source files | Domain | E03 eligible | |
| |---|---|---:|---|---:| |
| | CR_001 | `d870071152bd` | 1 | AWS secrets | yes | |
| | CR_002 | `b6dedc95bd2b` | 2 | Cache adapters | yes | |
| | CR_003 | `91dcb6c16bec` | 1 | Build logging | yes | |
| | CR_004 | `eeba44f142de` | 1 | Concrete cache policy | no | |
| | CR_005 | `f07534b7a203` | 2 | Artifact variables | yes | |
| | CR_006 | `e05f89e3ec2e` | 1 | Cancellation errors | yes | |
| | CR_007 | `b4b74fa3d7c8` | 2 | Trace features | yes | |
| | CR_008 | `002579e259d9` | 1 | Artifact upload | yes | |
| | CR_009 | `c9632925cb0c` | 1 | S3 presigning | yes | |
| | CR_010 | `3b1d17363021` | 1 | Kubernetes variables | no | |
| | CR_011 | `610441433600` | 1 | Artifact timeouts | no | |
| | CR_012 | `86c126f85257` | 1 | Gzip filenames | yes | |
| | CR_013 | `1a6fb708e571` | 3 | Cache sharding | yes | |
| | CR_014 | `213001f5e9e2` | 3 | After-script policy | no | |
| | CR_015 | `32a34faf2b81` | 2 | Step-runner errors | no | |
| |
| All 15 tasks enter E01, E02, E04, and E05. Ten tasks enter E03. |
| |
| ## End-to-end validation |
| |
| Validation used Go 1.26.5 on clean `git archive` snapshots. For each task, the |
| validator required: |
| |
| 1. the parent package suite to pass before hidden tests were added; |
| 2. the hidden test patch to apply and expose at least one failure; |
| 3. the hidden source patch to apply; and |
| 4. the package suite with hidden tests and the gold source patch to pass. |
| |
| Ten tasks met every condition. CR_004's upstream test change did not expose a |
| parent failure. CR_010, CR_011, CR_014, and CR_015 had failures in the original |
| package suites on this host, so they were excluded from E03 without changing |
| their retrieval/localization eligibility. The complete commands, output tails, |
| exit codes, and timing are retained in `tasks/validation`. |
|
|
| ## Scope limitations |
|
|
| The task sample is a purposive population of recent, compact, test-backed fixes |
| from one large Go repository. It supports paired causal comparisons among |
| harnesses on this fixed population, but it does not justify generalization to |
| other repositories, languages, models, or issue distributions. Public commits |
| also carry a residual training-contamination risk, although their 2026 dates |
| reduce that risk relative to older benchmark tasks. |
|
|
