docs/PROTOCOL_AMENDMENTS.md

# Protocol amendments

This log records every post-freeze implementation correction. Earlier raw artifacts are
retained for auditability and are excluded from confirmatory analyses when the research-code
revision differs from the declared experiment revision.

## PA-001 — invalid E02 model output must be a scored outcome

- Date: 2026-07-18
- Experiment: E02
- Trigger: after three development cells under revision
  `a9a6f8b6d67ce59eba97c7ccc8331662eb41e7b5`, the next model response selected paths outside
  the harness-provided candidate set. The constraint validator rejected the response, but the
  matrix runner terminated instead of recording the violation.
- Correction: preserve the raw response, record the validation message as
  `protocol_violation`, use an empty selected-file list, assign zero localization metrics, and
  continue the matrix. Transport, server, and repository failures still terminate the run.
- Scientific impact: this changes no retrieval, prompt, context, model, decoding parameter, or
  metric. It prevents informative model noncompliance from becoming missing data. The three
  earlier one-shot artifacts and one query-stage artifact remain immutable development records;
  all primary E02 cells will be regenerated under the amended revision.

## PA-002 — invalid iterative query uses the visible issue as fallback

- Date: 2026-07-18
- Experiment: E02 iterative-query phase
- Trigger: the second query-reformulation call under revision
  `b08e027a355a9e5008909f6735543cc57efd5182` exhausted its response budget during reasoning
  and did not emit valid query JSON. The runner terminated before writing an artifact.
- Correction: retain the unmodified model response, record `protocol_violation`, and use the
  visible issue statement as the deterministic refined-retrieval query. Transport failures still
  terminate the run.
- Scientific impact: no hidden label, gold file, or test information enters the fallback. The
  fallback is the same visible text available to every harness. The one valid development query
  artifact under the earlier revision remains immutable but is excluded; all iterative E02 cells
  will run under the amended revision. One-shot E02 cells are unaffected and remain under
  revision `b08e027a355a9e5008909f6735543cc57efd5182`.

## PA-003 — oracle target added by the gold patch

- Date: 2026-07-18
- Experiment: E03 H019 only
- Trigger: after 39 E03 cells, `TASK_CR_005` reached H019. Its manifest correctly names
  `ExpandValue` as a gold symbol, but that function is newly added by the held-out patch and
  therefore cannot be found in the base-commit Tree-sitter index. No model call or result was
  created for the failed cell.
- Correction: when and only when an H019 gold symbol is absent at the base commit, read the
  old-file hunk line number from the frozen gold diff metadata and provide a ±20-line base-source
  window around that location. The added lines themselves are never read into the prompt. Existing
  gold functions continue to use exact Tree-sitter function bodies.
- Scientific impact: this preserves H019 as an oracle edit-location control for additions while
  avoiding answer-content leakage. It affects no non-oracle harness. The 39 completed E03 cells
  remain immutable under `e78637b7e90b230881d621f423fd22b1b53d362e`; the missing H019 cell and
  remaining task blocks use the amended revision.

## PA-004 — repository-neutral dense-retrieval instruction for Study 2

- Date: 2026-07-18
- Experiment: E08, before any Study 2 outcome was generated or inspected
- Trigger: the pre-execution design audit found that inherited profile `EMB001` says
  “Retrieve GitLab Runner code,” which is inappropriate for the GitLab CLI and python-gitlab
  replication repositories.
- Correction: retain `EMB001` unchanged for Study 1 and add `EMB002`, which uses the same local
  Qwen3 Embedding 0.6B artifact, quantization, dimensions, chunking, and index settings but freezes
  the repository-neutral instruction “Retrieve repository code relevant to resolving the
  described software issue.” E08 references `EMB002`.
- Scientific impact: this removes a repository-name confound and changes no primary hypothesis,
  task, agent model, outcome, or analysis. The new embedding-profile hash is recorded in every E08
  run. No E08 embedding cache or LLM outcome existed when the correction was made.

## PA-005 — stochastic reliability profile

- Date: 2026-07-18
- Experiment: E08 secondary reliability audit, before any Study 2 outcome was generated or inspected
- Trigger: seeds 1 and 2 had originally been paired with temperature 0. Under greedy decoding,
  seed changes can be inert and would provide a weak estimate of trajectory stability.
- Correction: keep all 840 primary cells at temperature 0, top-p 1, seed 0. Run the 24 frozen,
  balanced non-oracle reliability cells at temperature 0.2, top-p 1, and seeds 0, 1, and 2. This
  produces 72 sensitivity cells rather than 48 and raises the total live-cell plan from 888 to 912.
- Scientific impact: the confirmatory primary contrast and all deterministic main-matrix
  estimates are unchanged. Reliability is explicitly a stochastic decoding sensitivity analysis,
  identified by a distinct model-configuration hash and repetition field; it is not pooled into
  the primary matrix.

## PA-006 — infrastructure attempts are retained and retried, not scored

- Date: 2026-07-18
- Experiment: E08, before any Study 2 outcome was generated or inspected
- Trigger: crash-path review found that a partially created immutable run directory would block an
  identical retry, and broad exception handling could score a hidden-test infrastructure failure
  as a model failure.
- Correction: only connection failures and HTTP 408/429/5xx responses receive the single allowed
  no-response retry. Model/protocol errors are not retried. Interrupted attempts are moved intact
  to `results/infrastructure_attempts/` with an archive record before the identical run identity is
  retried. Hidden-test infrastructure exceptions propagate and are rerun; model-produced empty or
  invalid patches remain scored failures. A runtime lease always unloads all model instances and
  stops the `lms` server on exit.
- Scientific impact: this implements the preregistered missingness rule and prevents differential
  infrastructure failures from entering the binary endpoint. All attempts remain auditable.

## PA-007 — Tree-sitter identifier extraction shadowing

- Date: 2026-07-18
- Experiment: E08 H011, before any Study 2 outcome was generated or inspected
- Trigger: final source review found that the generalized Go/Python parser reused the name
  `identifiers` for both the allowed Tree-sitter node-type set and one symbol's extracted names.
  After the first declaration in a file, later declarations could therefore lose graph edges.
- Correction: use distinct immutable `identifier_types` and per-symbol `referenced_identifiers`
  variables, with regression tests proving extraction persists across multiple declarations.
- Scientific impact: this restores the declared one-hop symbol-graph implementation for H011. It
  changes no hypothesis or analysis and was corrected before any E08 retrieval index or outcome
  existed.

## PA-008 — preserve the causal exception when LM Studio is already stopped

- Date: 2026-07-18
- Experiment: E08 infrastructure startup; zero completed E08 cells and zero inspected outcomes
- Trigger: the first matrix invocation exited before creating any cell artifact. During exception
  cleanup, `lms server stop` reported that the server was already stopped and masked the earlier
  causal exception.
- Correction: cleanup now records unload/stop errors, checks `lms server status` before stopping,
  treats an already-stopped server as a safe terminal state, and re-raises the original execution
  exception. Cleanup errors are fatal only when no earlier exception exists.
- Scientific impact: no task, treatment, model, outcome, generation parameter, or analysis changes.
  This is an infrastructure-diagnostics correction made with zero E08 outcomes.

## PA-009 — wait for REST readiness after `lms server start`

- Date: 2026-07-18
- Experiment: E08 infrastructure startup; zero completed E08 cells and zero inspected outcomes
- Trigger: direct CLI diagnosis showed that `lms server start/status` can report “running” before
  port 1234 accepts official REST requests. The lifecycle wrapper returned at the CLI-status edge,
  and the next `/api/v1/models` request received connection refused.
- Correction: after both newly started and already-running CLI states, poll
  `GET /api/v1/models` until it responds (an authentication response also proves readiness) or the
  frozen lifecycle timeout expires. Preflight cleanup now also tolerates an already-stopped server.
- Scientific impact: no treatment or outcome changes. This removes a local server race before any
  E08 cell or outcome existed.

## PA-010 — distinguish “not running” in `lms server status`

- Date: 2026-07-18
- Experiment: E08 infrastructure startup; zero completed E08 cells and zero inspected outcomes
- Trigger: the readiness-gated invocation showed that this `lms` build exits 0 and writes
  “The server is not running” on stderr. The parser's positive substring check incorrectly marked
  that message as running and therefore waited for an API that had never been started.
- Correction: a running state now requires the phrase “server is running” and explicitly rejects
  “server is not running,” with positive and negative stderr regression tests.
- Scientific impact: lifecycle parsing only; no E08 cell or outcome existed when corrected.

## PA-011 — post-outcome trajectory-derived tool-protocol diagnostic

- Date: 2026-07-18
- Experiment: E08 analysis, after all 912 outcomes were sealed and the registered endpoint analysis
  had been run
- Trigger: the registered failure taxonomy showed 420/420 empty patches for M003 despite many tool
  calls. A read-only audit was needed to distinguish model/tool-protocol incompatibility from an
  LM Studio identity, transport, or execution failure.
- Change: add a clearly labeled exploratory diagnostic that counts tool attempts, error events,
  patch-protocol errors, executor calls, and executor acceptances from the append-only
  `trajectory.jsonl` records, stratified by model and treatment. No run is retried or excluded and
  no confirmatory or secondary contrast changes.
- Telemetry caveat discovered: `final_metrics.tool_counts` increments failed tool calls both when
  attempted and again on the error path. The aggregate `final_metrics.tool_calls` agrees exactly
  with one `tool_call` trajectory event per invocation and is unaffected. Per-tool diagnostics now
  use trajectory events only. Correctness, localization, latency, token, endpoint, and paired-test
  fields are unaffected.
- Scientific impact: post-outcome exploratory mechanism analysis only. It may explain a result but
  may not upgrade, replace, or alter any registered inference.

## PA-012 — undefined chance-corrected agreement under universal failure

- Date: 2026-07-18
- Experiment: E08 reliability analysis, after all 912 outcomes were sealed
- Trigger: all 72 temperature-0.2 reliability observations occupy the failure category. The first
  implementation returned 1.0 for chance-corrected agreement when its denominator was zero.
- Correction: raw unanimity and pairwise agreement remain 1.0, but Fleiss' kappa, nominal
  Krippendorff alpha, and ICC(1,1) are now reported as undefined because the ratings have no
  marginal or between-cell variance. A regression test fixes this behavior.
- Scientific impact: reporting correction only. It changes no run, endpoint, contrast, confidence
  interval, or model fit. The reliability conclusion is restricted to reproducible failure on the
  24 selected cells; it is not generalized as stochastic robustness.

## PA-013 — score malformed raw-diff actions as tool failures

- Date: 2026-07-19
- Experiment: E09, after 5/540 cells completed and before any registered comparison, confidence
  interval, compatibility gate, or model-by-interface summary was computed
- Trigger: the sixth cell emitted a non-unified-diff `apply_patch` payload. The shared patch parser
  raised its typed `PatchOutputError`, but E09's interaction loop caught only `ValueError` and JSON
  decoding failures. This terminated the matrix instead of returning the preregistered ordinary
  tool-error observation to the model.
- Correction: include `PatchOutputError` in the existing tool-observation exception boundary and
  add a regression test for the exact malformed action. No prompt, task, model, interface,
  generation parameter, budget, endpoint, gate, or analysis changes. The 5 completed and 1
  incomplete cells from revision `0506013c0b8e302106b116ef9053a73ab6e78b70` are retained as an
  aborted infrastructure attempt outside scored raw results. All 540 cells restart under one new
  committed revision and the preflight is repeated.
- Scientific impact: post-outcome infrastructure amendment. The discarded pilot cells may not be
  analyzed or selectively reused. E09 remains prospective for its registered comparison but this
  amendment and the observed pilot count must be disclosed in the paper and release manifest.