docs/experiment_chain_v5_unified.md

# Experiment chain — unified-MM LLM (paper-grade, v5)

Single document that ties together every `.py` and `.sh` we run from
zero-shot bench to final SV-GSPO checkpoint. v4 (in
`experiment_chain.md`) covered the per-task LLM progression. v5 adds
the unified-multimodal stack and the post-bench training pipeline that
auto-fires after the bench grid finishes.

Run order is the same as the order of stages in
`/dev/shm/dnathinker/post_bench_pipeline.sh` — the H100 just reads
that script top-to-bottom, no SLURM dependencies.

## 0. Bench grid (zero-shot baselines)

| Stage | Script | Output | Purpose |
|-------|--------|--------|---------|
| ZS-T1 raw | `scripts/run_llm_benchmark_vllm.py --task enhancer_generation --prompt raw` | `runs/exp_t1_grid_*/zs_raw/{predictions,metrics}.json{,l}` | Paper Table 1 row 1 |
| ZS-T1 enriched | same w/ `--prompt enriched` | `runs/exp_t1_grid_*/zs_enriched/...` | Table 1 row 2 |
| ZS-T2 raw | `--task pair_prediction --prompt raw` | `runs/exp_t2_grid_*/zs_raw/...` | Table 1 row 1 (T2) |
| ZS-T2 enriched | same enriched | `runs/exp_t2_grid_*/zs_enriched/...` | Table 1 row 2 (T2) |
| ZS-T3 raw / enriched | `--task enhancer_editing` × {raw, enriched} | `runs/exp_t3_grid_*/...` | Table 1 rows 1–2 (T3) |

Driver: `/dev/shm/dnathinker/launch_bench_vllm.sh` runs the 6 vLLM
benches sequentially. When the orchestrator PID exits, an attached
watcher fires `post_bench_pipeline.sh`.

## 1. Post-bench pipeline (auto-triggered)

`/dev/shm/dnathinker/post_bench_pipeline.sh`. Each stage skip-checks
on its own output file, so re-runs are idempotent.

### Stage 0 — ZS scoring (early HF push)

* `scripts/run_generation_eval.py` → `genqual.json` (FBD / spec /
  argmax-acc / per-cell-type) for T1+T3 zs_raw / zs_enriched.
* `scripts/eval_t3_oracle.py` → `genqual_t3_oracle.json`
  (within-budget, length-preserved, objective-success per
  edit_type, per-cell-type) on T3 zs predictions.
* HF push of the partial bench results so lab can see numbers
  before training stages finish.

### Stages 1–4 — Fusion-SFT family (the headline)

Each `run_fusion` call invokes `scripts/train_fusion_sft.py` with
`--architecture-mode llava`, then **Stage Nb** invokes
`scripts/predict_fusion.py` on the trained adapter to get
predictions on the full test set, followed by
`run_generation_eval.py` (T1/T3) and `eval_t3_oracle.py` (T3 only).
These produce the `lora_raw` / `lora_enriched` rows in Table 1.

| Stage | Train script call | Inference + scoring | Paper row |
|-------|---|---|---|
| 1 | T1 fusion-SFT (n35k T1) | `score_adapter T1 ... raw / enriched` | T1 row 4 |
| 2 | T2 fusion-SFT (n35k T2 balanced) | `score_adapter T2 ... raw / enriched` | T2 row 4 |
| 3 | T3 fusion-SFT (n35k T3, heuristic gold) | `score_adapter T3 ... raw / enriched` | T3 row 4a |
| 3b | T3 reasoning-only SFT (`--mask-assistant-dna-span`) | same | T3 row 4b — paper ablation |
| 3c | T3 RFT (Stage A → K candidates → oracle-filter → re-SFT) | same | T3 row 4c — paper ablation |
| 4 | **Joint multitask** fusion-SFT (105k = 35k×3 balanced) | `score_adapter` × {T1,T2,T3} × {raw,enriched} | **headline row** — one model, three tasks |

`score_adapter` is defined inside `post_bench_pipeline.sh`. It exists
because `run_llm_benchmark.py --adapter-dir` expects PEFT format
(`adapter_model.bin` + `adapter_config.json`), and our
`FusionSFTTrainer` saves a **full** OneShotFusionLM state_dict (LLM +
LoRA + NTv3 projector + cell context encoder) via `torch.save`.
`predict_fusion.py` rebuilds the model and `load_state_dict`s it,
then runs `model.llm.generate` with the same prompt builder + parser
that `ZeroShotLLM.predict` uses, so `predictions.jsonl` is shape-
compatible with the genqual + T3-oracle scorers. This is the single
bridge between training output and the eval pipeline.

### Stages 5–6 — NTv3-only baselines

* Stage 5: `scripts/train_generation.py --head mdlm` (NTv3-MDLM on T1).
* Stage 6: `scripts/train_ntv3_direct.py` (NTv3-direct on T2).
* "no LLM" rows in Table 1 — proves the LLM contributes signal.

### Stage 7 — Aggregator + final HF push

* `aggregate_results.py` walks `runs/`, collapses
  `(task, mode, prompt)` and writes
  `/dev/shm/dnathinker/results/h100_snapshot.md`.
* HF push of metrics + genqual + h100_snapshot.md.

## 2. Where Loop-SFT fits

Loop-SFT (`scripts/train_loop_sft.py`) is **not redundant** with RFT.
The two filter on different signals:

* **RFT** (Stage 3c): filter by *output objective* — generate K
  candidates, keep ones whose **DNA sequence** satisfies budget + motif
  + activity-shift via the oracle. Improves the **final answer**.
* **Loop-SFT**: filter by *trajectory* — keep traces whose
  intermediate tool calls and reasoning chain are correct. Improves
  the **reasoning chain that leads to the answer**.

The full T3 stack the paper aims for:

```
Fusion-SFT (heuristic)  →  Loop-SFT (trajectory-filtered)  →  RFT (oracle-filtered)  →  SV-GSPO (RL)
       Stage A                       Stage A'                       Stage B               Stage C
```

Stage A' (Loop-SFT) is **deferred** to a follow-up run because the
trajectory-trace dataset (`16K v9` in `t3_evaluation_design.md` §10)
is the lab's, not the H100's. The H100 ships:
- Stage A (the three `run_fusion` calls)
- Stage A's reasoning-only ablation (3b) — equivalent to a
  cold-start Loop-SFT with no traces; an ablation that shows
  losing the heuristic DNA target doesn't tank the model
- Stage B (RFT, 3c)

When the lab finishes Loop-SFT on its side, the chain re-merges:
both teams point at the same `exp_t3_fusion_sft_*/final/pytorch_model.bin`,
the lab adds Loop-SFT on top, the H100 adds RFT on top, and we pick
whichever path scores higher on `eval_t3_oracle.py` for the paper.

## 3. Job map (current state, 2026-04-27 UTC)

```
H100 NVL
├── PID 100474  launch_bench_vllm.sh (orchestrator)
│   └── PID 121129  vLLM bench T2 zs_enriched (in flight)
│       queued: T3 zs_raw, T3 zs_enriched
└── PID 100544  watcher → post_bench_pipeline.sh (idle until 100474 exits)
```

ETAs (rough, post-T2 enriched completion):
* T3 raw + T3 enriched bench: ~5h each (10h total)
* Stage 0 + 0c (genqual + T3 oracle on zs preds): ~30 min
* Stages 1–3 fusion-SFT (3 × 35k × 1 epoch on H100 NVL): ~6–8h total
* Stage 3b reasoning-only: ~3h
* Stage 3c RFT generate + filter + re-SFT: ~5h
* Stage 4 joint multitask 105k: ~10h
* Stages 5–6 NTv3-only: ~2h each
* Stage 7 aggregator + HF push: minutes

Total post-bench ≈ 40 H100-hours. Tracked in
`runs/post_bench_pipeline.log` — `tail -f` for liveness.

## 4. Paper-table → script map (cheat sheet)

| Table 1 row | Numbers come from | Per-cell breakdown? |
|---|---|---|
| Row 1 (zs_raw) | `runs/exp_t{1,2,3}_grid_*/zs_raw/genqual/genqual.json` | yes |
| Row 2 (zs_enriched) | `.../zs_enriched/genqual/genqual.json` | yes |
| Row 3 (LoRA, no NTv3) | DEFERRED — not in current pipeline |  |
| Row 4 (Fusion-SFT, per-task) | `runs/exp_t{1,2,3}_fusion_sft_*/predict_t{1,2,3}_{raw,enriched}/genqual/*.json` | yes (T1/T3); T2 has no per-cell — pair_prediction is binary |
| Row 4b (T3 reasoning-only) | `runs/exp_t3_fusion_sft_reasonly_*/predict_t3_*/genqual/...` | yes |
| Row 4c (T3 RFT) | `runs/exp_t3_fusion_sft_rft_*/predict_t3_*/genqual/...` | yes |
| **Headline (joint multitask)** | `runs/exp_joint_multitask_*/predict_t{1,2,3}_*/genqual/...` | yes |
| Row 5 (Loop-SFT) | lab side, slurm |  |
| Row 6 (SV-GSPO) | lab side, slurm |  |

T3-specific paper section uses the **objective-satisfaction** metrics
from `eval_t3_oracle.py` (`within_budget`, `length_preserved`,
`objective_success_*`, `transfer_specificity`,
`in_budget_at_{5,10,20}pct`), not the heuristic-overlap genqual ones —
see `t3_evaluation_design.md` §2 for why.

## 5. Reasoning-trace augmentation (OpenRouter / Nemotron, free)

`scripts/build_reasoning_traces.py` rewrites the assistant turn in any
T1/T2/T3 SFT JSONL to include a single-shot rationale that wires the
enriched evidence (TFBS scan, expression context, motif hits) to the
gold answer. Output schema matches the parent project's existing
`pe_dataset_reasoning_expansion_*/jsonl/` files exactly:

```
<reasoning_start>RATIONALE</reasoning_end>
<enhancer_dna_start>SEQ</enhancer_dna_end>     # T1/T3
<pair_label>paired|not_paired</pair_label>     # T2
```

Reuses `regureasoner.loop.openrouter.OpenRouterClient` (same retry +
backoff client `expand_loop_trajectories.py` uses). Single API call
per row — the teacher only writes the *justification*, not the
answer, so small free-tier models (default
`nvidia/nemotron-nano-9b-v2:free`; switch to
`nvidia/llama-3.1-nemotron-70b-instruct:free` for richer rationales)
stay reliable.

**Resumable**: appends to the output JSONL; on startup it scans every
`id` already present and skips those rows in the source. Daily reruns
accumulate without overlap.

**Budget**: `--max-requests` (default 1000) is the per-invocation
cap. OpenRouter free tier = 1000 req/day per key. Multiple keys can
shard line-level via `--shard-index/--num-shards`.

**Daily-loop launcher**: `slurm/build_reasoning_traces_loop.sh` —
sources `OPENROUTER_API_KEY` from `/dev/shm/dnathinker/.env`, walks
T1/T2/T3 with `PER_TASK=333` each (≈1000/day total), and
optionally `--daemon`s into a 24h sleep loop. Zero GPU; runs
alongside any training stage.

**SFT integration**: when ≥N augmented rows accumulate per task, point
`scripts/train_fusion_sft.py --train-jsonl` at
`/dev/shm/dnathinker/data/reasoning_traces/train.<task>.reasoning.jsonl`.
Same collator, same trainer — the only difference is the assistant
turn now starts with `<reasoning_start>...</reasoning_end>`, so the
trained model emits explicit rationale + answer at inference time.
This is the **paper's "reasoning model" row** in T3's table; the
non-reasoning fusion-SFT runs (Stages 1–3) stay as the
no-rationale comparison.

**Per-task source JSONL — what the teacher justifies**:

| Task | Source JSONL | Why |
|---|---|---|
| T1 | `train.enhancer_generation.strat7c.n35k.jsonl` (heuristic gold) | The heuristic gold is the empirical paired enhancer; teacher justifies why it pairs in this cell type. |
| T2 | `train.pair_prediction.strat7c.n35k.jsonl` (observed positive + pseudo-negative) | Teacher justifies the binary label using shared-TFBS / GC / expression evidence. |
| T3 | **post-RFT** `train.t3_rft.jsonl` | The heuristic gold for T3 is a synthetic motif-implant (not unique GT — see `t3_evaluation_design.md` §1). RFT (Stage 3c) replaces it with an oracle-validated candidate. Reasoning expansion **must run on the post-RFT JSONL** so the rationale justifies a sequence the oracle has actually scored, not the heuristic. Order: Fusion-SFT → RFT → reasoning expansion → reasoning-augmented Fusion-SFT. |

The launcher `slurm/build_reasoning_traces_loop.sh` defaults to the
heuristic-gold JSONLs for T1/T2 and the heuristic-gold for T3, but
override `T3_SRC=/dev/shm/dnathinker/runs/exp_t3_fusion_sft_rft_${STAMP}/.../train.t3_rft.jsonl`
once Stage 3c finishes — the loop's resume logic handles a mid-run
source swap because the augmented output JSONL keeps row ids.

## 6. Input sanitisation — applied globally before any model sees text

`regureasoner/utils/input_sanitize.py` (used by `PromptBuilder.user()`
and `build_reasoning_traces._format_user`) strips three classes of
issue at read-time, so we don't need to regenerate the prod JSONLs:

1. **Label leaks** — `peak_name=chr…`, `enhancer_peak_name=chr…`,
   the "Peak coordinates parsed to chr…:…" sentence, the "Observed
   dataset row is a released paired/not_paired link …" sentence
   (T2's biggest leak), and `label_source=…` lines.
2. **Unexplained proxy scores** — `Evolution proxy score …
   (expression_stability_proxy_v1)`, `promoter_likeness_score=…`,
   `quality_score / repeat_fraction / kmer_entropy_norm` (these are
   ad-hoc internal scores the model can't ground; we omit rather
   than try to explain in-prompt).
3. **Cell-type abbreviations** expanded — `cell_type=Ex` →
   `cell_type=Excitatory neuron (Ex)` so the model knows the biology.

Applied before any model call. Idempotent — running it twice yields
the same string. 12 unit tests cover every leak/score family +
idempotency + cell-type expansion (`tests/test_input_sanitize.py`).

Why we don't run this *inside* `post_bench_pipeline.sh`: the script
is IO-bound (no GPU), capped at 1000 req/day, and meant to run for
**multiple days** in the background. Putting it in the GPU pipeline
would either waste a single 1000-call day or block the rest of the
pipeline waiting for accumulation. The right pattern is to launch
`build_reasoning_traces_loop.sh --daemon` once at the start of the
campaign and let it accumulate rows independently. When a critical
mass exists, fire a single fusion-SFT run on the augmented JSONL.