# Experiment chain — unified-MM LLM (paper-grade, v5) Single document that ties together every `.py` and `.sh` we run from zero-shot bench to final SV-GSPO checkpoint. v4 (in `experiment_chain.md`) covered the per-task LLM progression. v5 adds the unified-multimodal stack and the post-bench training pipeline that auto-fires after the bench grid finishes. Run order is the same as the order of stages in `/dev/shm/dnathinker/post_bench_pipeline.sh` — the H100 just reads that script top-to-bottom, no SLURM dependencies. ## 0. Bench grid (zero-shot baselines) | Stage | Script | Output | Purpose | |-------|--------|--------|---------| | ZS-T1 raw | `scripts/run_llm_benchmark_vllm.py --task enhancer_generation --prompt raw` | `runs/exp_t1_grid_*/zs_raw/{predictions,metrics}.json{,l}` | Paper Table 1 row 1 | | ZS-T1 enriched | same w/ `--prompt enriched` | `runs/exp_t1_grid_*/zs_enriched/...` | Table 1 row 2 | | ZS-T2 raw | `--task pair_prediction --prompt raw` | `runs/exp_t2_grid_*/zs_raw/...` | Table 1 row 1 (T2) | | ZS-T2 enriched | same enriched | `runs/exp_t2_grid_*/zs_enriched/...` | Table 1 row 2 (T2) | | ZS-T3 raw / enriched | `--task enhancer_editing` × {raw, enriched} | `runs/exp_t3_grid_*/...` | Table 1 rows 1–2 (T3) | Driver: `/dev/shm/dnathinker/launch_bench_vllm.sh` runs the 6 vLLM benches sequentially. When the orchestrator PID exits, an attached watcher fires `post_bench_pipeline.sh`. ## 1. Post-bench pipeline (auto-triggered) `/dev/shm/dnathinker/post_bench_pipeline.sh`. Each stage skip-checks on its own output file, so re-runs are idempotent. ### Stage 0 — ZS scoring (early HF push) * `scripts/run_generation_eval.py` → `genqual.json` (FBD / spec / argmax-acc / per-cell-type) for T1+T3 zs_raw / zs_enriched. * `scripts/eval_t3_oracle.py` → `genqual_t3_oracle.json` (within-budget, length-preserved, objective-success per edit_type, per-cell-type) on T3 zs predictions. * HF push of the partial bench results so lab can see numbers before training stages finish. ### Stages 1–4 — Fusion-SFT family (the headline) Each `run_fusion` call invokes `scripts/train_fusion_sft.py` with `--architecture-mode llava`, then **Stage Nb** invokes `scripts/predict_fusion.py` on the trained adapter to get predictions on the full test set, followed by `run_generation_eval.py` (T1/T3) and `eval_t3_oracle.py` (T3 only). These produce the `lora_raw` / `lora_enriched` rows in Table 1. | Stage | Train script call | Inference + scoring | Paper row | |-------|---|---|---| | 1 | T1 fusion-SFT (n35k T1) | `score_adapter T1 ... raw / enriched` | T1 row 4 | | 2 | T2 fusion-SFT (n35k T2 balanced) | `score_adapter T2 ... raw / enriched` | T2 row 4 | | 3 | T3 fusion-SFT (n35k T3, heuristic gold) | `score_adapter T3 ... raw / enriched` | T3 row 4a | | 3b | T3 reasoning-only SFT (`--mask-assistant-dna-span`) | same | T3 row 4b — paper ablation | | 3c | T3 RFT (Stage A → K candidates → oracle-filter → re-SFT) | same | T3 row 4c — paper ablation | | 4 | **Joint multitask** fusion-SFT (105k = 35k×3 balanced) | `score_adapter` × {T1,T2,T3} × {raw,enriched} | **headline row** — one model, three tasks | `score_adapter` is defined inside `post_bench_pipeline.sh`. It exists because `run_llm_benchmark.py --adapter-dir` expects PEFT format (`adapter_model.bin` + `adapter_config.json`), and our `FusionSFTTrainer` saves a **full** OneShotFusionLM state_dict (LLM + LoRA + NTv3 projector + cell context encoder) via `torch.save`. `predict_fusion.py` rebuilds the model and `load_state_dict`s it, then runs `model.llm.generate` with the same prompt builder + parser that `ZeroShotLLM.predict` uses, so `predictions.jsonl` is shape- compatible with the genqual + T3-oracle scorers. This is the single bridge between training output and the eval pipeline. ### Stages 5–6 — NTv3-only baselines * Stage 5: `scripts/train_generation.py --head mdlm` (NTv3-MDLM on T1). * Stage 6: `scripts/train_ntv3_direct.py` (NTv3-direct on T2). * "no LLM" rows in Table 1 — proves the LLM contributes signal. ### Stage 7 — Aggregator + final HF push * `aggregate_results.py` walks `runs/`, collapses `(task, mode, prompt)` and writes `/dev/shm/dnathinker/results/h100_snapshot.md`. * HF push of metrics + genqual + h100_snapshot.md. ## 2. Where Loop-SFT fits Loop-SFT (`scripts/train_loop_sft.py`) is **not redundant** with RFT. The two filter on different signals: * **RFT** (Stage 3c): filter by *output objective* — generate K candidates, keep ones whose **DNA sequence** satisfies budget + motif + activity-shift via the oracle. Improves the **final answer**. * **Loop-SFT**: filter by *trajectory* — keep traces whose intermediate tool calls and reasoning chain are correct. Improves the **reasoning chain that leads to the answer**. The full T3 stack the paper aims for: ``` Fusion-SFT (heuristic) → Loop-SFT (trajectory-filtered) → RFT (oracle-filtered) → SV-GSPO (RL) Stage A Stage A' Stage B Stage C ``` Stage A' (Loop-SFT) is **deferred** to a follow-up run because the trajectory-trace dataset (`16K v9` in `t3_evaluation_design.md` §10) is the lab's, not the H100's. The H100 ships: - Stage A (the three `run_fusion` calls) - Stage A's reasoning-only ablation (3b) — equivalent to a cold-start Loop-SFT with no traces; an ablation that shows losing the heuristic DNA target doesn't tank the model - Stage B (RFT, 3c) When the lab finishes Loop-SFT on its side, the chain re-merges: both teams point at the same `exp_t3_fusion_sft_*/final/pytorch_model.bin`, the lab adds Loop-SFT on top, the H100 adds RFT on top, and we pick whichever path scores higher on `eval_t3_oracle.py` for the paper. ## 3. Job map (current state, 2026-04-27 UTC) ``` H100 NVL ├── PID 100474 launch_bench_vllm.sh (orchestrator) │ └── PID 121129 vLLM bench T2 zs_enriched (in flight) │ queued: T3 zs_raw, T3 zs_enriched └── PID 100544 watcher → post_bench_pipeline.sh (idle until 100474 exits) ``` ETAs (rough, post-T2 enriched completion): * T3 raw + T3 enriched bench: ~5h each (10h total) * Stage 0 + 0c (genqual + T3 oracle on zs preds): ~30 min * Stages 1–3 fusion-SFT (3 × 35k × 1 epoch on H100 NVL): ~6–8h total * Stage 3b reasoning-only: ~3h * Stage 3c RFT generate + filter + re-SFT: ~5h * Stage 4 joint multitask 105k: ~10h * Stages 5–6 NTv3-only: ~2h each * Stage 7 aggregator + HF push: minutes Total post-bench ≈ 40 H100-hours. Tracked in `runs/post_bench_pipeline.log` — `tail -f` for liveness. ## 4. Paper-table → script map (cheat sheet) | Table 1 row | Numbers come from | Per-cell breakdown? | |---|---|---| | Row 1 (zs_raw) | `runs/exp_t{1,2,3}_grid_*/zs_raw/genqual/genqual.json` | yes | | Row 2 (zs_enriched) | `.../zs_enriched/genqual/genqual.json` | yes | | Row 3 (LoRA, no NTv3) | DEFERRED — not in current pipeline | | | Row 4 (Fusion-SFT, per-task) | `runs/exp_t{1,2,3}_fusion_sft_*/predict_t{1,2,3}_{raw,enriched}/genqual/*.json` | yes (T1/T3); T2 has no per-cell — pair_prediction is binary | | Row 4b (T3 reasoning-only) | `runs/exp_t3_fusion_sft_reasonly_*/predict_t3_*/genqual/...` | yes | | Row 4c (T3 RFT) | `runs/exp_t3_fusion_sft_rft_*/predict_t3_*/genqual/...` | yes | | **Headline (joint multitask)** | `runs/exp_joint_multitask_*/predict_t{1,2,3}_*/genqual/...` | yes | | Row 5 (Loop-SFT) | lab side, slurm | | | Row 6 (SV-GSPO) | lab side, slurm | | T3-specific paper section uses the **objective-satisfaction** metrics from `eval_t3_oracle.py` (`within_budget`, `length_preserved`, `objective_success_*`, `transfer_specificity`, `in_budget_at_{5,10,20}pct`), not the heuristic-overlap genqual ones — see `t3_evaluation_design.md` §2 for why. ## 5. Reasoning-trace augmentation (OpenRouter / Nemotron, free) `scripts/build_reasoning_traces.py` rewrites the assistant turn in any T1/T2/T3 SFT JSONL to include a single-shot rationale that wires the enriched evidence (TFBS scan, expression context, motif hits) to the gold answer. Output schema matches the parent project's existing `pe_dataset_reasoning_expansion_*/jsonl/` files exactly: ``` RATIONALE SEQ # T1/T3 paired|not_paired # T2 ``` Reuses `regureasoner.loop.openrouter.OpenRouterClient` (same retry + backoff client `expand_loop_trajectories.py` uses). Single API call per row — the teacher only writes the *justification*, not the answer, so small free-tier models (default `nvidia/nemotron-nano-9b-v2:free`; switch to `nvidia/llama-3.1-nemotron-70b-instruct:free` for richer rationales) stay reliable. **Resumable**: appends to the output JSONL; on startup it scans every `id` already present and skips those rows in the source. Daily reruns accumulate without overlap. **Budget**: `--max-requests` (default 1000) is the per-invocation cap. OpenRouter free tier = 1000 req/day per key. Multiple keys can shard line-level via `--shard-index/--num-shards`. **Daily-loop launcher**: `slurm/build_reasoning_traces_loop.sh` — sources `OPENROUTER_API_KEY` from `/dev/shm/dnathinker/.env`, walks T1/T2/T3 with `PER_TASK=333` each (≈1000/day total), and optionally `--daemon`s into a 24h sleep loop. Zero GPU; runs alongside any training stage. **SFT integration**: when ≥N augmented rows accumulate per task, point `scripts/train_fusion_sft.py --train-jsonl` at `/dev/shm/dnathinker/data/reasoning_traces/train..reasoning.jsonl`. Same collator, same trainer — the only difference is the assistant turn now starts with `...`, so the trained model emits explicit rationale + answer at inference time. This is the **paper's "reasoning model" row** in T3's table; the non-reasoning fusion-SFT runs (Stages 1–3) stay as the no-rationale comparison. **Per-task source JSONL — what the teacher justifies**: | Task | Source JSONL | Why | |---|---|---| | T1 | `train.enhancer_generation.strat7c.n35k.jsonl` (heuristic gold) | The heuristic gold is the empirical paired enhancer; teacher justifies why it pairs in this cell type. | | T2 | `train.pair_prediction.strat7c.n35k.jsonl` (observed positive + pseudo-negative) | Teacher justifies the binary label using shared-TFBS / GC / expression evidence. | | T3 | **post-RFT** `train.t3_rft.jsonl` | The heuristic gold for T3 is a synthetic motif-implant (not unique GT — see `t3_evaluation_design.md` §1). RFT (Stage 3c) replaces it with an oracle-validated candidate. Reasoning expansion **must run on the post-RFT JSONL** so the rationale justifies a sequence the oracle has actually scored, not the heuristic. Order: Fusion-SFT → RFT → reasoning expansion → reasoning-augmented Fusion-SFT. | The launcher `slurm/build_reasoning_traces_loop.sh` defaults to the heuristic-gold JSONLs for T1/T2 and the heuristic-gold for T3, but override `T3_SRC=/dev/shm/dnathinker/runs/exp_t3_fusion_sft_rft_${STAMP}/.../train.t3_rft.jsonl` once Stage 3c finishes — the loop's resume logic handles a mid-run source swap because the augmented output JSONL keeps row ids. ## 6. Input sanitisation — applied globally before any model sees text `regureasoner/utils/input_sanitize.py` (used by `PromptBuilder.user()` and `build_reasoning_traces._format_user`) strips three classes of issue at read-time, so we don't need to regenerate the prod JSONLs: 1. **Label leaks** — `peak_name=chr…`, `enhancer_peak_name=chr…`, the "Peak coordinates parsed to chr…:…" sentence, the "Observed dataset row is a released paired/not_paired link …" sentence (T2's biggest leak), and `label_source=…` lines. 2. **Unexplained proxy scores** — `Evolution proxy score … (expression_stability_proxy_v1)`, `promoter_likeness_score=…`, `quality_score / repeat_fraction / kmer_entropy_norm` (these are ad-hoc internal scores the model can't ground; we omit rather than try to explain in-prompt). 3. **Cell-type abbreviations** expanded — `cell_type=Ex` → `cell_type=Excitatory neuron (Ex)` so the model knows the biology. Applied before any model call. Idempotent — running it twice yields the same string. 12 unit tests cover every leak/score family + idempotency + cell-type expansion (`tests/test_input_sanitize.py`). Why we don't run this *inside* `post_bench_pipeline.sh`: the script is IO-bound (no GPU), capped at 1000 req/day, and meant to run for **multiple days** in the background. Putting it in the GPU pipeline would either waste a single 1000-call day or block the rest of the pipeline waiting for accumulation. The right pattern is to launch `build_reasoning_traces_loop.sh --daemon` once at the start of the campaign and let it accumulate rows independently. When a critical mass exists, fire a single fusion-SFT run on the augmented JSONL.