docs/t3_post_v5_followups.md

# T3 — post-v5 follow-ups (deep-think + work plan)

The first-pass v5 work (oracle eval, multi-turn RFT, reasoning expansion,
sanitiser) is shipped. This doc captures the deeper questions the T3
rethink raises, what's already been done about them, and what's
deliberately deferred.

## 1. RFT timing — when, relative to T1/T2 + joint training?

**TL;DR**: Current order (per-task SFT → T3 RFT → joint multitask) is
the right *minimum*. A second RFT pass after the joint adapter is a
worthwhile **ablation** but adds a serial dependency.

### The current order

```
Stage 1   T1 fusion-SFT (heuristic gold)
Stage 2   T2 fusion-SFT
Stage 3   T3 fusion-SFT (heuristic synthetic-edit gold)
Stage 3b  T3 reasoning-only SFT (paper ablation)
Stage 3c  RFT — generate K candidates from Stage-3 adapter,
          oracle-filter, retrain on filtered set
Stage 4   Joint multitask fusion-SFT (T1+T2+T3, 35k each, balanced)
```

### Why RFT can't move earlier

RFT needs a candidate generator that's already on-task — i.e., capable
of producing T3-shaped enhancer edits. The base Qwen3.5-2B can't do
that without prior T3 SFT (it doesn't know the
`<enhancer_dna_start>` schema, the cell-type grammar, or the edit
budget convention). So RFT must come **after at least one T3 SFT pass**.

### Why current order beats "RFT after joint"

The Stage-3 adapter is trained on 35k T3-only rows. The Stage-4 joint
adapter sees only 35k T3 rows out of a 105k mix (33% T3 share). For
candidate-generation specifically:

* **Stage-3 adapter**: more T3-faithful, fewer cross-task style
  artefacts. Higher per-row keep-rate when scoring against the T3
  oracle.
* **Stage-4 adapter**: better generalist, slightly weaker T3 grammar.

Empirical question — the **ablation** to run:

> RFT-from-Stage3 (current default) vs RFT-from-Stage4 (joint adapter)
> — does the joint adapter's regularisation produce candidates with
> higher mean objective margins, or do the format artefacts dominate?

This is a one-flag change: `--adapter-state-dict
runs/exp_joint_multitask_*/final/pytorch_model.bin` instead of the
T3-only path. Costs one RFT pass + one re-train. Worth a Table 3 row.

### Why a second RFT pass after joint isn't free

Re-RFT-ing the joint adapter would chain:

```
Stage 4 → Stage 4-RFT → retrain joint → ...
```

Each step is multi-hour on H100. Diminishing returns are likely (the
oracle is the same, the candidate distribution doesn't gain new
biology after the second pass). Defer to extended-paper revision.

## 2. T3-specific oracle?

**TL;DR**: We use the DeepSTARR-7cell oracle for T3 (and T1). A T3-
specific oracle is a paper-extension worth doing if reviewers push;
the minimum-publishable suite ships with the shared oracle.

### What a T3-specific oracle would predict

Two candidates:

* **"Edit-quality" oracle** — input: `(reference, edited, edit_distance,
  cell_type, edit_type)`. Output: `(satisfies_objective, margin)`.
  Trained on RFT-filtered (positive) + RFT-rejected (negative)
  candidates. Self-bootstrap risk: the oracle becomes circular if it
  only learns what the previous oracle's filter did.

* **"Pairing" oracle (transferred from T2)** — input:
  `(promoter, edited_enhancer, cell_type)`. Output: `pairing_score`.
  Trained on observed pairs (T2 dataset). For T3 we'd score `(promoter,
  edited)` and reward the model when this score increases over
  `(promoter, reference)`. This is biology-grounded and non-circular.

### Why we ship with DeepSTARR-7cell

The DeepSTARR-7cell oracle gives per-cell-type activity scores. For T3:

* `activity_boost` reduces to "did the activity in the source cell
  go up?" — `pred_activity_src > ref_activity_src`. Direct read from
  oracle.
* `cell_type_transfer` reduces to "did the activity shift toward target?"
  — `(pred_tgt - pred_src) - (ref_tgt - ref_src) > 0`. Direct read.
* `promoter_retarget` reduces to "is the new motif present?" — IUPAC
  scan, no oracle needed.

So **all three T3 objectives are computable from the existing oracle**
without a T3-specific one. The DeepSTARR-7cell oracle is weak
(`val_pearson_mean=0.136`) in absolute terms, but the metrics use
**deltas and shifts** — relative ranking, where weak oracles still
carry meaningful signal.

If the paper review pushes back on oracle weakness, the "T2-pairing-as-
T3-oracle" path is the right extension — concrete and publishable.

## 3. Loop-SFT — does it need T3-aware changes?

**TL;DR**: No code change. The data source for T3 trajectories should
swap to the post-RFT JSONL, but Loop-SFT itself is task-agnostic.

### What Loop-SFT does

`scripts/train_loop_sft.py` consumes JSONL records of the shape:

```
{"id": ..., "task_type": ...,
 "messages": [system, user],
 "trajectory": {"steps": [{kind, state, text, tool_name, tool_args, tool_result}, ...]}}
```

The collator renders the trajectory into a single assistant turn the
LLM is trained to emit. Task-type doesn't change anything — the
collator looks at trajectory steps, not task semantics.

### What changes for T3

The **trajectory dataset** for T3 is currently expanded from the
heuristic-gold T3 JSONL via `scripts/expand_loop_trajectories.py`. The
trajectory's final `kind="final"` step contains the gold enhancer
sequence — currently the heuristic synthetic-edit, **not** the
oracle-validated RFT candidate.

To align with v5:

```bash
# old:
python scripts/expand_loop_trajectories.py \
    --source data/prod_samples/train.enhancer_editing.strat7c.n35k.jsonl \
    --out    data/trajectories/train.enhancer_editing.jsonl

# new (post-RFT-aware):
python scripts/expand_loop_trajectories.py \
    --source runs/exp_t3_fusion_sft_${STAMP}/rft_filtered_train.jsonl \
    --out    data/trajectories/train.enhancer_editing.rft.jsonl
```

Same script, different `--source`. The expander reads the assistant's
gold answer from `messages[-1]["content"]`, which RFT replaced with
the candidate. So the trajectory's `final` step inherits the
candidate. No code change.

For the paper, we report Loop-SFT trained on:

* Heuristic-gold trajectories (matches the current pipeline default)
* **Post-RFT trajectories** (new; aligned with v5 evaluation)

These give two T3 Loop-SFT rows in Table 1 — a clean ablation.

## 4. SV-GSPO — does it need T3-aware changes?

**TL;DR**: **YES, and it's now done.** The T3 outcome-reward function
in `regureasoner/rl/reward_shaper.py` was trained on the **wrong**
objective (sequence-distance window) before this commit. Fixed.

### What was wrong

The previous `outcome_enhancer_editing(final, gold, min_edit=1,
max_edit=60)` returned 1.0 for any edit distance in `[1, 60]` and 0
for identity. That tells the agent "make a moderately-sized edit" —
which is exactly the wrong signal under the v5 framework where the
metric of record is *objective satisfaction*, not edit-distance window.

### What's changed

`outcome_enhancer_editing(final, gold)` now returns the average of three
binary checks aligned with `scripts/eval_t3_oracle.py`:

* `within_budget` — Hamming ≤ `gold["edit_budget"]` (or 5%-of-len(ref)
  fallback when no budget).
* `length_preserved` — `len(pred) == len(ref)`.
* `target_motif_present` — IUPAC regex (fwd + revcomp) for
  `gold["target_motif"]`.

Score ∈ {0, 1/3, 2/3, 1}. SV-GSPO group-normalisation handles the
discreteness; rollouts are expected to land between 0 and 1 with most
mass at 2/3 (budget + length usually pass; motif is the bottleneck).

### What's NOT done (deliberate)

The activity-based objectives (`activity_delta_src`,
`activity_relative_shift`) require an oracle forward-pass per rollout
— too slow for the hot RL loop. The evaluator-backed scorer
infrastructure in `reward_shaper.py` (the `OUTCOME_SCORERS_EVAL`
hook) is the right place to wire that in for **offline rescoring**
of completed rollouts; we'd then either include it as a `--reward-mode
oracle` flag or report both as a sanity check.

If reviewers push for "RL with the actual headline metric" we can:

1. Add an `OutcomeFnOracle` that takes the oracle as a closure and
   computes `activity_delta_src` per rollout.
2. Cache oracle outputs by `(seq, cell_type)` hash to amortise repeat
   eval across rollouts.

Estimated cost: ~50 LoC + a slow-but-correct ablation row.

## 5. External baselines — how do we compare against prior work?

**TL;DR**: Currently weak. Adding two strong external baselines (TACO
+ HyenaDNA / NT-v2) would harden the paper.

### What we have (internal)

* Zero-shot Qwen3.5-2B (raw + tool-enriched prompts)
* Fusion-SFT (NTv3-650M + LLM + cell context, our architecture)
* Loop-SFT (trajectory-augmented Fusion-SFT)
* SV-GSPO (RL on top of Loop-SFT)
* NTv3-MDLM T1, NTv3-direct T2 (no-LLM baselines, our architecture
  but no language model)

### What's missing (external SOTA)

| Model | Task fit | Comparable for | Effort to add | Priority |
|---|---|---|---|---|
| **TACO** (Lin et al. NeurIPS 2024) | T3 native | T3 (paper-precedent) | Medium — repo public; needs DeepSTARR oracle re-fit | **HIGH** |
| **HyenaDNA** (Nguyen et al. NeurIPS 2023) | T1 / T2 | T1 generation, T2 binary classification | Low — already wired as encoder; needs head training only | **HIGH** |
| **DNABERT-2 / NT-v2** | T1 / T2 / T3 | All three (small encoder baseline) | Low — `regureasoner_loop` has NT-v2 wired | MEDIUM |
| **CtrlDNA** | T1 conditional generation | T1 only | Medium — repo public, training data alignment needed | MEDIUM |
| **Evo / Evo2** | Generation, fluency | T1 (but they're 7B+, hard to run on H100) | High — vortex install on lab cluster | LOW |
| **Caduceus** | DNA encoder | Same as NT-v2; redundant | Low | LOW |
| **DeepSTARR** (predictor) | Activity prediction | Used as our oracle, NOT a baseline for our tasks | N/A | N/A |

**Recommended for the minimum-publishable submission**: add TACO (T3
paper precedent) + HyenaDNA (T2 fluency baseline). DNABERT-2 is
nice-to-have. The rest go into the extended-paper version.

The exact recipe:

* TACO: clone their repo, drop in our DeepSTARR-7cell oracle, run
  their trainer on our T3 train split. Eval with our
  `eval_t3_oracle.py`. Apples-to-apples.
* HyenaDNA: their HF model card already has T2-style heads. Wire as
  a `--encoder hyenadna` option in `run_genomefm_benchmark.py` and
  retrain the pair head.

Both ~1 day of work each.

## 6. Pipeline state — does any in-flight job need modification?

**No.** Audit:

* **Bench grid (in flight)** — vLLM zero-shot inference. T3 zs eval
  uses the heuristic gold's metadata only (target_motif, edit_budget)
  — no leakage from the v5 framework changes. Safe to let finish.
* **post_bench_pipeline.sh** — already updated with multi-turn RFT
  (commit `25504fd`) and Stage 3d post-RFT reasoning (commit `3e65c96`).
  Will pick up the changes when it auto-fires.
* **No fusion-SFT job is currently running.** Stages 1–4 fire only
  after the bench grid finishes.

The `outcome_enhancer_editing` reward fix lands in `regureasoner/rl/
reward_shaper.py` — used only by SV-GSPO (Stage C, not yet running on
H100). Lab-side SV-GSPO runs would need to pull this commit.

## 7. Concrete addition to the experiment suite

These lines belong in [`docs/minimal_publishable_suite.md`](minimal_publishable_suite.md)
once the H100 clears its current backlog:

```bash
# Phase B-T3-RFT-from-joint  (new ablation; 6h)
STAGE_4_FINAL=runs/exp_joint_multitask_${STAMP}/final/pytorch_model.bin
pixi run python scripts/rft_t3.py \
    --adapter-state-dict $STAGE_4_FINAL \
    --train-jsonl data/prod_samples/train.enhancer_editing.strat7c.n35k.jsonl \
    --oracle-path runs/exp_oracle_ds_7cell_min/oracle.pt \
    --output-jsonl runs/exp_t3_rft_from_joint_${STAMP}/rft_filtered_train.jsonl \
    --candidates 4 --rounds 4 --temp-ramp 0.15

# Phase B-T3-Loop-SFT-on-RFT  (new ablation; 4h)
pixi run python scripts/expand_loop_trajectories.py \
    --source runs/exp_t3_fusion_sft_${STAMP}/rft_filtered_train.jsonl \
    --out    data/trajectories/train.enhancer_editing.rft.jsonl
TASK=enhancer_editing \
TRAIN_JSONL=data/trajectories/train.enhancer_editing.rft.jsonl \
... \
bash slurm/run_train_loop_sft.sh
```

## TL;DR for the paper story

* **T3 dataset regen**: train labels, yes via RFT (in the pipeline).
  Test labels, no — eval ignores the heuristic gold.
* **T3 benchmark code**: clean. `eval_t3_oracle.py` is the new headline
  scorer; old `genqual.json` argmax_acc is informative-only.
* **T3 reward shaper**: fixed in this commit. SV-GSPO will now
  optimise the right objective.
* **Loop-SFT**: no code change; just point at the post-RFT JSONL.
* **External baselines**: TACO + HyenaDNA are the two we should add
  before submission. Both ~1 day each.