results/master_progress.md

# Master progress dashboard — 2026-04-27 ~04:40 UTC

Every experiment that's been run, is running, or is queued — across
H100 + lab cluster. Includes architecture-mode (LLaVA / unified-NTP /
unified-MDLM / diffusion) and contrastive (aux pair, aligner loss)
ablations.

## 1. Live processes (H100)

| PID | Job | Elapsed | ETA |
|---|---|---|---|
| 100474 | `launch_bench_vllm.sh` orchestrator | since Apr 26 | runs until last task completes |
| 137805 | T1 reasoning expansion (`build_reasoning_traces.py`) | 53 min | ~10 min remaining (281/333) |
| 139902 | T3 zs_raw vLLM bench | 35 min | ~4.5 h |
| 100544 | watcher → `post_bench_pipeline.sh` | idle | fires when bench grid exits |

## 2. T1 — enhancer_generation

| # | Variant | Host | n | parse | gc_err | len_ratio | Cells | Sample path |
|---|---|---|---|---|---|---|---|---|
| 1 | **zs_raw (full)** | H100 | 372,210 | 0.9996 | 0.116 | **1.64** | 7-cell | `runs/exp_t1_grid_separatedQA_20260426_h100_vllm_full/zs_raw/` |
| 2 | **zs_enriched (full)** | H100 | 372,210 | 0.9997 | 0.126 | **1.67** | 7-cell | `runs/exp_t1_grid_separatedQA_20260426_h100_vllm_full/zs_enriched/` |
| 3 | zs_raw TRUNCATED (max=200, **superseded**) | H100 | 372,210 | 0.9996 | 0.124 | 0.72 | 7-cell | `runs/exp_t1_grid_separatedQA_20260426_h100_vllm_TRUNCATED/zs_raw/` |
| 4 | zs_raw smoke (n=64 Ex) | lab | 64 | 1.0 | 0.093 | 1.83 | Ex | `_lab_results/runs/exp_t1_grid_separatedQA_20260424_154915/zs_raw/` |
| 5 | zs_enriched smoke (n=64) | lab | 64 | 1.0 | 0.096 | 1.62 | Ex | `…/zs_enriched/` |
| 6 | **lora_raw smoke [COLLAPSED]** | lab | 64 | 1.0 | 0.070 | **3.64** 🚨 | Ex | `…/lora_raw/` |
| 7 | **lora_enriched smoke [COLLAPSED]** | lab | 64 | 1.0 | 0.102 | **3.90** 🚨 | Ex | `…/lora_enriched/` |
| 8 | **fusion-SFT (Stage 1)** | H100 | — | — | — | — | — | **QUEUED** (auto post-bench) |
| 9 | **NTv3-MDLM (Stage 5)** | H100 | — | — | — | — | — | **QUEUED** |
| 10 | **Reasoning expansion (Tier 2)** | H100 | 281 / 333 | 0 leaks | rich rationales | — | 7-cell | `data/reasoning_traces/train.enhancer_generation.reasoning.jsonl` |

## 3. T2 — pair_prediction

| # | Variant | Host | n | accuracy | F1 | precision | recall | Cells | Sample path |
|---|---|---|---|---|---|---|---|---|---|
| 1 | **zs_raw (full)** | H100 | 744,420 | 0.500 | 0.0001 | 0.65 | ~0 | 7-cell | `runs/exp_t2_grid_separatedQA_20260426_h100_vllm_full/zs_raw/` |
| 2 | **zs_enriched (full)** | H100 | 744,420 | 0.500 | 0.002 | 0.58 | 0.001 | 7-cell | `runs/exp_t2_grid_separatedQA_20260426_h100_vllm_full/zs_enriched/` |
| 3 | asym pair NTv3+NT-v2 aux=**none** | lab | 128 | 0.773 | **0.808** | 0.701 | 0.953 | Ex | `_lab_results/runs/exp_t2_pair_aux_none_20260425_192434_prod/` |
| 4 | asym pair aux=**supcon_pair** | lab | 128 | 0.719 | 0.710 | 0.733 | 0.688 | Ex | `…/exp_t2_pair_aux_supcon_pair_20260425_192434_prod/` |
| 5 | asym pair aux=**tier_aware_supcon** | lab | 128 | 0.711 | 0.776 | 0.634 | **1.000** | Ex | `…/exp_t2_pair_aux_tier_aware_supcon_20260425_192434_prod/` |
| 6 | **fusion-SFT (Stage 2)** | H100 | — | — | — | — | — | — | **QUEUED** |
| 7 | **Galaxy regen (enhancer TFBS scan)** | lab/galaxy | — | — | — | — | — | — | **PROVISIONED** (lab patched script in `dec7a3e`); not yet launched |
| 8 | **Reasoning expansion (Tier 2)** | H100 | — | gated on #7 | — | — | — | — | **DEFERRED** (Stage 3e) |
| 9 | **NTv3-direct (Stage 6)** | H100 | — | — | — | — | — | — | **QUEUED** |

⚠️ **T2 zero-shot is degenerate**: model trivially predicts `not_paired` → recall ≈ 0. Tool-enriched gives marginal lift but the missing enhancer-side TFBS scan is the bottleneck. Lab's asym-pair smokes (n=128 Ex) reach F1=0.81 — proves the architecture works, full benchmark pending.

## 4. T3 — enhancer_editing

| # | Variant | Host | Status | Sample path |
|---|---|---|---|---|
| 1 | **zs_raw bench (full ~372k)** | H100 | **RUNNING** (PID 139902, 35 min in, ETA ~4.5 h) | `runs/exp_t3_grid_separatedQA_20260426_h100_vllm_full/zs_raw/` |
| 2 | zs_enriched bench (full ~372k) | H100 | queued behind #1 | same parent dir / `zs_enriched/` |
| 3 | fusion-SFT (Stage 3, heuristic gold) | H100 | queued (auto, post-bench) | `runs/exp_t3_fusion_sft_20260427_h100/` |
| 4 | **reasoning-only ablation (Stage 3b)** | H100 | queued | `runs/exp_t3_fusion_sft_reasonly_20260427_h100/` |
| 5 | **multi-turn RFT (Stage 3c, --rounds 4)** | H100 | queued | `runs/exp_t3_fusion_sft_rft_20260427_h100/` |
| 6 | **post-RFT reasoning expansion (Stage 3d)** | H100 | queued (gated on #5) | `data/reasoning_traces/train.enhancer_editing.reasoning.jsonl` |
| 7 | RFT-from-joint ablation | lab | proposed in `t3_post_v5_followups.md` §1 | — |
| 8 | Loop-SFT on post-RFT | lab | proposed | — |

## 5. Joint multitask (the headline)

| # | Variant | Host | n | Status | Path |
|---|---|---|---|---|---|
| 1 | **Joint multitask balanced 35k×3** (Stage 4) | H100 | 105k train | **QUEUED** | input: `data/prod_samples/train.joint_multitask_balanced.strat7c.n105k.jsonl` (992MB, 35k T1 + 35k T2 + 35k T3) |
| 2 | Score adapter on T1 raw / enriched | H100 | 372k | queued | `predict_t1_{raw,enriched}/genqual.json` |
| 3 | Score adapter on T2 raw / enriched | H100 | 744k | queued | `predict_t2_{raw,enriched}/metrics.json` |
| 4 | Score adapter on T3 raw / enriched | H100 | 372k | queued | `predict_t3_{raw,enriched}/genqual_t3_oracle.json` |

## 6. Architecture-mode ablation (Table 3 Phase 2 — `llava` vs `unified+ntp` vs `unified+mdlm` vs `diffusion`)

**Status: DEFERRED to lab cluster.**

The DNA-output-head ablation surface is wired (`scripts/train_fusion_sft.py
--architecture-mode {llava,unified,diffusion}`, `--dna-loss-kind {mdlm,ntp}`,
`--dna-loss-weight λ`) — see `docs/unified_multimodal_lm_survey.md` for the
survey behind it. Currently:

| Mode | Status | Where wired |
|---|---|---|
| `llava` (default; LLM head emits DNA as text tokens) | **In every fusion-SFT call on H100** | `slurm/post_bench_pipeline_h100_v5.sh` Stages 1/2/3/4 use `--architecture-mode llava` |
| `unified+ntp` (DNA head with plain CE on the DNA vocab) | **Wired but not launched** | `slurm/run_unified_arch_ablation.sh` |
| `unified+mdlm` (DNA head with LLaDA ELBO + 1/t reweight) | wired, not launched | same launcher |
| `diffusion` (LLaDA full diffusion) | **NOT YET WIRED** (per `train_fusion_sft.py:88`: "Phase 3 = diffusion (LLaDA, not yet wired)") | future |

**Lab action item** added: launch `slurm/run_unified_arch_ablation.sh`
on a non-H100 node (the H100 stays focused on the headline runs).
Three jobs in one sbatch: llava (control) / unified+ntp / unified+mdlm
on T1. ETA ~10h per arch on a lab GPU. Already documented in
`docs/minimal_publishable_suite.md §4e`.

## 7. Contrastive / aux-loss ablations

### 7a. T2 pair-aux contrastive (DONE — lab smoke)

3 variants (Table 1 sub-figure / Table 3 row), all smoke-tested at
n=128 Ex (rows 3–5 in the T2 table above). The full-set re-run will
fire after the galaxy regen lands so the new T2 enriched JSONL feeds
both the asym-pair model and the fusion-SFT stack.

### 7b. Aligner loss ablation (3 contrastive variants, T1 trimodal)

`slurm/run_aligner_loss_ablation.sh` — three loss variants
(infoNCE / supcon / tier-aware-supcon-style) for the trimodal aligner
(promoter↔enhancer↔expression). Status: **wired, not launched**. Lab
side. Documented in `docs/minimal_publishable_suite.md §4b`.

### 7c. Multi-encoder grid (NTv3 vs HyenaDNA vs Caduceus)

`slurm/run_multi_encoder_grid.sh` — DNA-encoder ablation at the T1 /
T2 layer. **Wired**; NTv3-650M is the current default everywhere.
Lab side, not launched.

## 8. Oracle + supporting infra

| Asset | Host | Status | Path |
|---|---|---|---|
| DeepSTARR-7cell oracle (`val_pearson_mean=0.136`, weak-but-aggregable) | lab | DONE | `_lab_results/runs/exp_oracle_ds_7cell_fdr_both_20260424_162210/oracle.pt` |
| Enformer oracle (Table 4 cross-oracle) | lab | not built | — |
| Sei oracle (Table 4) | lab | not built | — |
| Joint multitask balanced 105k JSONL | H100 | DONE (35k × 3 verified) | `data/prod_samples/train.joint_multitask_balanced.strat7c.n105k.jsonl` |
| Test JSONLs (T1+T2+T3 full) | H100 | DONE | `data/prod_full_test/jsonl/test.{enhancer_generation,pair_prediction,enhancer_editing}.jsonl` |

## 9. External baselines (the comparison gap)

| Model | Status | Priority | Doc |
|---|---|---|---|
| TACO (Lin et al. NeurIPS 2024) — T3 paper precedent | **NOT STARTED** | **HIGH** | `t3_post_v5_followups.md §5` |
| HyenaDNA — T2 fluency baseline | NOT STARTED | **HIGH** | same |
| DNABERT-2 / NT-v2 — encoder baselines | wired as encoders only; head not trained | MEDIUM | same |
| CtrlDNA — T1 conditional gen | NOT STARTED | MEDIUM | same |
| Evo / Evo2 — large fluency | NOT STARTED | LOW | same |

Lab action item: TACO + HyenaDNA, ~1 day each.

## 10. SV-GSPO (RL) + Loop-SFT — pipeline state

| Component | Status |
|---|---|
| SV-GSPO outcome reward for T3 (was buggy, **fixed in `e133cf1`**) | code synced, not yet trained |
| SV-GSPO ablation grid (Table 2: cost-aware / k₃-KL / DAPO / KL=0 / no-group-norm) | pipeline wired, **not yet launched** |
| Loop-SFT on heuristic-gold trajectories | pipeline wired, not launched |
| Loop-SFT on post-RFT trajectories (T3 only) | proposed in `t3_post_v5_followups.md §3` |

## 11. Branch + HF state

```
HEAD on mllm-integrate-server2: f304894 (merge lab's dec7a3e regen_t2 PYTHON_BIN fix)
                                4 commits ahead of mllm-integrate
                                0 commits behind  (lab fully caught up)

HF mirror: explcre/dnathinker-checkpoints (last push 04:10 UTC)
  runs/exp_t1_grid_*_full/zs_{raw,enriched}/metrics.json
  runs/exp_t2_grid_*_full/zs_{raw,enriched}/metrics.json
  data/reasoning_traces/train.enhancer_generation.reasoning.jsonl  (live, 281 rows)
  data/reasoning_traces/smoke_5rows_{t1,t2,t3}_postsanitize.jsonl
  data/reasoning_traces/post_rft_{contract_fixture,smoke}.jsonl
  docs/{lab_message_v2, t3_metrics_quickref, t3_post_v5_followups, experiment_chain_v5_unified}.md
  results/h100_snapshot.md
```

## 12. Total ETA to headline

| Step | Wall-clock |
|---|---|
| T3 zs_raw + enriched bench | ~10 h from now |
| Stage 0c oracle scoring on T3 zs preds | ~30 min after bench |
| Stages 1/2/3 + score-adapter (T1, T2, T3 fusion-SFT + per-cell oracle) | ~22 h |
| Stage 3b (T3 reasoning-only) | ~3 h |
| Stage 3c (T3 multi-turn RFT + retrain) | ~5 h |
| Stage 3d (T3 reasoning expansion 333 rows) | ~30 min IO-bound (in parallel with Stage 4) |
| Stage 3f (T1 reasoning) | continuous, +333/day |
| Stage 4 (joint multitask 105k) | ~10 h |
| Stages 5+6 (NTv3-only baselines) | ~4 h |
| Stage 7 (aggregator + final HF push) | minutes |

**Total H100 post-bench: ~36 h**. With lab cluster handling
arch-mode + aligner + contrastive + TACO + HyenaDNA in parallel,
the headline submission lands in **~3–4 days**.

## 13. Critical gates (what's blocking)

| Gate | Blocker | Unblocks |
|---|---|---|
| **Galaxy T2 enhancer regen** | lab launches `slurm/regen_t2_enriched_with_enhancer_scan.sh`; ~8h CPU | T2 bench rerun, T2 fusion-SFT, T2 reasoning expansion (Stage 3e), proper T2 row in Table 1 |
| **T3 RFT runs (Stage 3c)** | needs Stage 3 done | T3 reasoning expansion (Stage 3d), post-RFT row in Table 1 T3 column |
| **Reasoning accumulation (Tier 2)** | OpenRouter 1000/day cap per key | "Reasoning model" rows for T1 (now), T2 (after regen), T3 (after RFT). Multi-key parallel = lab side. |
| **TACO + HyenaDNA** | lab work | external baseline rows in Table 1 — reviewers will ask |