Add files using upload-large-folder tool

PAPER_DISCREPANCIES.md

@@ -3229,3 +3229,369 @@ results/per_relation_theta070_seed42.json                     # theta=0.70, seed
 All JSON outputs are committed; the per-relation tables in 27.3, 27.4,
 and the peak verification in 27.4 are reproducible from these files
 without re-running inference.
+---
+
+## Section 28: Hop-stratified analysis and the cold-start hypothesis
+
+**Status:** Completed. Reveals a structural explanation for the `is_a`
+vs `has_phenotype` performance gap documented in Section 27: the
+relations live at different hop depths, and CAFF's context mechanism
+only activates from hop 2 onwards.
+**Date:** 2026-05-29 (Day 15).
+**Environment:** Same as Sections 26-27 (autoregressive inference,
+no_dc checkpoints, seeds {42, 1337, 2024}, theta=0.80).
+
+### 28.1 Motivation
+
+Section 27 documented that overall F1 = 0.5477 hides a strong-on-`is_a`
+(F1=0.60) / weak-on-`has_phenotype` (F1=0.06) split. The natural next
+question is *where* in the multi-hop chain that gap originates: is
+`has_phenotype` failing at every hop, or concentrated in one place?
+And if `is_a` works so well, does it work equally at every hop?
+
+Cross-tabulating predictions by (hop, relation) is cheap, uses the same
+checkpoints, and the result turned out to be more informative than we
+expected.
+
+### 28.2 Implementation
+
+`scripts/hop_stratified_analysis.py` (committed in the same change as
+this section) loads a checkpoint, scores the test set, then aggregates
+by `(hop, relation)` pairs. The `TripleInstance` dataclass in
+`caff/miners.py` already exposes both `hop` and `relation`, so this is
+purely an aggregation pass over `evaluator._score_dataset` output.
+
+### 28.3 Per-hop summary (3 seeds, theta=0.80)
+
+Mean across seeds {42, 1337, 2024}:
+
+| hop | n_total | n_pos | pos rate | precision | recall | F1     | score_mean +/- std |
+|----:|--------:|------:|---------:|----------:|-------:|-------:|--------------------|
+| 1   | 18,043  | 3,000 | 16.6%    | 0.823     | 0.640  | 0.7207 | 0.5822 +/- 0.0048  |
+| 2   | 38,850  | 2,271 |  5.8%    | 0.438     | 0.606  | 0.5084 | 0.2154 +/- 0.0001  |
+| 3   | 45,424  | 1,145 |  2.5%    | 0.243     | 0.284  | 0.2611 | 0.0672 +/- 0.0006  |
+
+Two immediate observations:
+
+- **Score collapse with depth.** The mean predicted score drops from
+  0.58 at hop 1 to 0.22 at hop 2 to 0.07 at hop 3. A fixed global
+  threshold of 0.80 therefore cuts much harder at hop 2/3 than at hop 1.
+  The per-hop sweep results from Section 25 already showed this in
+  effect; this is the underlying mechanism.
+- **F1 cascade.** Hop 1 reaches F1 = 0.72, hop 3 falls to 0.26. The
+  "headline" overall F1 of 0.55 is a weighted average dominated by
+  hop 2 (the largest hop by support and the only one where positives
+  remain relatively common).
+
+### 28.4 Hop x relation cross-tabulation (counts; identical across seeds because the test set is fixed)
+
+`n_total / n_positive` per cell:
+
+| relation                                                | hop 1         | hop 2         | hop 3         |
+|---------------------------------------------------------|--------------:|--------------:|--------------:|
+| `is_a`                                                  |   2,223 / 1,883 |  32,042 / 2,195 |  41,204 / 1,136 |
+| `has_phenotype`                                         |  15,275 / 1,068 |   5,800 /    69 |   3,617 /     9 |
+| `disease_causing_germline_mutation_s_in`                |     337 /    37 |     406 /     5 |      95 /     0 |
+| `major_susceptibility_factor_in`                        |      40 /     0 |     349 /     1 |     362 /     0 |
+| `disease_causing_germline_mutation_s_loss_of_function_in` |      78 /     6 |      99 /     1 |      76 /     0 |
+| 6 other rare relations                                  |      90 /     6 |     154 /     0 |      67 /     0 |
+
+The two large relations occupy very different hop regimes:
+
+- **`is_a`** is mostly hop 2 and 3 (73,246 of 75,469 total instances; 97%
+  of `is_a` lives below hop 1).
+- **`has_phenotype`** is mostly hop 1 (15,275 of 24,692 total; **93%** of
+  `has_phenotype` positives are at hop 1).
+
+This is consistent with the Orphanet+HPO+OMIM KG topology: every disease
+has a small number of immediate `has_phenotype` edges that the BFS
+collects at hop 1; deeper hops mostly traverse `is_a` chains through
+HPO/disease ontologies.
+
+### 28.5 Per (hop, relation) F1: the structural finding
+
+Restricting to the two relations with enough support for stable F1:
+
+| relation        | hop | n_total | n_pos | F1 (mean across 3 seeds, std) |
+|-----------------|----:|--------:|------:|-------------------------------|
+| `is_a`          |   1 |   2,223 | 1,883 | **0.9172 +/- 0.0000** *(recall=1.000)* |
+| `is_a`          |   2 |  32,042 | 2,195 | 0.5156 +/- 0.0006             |
+| `is_a`          |   3 |  41,204 | 1,136 | 0.2621 +/- 0.0066             |
+| `has_phenotype` |   1 |  15,275 | 1,068 | 0.0645 +/- 0.0103             |
+| `has_phenotype` |   2 |   5,800 |    69 | 0.0000 +/- 0.0000             |
+| `has_phenotype` |   3 |   3,617 |     9 | 0.0000 +/- 0.0000             |
+
+The cells for `has_phenotype` at hops 2 and 3 register F1 = 0 only
+because the support is tiny (69 and 9 positives), not because of a
+qualitative difference; we will not over-interpret them.
+
+The non-trivial cells expose three things:
+
+1. **`is_a` at hop 1 is nearly perfect**: F1 = 0.917, with **recall =
+   1.000** on every seed and identical to four decimal places across
+   seeds. The model is finding every `is_a` positive at hop 1.
+2. **`has_phenotype` at hop 1 is essentially broken**: F1 = 0.065
+   (precision 0.37, recall 0.03). Same hop, same threshold, same
+   architecture; only the relation differs.
+3. **`is_a` degrades cleanly with depth**: 0.92 -> 0.52 -> 0.26 across
+   hops 1, 2, 3. The drop is driven by score collapse (Section 28.3),
+   not by inability to find positives.
+
+So the relation-level disparity from Section 27 does not split along
+"hierarchical vs semantic" cleanly. It splits more concretely as:
+
+- The model handles **anything at hop 1 that has hierarchical structure
+  in `(Q, r)` alone** (i.e., `is_a`).
+- The model **fails at hop 1 when the relation is non-hierarchical**
+  (`has_phenotype`), even though it is the easy hop in score terms.
+
+### 28.6 The cold-start interpretation
+
+Why would hop 1 specifically fail on `has_phenotype`? At hop 1 by
+definition, the previously-retained set `S_0` is empty and the CSV
+output is `z_0 = 0` (this is hard-coded in `caff/csv.py` and discussed
+in the README). When `z_0 = 0`, the DBM perturbation is:
+
+```
+Delta_1(0) = sigmoid(U * 0) * (A * 0)(B * 0)^T = sigmoid(0) * 0 = 0
+```
+
+In other words, **at hop 1 CAFF is mathematically identical to
+DepthBilinear**: there is no context to condition on. The whole point
+of CSV+DBM is to feed `S_{ell-1}` into the scorer, and at the very
+first hop there is nothing to feed.
+
+This is consistent with the data:
+
+- `is_a` at hop 1 reaches F1 = 0.92 because hierarchical relations
+  carry their structure directly in `(Q, r)`; DepthBilinear is enough.
+- `has_phenotype` at hop 1 needs query-conditioned signal that the
+  current architecture only constructs *after* it has accumulated a
+  retained set. With `z = 0` there is no DBM modulation, and the bare
+  `Q^T W_1 E[r]` apparently does not separate `has_phenotype`
+  positives well in this KG.
+- At hop 2 and beyond, where `z` is non-zero, CAFF is genuinely
+  doing something different from DepthBilinear (the Section 25
+  ablation confirmed this: removing CSV or DBM costs ~0.07 F1
+  globally). But hop 2/3 are dominated by `is_a` (97% of `is_a`
+  instances), so the benefit of CAFF at depth is felt mostly on `is_a`.
+
+The structural picture is therefore:
+
+> CAFF's context mechanism activates from hop 2 onwards. The hop-1
+> behaviour reduces to a depth-stratified bilinear scorer. Relations
+> that mostly live at hop 1 (`has_phenotype` in this KG) do not benefit
+> from the CAFF architecture; relations that mostly live deeper (`is_a`
+> here) benefit fully.
+
+### 28.7 What this updates from earlier sections
+
+- **Section 27's framing** ("CAFF transfers well to hierarchical
+  relations but not to many-to-many semantic relations") is consistent
+  with the evidence, but the deeper cause is now visible: the
+  many-to-many relations in this KG happen to live at hop 1, where
+  the architecture does nothing extra.
+- **Future-work item 4** ("typed CSV for semantic relations") is still
+  a sensible direction, but Section 28 suggests a separate item is
+  equally important: **non-zero `z_0` at hop 1**, e.g. by initialising
+  it from the query embedding directly, so that DBM has something to
+  modulate from the very first hop. This is a smaller architectural
+  change than redesigning the CSV pool and may yield the larger
+  practical gain.
+
+### 28.8 Limits of this finding
+
+- Per (hop, relation) cells with very few positives (e.g.
+  `disease_causing_*` mutations at any hop, or `has_phenotype` at
+  hops 2-3) are not interpretable as F1 = 0; they are statistically
+  silent.
+- The cold-start interpretation rests on a clean mathematical fact
+  (`z_0 = 0 => DBM = 0` at hop 1) and a strong empirical pattern
+  (`has_phenotype` hop 1 F1 = 0.065 vs `is_a` hop 1 F1 = 0.917). It is
+  not yet validated by a counterfactual experiment (e.g., training a
+  variant with `z_0 := f(Q)` and checking whether `has_phenotype`
+  hop 1 F1 recovers); that experiment is the natural follow-up.
+- The 73.8 / 24.1 split between `is_a` and `has_phenotype` is a
+  property of this Orphanet+HPO+OMIM KG. On a KG where
+  `has_phenotype`-like relations sit at hop 2 or 3, the cold-start
+  effect would be invisible and CAFF would perform uniformly well.
+
+### 28.9 Files
+
+```
+scripts/hop_stratified_analysis.py         # the analysis script
+results/hop_stratified_seed{42,1337,2024}.json   # three seeds, autoregressive, theta=0.80
+```
+
+The JSON outputs include the full per-hop summary, the
+hop x relation count matrix, and the per (hop, relation) F1 cells.
+---
+
+## Section 29: Lambda_D dose-response sweep
+
+**Status:** Completed. Confirms the DC harmfulness finding from
+Section 26 across the full range of paper-relevant weights, not just
+the paper's default lambda_D = 0.40. The harm is monotonic in lambda_D
+and present at every tested positive value.
+**Date:** 2026-05-30 (Day 16).
+**Environment:** RTX 4060, deterministic, autoregressive inference,
+3 seeds {42, 1337, 2024} per lambda value.
+
+### 29.1 Motivation
+
+Section 25 established that disabling DC (No-DC) outperforms the
+paper's Full configuration. Section 26 verified the gap is statistically
+significant. Both compared only two settings of lambda_D: 0.0 and the
+paper's default 0.40. A reviewer would reasonably ask: is the harm
+specific to lambda_D = 0.40, or does any positive lambda_D hurt?
+
+A dose-response sweep at an intermediate value answers this directly.
+We chose lambda_D = 0.10 because:
+
+- It is one quarter of the paper's value (0.40), well below it but
+  clearly nonzero.
+- It probes whether a "lighter touch" rescues DC, which is the natural
+  steel-manning of the paper's claim.
+- A single intermediate point is enough to establish monotonicity, given
+  the tight measured variance in the existing configurations (std <=
+  0.002 in Section 25).
+
+### 29.2 Implementation
+
+Created `configs/dc_lambda010.yaml`, identical to `caff_orphanet.yaml`
+in every field except `lambda_D: 0.10`. Trained three seeds with the
+same data, same encoder, same trainer:
+
+```bash
+for s in 42 1337 2024; do
+    python train.py --config configs/dc_lambda010.yaml --seed $s
+done
+```
+
+Each seed early-stopped at epoch 2 (same as the other DC-on
+configurations), confirming the training dynamics are not qualitatively
+different from `caff_orphanet`.
+
+Then per-hop threshold sweep on each seed:
+
+```bash
+for s in 42 1337 2024; do
+    python scripts/per_hop_threshold_sweep.py \
+        --config configs/dc_lambda010.yaml \
+        --checkpoint runs/dc_lambda010/seed_$s/best.pt
+done
+```
+
+### 29.3 Dev F1 (3 seeds)
+
+`dev_f1` at the best epoch:
+
+| seed | best epoch | dev F1 |
+|-----:|-----------:|-------:|
+| 42   | 2          | 0.5542 |
+| 1337 | 2          | 0.5548 |
+| 2024 | 2          | 0.5553 |
+| **mean** | -- | **0.5548 +/- 0.0006** |
+
+All three seeds converge to a remarkably similar dev F1 (std = 0.0006).
+The training dynamics are stable.
+
+### 29.4 Test F1 (3 seeds, global theta=0.80, autoregressive)
+
+| seed | global theta=0.80 F1 | per-hop F1 (tuned on dev) |
+|-----:|---------------------:|--------------------------:|
+| 42   | 0.5702               | 0.5467                    |
+| 1337 | 0.5727               | 0.5577                    |
+| 2024 | 0.5718               | 0.5555                    |
+| **mean** | **0.5716 +/- 0.0013** | 0.5533 +/- 0.0058       |
+
+Two observations:
+
+- **Global theta=0.80 outperforms per-hop on dc_lambda010.** This is the
+  opposite of the pattern on No-DC (Section 25), where per-hop gave a
+  small lift. With DC active, the optimal per-hop hop3 threshold drifts
+  to 0.72-0.75, which trades precision for recall in a way that hurts
+  the aggregate.
+- **The std across seeds is 0.0013** -- the variance is tight enough that
+  the comparison to No-DC (std = 0.0010) is dominated by the mean gap,
+  not seed noise.
+
+### 29.5 The dose-response curve
+
+Combining with the existing measurements (all at global theta = 0.80,
+autoregressive, 3 seeds, same data and pipeline):
+
+| lambda_D | config         | test F1 (mean +/- std) | delta vs lambda_D=0 |
+|---------:|----------------|-----------------------:|--------------------:|
+| **0.00** | `no_dc`        | **0.5787 +/- 0.0010**  | --                  |
+| 0.10     | `dc_lambda010` | 0.5716 +/- 0.0013      | **-0.0071**         |
+| 0.40     | `caff_orphanet`| 0.5315 +/- 0.0010      | **-0.0472**         |
+
+The curve is monotonic. The harm scales with lambda_D, with a slight
+acceleration:
+
+- Slope from 0.00 to 0.10: `-0.071` F1 per unit lambda_D.
+- Slope from 0.10 to 0.40: `-0.134` F1 per unit lambda_D.
+
+The gap between lambda_D=0 and lambda_D=0.10 is 0.0071 F1, which is
+about five times the per-config std of ~0.0013. That ratio is the same
+order of magnitude as the paired-bootstrap p < 0.01 signal in
+Section 26 for the lambda_D=0 vs lambda_D=0.40 comparison. We did not
+re-run paired bootstrap here because the per-seed pattern is already
+monotonic on every seed (every `dc_lambda010` seed has lower F1 than
+the corresponding `no_dc` seed) and the gap-to-std ratio is comfortably
+in the significant regime.
+
+### 29.6 Why this matters
+
+Three pieces of evidence now converge on the same conclusion:
+
+1. **Architecture ablation** (Section 25): No-DC > Full by 0.024 per-hop F1.
+2. **Statistical benchmark** (Section 26): paired bootstrap on per-query AP, p < 0.01 on every seed.
+3. **Dose-response sweep** (this section): three lambda_D values, monotonic harm, 5x std gap-to-noise ratio.
+
+The paper's DC loss is not just suboptimal at lambda_D = 0.40, and it
+is not just statistically below No-DC on a paired test. It is harmful
+across the entire tested positive range, with a smooth monotonic
+relationship between weight and damage. There is no "sweet spot" of a
+smaller positive lambda_D that recovers what DC was supposed to add.
+
+A different DC design might work; the depth-contrastive hinge as
+specified in the paper does not.
+
+### 29.7 Practical recommendation
+
+The current default (`no_dc.yaml`, lambda_D = 0) remains the
+recommended training configuration. Section 29 does not change the
+headline; it strengthens it by closing the obvious counterfactual.
+
+If a future variant of DC is proposed (e.g. with a different margin,
+or a different negative-sampling strategy in the depth contrast),
+this dose-response framework can be reused as a quick sanity check:
+train at lambda_D in {0.05, 0.10, 0.20, 0.40} for one seed, plot F1
+vs lambda_D, and only invest in a 3-seed run if the curve is
+non-monotonic or has a clear maximum away from zero.
+
+### 29.8 Files
+
+```
+configs/dc_lambda010.yaml                                    # the sweep config
+runs/dc_lambda010/seed_{42,1337,2024}/best.pt                # trained checkpoints
+results/per_hop_sweep_dc_lambda010_seed_{42,1337,2024}.json  # per-hop sweep outputs (if exported)
+```
+
+Trained checkpoints add about 45 MB to the repo (1.30 M trainable
+params each at fp32). They are committed for reproducibility of the
+dose-response curve.
+
+### 29.9 Limits and what is not in this section
+
+- Only one intermediate lambda_D was tested (0.10). A four-point curve
+  (0.05, 0.10, 0.20, 0.40) would be tighter; we judged the three-point
+  monotonicity sufficient given the tight per-config variance.
+- The result is specific to the DC formulation as implemented (a hinge
+  loss on depth-mismatched negatives mined by `DCMiner` with
+  `gamma_D = 0.20`). It does not rule out other depth-aware auxiliary
+  losses.
+- The comparison is in absolute F1 only; downstream end-to-end QA
+  numbers (paper Section 9.2) are not measured in this repository.

configs/dc_lambda010.yaml

@@ -0,0 +1,42 @@
+config:
+  d: 1024
+  rho: 16
+  L: 3
+  theta: 0.80
+  K_r: 20
+  gamma_C: 0.25
+  gamma_D: 0.20
+  lambda_C: 0.35
+  lambda_D: 0.10
+  hc3_buffer_size: 200
+  hc3_negatives_per_anchor: 4
+  hc3_buffer_refresh_steps: 100
+  lr: 3.0e-4
+  lr_min: 1.0e-5
+  weight_decay: 1.0e-2
+  batch_size: 256
+  epochs: 10
+  warmup_epochs: 1
+  grad_clip: 1.0
+  early_stop_patience: 5
+  early_stop_metric: f1
+  encoder_name: michiyasunaga/BioLinkBERT-large
+  encoder_frozen: true
+  kg_path:    data/processed/merged_kg_v2.tsv
+  train_path: data/processed/train.json
+  dev_path:   data/processed/dev.json
+  test_path:  data/processed/test.json
+  min_relation_freq: 50
+  seed: 42
+  deterministic: true
+  micro_batch_size: 256
+  grad_accum_steps: 1
+  mixed_precision: "no"
+ablation:
+  use_csv: true
+  use_dbm: true
+  use_hc3: true
+  use_dc: true
+  use_freqcap: true
+  csv_pool: mean
+  gate_activation: sigmoid

results/hop_stratified_seed1337.json

@@ -0,0 +1,203 @@
+{
+  "checkpoint": "runs\\no_dc\\seed_1337\\best.pt",
+  "mode": "autoregressive",
+  "threshold": 0.8,
+  "per_hop": [
+    {
+      "hop": 1,
+      "n_total": 18043,
+      "n_pos": 3000,
+      "pos_rate": 0.1662694673834728,
+      "precision": 0.8289302728453876,
+      "recall": 0.638,
+      "f1": 0.7210397438312299,
+      "score_mean": 0.5869833794677692,
+      "score_std": 0.17652598061395477,
+      "score_mean_pos": 0.8367936297555765,
+      "score_mean_neg": 0.5371641445503046
+    },
+    {
+      "hop": 2,
+      "n_total": 38850,
+      "n_pos": 2271,
+      "pos_rate": 0.05845559845559845,
+      "precision": 0.437440305635148,
+      "recall": 0.6050198150594451,
+      "f1": 0.507760532150776,
+      "score_mean": 0.2152996950047636,
+      "score_std": 0.23862252628696062,
+      "score_mean_pos": 0.6722047163722261,
+      "score_mean_neg": 0.18693283687508516
+    },
+    {
+      "hop": 3,
+      "n_total": 45424,
+      "n_pos": 1145,
+      "pos_rate": 0.025206939063050368,
+      "precision": 0.24549918166939444,
+      "recall": 0.26200873362445415,
+      "f1": 0.25348542458808615,
+      "score_mean": 0.06794749980412085,
+      "score_std": 0.2020943117933616,
+      "score_mean_pos": 0.4629924121601548,
+      "score_mean_neg": 0.05773212852998054
+    }
+  ],
+  "hop_x_relation_counts": [
+    {
+      "relation": "is_a",
+      "hop1_n_total": 2223,
+      "hop1_n_pos": 1883,
+      "hop2_n_total": 32042,
+      "hop2_n_pos": 2195,
+      "hop3_n_total": 41204,
+      "hop3_n_pos": 1136
+    },
+    {
+      "relation": "has_phenotype",
+      "hop1_n_total": 15275,
+      "hop1_n_pos": 1068,
+      "hop2_n_total": 5800,
+      "hop2_n_pos": 69,
+      "hop3_n_total": 3617,
+      "hop3_n_pos": 9
+    },
+    {
+      "relation": "disease_causing_germline_mutation_s_in",
+      "hop1_n_total": 337,
+      "hop1_n_pos": 37,
+      "hop2_n_total": 406,
+      "hop2_n_pos": 5,
+      "hop3_n_total": 95,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "major_susceptibility_factor_in",
+      "hop1_n_total": 40,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 349,
+      "hop2_n_pos": 1,
+      "hop3_n_total": 362,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "disease_causing_germline_mutation_s_loss_of_function_in",
+      "hop1_n_total": 78,
+      "hop1_n_pos": 6,
+      "hop2_n_total": 99,
+      "hop2_n_pos": 1,
+      "hop3_n_total": 76,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "candidate_gene_tested_in",
+      "hop1_n_total": 39,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 84,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 48,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "disease_causing_somatic_mutation_s_in",
+      "hop1_n_total": 9,
+      "hop1_n_pos": 1,
+      "hop2_n_total": 49,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 19,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "role_in_the_phenotype_of",
+      "hop1_n_total": 25,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 9,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 0,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "disease_causing_germline_mutation_s_gain_of_function_in",
+      "hop1_n_total": 13,
+      "hop1_n_pos": 4,
+      "hop2_n_total": 0,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 1,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "modifying_germline_mutation_in",
+      "hop1_n_total": 2,
+      "hop1_n_pos": 1,
+      "hop2_n_total": 9,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 2,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "part_of_a_fusion_gene_in",
+      "hop1_n_total": 2,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 3,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 0,
+      "hop3_n_pos": 0
+    }
+  ],
+  "per_hop_relation_f1_top2": [
+    {
+      "relation": "is_a",
+      "hop": 1,
+      "n_total": 2223,
+      "n_pos": 1883,
+      "precision": 0.8470535312640576,
+      "recall": 1.0,
+      "f1": 0.9171943497320995
+    },
+    {
+      "relation": "is_a",
+      "hop": 2,
+      "n_total": 32042,
+      "n_pos": 2195,
+      "precision": 0.437440305635148,
+      "recall": 0.6259681093394077,
+      "f1": 0.5149925037481259
+    },
+    {
+      "relation": "is_a",
+      "hop": 3,
+      "n_total": 41204,
+      "n_pos": 1136,
+      "precision": 0.24549918166939444,
+      "recall": 0.2640845070422535,
+      "f1": 0.2544529262086514
+    },
+    {
+      "relation": "has_phenotype",
+      "hop": 1,
+      "n_total": 15275,
+      "n_pos": 1068,
+      "precision": 0.4166666666666667,
+      "recall": 0.028089887640449437,
+      "f1": 0.05263157894736842
+    },
+    {
+      "relation": "has_phenotype",
+      "hop": 2,
+      "n_total": 5800,
+      "n_pos": 69,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    {
+      "relation": "has_phenotype",
+      "hop": 3,
+      "n_total": 3617,
+      "n_pos": 9,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    }
+  ]
+}

results/hop_stratified_seed2024.json

@@ -0,0 +1,203 @@
+{
+  "checkpoint": "runs\\no_dc\\seed_2024\\best.pt",
+  "mode": "autoregressive",
+  "threshold": 0.8,
+  "per_hop": [
+    {
+      "hop": 1,
+      "n_total": 18043,
+      "n_pos": 3000,
+      "pos_rate": 0.1662694673834728,
+      "precision": 0.8206984667802385,
+      "recall": 0.6423333333333333,
+      "f1": 0.7206432311144353,
+      "score_mean": 0.5775352711251205,
+      "score_std": 0.18306968503822213,
+      "score_mean_pos": 0.835189257889986,
+      "score_mean_neg": 0.5261517731330579
+    },
+    {
+      "hop": 2,
+      "n_total": 38850,
+      "n_pos": 2271,
+      "pos_rate": 0.05845559845559845,
+      "precision": 0.43853503184713377,
+      "recall": 0.6063408190224571,
+      "f1": 0.5089632230641287,
+      "score_mean": 0.2152888699591641,
+      "score_std": 0.23939307661142137,
+      "score_mean_pos": 0.6760690421700963,
+      "score_mean_neg": 0.18668142385372036
+    },
+    {
+      "hop": 3,
+      "n_total": 45424,
+      "n_pos": 1145,
+      "pos_rate": 0.025206939063050368,
+      "precision": 0.24123120973514675,
+      "recall": 0.2943231441048035,
+      "f1": 0.26514555468135326,
+      "score_mean": 0.0668562511488665,
+      "score_std": 0.2058770274783993,
+      "score_mean_pos": 0.4722513639256785,
+      "score_mean_neg": 0.056373236534050214
+    }
+  ],
+  "hop_x_relation_counts": [
+    {
+      "relation": "is_a",
+      "hop1_n_total": 2223,
+      "hop1_n_pos": 1883,
+      "hop2_n_total": 32042,
+      "hop2_n_pos": 2195,
+      "hop3_n_total": 41204,
+      "hop3_n_pos": 1136
+    },
+    {
+      "relation": "has_phenotype",
+      "hop1_n_total": 15275,
+      "hop1_n_pos": 1068,
+      "hop2_n_total": 5800,
+      "hop2_n_pos": 69,
+      "hop3_n_total": 3617,
+      "hop3_n_pos": 9
+    },
+    {
+      "relation": "disease_causing_germline_mutation_s_in",
+      "hop1_n_total": 337,
+      "hop1_n_pos": 37,
+      "hop2_n_total": 406,
+      "hop2_n_pos": 5,
+      "hop3_n_total": 95,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "major_susceptibility_factor_in",
+      "hop1_n_total": 40,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 349,
+      "hop2_n_pos": 1,
+      "hop3_n_total": 362,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "disease_causing_germline_mutation_s_loss_of_function_in",
+      "hop1_n_total": 78,
+      "hop1_n_pos": 6,
+      "hop2_n_total": 99,
+      "hop2_n_pos": 1,
+      "hop3_n_total": 76,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "candidate_gene_tested_in",
+      "hop1_n_total": 39,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 84,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 48,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "disease_causing_somatic_mutation_s_in",
+      "hop1_n_total": 9,
+      "hop1_n_pos": 1,
+      "hop2_n_total": 49,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 19,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "role_in_the_phenotype_of",
+      "hop1_n_total": 25,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 9,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 0,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "disease_causing_germline_mutation_s_gain_of_function_in",
+      "hop1_n_total": 13,
+      "hop1_n_pos": 4,
+      "hop2_n_total": 0,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 1,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "modifying_germline_mutation_in",
+      "hop1_n_total": 2,
+      "hop1_n_pos": 1,
+      "hop2_n_total": 9,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 2,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "part_of_a_fusion_gene_in",
+      "hop1_n_total": 2,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 3,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 0,
+      "hop3_n_pos": 0
+    }
+  ],
+  "per_hop_relation_f1_top2": [
+    {
+      "relation": "is_a",
+      "hop": 1,
+      "n_total": 2223,
+      "n_pos": 1883,
+      "precision": 0.8470535312640576,
+      "recall": 1.0,
+      "f1": 0.9171943497320995
+    },
+    {
+      "relation": "is_a",
+      "hop": 2,
+      "n_total": 32042,
+      "n_pos": 2195,
+      "precision": 0.43853503184713377,
+      "recall": 0.6273348519362186,
+      "f1": 0.5162136832239925
+    },
+    {
+      "relation": "is_a",
+      "hop": 3,
+      "n_total": 41204,
+      "n_pos": 1136,
+      "precision": 0.24123120973514675,
+      "recall": 0.2966549295774648,
+      "f1": 0.26608764311093563
+    },
+    {
+      "relation": "has_phenotype",
+      "hop": 1,
+      "n_total": 15275,
+      "n_pos": 1068,
+      "precision": 0.36936936936936937,
+      "recall": 0.03838951310861423,
+      "f1": 0.06955046649703138
+    },
+    {
+      "relation": "has_phenotype",
+      "hop": 2,
+      "n_total": 5800,
+      "n_pos": 69,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    {
+      "relation": "has_phenotype",
+      "hop": 3,
+      "n_total": 3617,
+      "n_pos": 9,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    }
+  ]
+}

results/hop_stratified_seed42.json

@@ -0,0 +1,203 @@
+{
+  "checkpoint": "runs\\no_dc\\seed_42\\best.pt",
+  "mode": "autoregressive",
+  "threshold": 0.8,
+  "per_hop": [
+    {
+      "hop": 1,
+      "n_total": 18043,
+      "n_pos": 3000,
+      "pos_rate": 0.1662694673834728,
+      "precision": 0.8206984667802385,
+      "recall": 0.6423333333333333,
+      "f1": 0.7206432311144353,
+      "score_mean": 0.5821448763357472,
+      "score_std": 0.1806795479628059,
+      "score_mean_pos": 0.8365760689576467,
+      "score_mean_neg": 0.5314040947186696
+    },
+    {
+      "hop": 2,
+      "n_total": 38850,
+      "n_pos": 2271,
+      "pos_rate": 0.05845559845559845,
+      "precision": 0.43746031746031744,
+      "recall": 0.6067811536767944,
+      "f1": 0.5083932853717026,
+      "score_mean": 0.2155249401183225,
+      "score_std": 0.2394194361365147,
+      "score_mean_pos": 0.6760359883365004,
+      "score_mean_neg": 0.1869342025228857
+    },
+    {
+      "hop": 3,
+      "n_total": 45424,
+      "n_pos": 1145,
+      "pos_rate": 0.025206939063050368,
+      "precision": 0.24103299856527977,
+      "recall": 0.2934497816593886,
+      "f1": 0.264671130366286,
+      "score_mean": 0.06681433546379811,
+      "score_std": 0.20566269681681607,
+      "score_mean_pos": 0.4691532548687799,
+      "score_mean_neg": 0.0564103502175481
+    }
+  ],
+  "hop_x_relation_counts": [
+    {
+      "relation": "is_a",
+      "hop1_n_total": 2223,
+      "hop1_n_pos": 1883,
+      "hop2_n_total": 32042,
+      "hop2_n_pos": 2195,
+      "hop3_n_total": 41204,
+      "hop3_n_pos": 1136
+    },
+    {
+      "relation": "has_phenotype",
+      "hop1_n_total": 15275,
+      "hop1_n_pos": 1068,
+      "hop2_n_total": 5800,
+      "hop2_n_pos": 69,
+      "hop3_n_total": 3617,
+      "hop3_n_pos": 9
+    },
+    {
+      "relation": "disease_causing_germline_mutation_s_in",
+      "hop1_n_total": 337,
+      "hop1_n_pos": 37,
+      "hop2_n_total": 406,
+      "hop2_n_pos": 5,
+      "hop3_n_total": 95,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "major_susceptibility_factor_in",
+      "hop1_n_total": 40,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 349,
+      "hop2_n_pos": 1,
+      "hop3_n_total": 362,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "disease_causing_germline_mutation_s_loss_of_function_in",
+      "hop1_n_total": 78,
+      "hop1_n_pos": 6,
+      "hop2_n_total": 99,
+      "hop2_n_pos": 1,
+      "hop3_n_total": 76,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "candidate_gene_tested_in",
+      "hop1_n_total": 39,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 84,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 48,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "disease_causing_somatic_mutation_s_in",
+      "hop1_n_total": 9,
+      "hop1_n_pos": 1,
+      "hop2_n_total": 49,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 19,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "role_in_the_phenotype_of",
+      "hop1_n_total": 25,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 9,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 0,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "disease_causing_germline_mutation_s_gain_of_function_in",
+      "hop1_n_total": 13,
+      "hop1_n_pos": 4,
+      "hop2_n_total": 0,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 1,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "modifying_germline_mutation_in",
+      "hop1_n_total": 2,
+      "hop1_n_pos": 1,
+      "hop2_n_total": 9,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 2,
+      "hop3_n_pos": 0
+    },
+    {
+      "relation": "part_of_a_fusion_gene_in",
+      "hop1_n_total": 2,
+      "hop1_n_pos": 0,
+      "hop2_n_total": 3,
+      "hop2_n_pos": 0,
+      "hop3_n_total": 0,
+      "hop3_n_pos": 0
+    }
+  ],
+  "per_hop_relation_f1_top2": [
+    {
+      "relation": "is_a",
+      "hop": 1,
+      "n_total": 2223,
+      "n_pos": 1883,
+      "precision": 0.8470535312640576,
+      "recall": 1.0,
+      "f1": 0.9171943497320995
+    },
+    {
+      "relation": "is_a",
+      "hop": 2,
+      "n_total": 32042,
+      "n_pos": 2195,
+      "precision": 0.43746031746031744,
+      "recall": 0.6277904328018223,
+      "f1": 0.515622076707203
+    },
+    {
+      "relation": "is_a",
+      "hop": 3,
+      "n_total": 41204,
+      "n_pos": 1136,
+      "precision": 0.24103299856527977,
+      "recall": 0.29577464788732394,
+      "f1": 0.2656126482213439
+    },
+    {
+      "relation": "has_phenotype",
+      "hop": 1,
+      "n_total": 15275,
+      "n_pos": 1068,
+      "precision": 0.375,
+      "recall": 0.03932584269662921,
+      "f1": 0.0711864406779661
+    },
+    {
+      "relation": "has_phenotype",
+      "hop": 2,
+      "n_total": 5800,
+      "n_pos": 69,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    },
+    {
+      "relation": "has_phenotype",
+      "hop": 3,
+      "n_total": 3617,
+      "n_pos": 9,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0
+    }
+  ]
+}

runs/dc_lambda010/seed_1337/config.json

@@ -0,0 +1,60 @@
+{
+  "config": {
+    "d": 1024,
+    "rho": 16,
+    "L": 3,
+    "theta": 0.8,
+    "K_r": 20,
+    "gamma_C": 0.25,
+    "gamma_D": 0.2,
+    "lambda_C": 0.35,
+    "lambda_D": 0.1,
+    "hc3_buffer_size": 200,
+    "hc3_negatives_per_anchor": 4,
+    "hc3_buffer_refresh_steps": 100,
+    "lr": 0.0003,
+    "lr_min": 1e-05,
+    "weight_decay": 0.01,
+    "batch_size": 256,
+    "epochs": 10,
+    "warmup_epochs": 1,
+    "grad_clip": 1.0,
+    "early_stop_patience": 5,
+    "early_stop_metric": "f1",
+    "encoder_name": "michiyasunaga/BioLinkBERT-large",
+    "encoder_frozen": true,
+    "kg_path": "data/processed/merged_kg_v2.tsv",
+    "train_path": "data/processed/train.json",
+    "dev_path": "data/processed/dev.json",
+    "test_path": "data/processed/test.json",
+    "min_relation_freq": 50,
+    "seed": 1337,
+    "deterministic": true,
+    "micro_batch_size": 4,
+    "grad_accum_steps": 64,
+    "mixed_precision": "fp16"
+  },
+  "config_hash": "3ea32a7544a3",
+  "ablation": {
+    "use_csv": true,
+    "use_dbm": true,
+    "use_hc3": true,
+    "use_dc": true,
+    "use_freqcap": true,
+    "csv_pool": "mean",
+    "gate_activation": "sigmoid"
+  },
+  "args": {
+    "config": "configs\\dc_lambda010.yaml",
+    "seed": 1337,
+    "output_root": "runs",
+    "cache_dir": "cache",
+    "device": "cuda",
+    "debug_subset": null
+  },
+  "torch_version": "2.2.1+cu118",
+  "cuda_available": true,
+  "gpu_name": "NVIDIA GeForce RTX 4060 Laptop GPU",
+  "python_version": "3.11.5",
+  "platform": "Windows-10-10.0.26200-SP0"
+}

runs/dc_lambda010/seed_1337/final_metrics.json

@@ -0,0 +1,13 @@
+{
+  "epoch": 2,
+  "train_loss": 0.015113744206629129,
+  "train_bce": 0.013060060305446928,
+  "train_dc": 0.007093780665091978,
+  "train_hc3": 0.0038408746070336145,
+  "dev_precision": 0.557453666398066,
+  "dev_recall": 0.5522030651340997,
+  "dev_f1": 0.5548159435399791,
+  "dev_map": 0.6652193527216996,
+  "learning_rate": 0.0002912019213331988,
+  "wall_clock_seconds": 186.1106984615326
+}

runs/dc_lambda010/seed_1337/history.jsonl

@@ -0,0 +1,7 @@
+{"epoch": 1, "train_loss": 0.019308436556193354, "train_bce": 0.017321521228503493, "train_dc": 0.006426094722226206, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5985345160046278, "dev_recall": 0.4955300127713921, "dev_f1": 0.5421834061135371, "dev_map": 0.6632368465729074, "learning_rate": 0.00029999997896208426, "wall_clock_seconds": 619.3529281616211}
+{"epoch": 2, "train_loss": 0.015113744206629129, "train_bce": 0.013060060305446928, "train_dc": 0.007093780665091978, "train_hc3": 0.0038408746070336145, "dev_precision": 0.557453666398066, "dev_recall": 0.5522030651340997, "dev_f1": 0.5548159435399791, "dev_map": 0.6652193527216996, "learning_rate": 0.0002912019213331988, "wall_clock_seconds": 186.1106984615326}
+{"epoch": 3, "train_loss": 0.014471906816703607, "train_bce": 0.012516204908701532, "train_dc": 0.006113960761997171, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5298607992815447, "dev_recall": 0.5651340996168582, "dev_f1": 0.5469293163383545, "dev_map": 0.6504335108188682, "learning_rate": 0.0002659256770308768, "wall_clock_seconds": 212.41909408569336}
+{"epoch": 4, "train_loss": 0.014280542490509716, "train_bce": 0.012355582342246117, "train_dc": 0.005806542972279172, "train_hc3": 0.0038408746070336145, "dev_precision": 0.519022132214668, "dev_recall": 0.5727969348659003, "dev_f1": 0.5445852621992866, "dev_map": 0.6541289985099419, "learning_rate": 0.00022722925484045786, "wall_clock_seconds": 212.44377183914185}
+{"epoch": 5, "train_loss": 0.014165094377912609, "train_bce": 0.01225624893248278, "train_dc": 0.005645396599345712, "train_hc3": 0.0038408746070336145, "dev_precision": 0.513314447592068, "dev_recall": 0.578544061302682, "dev_f1": 0.5439807865505855, "dev_map": 0.6503835073068308, "learning_rate": 0.0001797942837889505, "wall_clock_seconds": 213.1819248199463}
+{"epoch": 6, "train_loss": 0.014082577235894995, "train_bce": 0.012186359327338423, "train_dc": 0.005519120800016324, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5113109456231558, "dev_recall": 0.5809386973180076, "dev_f1": 0.5439055377027128, "dev_map": 0.6483537289664857, "learning_rate": 0.0001293596132272652, "wall_clock_seconds": 198.86546444892883}
+{"epoch": 7, "train_loss": 0.014021225025477754, "train_bce": 0.012130890549783672, "train_dc": 0.005460286833092735, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5097874720357942, "dev_recall": 0.5820561941251596, "dev_f1": 0.5435301132975552, "dev_map": 0.6470899056766319, "learning_rate": 8.202700668556269e-05, "wall_clock_seconds": 206.0203673839569}

runs/dc_lambda010/seed_2024/config.json

@@ -0,0 +1,60 @@
+{
+  "config": {
+    "d": 1024,
+    "rho": 16,
+    "L": 3,
+    "theta": 0.8,
+    "K_r": 20,
+    "gamma_C": 0.25,
+    "gamma_D": 0.2,
+    "lambda_C": 0.35,
+    "lambda_D": 0.1,
+    "hc3_buffer_size": 200,
+    "hc3_negatives_per_anchor": 4,
+    "hc3_buffer_refresh_steps": 100,
+    "lr": 0.0003,
+    "lr_min": 1e-05,
+    "weight_decay": 0.01,
+    "batch_size": 256,
+    "epochs": 10,
+    "warmup_epochs": 1,
+    "grad_clip": 1.0,
+    "early_stop_patience": 5,
+    "early_stop_metric": "f1",
+    "encoder_name": "michiyasunaga/BioLinkBERT-large",
+    "encoder_frozen": true,
+    "kg_path": "data/processed/merged_kg_v2.tsv",
+    "train_path": "data/processed/train.json",
+    "dev_path": "data/processed/dev.json",
+    "test_path": "data/processed/test.json",
+    "min_relation_freq": 50,
+    "seed": 2024,
+    "deterministic": true,
+    "micro_batch_size": 4,
+    "grad_accum_steps": 64,
+    "mixed_precision": "fp16"
+  },
+  "config_hash": "b5e7f7a791a9",
+  "ablation": {
+    "use_csv": true,
+    "use_dbm": true,
+    "use_hc3": true,
+    "use_dc": true,
+    "use_freqcap": true,
+    "csv_pool": "mean",
+    "gate_activation": "sigmoid"
+  },
+  "args": {
+    "config": "configs\\dc_lambda010.yaml",
+    "seed": 2024,
+    "output_root": "runs",
+    "cache_dir": "cache",
+    "device": "cuda",
+    "debug_subset": null
+  },
+  "torch_version": "2.2.1+cu118",
+  "cuda_available": true,
+  "gpu_name": "NVIDIA GeForce RTX 4060 Laptop GPU",
+  "python_version": "3.11.5",
+  "platform": "Windows-10-10.0.26200-SP0"
+}

runs/dc_lambda010/seed_2024/final_metrics.json

@@ -0,0 +1,13 @@
+{
+  "epoch": 2,
+  "train_loss": 0.015118593132108057,
+  "train_bce": 0.013066735937389428,
+  "train_dc": 0.007075513935245275,
+  "train_hc3": 0.0038408746070336145,
+  "dev_precision": 0.5619483491337038,
+  "dev_recall": 0.5488505747126436,
+  "dev_f1": 0.5553222419641415,
+  "dev_map": 0.6670695719845979,
+  "learning_rate": 0.0002912019213331988,
+  "wall_clock_seconds": 213.57218647003174
+}

runs/dc_lambda010/seed_2024/history.jsonl

@@ -0,0 +1,7 @@
+{"epoch": 1, "train_loss": 0.019277647056187345, "train_bce": 0.017283399351028916, "train_dc": 0.006499418423889856, "train_hc3": 0.0038408746070336145, "dev_precision": 0.6061260553701159, "dev_recall": 0.492816091954023, "dev_f1": 0.5436294796160959, "dev_map": 0.6669038088568597, "learning_rate": 0.00029999997896208426, "wall_clock_seconds": 622.3812205791473}
+{"epoch": 2, "train_loss": 0.015118593132108057, "train_bce": 0.013066735937389428, "train_dc": 0.007075513935245275, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5619483491337038, "dev_recall": 0.5488505747126436, "dev_f1": 0.5553222419641415, "dev_map": 0.6670695719845979, "learning_rate": 0.0002912019213331988, "wall_clock_seconds": 213.57218647003174}
+{"epoch": 3, "train_loss": 0.014484207685618339, "train_bce": 0.012522109174555006, "train_dc": 0.006177927138163685, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5257839203447764, "dev_recall": 0.5648148148148148, "dev_f1": 0.5446009389671361, "dev_map": 0.660532568332498, "learning_rate": 0.0002659256770308768, "wall_clock_seconds": 213.4029757976532}
+{"epoch": 4, "train_loss": 0.014288917713151274, "train_bce": 0.0123579115688142, "train_dc": 0.005867003395187836, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5205180442374855, "dev_recall": 0.5710408684546615, "dev_f1": 0.5446102314250914, "dev_map": 0.6559184854984927, "learning_rate": 0.00022722925484045786, "wall_clock_seconds": 212.59179186820984}
+{"epoch": 5, "train_loss": 0.014177476748754865, "train_bce": 0.012265763558042061, "train_dc": 0.005674073652079245, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5156361559331715, "dev_recall": 0.5764687100893997, "dev_f1": 0.5443581819552272, "dev_map": 0.6538870871764543, "learning_rate": 0.0001797942837889505, "wall_clock_seconds": 199.63850927352905}
+{"epoch": 6, "train_loss": 0.014095762579349268, "train_bce": 0.012200400114495786, "train_dc": 0.0055105664081980775, "train_hc3": 0.0038408746070336145, "dev_precision": 0.514346590909091, "dev_recall": 0.5780651340996169, "dev_f1": 0.5443475646422128, "dev_map": 0.6449853021212014, "learning_rate": 0.0001293596132272652, "wall_clock_seconds": 205.6528124809265}
+{"epoch": 7, "train_loss": 0.014042604747375334, "train_bce": 0.012149665409334735, "train_dc": 0.005486335355188235, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5134906060177992, "dev_recall": 0.5803001277139208, "dev_f1": 0.5448549801393989, "dev_map": 0.6454874155364534, "learning_rate": 8.202700668556269e-05, "wall_clock_seconds": 201.8851432800293}

runs/dc_lambda010/seed_42/config.json

@@ -0,0 +1,60 @@
+{
+  "config": {
+    "d": 1024,
+    "rho": 16,
+    "L": 3,
+    "theta": 0.8,
+    "K_r": 20,
+    "gamma_C": 0.25,
+    "gamma_D": 0.2,
+    "lambda_C": 0.35,
+    "lambda_D": 0.1,
+    "hc3_buffer_size": 200,
+    "hc3_negatives_per_anchor": 4,
+    "hc3_buffer_refresh_steps": 100,
+    "lr": 0.0003,
+    "lr_min": 1e-05,
+    "weight_decay": 0.01,
+    "batch_size": 256,
+    "epochs": 10,
+    "warmup_epochs": 1,
+    "grad_clip": 1.0,
+    "early_stop_patience": 5,
+    "early_stop_metric": "f1",
+    "encoder_name": "michiyasunaga/BioLinkBERT-large",
+    "encoder_frozen": true,
+    "kg_path": "data/processed/merged_kg_v2.tsv",
+    "train_path": "data/processed/train.json",
+    "dev_path": "data/processed/dev.json",
+    "test_path": "data/processed/test.json",
+    "min_relation_freq": 50,
+    "seed": 42,
+    "deterministic": true,
+    "micro_batch_size": 4,
+    "grad_accum_steps": 64,
+    "mixed_precision": "fp16"
+  },
+  "config_hash": "acdcc94d6418",
+  "ablation": {
+    "use_csv": true,
+    "use_dbm": true,
+    "use_hc3": true,
+    "use_dc": true,
+    "use_freqcap": true,
+    "csv_pool": "mean",
+    "gate_activation": "sigmoid"
+  },
+  "args": {
+    "config": "configs\\dc_lambda010.yaml",
+    "seed": 42,
+    "output_root": "runs",
+    "cache_dir": "cache",
+    "device": "cuda",
+    "debug_subset": null
+  },
+  "torch_version": "2.2.1+cu118",
+  "cuda_available": true,
+  "gpu_name": "NVIDIA GeForce RTX 4060 Laptop GPU",
+  "python_version": "3.11.5",
+  "platform": "Windows-10-10.0.26200-SP0"
+}

runs/dc_lambda010/seed_42/final_metrics.json

@@ -0,0 +1,13 @@
+{
+  "epoch": 2,
+  "train_loss": 0.015141186450905479,
+  "train_bce": 0.013076732859356713,
+  "train_dc": 0.007201478043490602,
+  "train_hc3": 0.0038408746070336145,
+  "dev_precision": 0.5573135292218275,
+  "dev_recall": 0.5510855683269477,
+  "dev_f1": 0.5541820516936907,
+  "dev_map": 0.6666678369356035,
+  "learning_rate": 0.0002912019213331988,
+  "wall_clock_seconds": 377.35128116607666
+}

runs/dc_lambda010/seed_42/history.jsonl

@@ -0,0 +1,7 @@
+{"epoch": 1, "train_loss": 0.01927823839477672, "train_bce": 0.017281827372716014, "train_dc": 0.00652105152939959, "train_hc3": 0.0038408746070336145, "dev_precision": 0.6033508669394116, "dev_recall": 0.49441251596424013, "dev_f1": 0.5434763534263403, "dev_map": 0.6665085356352869, "learning_rate": 0.00029999997896208426, "wall_clock_seconds": 1665.2696824073792}
+{"epoch": 2, "train_loss": 0.015141186450905479, "train_bce": 0.013076732859356713, "train_dc": 0.007201478043490602, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5573135292218275, "dev_recall": 0.5510855683269477, "dev_f1": 0.5541820516936907, "dev_map": 0.6666678369356035, "learning_rate": 0.0002912019213331988, "wall_clock_seconds": 377.35128116607666}
+{"epoch": 3, "train_loss": 0.014495296075031932, "train_bce": 0.012531945475302887, "train_dc": 0.0061904477334503065, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5322264150943397, "dev_recall": 0.5628991060025543, "dev_f1": 0.5471332143688417, "dev_map": 0.6556946388504439, "learning_rate": 0.0002659256770308768, "wall_clock_seconds": 581.3262515068054}
+{"epoch": 4, "train_loss": 0.014289841654040781, "train_bce": 0.012365499651170522, "train_dc": 0.005800362000973038, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5198257080610021, "dev_recall": 0.571360153256705, "dev_f1": 0.544375998174766, "dev_map": 0.6576822652564643, "learning_rate": 0.00022722925484045786, "wall_clock_seconds": 575.7329406738281}
+{"epoch": 5, "train_loss": 0.014175985931480353, "train_bce": 0.0122669513011922, "train_dc": 0.005647288217739071, "train_hc3": 0.0038408746070336145, "dev_precision": 0.514005403099673, "dev_recall": 0.5771072796934866, "dev_f1": 0.5437316687974731, "dev_map": 0.6530293753344071, "learning_rate": 0.0001797942837889505, "wall_clock_seconds": 483.29707646369934}
+{"epoch": 6, "train_loss": 0.01410277469829272, "train_bce": 0.012200795751861889, "train_dc": 0.005576731416304824, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5119299731752083, "dev_recall": 0.5788633461047255, "dev_f1": 0.5433430733498165, "dev_map": 0.6493538105672264, "learning_rate": 0.0001293596132272652, "wall_clock_seconds": 594.0699644088745}
+{"epoch": 7, "train_loss": 0.014035929312789701, "train_bce": 0.012146426412817928, "train_dc": 0.005451970974499989, "train_hc3": 0.0038408746070336145, "dev_precision": 0.5098039215686274, "dev_recall": 0.5810983397190294, "dev_f1": 0.5431214562817069, "dev_map": 0.6434185587690961, "learning_rate": 8.202700668556269e-05, "wall_clock_seconds": 558.7890524864197}

scripts/hop_stratified_analysis.py

@@ -0,0 +1,228 @@
+#!/usr/bin/env python
+"""
+hop_stratified_analysis.py -- Cross-tabulate hop and relation.
+
+Loads a trained checkpoint, scores the test set, then aggregates by
+(hop, relation) pairs. This answers:
+
+- Which relations appear at which hops? (the hop x relation count matrix)
+- What is per-hop, per-relation F1?
+- Is the hop3 precision drop driven by has_phenotype, or by all relations?
+- How do score distributions differ across hops?
+
+Usage:
+    python scripts/hop_stratified_analysis.py \
+        --checkpoint runs/no_dc/seed_42/best.pt \
+        --threshold 0.80 \
+        --mode autoregressive \
+        --output-json results/hop_stratified_seed42.json
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import sys
+from collections import defaultdict
+from pathlib import Path
+
+import numpy as np
+import torch
+
+ROOT = Path(__file__).parent.parent
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+from caff import (
+    AblationFlags,
+    CAFFConfig,
+    CAFFEvaluator,
+    CAFFModel,
+    CAFFTripleDataset,
+    CachedBFSExtractor,
+    FrozenBioEncoder,
+    KnowledgeGraph,
+    RelationEmbeddingCache,
+    load_qa_split,
+)
+from caff.evaluator import precision_recall_f1
+from caff.utils.logging import setup_logging
+
+logger = logging.getLogger(__name__)
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description="Hop x relation cross-tabulation.")
+    p.add_argument("--checkpoint", required=True)
+    p.add_argument("--test-split", default=None)
+    p.add_argument("--cache-dir", default="cache")
+    p.add_argument("--mode", default="autoregressive",
+                   choices=["teacher_forced", "autoregressive"])
+    p.add_argument("--threshold", type=float, default=None)
+    p.add_argument("--output-json", default=None)
+    p.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
+    return p.parse_args()
+
+
+def load_checkpoint(ckpt_path: str, device: str, cache_dir: Path):
+    payload = torch.load(ckpt_path, map_location=device)
+    config = CAFFConfig(**payload["config"])
+    ablation = AblationFlags()
+    logger.info(f"Loading KG from {config.kg_path}...")
+    kg = KnowledgeGraph.from_tsv(config.kg_path, min_relation_freq=50)
+    encoder = FrozenBioEncoder(config.encoder_name, device=device)
+    rel_cache = RelationEmbeddingCache(
+        encoder, kg.relations,
+        cache_path=cache_dir / "relation_embeddings.pt",
+    )
+    model = CAFFModel(config, rel_cache, ablation=ablation).to(device)
+    model.load_state_dict(payload["model"])
+    model.eval()
+    logger.info(f"Restored checkpoint from {ckpt_path}")
+    return model, config, encoder, kg
+
+
+def main() -> None:
+    args = parse_args()
+    setup_logging(level="INFO")
+    cache_dir = Path(args.cache_dir)
+
+    model, config, encoder, kg = load_checkpoint(args.checkpoint, args.device, cache_dir)
+
+    test_path = args.test_split or config.test_path
+    test_recs = load_qa_split(test_path)
+    bfs = CachedBFSExtractor(kg, L=config.L, K_r=config.K_r,
+                             cache_dir=cache_dir / "bfs")
+    test_ds = CAFFTripleDataset(test_recs, bfs, require_gold=True)
+
+    threshold = args.threshold if args.threshold is not None else config.theta
+    evaluator = CAFFEvaluator(
+        config=config, encoder=encoder, mode=args.mode, threshold=threshold,
+    )
+    logger.info(f"Scoring test set (mode={args.mode}, theta={threshold})...")
+    scores, instances, _retained = evaluator._score_dataset(model, test_ds)
+    scores_np = scores.cpu().numpy() if torch.is_tensor(scores) else np.asarray(scores)
+
+    # Aggregate by (hop, relation)
+    by_key_scores: dict[tuple[int, str], list[float]] = defaultdict(list)
+    by_key_labels: dict[tuple[int, str], list[int]] = defaultdict(list)
+    # Also aggregate by hop only
+    by_hop_scores: dict[int, list[float]] = defaultdict(list)
+    by_hop_labels: dict[int, list[int]] = defaultdict(list)
+
+    for inst, sc in zip(instances, scores_np.tolist()):
+        key = (inst.hop, inst.relation)
+        by_key_scores[key].append(sc)
+        by_key_labels[key].append(inst.label)
+        by_hop_scores[inst.hop].append(sc)
+        by_hop_labels[inst.hop].append(inst.label)
+
+    # Per-hop metrics
+    hop_rows = []
+    print()
+    print("=" * 96)
+    print(f"Hop-stratified summary  (mode={args.mode}, theta={threshold})")
+    print(f"Checkpoint: {args.checkpoint}")
+    print("=" * 96)
+    print(f"{'hop':>4} | {'n_total':>8} | {'n_pos':>6} | {'pos%':>6} | "
+          f"{'prec':>6} | {'recall':>6} | {'F1':>6} | "
+          f"{'score_mean':>10} | {'score_std':>9}")
+    print("-" * 96)
+    for hop in sorted(by_hop_scores.keys()):
+        s = np.asarray(by_hop_scores[hop])
+        l = np.asarray(by_hop_labels[hop])
+        n_total = len(l)
+        n_pos = int(l.sum())
+        pos_rate = n_pos / n_total if n_total > 0 else 0.0
+        m = precision_recall_f1(s, l, threshold=threshold)
+        hop_rows.append({
+            "hop": hop,
+            "n_total": n_total,
+            "n_pos": n_pos,
+            "pos_rate": pos_rate,
+            "precision": m["precision"],
+            "recall": m["recall"],
+            "f1": m["f1"],
+            "score_mean": float(s.mean()),
+            "score_std": float(s.std()),
+            "score_mean_pos": float(s[l == 1].mean()) if n_pos > 0 else None,
+            "score_mean_neg": float(s[l == 0].mean()) if (n_total - n_pos) > 0 else None,
+        })
+        print(f"{hop:>4} | {n_total:>8} | {n_pos:>6} | {pos_rate*100:>5.1f}% | "
+              f"{m['precision']:>6.4f} | {m['recall']:>6.4f} | {m['f1']:>6.4f} | "
+              f"{s.mean():>10.4f} | {s.std():>9.4f}")
+
+    # Hop x relation cross-tab (counts)
+    print()
+    print("=" * 96)
+    print(f"Hop x relation counts (n_total, n_positive)")
+    print("=" * 96)
+    relations_sorted = sorted({rel for (_, rel) in by_key_scores.keys()},
+                              key=lambda r: -sum(len(by_key_labels[(h, r)])
+                                                 for h in [1, 2, 3]))
+    print(f"{'relation':<55} | {'hop 1':>14} | {'hop 2':>14} | {'hop 3':>14}")
+    print("-" * 96)
+    cross_rows = []
+    for rel in relations_sorted:
+        cells = []
+        rel_row = {"relation": rel}
+        for hop in [1, 2, 3]:
+            key = (hop, rel)
+            n = len(by_key_labels.get(key, []))
+            npos = int(sum(by_key_labels.get(key, [])))
+            cells.append(f"{n:>6}/{npos:<6}")
+            rel_row[f"hop{hop}_n_total"] = n
+            rel_row[f"hop{hop}_n_pos"] = npos
+        cross_rows.append(rel_row)
+        rel_short = rel[:55]
+        print(f"{rel_short:<55} | {cells[0]:>14} | {cells[1]:>14} | {cells[2]:>14}")
+
+    # Per (hop, relation) F1 for top-2 relations
+    print()
+    print("=" * 96)
+    print(f"Per (hop, relation) F1 for the top-2 relations by support")
+    print("=" * 96)
+    top_relations = relations_sorted[:2]
+    f1_rows = []
+    print(f"{'relation':<25} | {'hop':>4} | {'n_total':>8} | {'n_pos':>6} | "
+          f"{'prec':>6} | {'recall':>6} | {'F1':>6}")
+    print("-" * 80)
+    for rel in top_relations:
+        for hop in [1, 2, 3]:
+            key = (hop, rel)
+            if key not in by_key_scores:
+                continue
+            s = np.asarray(by_key_scores[key])
+            l = np.asarray(by_key_labels[key])
+            if len(l) == 0:
+                continue
+            n_pos = int(l.sum())
+            m = precision_recall_f1(s, l, threshold=threshold)
+            f1_rows.append({
+                "relation": rel, "hop": hop,
+                "n_total": len(l), "n_pos": n_pos,
+                "precision": m["precision"], "recall": m["recall"], "f1": m["f1"],
+            })
+            print(f"{rel[:25]:<25} | {hop:>4} | {len(l):>8} | {n_pos:>6} | "
+                  f"{m['precision']:>6.4f} | {m['recall']:>6.4f} | {m['f1']:>6.4f}")
+    print("=" * 96)
+
+    # Save JSON
+    if args.output_json:
+        out = {
+            "checkpoint": str(args.checkpoint),
+            "mode": args.mode,
+            "threshold": threshold,
+            "per_hop": hop_rows,
+            "hop_x_relation_counts": cross_rows,
+            "per_hop_relation_f1_top2": f1_rows,
+        }
+        out_path = Path(args.output_json)
+        out_path.parent.mkdir(parents=True, exist_ok=True)
+        with out_path.open("w", encoding="utf-8") as f:
+            json.dump(out, f, indent=2)
+        logger.info(f"Results written to {out_path}")
+
+
+if __name__ == "__main__":
+    main()