README: document RelBench leaderboard checkpoints
#3
by rishabh-ranjan - opened
README.md
CHANGED
|
@@ -64,4 +64,23 @@ The architecture and training loop build on the [Relational Transformer](https:/
|
|
| 64 |
huggingface-cli download kvignesh1420/relational-transformer-plurel \
|
| 65 |
--repo-type model \
|
| 66 |
--local-dir ~/scratch/rt_hf_ckpts
|
| 67 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
huggingface-cli download kvignesh1420/relational-transformer-plurel \
|
| 65 |
--repo-type model \
|
| 66 |
--local-dir ~/scratch/rt_hf_ckpts
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
## RelBench leaderboard checkpoints (added 2026-06)
|
| 71 |
+
|
| 72 |
+
These files back the PluRel numbers on the [RelBench leaderboard](https://relbench.stanford.edu/leaderboard/).
|
| 73 |
+
Protocols follow the repo's continued-pretraining script (50k steps, batch 128,
|
| 74 |
+
lr 5e-4 cosine, from `synthetic-pretrain_rdb_1024_size_4b.pt`) and the RT
|
| 75 |
+
`example_finetune` protocol (lr 1e-4, batch 32, 2^15+1 steps), with regression
|
| 76 |
+
best-checkpoint selection by val NMAE (MAE / train-split std, ddof=1) — the
|
| 77 |
+
leaderboard metric — instead of R². Evaluation = full official test split.
|
| 78 |
+
|
| 79 |
+
- `cntd-pretrain_<db>_<task>.pt` — synthetic+real continued pretraining, one
|
| 80 |
+
leave-one-DB-out run per database (incl. rel-event), per-task best checkpoint.
|
| 81 |
+
These produce the "PluRel | synthetic + real" zero-shot regression and
|
| 82 |
+
rel-event cells.
|
| 83 |
+
- `leaderboard-finetuned/finetune_<db>_<task>.pt` — fine-tuned from the matching
|
| 84 |
+
`cntd-pretrain` checkpoint (chosen over synthetic-only by val zero-shot, which
|
| 85 |
+
the synthetic+real checkpoint won on every task). These produce the
|
| 86 |
+
"PluRel | pretrained + fine-tuned" leaderboard row.
|