Merge stanford-star/rt-p into this repo; move original PluRel-paper checkpoints under paper/
2ed96ef verified | datasets: | |
| - kvignesh1420/plurel | |
| library_name: pytorch | |
| license: mit | |
| pipeline_tag: tabular-classification | |
| metrics: | |
| - roc_auc | |
| - r_squared | |
| tags: | |
| - relational-data | |
| - tabular | |
| - foundation-model | |
| - pretraining | |
| - relational-transformer | |
| - relbench | |
| - synthetic-data | |
| - tabular-regression | |
| > **Note (2026-08):** these are the original PluRel-paper checkpoints, moved from this repository's root into `paper/` when the current RT-PluRel checkpoint pair (formerly `stanford-star/rt-p`) was merged into this repository. See the root `README.md`. | |
| # Relational Transformer — PluRel Checkpoints | |
| Relational Transformer (RT) model checkpoints pretrained on synthetic relational databases generated by [PluRel](https://huggingface.co/datasets/kvignesh1420/plurel). | |
| Relational Transformer is a foundation model architecture for relational data that enables zero-shot transfer across heterogeneous schemas and tasks. It was introduced in: | |
| > **Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data** | |
| > Rishabh Ranjan, Valter Hudovernik, Mark Znidar, Charilaos Kanatsoulis, Roshan Upendra, Mahmoud Mohammadi, Joe Meyer, Tom Palczewski, Carlos Guestrin, Jure Leskovec — [arXiv:2510.06377](https://arxiv.org/abs/2510.06377) (ICLR 2026) | |
| The checkpoints provided in this repository were trained using the methodology described in: | |
| > **PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models** | |
| > Kothapalli, Ranjan, Hudovernik, Dwivedi, Hoffart, Guestrin, Leskovec — [arXiv:2602.04029](https://arxiv.org/abs/2602.04029) (2026) | |
| [](https://arxiv.org/abs/2510.06377) | |
| [](https://github.com/snap-stanford/relational-transformer) | |
| [](https://arxiv.org/abs/2602.04029) | |
| [](https://snap-stanford.github.io/plurel/) | |
| [](https://github.com/snap-stanford/plurel) | |
| [](https://huggingface.co/datasets/kvignesh1420/plurel) | |
| --- | |
| ## Model Architecture | |
| The Relational Transformer operates on multi-tabular relational databases, treating rows across linked tables as a sequence via BFS-ordered context sampling. It utilizes a Relational Attention mechanism over columns, rows, and primary-foreign key links. | |
| | Hyperparameter | Value | | |
| |----------------|-------| | |
| | Transformer blocks | 12 | | |
| | Model dimension (`d_model`) | 256 | | |
| | Attention heads | 8 | | |
| | FFN dimension (`d_ff`) | 1,024 | | |
| | Context length | 1,024 tokens | | |
| | Text encoder | `all-MiniLM-L12-v2` (d_text = 384) | | |
| | Max BFS width | 128 | | |
| The architecture and training loop build on the [Relational Transformer](https://github.com/snap-stanford/relational-transformer) codebase. | |
| --- | |
| ## Download | |
| Single checkpoint (Python) — fetch `config.json` alongside the weights; it carries | |
| the model architecture and is what the Hub uses to count downloads: | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| config = hf_hub_download("stanford-star/rt-plurel", "paper/config.json") | |
| ckpt = hf_hub_download("stanford-star/rt-plurel", "paper/synthetic-pretrain_rdb_1024_size_4b.pt") | |
| ``` | |
| Full repository (CLI): | |
| ```bash | |
| hf download stanford-star/rt-plurel \ | |
| --repo-type model \ | |
| --local-dir ~/scratch/rt_hf_ckpts | |
| ``` | |
| ## RelBench leaderboard checkpoints (added 2026-06) | |
| Protocols follow the repo's continued-pretraining script (50k steps, batch 128, | |
| lr 5e-4 cosine, from `synthetic-pretrain_rdb_1024_size_4b.pt`) and the RT | |
| `example_finetune` protocol (lr 1e-4, batch 32, 2^15+1 steps), with regression | |
| best-checkpoint selection by val NMAE (MAE / train-split std, ddof=1) — the | |
| leaderboard metric — instead of R². Evaluation = full official test split. | |
| - `cntd-pretrain_<db>_<task>.pt` — synthetic+real continued pretraining, one | |
| leave-one-DB-out run per database (incl. rel-event), per-task best checkpoint. | |
| These produce the "PluRel | synthetic + real" zero-shot regression and | |
| rel-event cells. | |
| - `finetune_<db>_<task>.pt` — fine-tuned from the matching | |
| `cntd-pretrain` checkpoint (chosen over synthetic-only by val zero-shot, which | |
| the synthetic+real checkpoint won on every task). These produce the | |
| "PluRel | pretrained + fine-tuned" leaderboard row. |