diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..7cdbe0b482f604a06a0988dad8877ae8d9257f7d --- /dev/null +++ b/LICENSE @@ -0,0 +1,438 @@ +Attribution-NonCommercial-ShareAlike 4.0 International + +======================================================================= + +Creative Commons Corporation ("Creative Commons") is not a law firm and +does not provide legal services or legal advice. Distribution of +Creative Commons public licenses does not create a lawyer-client or +other relationship. Creative Commons makes its licenses and related +information available on an "as-is" basis. Creative Commons gives no +warranties regarding its licenses, any material licensed under their +terms and conditions, or any related information. Creative Commons +disclaims all liability for damages resulting from their use to the +fullest extent possible. + +Using Creative Commons Public Licenses + +Creative Commons public licenses provide a standard set of terms and +conditions that creators and other rights holders may use to share +original works of authorship and other material subject to copyright +and certain other rights specified in the public license below. The +following considerations are for informational purposes only, are not +exhaustive, and do not form part of our licenses. + + Considerations for licensors: Our public licenses are + intended for use by those authorized to give the public + permission to use material in ways otherwise restricted by + copyright and certain other rights. Our licenses are + irrevocable. Licensors should read and understand the terms + and conditions of the license they choose before applying it. + Licensors should also secure all rights necessary before + applying our licenses so that the public can reuse the + material as expected. Licensors should clearly mark any + material not subject to the license. This includes other CC- + licensed material, or material used under an exception or + limitation to copyright. More considerations for licensors: + wiki.creativecommons.org/Considerations_for_licensors + + Considerations for the public: By using one of our public + licenses, a licensor grants the public permission to use the + licensed material under specified terms and conditions. If + the licensor's permission is not necessary for any reason--for + example, because of any applicable exception or limitation to + copyright--then that use is not regulated by the license. Our + licenses grant only permissions under copyright and certain + other rights that a licensor has authority to grant. Use of + the licensed material may still be restricted for other + reasons, including because others have copyright or other + rights in the material. A licensor may make special requests, + such as asking that all changes be marked or described. + Although not required by our licenses, you are encouraged to + respect those requests where reasonable. More considerations + for the public: + wiki.creativecommons.org/Considerations_for_licensees + +======================================================================= + +Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International +Public License + +By exercising the Licensed Rights (defined below), You accept and agree +to be bound by the terms and conditions of this Creative Commons +Attribution-NonCommercial-ShareAlike 4.0 International Public License +("Public License"). To the extent this Public License may be +interpreted as a contract, You are granted the Licensed Rights in +consideration of Your acceptance of these terms and conditions, and the +Licensor grants You such rights in consideration of benefits the +Licensor receives from making the Licensed Material available under +these terms and conditions. + + +Section 1 -- Definitions. + + a. Adapted Material means material subject to Copyright and Similar + Rights that is derived from or based upon the Licensed Material + and in which the Licensed Material is translated, altered, + arranged, transformed, or otherwise modified in a manner requiring + permission under the Copyright and Similar Rights held by the + Licensor. For purposes of this Public License, where the Licensed + Material is a musical work, performance, or sound recording, + Adapted Material is always produced where the Licensed Material is + synched in timed relation with a moving image. + + b. Adapter's License means the license You apply to Your Copyright + and Similar Rights in Your contributions to Adapted Material in + accordance with the terms and conditions of this Public License. + + c. BY-NC-SA Compatible License means a license listed at + creativecommons.org/compatiblelicenses, approved by Creative + Commons as essentially the equivalent of this Public License. + + d. Copyright and Similar Rights means copyright and/or similar rights + closely related to copyright including, without limitation, + performance, broadcast, sound recording, and Sui Generis Database + Rights, without regard to how the rights are labeled or + categorized. For purposes of this Public License, the rights + specified in Section 2(b)(1)-(2) are not Copyright and Similar + Rights. + + e. Effective Technological Measures means those measures that, in the + absence of proper authority, may not be circumvented under laws + fulfilling obligations under Article 11 of the WIPO Copyright + Treaty adopted on December 20, 1996, and/or similar international + agreements. + + f. Exceptions and Limitations means fair use, fair dealing, and/or + any other exception or limitation to Copyright and Similar Rights + that applies to Your use of the Licensed Material. + + g. License Elements means the license attributes listed in the name + of a Creative Commons Public License. The License Elements of this + Public License are Attribution, NonCommercial, and ShareAlike. + + h. Licensed Material means the artistic or literary work, database, + or other material to which the Licensor applied this Public + License. + + i. Licensed Rights means the rights granted to You subject to the + terms and conditions of this Public License, which are limited to + all Copyright and Similar Rights that apply to Your use of the + Licensed Material and that the Licensor has authority to license. + + j. Licensor means the individual(s) or entity(ies) granting rights + under this Public License. + + k. NonCommercial means not primarily intended for or directed towards + commercial advantage or monetary compensation. For purposes of + this Public License, the exchange of the Licensed Material for + other material subject to Copyright and Similar Rights by digital + file-sharing or similar means is NonCommercial provided there is + no payment of monetary compensation in connection with the + exchange. + + l. Share means to provide material to the public by any means or + process that requires permission under the Licensed Rights, such + as reproduction, public display, public performance, distribution, + dissemination, communication, or importation, and to make material + available to the public including in ways that members of the + public may access the material from a place and at a time + individually chosen by them. + + m. Sui Generis Database Rights means rights other than copyright + resulting from Directive 96/9/EC of the European Parliament and of + the Council of 11 March 1996 on the legal protection of databases, + as amended and/or succeeded, as well as other essentially + equivalent rights anywhere in the world. + + n. You means the individual or entity exercising the Licensed Rights + under this Public License. Your has a corresponding meaning. + + +Section 2 -- Scope. + + a. License grant. + + 1. Subject to the terms and conditions of this Public License, + the Licensor hereby grants You a worldwide, royalty-free, + non-sublicensable, non-exclusive, irrevocable license to + exercise the Licensed Rights in the Licensed Material to: + + a. reproduce and Share the Licensed Material, in whole or + in part, for NonCommercial purposes only; and + + b. produce, reproduce, and Share Adapted Material for + NonCommercial purposes only. + + 2. Exceptions and Limitations. For the avoidance of doubt, where + Exceptions and Limitations apply to Your use, this Public + License does not apply, and You do not need to comply with + its terms and conditions. + + 3. Term. The term of this Public License is specified in Section + 6(a). + + 4. Media and formats; technical modifications allowed. The + Licensor authorizes You to exercise the Licensed Rights in + all media and formats whether now known or hereafter created, + and to make technical modifications necessary to do so. The + Licensor waives and/or agrees not to assert any right or + authority to forbid You from making technical modifications + necessary to exercise the Licensed Rights, including + technical modifications necessary to circumvent Effective + Technological Measures. For purposes of this Public License, + simply making modifications authorized by this Section 2(a) + (4) never produces Adapted Material. + + 5. Downstream recipients. + + a. Offer from the Licensor -- Licensed Material. Every + recipient of the Licensed Material automatically + receives an offer from the Licensor to exercise the + Licensed Rights under the terms and conditions of this + Public License. + + b. Additional offer from the Licensor -- Adapted Material. + Every recipient of Adapted Material from You + automatically receives an offer from the Licensor to + exercise the Licensed Rights in the Adapted Material + under the conditions of the Adapter's License You apply. + + c. No downstream restrictions. You may not offer or impose + any additional or different terms or conditions on, or + apply any Effective Technological Measures to, the + Licensed Material if doing so restricts exercise of the + Licensed Rights by any recipient of the Licensed + Material. + + 6. No endorsement. Nothing in this Public License constitutes or + may be construed as permission to assert or imply that You + are, or that Your use of the Licensed Material is, connected + with, or sponsored, endorsed, or granted official status by, + the Licensor or others designated to receive attribution as + provided in Section 3(a)(1)(A)(i). + + b. Other rights. + + 1. Moral rights, such as the right of integrity, are not + licensed under this Public License, nor are publicity, + privacy, and/or other similar personality rights; however, to + the extent possible, the Licensor waives and/or agrees not to + assert any such rights held by the Licensor to the limited + extent necessary to allow You to exercise the Licensed + Rights, but not otherwise. + + 2. Patent and trademark rights are not licensed under this + Public License. + + 3. To the extent possible, the Licensor waives any right to + collect royalties from You for the exercise of the Licensed + Rights, whether directly or through a collecting society + under any voluntary or waivable statutory or compulsory + licensing scheme. In all other cases the Licensor expressly + reserves any right to collect such royalties, including when + the Licensed Material is used other than for NonCommercial + purposes. + + +Section 3 -- License Conditions. + +Your exercise of the Licensed Rights is expressly made subject to the +following conditions. + + a. Attribution. + + 1. If You Share the Licensed Material (including in modified + form), You must: + + a. retain the following if it is supplied by the Licensor + with the Licensed Material: + + i. identification of the creator(s) of the Licensed + Material and any others designated to receive + attribution, in any reasonable manner requested by + the Licensor (including by pseudonym if + designated); + + ii. a copyright notice; + + iii. a notice that refers to this Public License; + + iv. a notice that refers to the disclaimer of + warranties; + + v. a URI or hyperlink to the Licensed Material to the + extent reasonably practicable; + + b. indicate if You modified the Licensed Material and + retain an indication of any previous modifications; and + + c. indicate the Licensed Material is licensed under this + Public License, and include the text of, or the URI or + hyperlink to, this Public License. + + 2. You may satisfy the conditions in Section 3(a)(1) in any + reasonable manner based on the medium, means, and context in + which You Share the Licensed Material. For example, it may be + reasonable to satisfy the conditions by providing a URI or + hyperlink to a resource that includes the required + information. + 3. If requested by the Licensor, You must remove any of the + information required by Section 3(a)(1)(A) to the extent + reasonably practicable. + + b. ShareAlike. + + In addition to the conditions in Section 3(a), if You Share + Adapted Material You produce, the following conditions also apply. + + 1. The Adapter's License You apply must be a Creative Commons + license with the same License Elements, this version or + later, or a BY-NC-SA Compatible License. + + 2. You must include the text of, or the URI or hyperlink to, the + Adapter's License You apply. You may satisfy this condition + in any reasonable manner based on the medium, means, and + context in which You Share Adapted Material. + + 3. You may not offer or impose any additional or different terms + or conditions on, or apply any Effective Technological + Measures to, Adapted Material that restrict exercise of the + rights granted under the Adapter's License You apply. + + +Section 4 -- Sui Generis Database Rights. + +Where the Licensed Rights include Sui Generis Database Rights that +apply to Your use of the Licensed Material: + + a. for the avoidance of doubt, Section 2(a)(1) grants You the right + to extract, reuse, reproduce, and Share all or a substantial + portion of the contents of the database for NonCommercial purposes + only; + + b. if You include all or a substantial portion of the database + contents in a database in which You have Sui Generis Database + Rights, then the database in which You have Sui Generis Database + Rights (but not its individual contents) is Adapted Material, + including for purposes of Section 3(b); and + + c. You must comply with the conditions in Section 3(a) if You Share + all or a substantial portion of the contents of the database. + +For the avoidance of doubt, this Section 4 supplements and does not +replace Your obligations under this Public License where the Licensed +Rights include other Copyright and Similar Rights. + + +Section 5 -- Disclaimer of Warranties and Limitation of Liability. + + a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE + EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS + AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF + ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, + IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, + WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR + PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, + ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT + KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT + ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. + + b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE + TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, + NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, + INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, + COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR + USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN + ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR + DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR + IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. + + c. The disclaimer of warranties and limitation of liability provided + above shall be interpreted in a manner that, to the extent + possible, most closely approximates an absolute disclaimer and + waiver of all liability. + + +Section 6 -- Term and Termination. + + a. This Public License applies for the term of the Copyright and + Similar Rights licensed here. However, if You fail to comply with + this Public License, then Your rights under this Public License + terminate automatically. + + b. Where Your right to use the Licensed Material has terminated under + Section 6(a), it reinstates: + + 1. automatically as of the date the violation is cured, provided + it is cured within 30 days of Your discovery of the + violation; or + + 2. upon express reinstatement by the Licensor. + + For the avoidance of doubt, this Section 6(b) does not affect any + right the Licensor may have to seek remedies for Your violations + of this Public License. + + c. For the avoidance of doubt, the Licensor may also offer the + Licensed Material under separate terms or conditions or stop + distributing the Licensed Material at any time; however, doing so + will not terminate this Public License. + + d. Sections 1, 5, 6, 7, and 8 survive termination of this Public + License. + + +Section 7 -- Other Terms and Conditions. + + a. The Licensor shall not be bound by any additional or different + terms or conditions communicated by You unless expressly agreed. + + b. Any arrangements, understandings, or agreements regarding the + Licensed Material not stated herein are separate from and + independent of the terms and conditions of this Public License. + + +Section 8 -- Interpretation. + + a. For the avoidance of doubt, this Public License does not, and + shall not be interpreted to, reduce, limit, restrict, or impose + conditions on any use of the Licensed Material that could lawfully + be made without permission under this Public License. + + b. To the extent possible, if any provision of this Public License is + deemed unenforceable, it shall be automatically reformed to the + minimum extent necessary to make it enforceable. If the provision + cannot be reformed, it shall be severed from this Public License + without affecting the enforceability of the remaining terms and + conditions. + + c. No term or condition of this Public License will be waived and no + failure to comply consented to unless expressly agreed to by the + Licensor. + + d. Nothing in this Public License constitutes or may be interpreted + as a limitation upon, or waiver of, any privileges and immunities + that apply to the Licensor or You, including from the legal + processes of any jurisdiction or authority. + +======================================================================= + +Creative Commons is not a party to its public +licenses. Notwithstanding, Creative Commons may elect to apply one of +its public licenses to material it publishes and in those instances +will be considered the “Licensor.” The text of the Creative Commons +public licenses is dedicated to the public domain under the CC0 Public +Domain Dedication. Except for the limited purpose of indicating that +material is shared under a Creative Commons public license or as +otherwise permitted by the Creative Commons policies published at +creativecommons.org/policies, Creative Commons does not authorize the +use of the trademark "Creative Commons" or any other trademark or logo +of Creative Commons without its prior written consent including, +without limitation, in connection with any unauthorized modifications +to any of its public licenses or any other arrangements, +understandings, or agreements concerning use of licensed material. For +the avoidance of doubt, this paragraph does not form part of the +public licenses. + +Creative Commons may be contacted at creativecommons.org. + diff --git a/README.md b/README.md index 056fc354577ff493dc4f56fa459bfbe249d77820..c524e8d6adb181491c31c55415e9caac09e5869e 100644 --- a/README.md +++ b/README.md @@ -1,98 +1,95 @@ --- +license: cc-by-nc-sa-4.0 +tags: + - relational-deep-learning + - relational-databases + - relational-data + - tabular + - tabular-classification + - tabular-regression + - foundation-model + - in-context-learning + - few-shot + - pretraining + - synthetic-data + - relbench + - relational-transformer datasets: -- kvignesh1420/plurel -library_name: pytorch -license: mit + - stanford-star/the-join + - stanford-star/relbench + - kvignesh1420/plurel pipeline_tag: tabular-classification metrics: -- roc_auc -- r_squared -tags: -- relational-data -- tabular -- foundation-model -- pretraining -- relational-transformer -- relbench -- synthetic-data -- tabular-regression + - roc_auc + - r_squared +library_name: pytorch --- -# Relational Transformer — PluRel Checkpoints - -Relational Transformer (RT) model checkpoints pretrained on synthetic relational databases generated by [PluRel](https://huggingface.co/datasets/kvignesh1420/plurel). +# RT-PluRel (`rt-plurel`) -Relational Transformer is a foundation model architecture for relational data that enables zero-shot transfer across heterogeneous schemas and tasks. It was introduced in: +RT-PluRel is a Relational Transformer checkpoint pair for **in-context / +few-shot entity prediction** over multi-table relational databases (no per-task +gradient training). Same architecture and file layout as +[`stanford-star/rt-j`](https://huggingface.co/stanford-star/rt-j) — drop-in +replacement. -> **Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data** -> Rishabh Ranjan, Valter Hudovernik, Mark Znidar, Charilaos Kanatsoulis, Roshan Upendra, Mahmoud Mohammadi, Joe Meyer, Tom Palczewski, Carlos Guestrin, Jure Leskovec — [arXiv:2510.06377](https://arxiv.org/abs/2510.06377) (ICLR 2026) +> **Repository layout note (2026-08).** This repository now hosts two +> generations of RT-PluRel checkpoints. The current checkpoint pair (formerly +> published as `stanford-star/rt-p`, which this repository replaces) lives at +> the repository root, described below. The original checkpoints of the PluRel +> paper ([arXiv:2602.04029](https://arxiv.org/abs/2602.04029)), previously at +> this repository's root, are preserved unchanged under +> [`paper/`](./tree/main/paper) — see [`paper/README.md`](./blob/main/paper/README.md). -The checkpoints provided in this repository were trained using the methodology described in: - -> **PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models** -> Kothapalli, Ranjan, Hudovernik, Dwivedi, Hoffart, Guestrin, Leskovec — [arXiv:2602.04029](https://arxiv.org/abs/2602.04029) (2026) - -[![arXiv (RT)](https://img.shields.io/badge/arXiv-2510.06377-b31b1b?style=flat&logo=arxiv)](https://arxiv.org/abs/2510.06377) -[![GitHub (RT)](https://img.shields.io/badge/Code-RT_GitHub-black?style=flat&logo=github)](https://github.com/snap-stanford/relational-transformer) -[![arXiv (PluRel)](https://img.shields.io/badge/arXiv-2602.04029-b31b1b?style=flat&logo=arxiv)](https://arxiv.org/abs/2602.04029) -[![Project Page (PluRel)](https://img.shields.io/badge/Project-Page-blue?style=flat&logo=github)](https://snap-stanford.github.io/plurel/) -[![GitHub (PluRel)](https://img.shields.io/badge/Code-PluRel_GitHub-black?style=flat&logo=github)](https://github.com/snap-stanford/plurel) -[![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-yellow?style=flat&logo=huggingface)](https://huggingface.co/datasets/kvignesh1420/plurel) - ---- +| Variant | Folder | Task | +|---|---|---| +| Classifier | `classification/` | binary entity classification | +| Regressor | `regression/` | entity regression | -## Model Architecture +Architecture: ~85.6M params · bfloat16 · 12 blocks, d_model 512, 8 heads, d_ff 2048 · +text columns embedded with `all-MiniLM-L12-v2` (d_text 384). +Each folder contains `model.safetensors` (weights) and `config.json` +(dims + text-embedding model). -The Relational Transformer operates on multi-tabular relational databases, treating rows across linked tables as a sequence via BFS-ordered context sampling. It utilizes a Relational Attention mechanism over columns, rows, and primary-foreign key links. +## Usage -| Hyperparameter | Value | -|----------------|-------| -| Transformer blocks | 12 | -| Model dimension (`d_model`) | 256 | -| Attention heads | 8 | -| FFN dimension (`d_ff`) | 1,024 | -| Context length | 1,024 tokens | -| Text encoder | `all-MiniLM-L12-v2` (d_text = 384) | -| Max BFS width | 128 | +```python +from rt.model import RelationalTransformer -The architecture and training loop build on the [Relational Transformer](https://github.com/snap-stanford/relational-transformer) codebase. +model = RelationalTransformer.from_pretrained("stanford-star/rt-plurel/classification") +# or: RelationalTransformer.from_pretrained("stanford-star/rt-plurel", subfolder="regression") +``` ---- +```bash +pixi run eval --model.load-ckpt-path stanford-star/rt-plurel/classification ... +``` -## Download +## `paper/`: original PluRel-paper checkpoints -Single checkpoint (Python) — fetch `config.json` alongside the weights; it carries -the model architecture and is what the Hub uses to count downloads: +The `paper/` subdirectory preserves the earlier RT-PluRel release: `.pt` +checkpoints (12 blocks, d_model 256, d_ff 1024) pretrained on synthetic +relational databases generated by +[PluRel](https://huggingface.co/datasets/kvignesh1420/plurel), plus the +continued-pretraining and fine-tuned RelBench leaderboard checkpoints. Details, +protocols, and download snippets: [`paper/README.md`](./blob/main/paper/README.md). +Paths from before the reorganization gain a `paper/` prefix, e.g. ```python from huggingface_hub import hf_hub_download -config = hf_hub_download("stanford-star/rt-plurel", "config.json") -ckpt = hf_hub_download("stanford-star/rt-plurel", "synthetic-pretrain_rdb_1024_size_4b.pt") -``` - -Full repository (CLI): - -```bash -hf download stanford-star/rt-plurel \ - --repo-type model \ - --local-dir ~/scratch/rt_hf_ckpts +ckpt = hf_hub_download("stanford-star/rt-plurel", "paper/synthetic-pretrain_rdb_1024_size_4b.pt") ``` +## License -## RelBench leaderboard checkpoints (added 2026-06) +The root checkpoint pair (`classification/`, `regression/`) is released under +CC BY-NC-SA 4.0 (see `LICENSE`). The files under `paper/` retain their original +MIT license. -Protocols follow the repo's continued-pretraining script (50k steps, batch 128, -lr 5e-4 cosine, from `synthetic-pretrain_rdb_1024_size_4b.pt`) and the RT -`example_finetune` protocol (lr 1e-4, batch 32, 2^15+1 steps), with regression -best-checkpoint selection by val NMAE (MAE / train-split std, ddof=1) — the -leaderboard metric — instead of R². Evaluation = full official test split. +## Related -- `cntd-pretrain__.pt` — synthetic+real continued pretraining, one - leave-one-DB-out run per database (incl. rel-event), per-task best checkpoint. - These produce the "PluRel | synthetic + real" zero-shot regression and - rel-event cells. -- `finetune__.pt` — fine-tuned from the matching - `cntd-pretrain` checkpoint (chosen over synthetic-only by val zero-shot, which - the synthetic+real checkpoint won on every task). These produce the - "PluRel | pretrained + fine-tuned" leaderboard row. \ No newline at end of file +- Training data: [stanford-star/the-join](https://huggingface.co/datasets/stanford-star/the-join) +- Evaluation: [RelBench](https://huggingface.co/datasets/stanford-star/relbench) +- Reference model: [stanford-star/rt-j](https://huggingface.co/stanford-star/rt-j) +- Papers: [Relational Transformer (arXiv:2510.06377)](https://arxiv.org/abs/2510.06377) · + [PluRel (arXiv:2602.04029)](https://arxiv.org/abs/2602.04029) diff --git a/classification/config.json b/classification/config.json new file mode 100644 index 0000000000000000000000000000000000000000..8e914c9dff33a928d36fdcdb73a9df2b8a90bf49 --- /dev/null +++ b/classification/config.json @@ -0,0 +1,16 @@ +{ + "codename": "rt-p-clf", + "name": "RT-P Classifier", + "task_type": "clf", + "checkpoint_file": "model.safetensors", + "embedding_model": "all-MiniLM-L12-v2", + "d_text": 384, + "model": { + "num_blocks": 12, + "d_model": 512, + "d_text": 384, + "num_heads": 8, + "d_ff": 2048, + "materialize_attn_masks": true + } +} diff --git a/classification/model.safetensors b/classification/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..a69c361358ec6f5f05b0662943f8f79e7b46d3d1 --- /dev/null +++ b/classification/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4a369f065d8f69e1da11ee4326ceb9d32cf78ee69255a438449e971f681b7f83 +size 171169918 diff --git a/config.json b/config.json index bc16a8927e954ba961731f4155598e64024dbf36..8d43c20c752d922aed630ff0942bc9f7aadaf78b 100644 --- a/config.json +++ b/config.json @@ -1,11 +1,24 @@ { - "model_type": "relational-transformer", - "num_blocks": 12, - "d_model": 256, + "codename": "rt-plurel", + "name": "RT-PluRel", + "embedding_model": "all-MiniLM-L12-v2", "d_text": 384, - "num_heads": 8, - "d_ff": 1024, - "ctx_len": 1024, - "max_bfs_width": 256, - "embedding_model": "all-MiniLM-L12-v2" -} \ No newline at end of file + "model": { + "num_blocks": 12, + "d_model": 512, + "d_text": 384, + "num_heads": 8, + "d_ff": 2048, + "materialize_attn_masks": true + }, + "variants": { + "clf": { + "path": "classification", + "task_type": "clf" + }, + "reg": { + "path": "regression", + "task_type": "reg" + } + } +} diff --git a/paper/README.md b/paper/README.md new file mode 100644 index 0000000000000000000000000000000000000000..06520da14523e506994c7b28c76c72bd5bbe771b --- /dev/null +++ b/paper/README.md @@ -0,0 +1,100 @@ +--- +datasets: +- kvignesh1420/plurel +library_name: pytorch +license: mit +pipeline_tag: tabular-classification +metrics: +- roc_auc +- r_squared +tags: +- relational-data +- tabular +- foundation-model +- pretraining +- relational-transformer +- relbench +- synthetic-data +- tabular-regression +--- + +> **Note (2026-08):** these are the original PluRel-paper checkpoints, moved from this repository's root into `paper/` when the current RT-PluRel checkpoint pair (formerly `stanford-star/rt-p`) was merged into this repository. See the root `README.md`. + +# Relational Transformer — PluRel Checkpoints + +Relational Transformer (RT) model checkpoints pretrained on synthetic relational databases generated by [PluRel](https://huggingface.co/datasets/kvignesh1420/plurel). + +Relational Transformer is a foundation model architecture for relational data that enables zero-shot transfer across heterogeneous schemas and tasks. It was introduced in: + +> **Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data** +> Rishabh Ranjan, Valter Hudovernik, Mark Znidar, Charilaos Kanatsoulis, Roshan Upendra, Mahmoud Mohammadi, Joe Meyer, Tom Palczewski, Carlos Guestrin, Jure Leskovec — [arXiv:2510.06377](https://arxiv.org/abs/2510.06377) (ICLR 2026) + +The checkpoints provided in this repository were trained using the methodology described in: + +> **PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models** +> Kothapalli, Ranjan, Hudovernik, Dwivedi, Hoffart, Guestrin, Leskovec — [arXiv:2602.04029](https://arxiv.org/abs/2602.04029) (2026) + +[![arXiv (RT)](https://img.shields.io/badge/arXiv-2510.06377-b31b1b?style=flat&logo=arxiv)](https://arxiv.org/abs/2510.06377) +[![GitHub (RT)](https://img.shields.io/badge/Code-RT_GitHub-black?style=flat&logo=github)](https://github.com/snap-stanford/relational-transformer) +[![arXiv (PluRel)](https://img.shields.io/badge/arXiv-2602.04029-b31b1b?style=flat&logo=arxiv)](https://arxiv.org/abs/2602.04029) +[![Project Page (PluRel)](https://img.shields.io/badge/Project-Page-blue?style=flat&logo=github)](https://snap-stanford.github.io/plurel/) +[![GitHub (PluRel)](https://img.shields.io/badge/Code-PluRel_GitHub-black?style=flat&logo=github)](https://github.com/snap-stanford/plurel) +[![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-yellow?style=flat&logo=huggingface)](https://huggingface.co/datasets/kvignesh1420/plurel) + +--- + +## Model Architecture + +The Relational Transformer operates on multi-tabular relational databases, treating rows across linked tables as a sequence via BFS-ordered context sampling. It utilizes a Relational Attention mechanism over columns, rows, and primary-foreign key links. + +| Hyperparameter | Value | +|----------------|-------| +| Transformer blocks | 12 | +| Model dimension (`d_model`) | 256 | +| Attention heads | 8 | +| FFN dimension (`d_ff`) | 1,024 | +| Context length | 1,024 tokens | +| Text encoder | `all-MiniLM-L12-v2` (d_text = 384) | +| Max BFS width | 128 | + +The architecture and training loop build on the [Relational Transformer](https://github.com/snap-stanford/relational-transformer) codebase. + +--- + +## Download + +Single checkpoint (Python) — fetch `config.json` alongside the weights; it carries +the model architecture and is what the Hub uses to count downloads: + +```python +from huggingface_hub import hf_hub_download + +config = hf_hub_download("stanford-star/rt-plurel", "paper/config.json") +ckpt = hf_hub_download("stanford-star/rt-plurel", "paper/synthetic-pretrain_rdb_1024_size_4b.pt") +``` + +Full repository (CLI): + +```bash +hf download stanford-star/rt-plurel \ + --repo-type model \ + --local-dir ~/scratch/rt_hf_ckpts +``` + + +## RelBench leaderboard checkpoints (added 2026-06) + +Protocols follow the repo's continued-pretraining script (50k steps, batch 128, +lr 5e-4 cosine, from `synthetic-pretrain_rdb_1024_size_4b.pt`) and the RT +`example_finetune` protocol (lr 1e-4, batch 32, 2^15+1 steps), with regression +best-checkpoint selection by val NMAE (MAE / train-split std, ddof=1) — the +leaderboard metric — instead of R². Evaluation = full official test split. + +- `cntd-pretrain__.pt` — synthetic+real continued pretraining, one + leave-one-DB-out run per database (incl. rel-event), per-task best checkpoint. + These produce the "PluRel | synthetic + real" zero-shot regression and + rel-event cells. +- `finetune__.pt` — fine-tuned from the matching + `cntd-pretrain` checkpoint (chosen over synthetic-only by val zero-shot, which + the synthetic+real checkpoint won on every task). These produce the + "PluRel | pretrained + fine-tuned" leaderboard row. \ No newline at end of file diff --git a/cntd-pretrain_rel-amazon_item-churn.pt b/paper/cntd-pretrain_rel-amazon_item-churn.pt similarity index 100% rename from cntd-pretrain_rel-amazon_item-churn.pt rename to paper/cntd-pretrain_rel-amazon_item-churn.pt diff --git a/cntd-pretrain_rel-amazon_item-ltv.pt b/paper/cntd-pretrain_rel-amazon_item-ltv.pt similarity index 100% rename from cntd-pretrain_rel-amazon_item-ltv.pt rename to paper/cntd-pretrain_rel-amazon_item-ltv.pt diff --git a/cntd-pretrain_rel-amazon_user-churn.pt b/paper/cntd-pretrain_rel-amazon_user-churn.pt similarity index 100% rename from cntd-pretrain_rel-amazon_user-churn.pt rename to paper/cntd-pretrain_rel-amazon_user-churn.pt diff --git a/cntd-pretrain_rel-amazon_user-ltv.pt b/paper/cntd-pretrain_rel-amazon_user-ltv.pt similarity index 100% rename from cntd-pretrain_rel-amazon_user-ltv.pt rename to paper/cntd-pretrain_rel-amazon_user-ltv.pt diff --git a/cntd-pretrain_rel-avito_ad-ctr.pt b/paper/cntd-pretrain_rel-avito_ad-ctr.pt similarity index 100% rename from cntd-pretrain_rel-avito_ad-ctr.pt rename to paper/cntd-pretrain_rel-avito_ad-ctr.pt diff --git a/cntd-pretrain_rel-avito_user-clicks.pt b/paper/cntd-pretrain_rel-avito_user-clicks.pt similarity index 100% rename from cntd-pretrain_rel-avito_user-clicks.pt rename to paper/cntd-pretrain_rel-avito_user-clicks.pt diff --git a/cntd-pretrain_rel-avito_user-visits.pt b/paper/cntd-pretrain_rel-avito_user-visits.pt similarity index 100% rename from cntd-pretrain_rel-avito_user-visits.pt rename to paper/cntd-pretrain_rel-avito_user-visits.pt diff --git a/cntd-pretrain_rel-event_user-attendance.pt b/paper/cntd-pretrain_rel-event_user-attendance.pt similarity index 100% rename from cntd-pretrain_rel-event_user-attendance.pt rename to paper/cntd-pretrain_rel-event_user-attendance.pt diff --git a/cntd-pretrain_rel-event_user-ignore.pt b/paper/cntd-pretrain_rel-event_user-ignore.pt similarity index 100% rename from cntd-pretrain_rel-event_user-ignore.pt rename to paper/cntd-pretrain_rel-event_user-ignore.pt diff --git a/cntd-pretrain_rel-event_user-repeat.pt b/paper/cntd-pretrain_rel-event_user-repeat.pt similarity index 100% rename from cntd-pretrain_rel-event_user-repeat.pt rename to paper/cntd-pretrain_rel-event_user-repeat.pt diff --git a/cntd-pretrain_rel-f1_driver-dnf.pt b/paper/cntd-pretrain_rel-f1_driver-dnf.pt similarity index 100% rename from cntd-pretrain_rel-f1_driver-dnf.pt rename to paper/cntd-pretrain_rel-f1_driver-dnf.pt diff --git a/cntd-pretrain_rel-f1_driver-position.pt b/paper/cntd-pretrain_rel-f1_driver-position.pt similarity index 100% rename from cntd-pretrain_rel-f1_driver-position.pt rename to paper/cntd-pretrain_rel-f1_driver-position.pt diff --git a/cntd-pretrain_rel-f1_driver-top3.pt b/paper/cntd-pretrain_rel-f1_driver-top3.pt similarity index 100% rename from cntd-pretrain_rel-f1_driver-top3.pt rename to paper/cntd-pretrain_rel-f1_driver-top3.pt diff --git a/cntd-pretrain_rel-hm_item-sales.pt b/paper/cntd-pretrain_rel-hm_item-sales.pt similarity index 100% rename from cntd-pretrain_rel-hm_item-sales.pt rename to paper/cntd-pretrain_rel-hm_item-sales.pt diff --git a/cntd-pretrain_rel-hm_user-churn.pt b/paper/cntd-pretrain_rel-hm_user-churn.pt similarity index 100% rename from cntd-pretrain_rel-hm_user-churn.pt rename to paper/cntd-pretrain_rel-hm_user-churn.pt diff --git a/cntd-pretrain_rel-stack_post-votes.pt b/paper/cntd-pretrain_rel-stack_post-votes.pt similarity index 100% rename from cntd-pretrain_rel-stack_post-votes.pt rename to paper/cntd-pretrain_rel-stack_post-votes.pt diff --git a/cntd-pretrain_rel-stack_user-badge.pt b/paper/cntd-pretrain_rel-stack_user-badge.pt similarity index 100% rename from cntd-pretrain_rel-stack_user-badge.pt rename to paper/cntd-pretrain_rel-stack_user-badge.pt diff --git a/cntd-pretrain_rel-stack_user-engagement.pt b/paper/cntd-pretrain_rel-stack_user-engagement.pt similarity index 100% rename from cntd-pretrain_rel-stack_user-engagement.pt rename to paper/cntd-pretrain_rel-stack_user-engagement.pt diff --git a/cntd-pretrain_rel-trial_site-success.pt b/paper/cntd-pretrain_rel-trial_site-success.pt similarity index 100% rename from cntd-pretrain_rel-trial_site-success.pt rename to paper/cntd-pretrain_rel-trial_site-success.pt diff --git a/cntd-pretrain_rel-trial_study-adverse.pt b/paper/cntd-pretrain_rel-trial_study-adverse.pt similarity index 100% rename from cntd-pretrain_rel-trial_study-adverse.pt rename to paper/cntd-pretrain_rel-trial_study-adverse.pt diff --git a/cntd-pretrain_rel-trial_study-outcome.pt b/paper/cntd-pretrain_rel-trial_study-outcome.pt similarity index 100% rename from cntd-pretrain_rel-trial_study-outcome.pt rename to paper/cntd-pretrain_rel-trial_study-outcome.pt diff --git a/paper/config.json b/paper/config.json new file mode 100644 index 0000000000000000000000000000000000000000..bc16a8927e954ba961731f4155598e64024dbf36 --- /dev/null +++ b/paper/config.json @@ -0,0 +1,11 @@ +{ + "model_type": "relational-transformer", + "num_blocks": 12, + "d_model": 256, + "d_text": 384, + "num_heads": 8, + "d_ff": 1024, + "ctx_len": 1024, + "max_bfs_width": 256, + "embedding_model": "all-MiniLM-L12-v2" +} \ No newline at end of file diff --git a/finetune_rel-amazon_item-churn.pt b/paper/finetune_rel-amazon_item-churn.pt similarity index 100% rename from finetune_rel-amazon_item-churn.pt rename to paper/finetune_rel-amazon_item-churn.pt diff --git a/finetune_rel-amazon_item-ltv.pt b/paper/finetune_rel-amazon_item-ltv.pt similarity index 100% rename from finetune_rel-amazon_item-ltv.pt rename to paper/finetune_rel-amazon_item-ltv.pt diff --git a/finetune_rel-amazon_user-churn.pt b/paper/finetune_rel-amazon_user-churn.pt similarity index 100% rename from finetune_rel-amazon_user-churn.pt rename to paper/finetune_rel-amazon_user-churn.pt diff --git a/finetune_rel-amazon_user-ltv.pt b/paper/finetune_rel-amazon_user-ltv.pt similarity index 100% rename from finetune_rel-amazon_user-ltv.pt rename to paper/finetune_rel-amazon_user-ltv.pt diff --git a/finetune_rel-avito_ad-ctr.pt b/paper/finetune_rel-avito_ad-ctr.pt similarity index 100% rename from finetune_rel-avito_ad-ctr.pt rename to paper/finetune_rel-avito_ad-ctr.pt diff --git a/finetune_rel-avito_user-clicks.pt b/paper/finetune_rel-avito_user-clicks.pt similarity index 100% rename from finetune_rel-avito_user-clicks.pt rename to paper/finetune_rel-avito_user-clicks.pt diff --git a/finetune_rel-avito_user-visits.pt b/paper/finetune_rel-avito_user-visits.pt similarity index 100% rename from finetune_rel-avito_user-visits.pt rename to paper/finetune_rel-avito_user-visits.pt diff --git a/finetune_rel-event_user-attendance.pt b/paper/finetune_rel-event_user-attendance.pt similarity index 100% rename from finetune_rel-event_user-attendance.pt rename to paper/finetune_rel-event_user-attendance.pt diff --git a/finetune_rel-event_user-ignore.pt b/paper/finetune_rel-event_user-ignore.pt similarity index 100% rename from finetune_rel-event_user-ignore.pt rename to paper/finetune_rel-event_user-ignore.pt diff --git a/finetune_rel-event_user-repeat.pt b/paper/finetune_rel-event_user-repeat.pt similarity index 100% rename from finetune_rel-event_user-repeat.pt rename to paper/finetune_rel-event_user-repeat.pt diff --git a/finetune_rel-f1_driver-dnf.pt b/paper/finetune_rel-f1_driver-dnf.pt similarity index 100% rename from finetune_rel-f1_driver-dnf.pt rename to paper/finetune_rel-f1_driver-dnf.pt diff --git a/finetune_rel-f1_driver-position.pt b/paper/finetune_rel-f1_driver-position.pt similarity index 100% rename from finetune_rel-f1_driver-position.pt rename to paper/finetune_rel-f1_driver-position.pt diff --git a/finetune_rel-f1_driver-top3.pt b/paper/finetune_rel-f1_driver-top3.pt similarity index 100% rename from finetune_rel-f1_driver-top3.pt rename to paper/finetune_rel-f1_driver-top3.pt diff --git a/finetune_rel-hm_item-sales.pt b/paper/finetune_rel-hm_item-sales.pt similarity index 100% rename from finetune_rel-hm_item-sales.pt rename to paper/finetune_rel-hm_item-sales.pt diff --git a/finetune_rel-hm_user-churn.pt b/paper/finetune_rel-hm_user-churn.pt similarity index 100% rename from finetune_rel-hm_user-churn.pt rename to paper/finetune_rel-hm_user-churn.pt diff --git a/finetune_rel-stack_post-votes.pt b/paper/finetune_rel-stack_post-votes.pt similarity index 100% rename from finetune_rel-stack_post-votes.pt rename to paper/finetune_rel-stack_post-votes.pt diff --git a/finetune_rel-stack_user-badge.pt b/paper/finetune_rel-stack_user-badge.pt similarity index 100% rename from finetune_rel-stack_user-badge.pt rename to paper/finetune_rel-stack_user-badge.pt diff --git a/finetune_rel-stack_user-engagement.pt b/paper/finetune_rel-stack_user-engagement.pt similarity index 100% rename from finetune_rel-stack_user-engagement.pt rename to paper/finetune_rel-stack_user-engagement.pt diff --git a/finetune_rel-trial_site-success.pt b/paper/finetune_rel-trial_site-success.pt similarity index 100% rename from finetune_rel-trial_site-success.pt rename to paper/finetune_rel-trial_site-success.pt diff --git a/finetune_rel-trial_study-adverse.pt b/paper/finetune_rel-trial_study-adverse.pt similarity index 100% rename from finetune_rel-trial_study-adverse.pt rename to paper/finetune_rel-trial_study-adverse.pt diff --git a/finetune_rel-trial_study-outcome.pt b/paper/finetune_rel-trial_study-outcome.pt similarity index 100% rename from finetune_rel-trial_study-outcome.pt rename to paper/finetune_rel-trial_study-outcome.pt diff --git a/synthetic-pretrain_rdb_1024_size_4b.pt b/paper/synthetic-pretrain_rdb_1024_size_4b.pt similarity index 100% rename from synthetic-pretrain_rdb_1024_size_4b.pt rename to paper/synthetic-pretrain_rdb_1024_size_4b.pt diff --git a/synthetic-pretrain_rdb_512_size_16b.pt b/paper/synthetic-pretrain_rdb_512_size_16b.pt similarity index 100% rename from synthetic-pretrain_rdb_512_size_16b.pt rename to paper/synthetic-pretrain_rdb_512_size_16b.pt diff --git a/regression/config.json b/regression/config.json new file mode 100644 index 0000000000000000000000000000000000000000..2e5cf9f61e5a215d6b8db923d528f7a4fb83c498 --- /dev/null +++ b/regression/config.json @@ -0,0 +1,16 @@ +{ + "codename": "rt-p-reg", + "name": "RT-P Regressor", + "task_type": "reg", + "checkpoint_file": "model.safetensors", + "embedding_model": "all-MiniLM-L12-v2", + "d_text": 384, + "model": { + "num_blocks": 12, + "d_model": 512, + "d_text": 384, + "num_heads": 8, + "d_ff": 2048, + "materialize_attn_masks": true + } +} diff --git a/regression/model.safetensors b/regression/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..57088019d2c20424e4a1f7a96a7c11c13f37de21 --- /dev/null +++ b/regression/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a87b467e8fd09a9e355d4da26fafc92b61feb8e6c546b5268e274ac5b956b917 +size 171169918