README.md

---
license: cc-by-nc-sa-4.0
tags:
  - relational-deep-learning
  - relational-databases
  - tabular
  - tabular-classification
  - tabular-regression
  - foundation-model
  - in-context-learning
  - few-shot
  - relbench
  - relational-transformer
datasets:
  - stanford-star/the-join
  - stanford-star/relbench
metrics:
  - roc_auc
  - mae
pipeline_tag: tabular-classification
model-index:
  - name: rt-j
    results:
      - task:
          type: tabular-classification
          name: Relational entity classification (in-context, RelBench)
        dataset:
          type: stanford-star/relbench
          name: RelBench (12 binary classification tasks)
        metrics:
          - type: roc_auc
            value: 0.7310
            name: Mean AUROC (single-context, L=8k, full test split)
      - task:
          type: tabular-regression
          name: Relational entity regression (in-context, RelBench)
        dataset:
          type: stanford-star/relbench
          name: RelBench (9 regression tasks)
        metrics:
          - type: mae
            value: 0.2677
            name: Mean MAE (single-context, L=8k, full test split)
---

# RT-J (`rt-j`)

RT-J is a Relational Transformer foundation model for **in-context / few-shot entity
prediction** over multi-table relational databases (no per-task gradient training).
This repository hosts both task-head variants:

| Variant | Folder | Task | Metric | Selected checkpoint |
|---|---|---|---|---|
| Classifier | [`classification/`](./tree/main/classification) | binary entity classification | AUROC ↑ (mean 0.7310 on 12 RelBench tasks) | SWA @ step 58,000 (best val AUROC) |
| Regressor | [`regression/`](./tree/main/regression) | entity regression | MAE ↓ (mean 0.2677 on 9 RelBench tasks, Z-scored) | SWA @ step 18,000 (best val MAE) |

Both variants share the same architecture and training recipe:
~85.6M params · bfloat16 · 12 blocks, d_model 512, 8 heads, d_ff 2048 ·
text columns embedded with `all-MiniLM-L12-v2` (d_text 384).
Each folder contains `model.safetensors` (weights) and `config.json`
(dims + text-embedding model + provenance).

## Usage

```python
from huggingface_hub import hf_hub_download

# shared architecture + variant map (also what the Hub counts as a download)
config = hf_hub_download("stanford-star/rt-j", "config.json")

# pick the variant: "classification" or "regression"
weights = hf_hub_download("stanford-star/rt-j", "classification/model.safetensors")
variant_config = hf_hub_download("stanford-star/rt-j", "classification/config.json")
```

## Related

- Training data: [stanford-star/the-join](https://huggingface.co/datasets/stanford-star/the-join)
- Evaluation: [RelBench](https://huggingface.co/datasets/stanford-star/relbench)
- More models from the team: [stanford-star](https://huggingface.co/stanford-star)