| --- |
| license: cc-by-4.0 |
| datasets: |
| - ai-lab/MBD-mini |
| tags: |
| - event-sequences |
| - banking |
| - transactions |
| - time-series |
| - foundation-model |
| - pytorch |
| library_name: pytorch |
| --- |
| |
| # EventGPT |
|
|
| EventGPT is an open generative pretrained transformer for banking event sequences. It reads a customer's |
| history of transactions and other events, predicts the next event token by token, and gives one |
| embedding per customer and point in time. It was pretrained on the public Multimodal Banking Dataset |
| (MBD-mini: 100,224 clients, 38.8 million transactions and 66.3 million geolocation pings). |
|
|
| - Paper: *EventGPT: Next-Event Pretraining on Banking Customer Histories* (Burakov, 2026), https://arxiv.org/abs/2611.03302 |
| - Code: https://github.com/deburky/EventGPT |
|
|
| ## Model |
|
|
| EventGPT is hierarchical. An event encoder turns the tokens of one event (name, type, amount bin, time gap |
| and other fields) into a vector; a causal history decoder reads these vectors with rotary positions taken |
| from elapsed time since the client's first event; a token decoder predicts the next event's tokens from |
| the history state. The models here have 23 million parameters (width 384) and were trained on 655 |
| million tokens. |
|
|
| ## Variants |
|
|
| Each variant is a subfolder with `model.safetensors`, `config.json` and `vocab.json` (the token table and |
| amount-bin edges the model was trained on; histories must be tokenized with it). |
|
|
| | variant | events | training windows | official mean ROC-AUC | |
| |---|---|---|---| |
| | `aligned-s42`, `aligned-s43` | transactions and geolocation | aligned, query-key normalization | 0.786, 0.788 | |
| | `aligned-profile-s42`, `aligned-profile-s43` | as above, with early fusion of the profile fields | aligned, query-key normalization | 0.803, 0.807 with late fusion of the profile fields | |
| | `trx-s42`, `trx-s43` | transactions only | random offsets | 0.794, 0.791 | |
| | `trx-oot-s42`, `trx-oot-s43` | transactions only, pretrained on events before February 2022 | random offsets | 0.790, 0.788 | |
|
|
| The official mean is the mean ROC-AUC over MBD's four product targets on test fold 0, with a logistic |
| head on the frozen embedding. On the same rows, CatBoost on engineered features reaches 0.816 and on |
| token counts over the full history 0.803. A rank-averaged ensemble of EventGPT (with late fusion of the |
| profile fields and token counts) and the engineered features reaches 0.830. |
|
|
| The paper repeats this comparison with each of MBD's five folds as the test fold, pretraining new models |
| on the other four. Averaged over the five folds, the embedding scores 0.787 against 0.808 for the |
| engineered features, 0.804 with the profile fields and token counts, and the ensemble 0.824. The ensemble |
| is above the engineered features on every fold and seed. |
|
|
| Without labels, the next-event distribution scores five "first transaction of a kind next month" targets |
| at ROC-AUC 0.78 to 0.91 (aligned variants, mean of two seeds). |
|
|
| ## Use |
|
|
| ```python |
| from eventgpt.hub import from_pretrained |
| |
| model, vocab = from_pretrained("deburky/EventGPT", "aligned-profile-s42") |
| ``` |
|
|
| Install the code from the GitHub repository. Tokenizing MBD-mini (`egpt prepare`), embedding histories |
| (`egpt probe`) and zero-shot scoring (`egpt zero-shot`) are commands of the `egpt` CLI. |
|
|
| ## Limitations |
|
|
| The vocabulary is MBD's event schema. Another institution's events need their own tokenization and their |
| own pretraining; these weights serve to reproduce and extend the paper's results on MBD. The models were |
| pretrained on the pretraining clients' events from 2021 and 2022, which include the months in which the |
| test clients' labels fall (never the test clients themselves); pretraining only on transactions before |
| February 2022 (`trx-oot`) lowers the official mean by 0.004 to 0.006. The weights here are pretrained with fold 0 |
| held out; the paper's models for the other four test folds are not released. |
|
|
| ## License and citation |
|
|
| The weights are released under CC BY 4.0, the license of MBD-mini, on which they were trained. Please cite |
| the paper and MBD (Mollaev et al., 2024). |
|
|
| ```bibtex |
| @misc{burakov2026eventgpt, |
| title = {EventGPT: Next-Event Pretraining on Banking Customer Histories}, |
| author = {Burakov, Denis}, |
| year = {2026}, |
| eprint = {2611.03302}, |
| archivePrefix = {arXiv}, |
| primaryClass = {stat.ML}, |
| url = {https://arxiv.org/abs/2611.03302} |
| } |
| ``` |
|
|
