| | --- |
| | language: en |
| | tags: |
| | - longformer |
| | - cdlm |
| | license: apache-2.0 |
| | inference: false |
| |
|
| | --- |
| | |
| |
|
| | # Cross-Document Language Modeling |
| |
|
| | CDLM: Cross-Document Language Modeling. |
| | Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan and Ido Dagan. In EMNLP Findings, 2021. [PDF](https://arxiv.org/pdf/2101.00406.pdf) |
| |
|
| |
|
| | Please note that during our pretraining we used the document and sentence separators, which you might want to add to your data. The document and sentence separators are `<doc-s>`, `</doc-s>` (the last two tokens in the vocabulary), and `<s>`, `</s>`, respectively. |
| |
|
| |
|
| | ```python |
| | from transformers import AutoTokenizer, AutoModel |
| | # load model and tokenizer |
| | tokenizer = AutoTokenizer.from_pretrained('biu-nlp/cdlm') |
| | model = AutoModel.from_pretrained('biu-nlp/cdlm') |
| | ``` |
| |
|
| | The original repo is [here](https://github.com/aviclu/CDLM). |
| |
|
| | If you find our work useful, please cite the paper as: |
| |
|
| | ```python |
| | @article{caciularu2021cross, |
| | title={Cross-Document Language Modeling}, |
| | author={Caciularu, Avi and Cohan, Arman and Beltagy, Iz and Peters, Matthew E and Cattan, Arie and Dagan, Ido}, |
| | journal={Findings of the Association for Computational Linguistics: EMNLP 2021}, |
| | year={2021} |
| | } |
| | ``` |
