| | --- |
| | language: |
| | - en |
| | tags: |
| | - sentence-similarity |
| | - text-classification |
| | datasets: |
| | - dennlinger/wiki-paragraphs |
| | metrics: |
| | - f1 |
| | license: mit |
| | --- |
| | |
| | # BERT-Wiki-Paragraphs |
| |
|
| | Authors: Satya Almasian\*, Dennis Aumiller\*, Lucienne-Sophie Marmé, Michael Gertz |
| | Contact us at `<lastname>@informatik.uni-heidelberg.de` |
| | Details for the training method can be found in our work [Structural Text Segmentation of Legal Documents](https://arxiv.org/abs/2012.03619). |
| | The training procedure follows the same setup, but we substitute legal documents for Wikipedia in this model. |
| | Find the associated training data here: [wiki-paragraphs](https://huggingface.co/datasets/dennlinger/wiki-paragraphs) |
| |
|
| | Training is performed in a form of weakly-supervised fashion to determine whether paragraphs topically belong together or not. |
| | We utilize automatically generated samples from Wikipedia for training, where paragraphs from within the same section are assumed to be topically coherent. |
| | We use the same articles as ([Koshorek et al., 2018](https://arxiv.org/abs/1803.09337)), |
| | albeit from a 2021 dump of Wikpeida, and split at paragraph boundaries instead of the sentence level. |
| |
|
| | ## Usage |
| | Preferred usage is through `transformers.pipeline`: |
| |
|
| | ```python |
| | from transformers import pipeline |
| | pipe = pipeline("text-classification", model="dennlinger/bert-wiki-paragraphs") |
| | |
| | pipe("{First paragraph} [SEP] {Second paragraph}") |
| | ``` |
| | A predicted "1" means that paragraphs belong to the same topic, a "0" indicates a disconnect. |
| |
|
| | ## Training Setup |
| | The model was trained for 3 epochs from `bert-base-uncased` on paragraph pairs (limited to 512 subwork with the `longest_first` truncation strategy). |
| | We use a batch size of 24 wit 2 iterations gradient accumulation (effective batch size of 48), and a learning rate of 1e-4, with gradient clipping at 5. |
| | Training was performed on a single Titan RTX GPU over the duration of 3 weeks. |
| |
|
