| --- |
| base_model: TextMachineProject/NewsBERT_1800-1920 |
| library_name: peft |
| tags: |
| - lora |
| - bert |
| - masked-language-modeling |
| --- |
| |
| # NewsBERT post-1900 LoRA adapter (3 epochs) |
|
|
| A LoRA adapter for [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920), |
| fine-tuned for three epochs on newspaper text (post-1900) |
| from the [Heritage Made Digital (HMD14)](https://www.bl.uk/collection-guides/heritage-made-digital) |
| and [Living with Machines (LwM)](https://livingwithmachines.ac.uk/) collections. |
|
|
| ## Training details |
|
|
| - **Period**: post-1900 |
| - **Base model**: `TextMachineProject/NewsBERT_1800-1920` |
| - **Method**: LoRA (PEFT), target modules: `query`, `value`, `word_embeddings` |
| - **LoRA rank**: 16, alpha: 32, dropout: 0.05 |
| - **Task**: Masked Language Modelling (15% masking probability) |
| - **Sequence length**: 128 tokens (sliding window, stride 96) |
| - **Epochs**: 3 |
| - **Batch size**: 256 |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForMaskedLM |
| from peft import PeftModel |
| |
| base = AutoModelForMaskedLM.from_pretrained("TextMachineProject/NewsBERT_1800-1920") |
| tokenizer = AutoTokenizer.from_pretrained("TextMachineProject/NewsBERT_1800-1920") |
| model = PeftModel.from_pretrained(base, "TextMachineProject/NewsBERT_post_1900_lora_3epochs") |
| ``` |
|
|
