| --- |
| base_model: TextMachineProject/NewsBERT_1800-1920 |
| library_name: peft |
| tags: |
| - lora |
| - bert |
| - masked-language-modeling |
| --- |
| |
| # NewsBERT pre-1850 LoRA adapter (1 epoch) |
|
|
| A LoRA adapter for [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920), |
| fine-tuned for one epoch on newspaper text (pre-1850) |
| from the [Heritage Made Digital (HMD14)](https://www.bl.uk/collection-guides/heritage-made-digital) |
| and [Living with Machines (LwM)](https://livingwithmachines.ac.uk/) collections. |
|
|
| ## Training details |
|
|
| - **Period**: pre-1850 |
| - **Base model**: `TextMachineProject/NewsBERT_1800-1920` |
| - **Method**: LoRA (PEFT), target modules: `query`, `value`, `word_embeddings` |
| - **LoRA rank**: 16, alpha: 32, dropout: 0.05 |
| - **Task**: Masked Language Modelling (15% masking probability) |
| - **Sequence length**: 128 tokens (sliding window, stride 96) |
| - **Epochs**: 1 |
| - **Batch size**: 256 |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForMaskedLM |
| from peft import PeftModel |
| |
| base = AutoModelForMaskedLM.from_pretrained("TextMachineProject/NewsBERT_1800-1920") |
| tokenizer = AutoTokenizer.from_pretrained("TextMachineProject/NewsBERT_1800-1920") |
| model = PeftModel.from_pretrained(base, "TextMachineProject/NewsBERT_pre_1850_lora_1epoch") |
| ``` |
|
|
| ## Notes |
|
|
| This is a 1-epoch checkpoint uploaded for evaluation purposes. Further training |
| is ongoing; updated adapters will be released separately. |
|
|
