---
base_model: TextMachineProject/NewsBERT_1800-1920
library_name: peft
tags:
  - lora
  - bert
  - masked-language-modeling
---

# NewsBERT pre-1850 LoRA adapter (3 epochs)

A LoRA adapter for [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920),
fine-tuned for three epochs on newspaper text (pre-1850)
from the [Heritage Made Digital (HMD14)](https://www.bl.uk/collection-guides/heritage-made-digital)
and [Living with Machines (LwM)](https://livingwithmachines.ac.uk/) collections.

## Training details

- **Period**: pre-1850
- **Base model**: `TextMachineProject/NewsBERT_1800-1920`
- **Method**: LoRA (PEFT), target modules: `query`, `value`, `word_embeddings`
- **LoRA rank**: 16, alpha: 32, dropout: 0.05
- **Task**: Masked Language Modelling (15% masking probability)
- **Sequence length**: 128 tokens (sliding window, stride 96)
- **Epochs**: 3
- **Batch size**: 256

## Usage

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM
from peft import PeftModel

base = AutoModelForMaskedLM.from_pretrained("TextMachineProject/NewsBERT_1800-1920")
tokenizer = AutoTokenizer.from_pretrained("TextMachineProject/NewsBERT_1800-1920")
model = PeftModel.from_pretrained(base, "TextMachineProject/NewsBERT_pre_1850_lora_3epochs")
```