---
base_model: TextMachineProject/NewsBERT_1800-1920
library_name: peft
tags:
  - lora
  - bert
  - masked-language-modeling
---

# NewsBERT post-1900 LoRA adapter (1 epoch)

A LoRA adapter for [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920),
fine-tuned for one epoch on newspaper text (post-1900)
from the [Heritage Made Digital (HMD14)](https://www.bl.uk/collection-guides/heritage-made-digital)
and [Living with Machines (LwM)](https://livingwithmachines.ac.uk/) collections.

## Training details

- **Period**: post-1900
- **Base model**: `TextMachineProject/NewsBERT_1800-1920`
- **Method**: LoRA (PEFT), target modules: `query`, `value`, `word_embeddings`
- **LoRA rank**: 16, alpha: 32, dropout: 0.05
- **Task**: Masked Language Modelling (15% masking probability)
- **Sequence length**: 128 tokens (sliding window, stride 96)
- **Epochs**: 1
- **Batch size**: 256

## Usage

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM
from peft import PeftModel

base = AutoModelForMaskedLM.from_pretrained("TextMachineProject/NewsBERT_1800-1920")
tokenizer = AutoTokenizer.from_pretrained("TextMachineProject/NewsBERT_1800-1920")
model = PeftModel.from_pretrained(base, "TextMachineProject/NewsBERT_post_1900_lora_1epoch")
```

## Notes

This is a 1-epoch checkpoint uploaded for evaluation purposes. Further training
is ongoing; updated adapters will be released separately.