metadata
base_model: TextMachineProject/NewsBERT_1800-1920
library_name: peft
tags:
- lora
- bert
- masked-language-modeling
NewsBERT pre-1850 LoRA adapter (1 epoch)
A LoRA adapter for TextMachineProject/NewsBERT_1800-1920, fine-tuned for one epoch on newspaper text (pre-1850) from the Heritage Made Digital (HMD14) and Living with Machines (LwM) collections.
Training details
- Period: pre-1850
- Base model:
TextMachineProject/NewsBERT_1800-1920 - Method: LoRA (PEFT), target modules:
query,value,word_embeddings - LoRA rank: 16, alpha: 32, dropout: 0.05
- Task: Masked Language Modelling (15% masking probability)
- Sequence length: 128 tokens (sliding window, stride 96)
- Epochs: 1
- Batch size: 256
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
from peft import PeftModel
base = AutoModelForMaskedLM.from_pretrained("TextMachineProject/NewsBERT_1800-1920")
tokenizer = AutoTokenizer.from_pretrained("TextMachineProject/NewsBERT_1800-1920")
model = PeftModel.from_pretrained(base, "TextMachineProject/NewsBERT_pre_1850_lora_1epoch")
Notes
This is a 1-epoch checkpoint uploaded for evaluation purposes. Further training is ongoing; updated adapters will be released separately.