AnasBelfathi/bert-legal-topic

BERT Legal (Topic-masked, continual pretraining)

bert-base-uncased continually pretrained (MLM) on a legal corpus, using topic-based selective masking (tf-idf weighted) instead of random masking: during training, content words that distinguish a document within the domain (e.g. "contract", "asylum", "tax") are masked preferentially.

Training data

A subset of lexlms/lex_files covering ECtHR case law, EU case law, and Indian case law.

Training

• Base model: bert-base-uncased
• Objective: masked language modeling, 15% masking rate, words sampled proportionally to their tf-idf weight
• 10 epochs, chunk size 512, effective batch size 1024, learning rate 2e-5

Method

The masking strategy follows:

Belfathi, A., Gallina, Y., Hernandez, N., Monceaux, L., & Dufour, R. (2025). Is Selective Masking A Key to Improving Domain Adaptation for Masked Language Model? ICAIL 2025. doi:10.1145/3769126.3769216

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("AnasBelfathi/bert-legal-topic")
model = AutoModelForMaskedLM.from_pretrained("AnasBelfathi/bert-legal-topic")