latincy/latin-bert

Latin BERT (Bamman & Burns 2020)

HuggingFace-compatible packaging of the Latin BERT model from:

Bamman, D., & Burns, P.J. (2020). Latin BERT: A Contextual Language Model for Classical Philology. arXiv preprint arXiv:2009.10053.

The original model and training code are available at github.com/dbamman/latin-bert. This repo repackages the same weights for use with HuggingFace transformers.

Note: This is an experimental repackaging. If you encounter any issues, please open a thread in the Discussion tab.

Model Details

• Version: 1.2.1 (no-code tokenizer — see Changelog)
• Architecture: BERT-base (12 layers, 768 hidden, 12 attention heads)
• Parameters: ~111M
• Vocab size: 32,900 (SubwordTextEncoder)
• Max sequence length: 512
• Training data: Latin texts (see paper for details)

Install

pip install transformers torch

Usage

Basic: Get contextual embeddings

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("latincy/latin-bert")
model = AutoModel.from_pretrained("latincy/latin-bert")

inputs = tokenizer("Gallia est omnis divisa in partes tres", return_tensors="pt")
outputs = model(**inputs)

# outputs.last_hidden_state: (batch, seq_len, 768)

Masked language model (fill-mask)

from transformers import AutoTokenizer, BertForMaskedLM
import torch

tokenizer = AutoTokenizer.from_pretrained("latincy/latin-bert")
model = BertForMaskedLM.from_pretrained("latincy/latin-bert")

text = "Gallia est omnis [MASK] in partes tres"
inputs = tokenizer(text, return_tensors="pt")
mask_idx = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]

with torch.no_grad():
    logits = model(**inputs).logits

top5 = logits[0, mask_idx, :].topk(5).indices.squeeze()
for token_id in top5:
    print(tokenizer.decode([token_id.item()]))

Tokenizer

The original Latin BERT uses a tensor2tensor SubwordTextEncoder, not standard WordPiece. As of v1.2.0 this repo ships a pure-JSON tokenizer.json that replicates it exactly using only built-in tokenizers components, so no trust_remote_code is needed — AutoTokenizer.from_pretrained just works, and the tokenizer is usable from any library that consumes tokenizer.json directly (Rust, JavaScript, ONNX runtimes, Text Embeddings Inference).

The Python reference implementations (tokenization_latin_bert.py, tokenization_latin_bert_fast.py) remain in the repo; code pinned to trust_remote_code=True keeps working and produces identical IDs.

Two edge cases diverge from the original tensor2tensor encoder, neither reachable with real text: characters outside the model's 2,187-character alphabet (which covers Latin, Greek, Cyrillic, digits, and punctuation) map to [UNK] instead of ordinal escapes, and the control character U+0001 is stripped from input. The [UNK] mapping is arguably an improvement rather than a regression — the original escapes such characters to digit-sequence subtokens (e.g. ἓ → \ 79 55 ;) that carry no meaning for the model, whereas [UNK] is BERT's intended representation for unknown input — but it is a difference, and it is reported as one.

Reproduces the Paper's Case Studies

All results below were produced through the native HF pipeline (AutoModel + AutoTokenizer) and match Bamman & Burns (2020):

POS tagging (Table 1)

Treebank	Accuracy
Perseus	95.2%
PROIEL	98.2%
ITTB	99.2%

Masked word prediction (Table 3)

Metric	Score
P@1	33.1%
P@10	62.2%
P@50	74.0%

Word sense disambiguation (Table 2)

Metric	Score
Accuracy	0.764

Contextual nearest neighbors (Tables 6–7)

Query	Top cosine	Retrieved
in (Caes. BG 1.1)	0.835	same-sense instances of in, as in the paper
audentes (Verg. Aen. 10.284)	0.919	lexical variants audentis, audaces, audentem

spaCy Integration

Works with spacy-transformers:

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "latincy/latin-bert"

No tokenizer_config overrides are needed as of v1.2.0 — the default fast tokenizer loads from tokenizer.json with no custom code.

Changelog

v1.2.1

• Fix: trust_remote_code=True loads failed under tokenizers 0.23 (TypeError: BertProcessing.__new__() got an unexpected keyword argument 'cls'); discussion #4.
• BertProcessing now takes positional args in tokenization_latin_bert_fast.py and scripts/build_tokenizer_json.py; works on tokenizers 0.22.2 and 0.23.3.
• tokenizer.json byte-identical; IDs and weights unchanged.

v1.2.0

• AutoTokenizer.from_pretrained("latincy/latin-bert") no longer needs trust_remote_code: new tokenizer.json (pure tokenizers components), tokenizer_class = PreTrainedTokenizerFast in tokenizer_config.json.
• Loaders with no revision now get the JSON tokenizer on main, which was v1.1.2 (remote-code). Under transformers 5.x, plain loads prefer tokenizer.json.
• IDs identical to v1.1.2 on all 15,444 infilling texts, all 47,743 WSD texts, all 2,187 alphabet characters, and 29,983 treebank sentences except one wordform. Weights unchanged.
• Divergence 1: characters outside the alphabet give [UNK] instead of \<ord>; digit escapes (seen: ἓν, U+1F13).
• Divergence 2: literal U+0001 is stripped from input.
• trust_remote_code=True still works and gives identical IDs; the Python tokenizer classes stay as the reference.
• scripts/build_tokenizer_json.py regenerates tokenizer.json; --check runs in the tests.
• transformers 5.x ignores use_fast=False for remote-code tokenizers; import LatinBertTokenizer directly for the slow class.

v1.1.2 — Bug fix: whitespace-split before subword encoding

The HF tokenizer's full-string path did character-class splitting on the entire string. Unlike the original tensor2tensor tokenizer.encode (and the GitHub gen_berts.py LatinTokenizer, which splits on whitespace first), it did not drop inter-word spaces, so it escaped every space to \32;_ and injected spurious subtokens between every word. This corrupted masked-word predictions and broke subword→word alignment when calling tokenizer("full sentence") (the documented usage). Both the slow and fast tokenizers now whitespace-split first, so the native string API reproduces the original per-word tokenization with no is_split_into_words needed. The fast tokenizer also gained the do_lower_case normalizer it was previously missing (it had ignored the flag, escaping capitals to codepoints exactly as the slow tokenizer did before v1.1.1). Decode now renders word boundaries as spaces (BERT-style; "tres." → "tres .").

Verified: the native full-string path reproduces the paper's infilling metrics exactly (P@1 33.1% / P@10 62.2% / P@50 74.0%, n=1161), identical to the per-word path, and recovers the reported masked words (e.g. Caesar aliam, Livy primo). The full case-study suite reproduces through the native path — POS tagging (95.2 / 98.2 / 99.2%), WSD (0.764), and the §4.4 contextual nearest neighbors (Tables 6–7: top cosine 0.835 for in and 0.919 for audentes, with the paper's lexical variants audentis / audaces / audentem retrieved).

Responds to discussion #3 — thanks to aimgo for the report.

v1.1.1 — Bug fix: add do_lower_case=True to tokenizer

The original Latin BERT vocabulary was trained on lowercased text. All original case studies (POS tagging, WSD, infilling) explicitly called .lower() before tokenizing. The HF PreTrainedTokenizer wrapper was missing this step, causing uppercase characters to be escaped to their ASCII codepoints (e.g. C → \67;), inflating token counts ~4x and producing embeddings the model was never trained on. The tokenizer now lowercases input by default (do_lower_case=True), matching the original pipeline behavior.

v1.1.0 — HuggingFace repackaging

Repackaged the original tensor2tensor SubwordTextEncoder tokenizer and PyTorch weights as a HuggingFace PreTrainedTokenizer + safetensors model.

v1.0.0 — Original model

Bamman & Burns (2020) Latin BERT weights and tensor2tensor tokenizer.

Citation

@article{bamman2020latin,
  title={Latin BERT: A Contextual Language Model for Classical Philology},
  author={Bamman, David and Burns, Patrick J},
  journal={arXiv preprint arXiv:2009.10053},
  year={2020}
}

Acknowledgments

This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.

Issue reports

• aimgo: tokenizers 0.23 BertProcessing error (#4), fixed in v1.2.1
• aimgo: word alignment issue (#3), fixed in v1.1.2
• thiagotps: do_lower_case=True not working (#2), fixed in v1.1.1 (tag re-cut 2026-05-04)