Koine-Greek-BERT (v1.0.1)

A domain-adapted BERT for Koine Greek, continued-pretrained from pranaydeeps/Ancient-Greek-BERT with a 50,000-token vocabulary extended for Koine word forms.

The model is accent-insensitive by design. Polytonic and monotonic input are read interchangeably (λόγος and λογος give the same token id), which makes it robust to differences of editorial accentuation. Diacritics are discarded during normalisation, so the model does not represent accentuation and its output cannot recover it.

Notices

• Tokenizer correction, 29 September 2026 (v1.0.1; weights unchanged). In v1.0, 2,476 of the 15,000 added tokens become identical to another added token once accents are stripped. Transformers 4.x and 5.0–5.15 resolve each collision to the lowest id, the tokenisation used in training and evaluation. From transformers 5.16 (tested: 5.16.0, 5.17.0, 5.20.0) the resolution depends on Python's hash seed, so the same word can receive a different id in every process; under 5.17 the v1.0 files give an evaluation perplexity of about 4,460 instead of about 10. v1.0.1 retires the 2,476 duplicates (ids kept, labels replaced by [unused_v1.0.1_<id>]), so the lowest id survives under every version. Verified: identical input ids for v1.0.1 under 5.17 and 4.57.3 and for v1.0 under 4.57.3, on 3,000 validation lines and 77,700 words of New Testament text; identical ids for v1.0.1 under different hash seeds; and identical ids for v1.0 and v1.0.1 under 5.0.0 (the version recorded for training) on all 8,881 validation lines and different hash seeds. With transformers 5.16 or later, update the v1.0 tokenizer files and recompute results obtained with them.
• Usage note, 30 September 2026. Added tokens are also matched inside words (see Known limitations), so a sentence tokenised as one string yields word_ids() in which the fragments of a split word count as separate words; code that expects one vector per whitespace-separated word will fail or silently misalign. Pass pre-split words with is_split_into_words=True (see How to use). Input ids are identical in both modes.
• Documentation correction, 23 September 2026. Earlier versions of this card described the vocabulary as "50,000 (Polytonic)" and the model as adapted to "full polytonic accentuation". This was inaccurate: the 15,000 added tokens were trained on polytonic text but registered with normalized: true, so their accent-stripped forms are what the tokenizer matches.

Model details

Base model	pranaydeeps/Ancient-Greek-BERT
Architecture	BERT, 12 layers, 768 hidden, 12 attention heads, 512 max positions
Parameters	124,492,880 (float32)
Vocabulary	50,000 ids: 35,000 base WordPiece entries (including the 5 special tokens) and 15,000 added; 2,476 duplicate added ids retired in v1.0.1
Objective	Masked language modelling
Language	Koine Greek (grc)

Tokenisation

The normaliser is BertNormalizer with lowercase=true; strip_accents is unset and therefore follows lowercase, so all diacritics are removed before matching. The added tokens carry normalized: true and are matched in accent-stripped form too.

v0.1 rejected accented input (λόγος → ['[UNK]']; ἐν ἀρχῇ ἦν ὁ λόγος → five [UNK]) and fragmented the discourse particles (ουν → ['ο','##υ','##ν'], μεν → ['με','##ν']). v1.0 accepts both forms (['λογος']; ['εν','αρχη','ην','ο','λογος']) and gives οὖν, μέν, οὐ, τίς and ἦν whole-word tokens, which matters for analyses of function words and particles.

Known limitations

Of the 15,000 added tokens, 11,016 are whole-word strings and 3,984 begin with ##. After accent-stripping:

• 12,524 are distinct forms and 2,476 duplicate another added token (1,759 whole-word duplicates in 1,285 collision groups, and 717 ## duplicates); v1.0.1 retires the duplicates.
• 2,814 distinct forms coincide with a base-vocabulary entry; for the 2,183 whole-word forms among them the base id is no longer produced. 9,710 are new forms.
• The 3,984 ## entries are never matched, since added tokens are matched as literal strings in normalised text, where ## does not occur; the base WordPiece continuation pieces are unaffected.
• Added tokens are matched inside words, splitting the word before WordPiece applies, e.g. κλητὸς → ['κλη', 'το', 'ς'], ἀφωρισμένος → ['αφωρισ', 'μεν', 'ος']; 20.4% of the 77,700 words of the New Testament letters tested are split in this way. The model was trained with this behaviour, so v1.0.1 keeps it: changing it (e.g. single_word registration) would alter the input ids and move the input away from the training distribution.
• Word-level work. On 518 windows of 150 words from the New Testament letters, string-mode tokenisation yields 209 word units per window on average and a mismatch in every window (v0.1: exactly 150); with is_split_into_words=True the word mapping is correct and the input ids are unchanged in all 518 windows (tokenizers 0.23, 30 September 2026). The problem was reported for a word-alignment pipeline tokenising whole verses (491 of 552 sentences affected).
• Token labels. The label of the surviving (lowest) id need not be the most frequent spelling: καὶ (35244), καί (35583), καῖ (37615), καϊ (40692) and καἰ (44466) collide, and 35244 is produced (an earlier version of this card gave 40692, the outcome of one transformers 5.17 process). Decoded output is not evidence about accentuation; tables built from convert_ids_to_tokens should be normalised before publication.

How to use

Plain strings suffice for sentence-level use (masked-language modelling, pooled vectors). To map sub-word vectors back to words, pass pre-split words:

from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("ABeZet/Koine-Greek-BERT", revision="v1.0.1")
model = AutoModel.from_pretrained("ABeZet/Koine-Greek-BERT", revision="v1.0.1")
words = "Παῦλος κλητὸς ἀπόστολος ἀφωρισμένος".split()
enc = tok(words, is_split_into_words=True, return_tensors="pt")
enc.word_ids()   # [None, 0, 1, 1, 1, 2, 3, 3, 3, None]: one index per input word

Training data

An assembled corpus of 177,608 lines (168,727 train, 8,881 validation). It is not published: its sources carry different licences, including non-commercial and share-alike ones, and redistribution has not been cleared. The tables below allow it to be assessed and reconstructed from the sources.

Source	Contribution	Licence
MorphGNT / SBLGNT	Greek New Testament	SBLGNT text CC BY 4.0; MorphGNT analysis CC BY-SA 3.0
Byzantine Majority Text	Greek New Testament, Byzantine tradition	Public domain (Unlicense)
Swete Septuagint	Greek Old Testament	CC BY-SA 4.0
First1KGreek	Philo, Apostolic Fathers, Origen, Eusebius, Clement of Alexandria, Athanasius, Gregory of Nazianzus, Epiphanius	CC BY-SA 4.0
Perseus Digital Library	Josephus, Polybius, Epictetus, Appian, Diodorus Siculus, Strabo	CC BY-SA 4.0
Gorman Dependency Trees	Classical Attic and Ionic prose; passages of Polybius, Diodorus Siculus, Josephus, Appian, Plutarch, Dionysius of Halicarnassus and Athenaeus	CC BY-NC-SA 4.0 (repository README; its licence file states CC BY-SA 4.0)
PROIEL Treebank	Greek New Testament, Herodotus, a Byzantine chronicle	CC BY-NC-SA 3.0 (US)

Composition by content, computed from the corpus manifest:

Component	Lines	Share
Patristic (Origen, Eusebius, Epiphanius, Athanasius, Clement of Alexandria, Gregory of Nazianzus)	33,714	18.98%
Secular Koine (Polybius, Diodorus Siculus, Strabo, Appian, Epictetus)	33,019	18.59%
Septuagint (Swete)	29,023	16.34%
New Testament (three text forms)	27,400	15.43%
Classical Attic and Ionic prose (Herodotus, Xenophon, Demosthenes, Thucydides, Plato, Aristotle, Attic orators)	23,362	13.15%
Jewish Koine (Philo, Josephus)	21,773	12.26%
Atticising prose of the Roman period (Plutarch, Dionysius of Halicarnassus, Athenaeus)	5,033	2.83%
Apostolic Fathers (1 Clement, Hermas, Ignatius, Barnabas, Didache, Polycarp)	3,259	1.83%
Byzantine Greek (a chronicle internally dated to ca. 1400 CE)	1,008	0.57%
Latin (Cicero, Ad Atticum, ingested in error from the multilingual PROIEL corpus)	17	0.01%

The New Testament is 15.43% of the training data, in three text forms: PROIEL (6.48%), the Byzantine majority text (4.49%) and SBLGNT/MorphGNT (4.46%). The base model, Ancient-Greek-BERT, was also pre-trained on data that include PROIEL and therefore the Greek New Testament. Without explicit caveat, the model is not suitable for authorship attribution, interpolation detection or any other evaluation on New Testament text, since such an evaluation cannot separate stylistic signal from memorisation.

About a sixth of the corpus is not Koine: classical Attic and Ionic prose (13.15%) and atticising prose of the Roman period (2.83%), included deliberately to broaden syntactic coverage. The model's name is not a statement about the composition of its training data.

Training procedure

Two-phase domain-adaptive pre-training (DAPT) on 2 × T4 GPUs. The configuration files saved at the end of both phases record transformers 5.0.0 (the published config.json records 4.57.3, the version used for the upload). In both phases all weights were trained (no layer frozen), with 15% masking, one example per corpus line truncated at 512 tokens, effective batch size 64, a linear learning-rate schedule, weight decay 0.01 and seed 42.

• New embeddings. Of the 15,000 added tokens, the 3,964 whose accent-stripped form is a base-vocabulary entry (2,814 distinct forms) were initialised as a copy of that entry's embedding; the other 11,036 as the mean of the 35,000 base embeddings plus Gaussian noise (s.d. 0.02).
• Phase 1: up to 4 epochs with early stopping on validation loss (best checkpoint kept), learning rate 2e-4, 500 warm-up steps, float16 mixed precision (batch 8, gradient accumulation 4).
• Phase 2: from the phase-1 model, 4 epochs, learning rate 5e-5, 200 warm-up steps, float32 (batch 4, gradient accumulation 8); float16 was abandoned in this phase after NaN gradient overflow on T4.

Evaluation

Metric	Phase 1	Phase 2 (v1.0)
Evaluation loss	2.5946	2.3423
Perplexity	13.39	10.41

On val_v1.txt (8,881 lines), CPU, single device, transformers 4.57. Check of the tokenizer correction (29 September 2026; same weights, transformers 5.17):

Tokenizer	Validation lines	Evaluation loss	Perplexity
v1.0 files (hash seed 1)	400	8.40	≈ 4,460
v1.0.1 files	400	2.30	10.00
v1.0.1 files	8,881 (full set)	2.3448	10.43

The full-set figure follows the original procedure (maximum length 512, batches of 16, 15% random masking seeded by batch number, mean of batch losses) and reproduces the v1.0 result of May 2026 within the variation of random masking.

Caveat on the split. The validation set is a random 5% of lines, drawn from the same works as the training set, and was also used for checkpoint selection and early stopping in phase 1. The perplexity therefore measures fit to the training distribution, not generalisation; a split held out by work would give a higher and more informative figure.

Zero-shot cloze probes (v1.0.1, transformers 5.17): ἐν ἀρχῇ ἦν ὁ λόγος , καὶ ὁ λόγος ἦν πρὸς τὸν θεόν , καὶ θεὸς ἦν ὁ [MASK] . → λόγος at rank 1; ἀγαπήσεις τὸν [MASK] σου ὡς σεαυτόν . → πλησίον at rank 1. The accents shown are stored labels, not predictions.

Intended use

Suitable for masked-language modelling over Koine and related Ancient Greek prose, feature extraction, and fine-tuning on downstream tasks over non-New-Testament Greek. Not suitable, without additional controls, for evaluation on New Testament text (training-data overlap), tasks requiring accentual or prosodic information, or faithful orthographic reconstruction from token labels.

Licence

CC BY-NC-SA 4.0 (attribution, non-commercial, share-alike), inherited from the training data: PROIEL (18,190 lines, 10.24% of the corpus) is CC BY-NC-SA 3.0 (US), and the README of the Gorman Dependency Trees (29,547 lines, 16.64%) states CC BY-NC-SA 4.0. Non-commercial use only.

Roadmap

A separate control model, Koine-Greek-BERT-noNT, is planned within the NEURAL SCRIBE project (NCN SONATINA 10). It will exclude the whole New Testament from its training data, so that New Testament text remains available for evaluation, and will contain no non-commercially licensed sources. No release date is fixed.

Version history

• Model card, 16 October 2026. Corrected: training procedure (no layer frozen in phase 1; phase-1 learning rate, warm-up, precision and early stopping), initialisation of the new embeddings, transformers version recorded for training, releases affected by the tokenizer collisions (5.0, used in training, is not), composition table, the New Testament in the base model's data, and the licences of the SBLGNT, Byzantine, Perseus, Gorman and PROIEL sources.
• Model card, 30 September 2026. Usage note and How to use on word-level tokenisation. Neither card revision changes the model or tokenizer files.
• v1.0.1 (current, 29 September 2026). Tokenizer files corrected so that collisions resolve deterministically to the lowest id under every transformers version; 2,476 duplicate entries retired (ids kept, labels replaced by placeholders). Weights and configuration unchanged.
• v1.0. Base pranaydeeps/Ancient-Greek-BERT; vocabulary extended to 50,000, new embeddings copied from the base vocabulary where the form exists, otherwise mean plus noise; two-phase DAPT. Documentation corrected 23 September 2026.
• v0.1 (deprecated). Proof of concept on nlpaueb/bert-base-greek-uncased-v1 (Modern Greek BERT), 35,000-token vocabulary, polytonic input not accepted.

Citation

@misc{zieminska_koine_greek_bert,
  author       = {Ziemińska, Agnieszka},
  title        = {Koine-Greek-BERT},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10565},
  howpublished = {\url{https://huggingface.co/ABeZet/Koine-Greek-BERT}}
}