| --- |
| language: |
| - orv |
| - cu |
| tags: |
| - masked-language-modeling |
| - old-slavonic |
| - old-russian |
| - birchbark |
| - historical-nlp |
| - dual-embeddings |
| license: apache-2.0 |
| --- |
| |
| # DualEmbLM |
|
|
| A masked language model trained from scratch on Old East Slavic and Old Church Slavonic texts, |
| with dual character-level + word-level embeddings. |
|
|
| ## Architecture |
|
|
| DualEmbLM combines: |
| - **Character-level tokenisation** (1 character = 1 token) — enables precise lacuna restoration at the character level |
| - **Word-level context embeddings** — provides morphological and lexical context via a 50k word vocabulary |
| - **Transformer encoder** (BERT architecture, trained from scratch) — 6 layers, hidden size 512, 8 attention heads |
|
|
| The dual embeddings are concatenated and projected into the shared |
| hidden space before being passed to the transformer encoder. |
|
|
| ## Training |
|
|
| The model was trained on a corpus of Old Russian and Church Slavonic texts assembled from the following sources: |
|
|
| | Source | Language | Word Tokens | Link | |
| |--------|----------|--------|------| |
| | Birchbark manuscripts | Old Novgorodian (mostly) | 21,464 | [gramoty.ru](https://gramoty.ru) | |
| | Epigraphy | Old Church Slavonic (mostly) | 8,102 | [epigraphica.ru](https://epigraphica.ru) | |
| | DIACU | Old Church Slavonic; Church Slavonic (Old Russian, Middle Bulgarian, Serbian, Resava recensions); Middle Russian | 1,683,307 | [ACL Anthology](https://aclanthology.org/2025.bsnlp-1.12/) | |
| | TOROT | Old Russian; Church Slavonic | 682,430 | [torottreebank.github.io](https://torottreebank.github.io) | |
| | Bible (Ponomar) | Church Slavonic | 603,047 | [GitHub](https://github.com/typiconman/ponomar/tree/master/Ponomar/languages/cu/bible/elis) | |
| | Byliny | Old Russian (XI–XVII c.) | 430,103 | [rusneb.ru](https://rusneb.ru/catalog/000199_000009_003636356/) | |
| | Pushkin House | Old Russian | 256,503 | [lib2.pushkinskijdom.ru](https://lib2.pushkinskijdom.ru) | |
| | Military Statute (Part 2) | Old Russian | 49,787 | [rusneb.ru](https://rusneb.ru/catalog/000199_000009_004093983/) | |
| | NKRYA (historical) | Old Russian; Old Rus (XI–XVIII c.) | 42,412 | [ruscorpora.ru](https://ruscorpora.ru) | |
|
|
| Masking details: MLM probability 8%, span masking, edge masking, random gap augmentation. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForMaskedLM |
| |
| model = AutoModelForMaskedLM.from_pretrained( |
| "MaximEremeev/DualEmb-slav", |
| trust_remote_code=True, |
| ) |
| ``` |
|
|
| ## Tasks |
|
|
| - **Generated lacunae restoration** (Test A Hit@1: 0.817, CER: 0.183) |
| - **Real lacunae restoration** (Test B char Hit@1: 0.466, span Hit@1: 0.222) |
|
|
| ## Contact |
|
|
| Maxim Eremeev, maeremeev@edu.hse.ru |
