---
language:
  - orv
  - cu
tags:
  - masked-language-modeling
  - old-slavonic
  - old-russian
  - birchbark
  - historical-nlp
  - roformer
  - rope
  - bpe
license: apache-2.0
---

# RoFormerBPE

A masked language model trained from scratch on Old East Slavic and Old Church Slavonic texts,
using a RoFormer architecture with BPE tokenisation. Based on [mini-roformer-ancient-rus-v2](https://huggingface.co/AlexSychovUN/mini-roformer-ancient-rus-v2).

Note: BPE token boundaries do not always align with lacuna boundaries in editorial markup,
which inflates span-level CER. For character-level restoration tasks consider using
[DualEmbLM](https://huggingface.co/MaximEremeev/DualEmb-slav) instead.

## Architecture

- **Tokenisation**: BPE (Byte Pair Encoding), vocabulary size 50k
- **Architecture**: RoFormer encoder with Rotary Position Embeddings (RoPE)
- **Size**: 6 layers, hidden size 512, 8 attention heads


## Training

The model was trained on a corpus of Old Russian and Church Slavonic texts assembled from the following sources:

| Source | Language | Word Tokens | Link |
|--------|----------|--------|------|
| Birchbark manuscripts | Old Novgorodian (mostly) | 21,464 | [gramoty.ru](https://gramoty.ru) |
| Epigraphy | Old Church Slavonic (mostly) | 8,102 | [epigraphica.ru](https://epigraphica.ru) |
| DIACU | Old Church Slavonic; Church Slavonic (Old Russian, Middle Bulgarian, Serbian, Resava recensions); Middle Russian | 1,683,307 | [ACL Anthology](https://aclanthology.org/2025.bsnlp-1.12/) |
| TOROT | Old Russian; Church Slavonic | 682,430 | [torottreebank.github.io](https://torottreebank.github.io) |
| Bible (Ponomar) | Church Slavonic | 603,047 | [GitHub](https://github.com/typiconman/ponomar/tree/master/Ponomar/languages/cu/bible/elis) |
| Byliny | Old Russian (11th–17th c.) | 430,103 | [rusneb.ru](https://rusneb.ru/catalog/000199_000009_003636356/) |
| Pushkin House | Old Russian | 256,503 | [lib2.pushkinskijdom.ru](https://lib2.pushkinskijdom.ru) |
| Military Statute (Part 2) | Old Russian | 49,787 | [rusneb.ru](https://rusneb.ru/catalog/000199_000009_004093983/) |
| NKRYA (historical) | Old Russian (11th–18th c.) | 42,412 | [ruscorpora.ru](https://ruscorpora.ru) |

Masking details: MLM probability 8%, span masking, edge masking, random gap augmentation.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained(
    "MaximEremeev/RoFormer-slav",
    trust_remote_code=True,
)
model = AutoModelForMaskedLM.from_pretrained(
    "MaximEremeev/RoFormer-slav",
    trust_remote_code=True,
)
```

## Tasks

- **Generated lacunae restoration** (Test A Hit@1: 0.281, CER: 0.839)
- **Real lacunae restoration** (Test B char Hit@1: 0.145, span Hit@1: 0.021)

## Contact

Maxim Eremeev, maeremeev@edu.hse.ru