---
language:
- ba
license: other
pretty_name: BashkirRoBERTa
library_name: transformers
pipeline_tag: fill-mask
tags:
- bashkir
- masked-language-modeling
- roberta
- sentencepiece
- custom-code
- onnx
- onnxruntime
---

# BashkirRoBERTa

> A masked language model for Bashkir, for fill-mask, spellchecking and
> foundation fine-tuning.

## Overview

A masked language model for Bashkir. Given a sentence with one `[MASK]` token, it
predicts the most probable missing Bashkir token from context. The model is useful
for fill-mask experiments, spellchecking and as a foundation for further
fine-tuning. It preserves a custom Pre-LayerNorm architecture rather than the
stock post-LayerNorm RoBERTa implementation.

| At a glance | |
| --- | --- |
| Task | Masked language modelling / fill-mask |
| Default artifact | `model.safetensors` (Transformers) or `onnx/model_int8.onnx` (ONNX) |
| Source | A monolingual Bashkir-language dataset |
| Version / license | v1 / custom terms (`other`) |

## Contents

### Files and Configurations

| File | Purpose | Size |
| --- | --- | ---: |
| `model.safetensors` | PyTorch weights for Transformers | 200.2 MB |
| `onnx/model_fp16.onnx` | FP16 ONNX model for GPU / DirectML | 121.2 MB |
| `onnx/model_int8.onnx` | INT8 ONNX model for fast CPU / mobile | 60.9 MB |
| `spm_bashkir_bert_16k.model` | SentencePiece tokenizer | — |
| `config.json` | Model configuration (`auto_map` for custom code) | — |
| `configuration_bashkir_roberta.py`, `modeling_bashkir_roberta.py`, `tokenization_bashkir_roberta.py` | Custom Pre-LayerNorm implementation | — |
| `tokenizer_config.json` | Tokenizer configuration | — |
| `META.json` | Release passport and artifact hashes | — |
| `SHA256SUMS` | Release checksums | — |

### Model Architecture

| Property | Value |
| --- | --- |
| Task | Masked language modelling / fill-mask |
| Architecture | Pre-LayerNorm Transformer encoder |
| Transformer blocks | 8 |
| Hidden size / attention heads | 640 / 10 |
| Feed-forward size | 2,560 |
| Context window | 256 subword tokens |
| Parameters | 50.04M |
| Tokenizer | SentencePiece BPE, 16,384 tokens |

The output embedding matrix is tied to the input word embeddings. Token IDs are
fixed: `<pad>` 0, `<unk>` 1, `<s>` 2, `</s>` 3, `[CLS]` 4, `[SEP]` 5 and
`[MASK]` 6.

### Examples

Outputs from the INT8 ONNX model on CPU:

| Input | Top prediction |
| --- | --- |
| `Мин башҡорт телен [MASK].` | `яратам` |
| `Башҡортостан — беҙҙең [MASK].` | `республика` |
| `Өфө — ҙур [MASK].` | `ҡала` |
| `Бөгөн Өфөлә яңы [MASK] асылды.` | `мәсет` |

## Method

The model was pretrained with dynamic masked-language modelling on a monolingual
Bashkir-language dataset assembled from encyclopedic, periodical and literary
sources. The source texts are not distributed in this repository.

### Evaluation

On a held-out Bashkir encyclopedic evaluation set the project reports **24.7%
top-1** and **54.0% top-5** accuracy for masked subword prediction. These are
diagnostic MLM results, not a general-purpose language-understanding score: a mask
may represent a whole word or a SentencePiece subword fragment.

## Quality and Use

This is a research model, not a production language service. Fill-mask predictions
are ranking suggestions that require context-appropriate review, especially for
ambiguous or short contexts. The checkpoint is released under custom terms while
the source-rights audit is completed.

### Limitations

- Diagnostic MLM accuracy only; not fine-tuned for any downstream task.
- A mask may correspond to a partial subword, not always a full word.
- Predictions reflect the training corpus and may prefer frequent or encyclopedic phrasing.
- No training texts are redistributed; provenance or removal requests go through the maintainer.

## Usage

```bash
pip install transformers torch huggingface_hub
```

PyTorch (Transformers), which requires `trust_remote_code=True` because of the
custom Pre-LayerNorm architecture:

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "failed09/bashkir-roberta"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True)

inputs = tokenizer("Мин башҡорт телен [MASK].", return_tensors="pt")
logits = model(**inputs).logits
mask_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
prediction_id = logits[0, mask_index].argmax().item()
print(tokenizer.decode([prediction_id]))  # яратам
```

ONNX Runtime for CPU and edge deployment:

```python
import numpy as np
import onnxruntime as ort
import sentencepiece as spm
from huggingface_hub import hf_hub_download

model_path = hf_hub_download("failed09/bashkir-roberta", "onnx/model_int8.onnx")
sp_path = hf_hub_download("failed09/bashkir-roberta", "spm_bashkir_bert_16k.model")

session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
sp = spm.SentencePieceProcessor(model_file=sp_path)

tokens = [2] + sp.encode("Мин башҡорт телен ") + [6] + sp.encode(".") + [3]
mask_idx = tokens.index(6)
logits = session.run(None, {"input_ids": np.array([tokens], dtype=np.int64)})[0][0, mask_idx]
top_tokens = np.argsort(logits)[::-1][:5]
print([sp.decode([int(t)]) for t in top_tokens])  # ['яратам', 'беләм', 'өйрәнә', ...]
```

## License

The checkpoint is released under custom terms (`other` on the Hub) while the
source-rights audit is completed. No training texts are redistributed. For
provenance or removal requests, contact the maintainer through the Hub.

## Citation

```bibtex
@software{failed09_bashkir_roberta_2026,
  title = {BashkirRoBERTa},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/failed09/bashkir-roberta},
  note = {Masked language model for Bashkir}
}
```

## Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development,
preservation and practical use of the Bashkir language. Other related models,
datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other
materials used to derive this release; rights and licensing remain with the
original authors, publishers and dataset providers. Source texts are not
redistributed in this repository, so users should follow the licenses and
attribution requirements of the relevant upstream resources.