Card: unified structure and packaging
Browse files- README.md +115 -70
- SHA256SUMS +5 -4
README.md
CHANGED
|
@@ -2,6 +2,8 @@
|
|
| 2 |
language:
|
| 3 |
- ba
|
| 4 |
license: apache-2.0
|
|
|
|
|
|
|
| 5 |
pipeline_tag: text-classification
|
| 6 |
tags:
|
| 7 |
- bashkir
|
|
@@ -14,12 +16,41 @@ tags:
|
|
| 14 |
|
| 15 |
# Bashkir LID (Binary BA vs NON_BA)
|
| 16 |
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
CPU inference. The model was trained with hard negatives from sister Turkic
|
| 20 |
-
languages, Russian and technical/noisy text, with special attention to Tatar.
|
| 21 |
|
| 22 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
| Property | Description |
|
| 25 |
| --- | --- |
|
|
@@ -30,30 +61,79 @@ languages, Russian and technical/noisy text, with special attention to Tatar.
|
|
| 30 |
| Preprocessing | Unicode lowercase and dual-channel sparse features, provided by `lid.py` |
|
| 31 |
| Outputs | Class labels (`ba`, `non_ba`) and probabilities |
|
| 32 |
|
| 33 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
| BA probability threshold | BA recall | Non-BA false acceptance |
|
| 40 |
| ---: | ---: | ---: |
|
| 41 |
| 0.50 | 73.0% (730/1,000) | 0/3,000 |
|
| 42 |
-
| 0.40 | 78.3% (783/1,000) | 2/3,000
|
| 43 |
-
| 0.35 | 80.8% (808/1,000) | 4/3,000
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
|
| 45 |
-
|
| 46 |
-
when preserving more Bashkir text matters, preferably with dictionary and
|
| 47 |
-
quality checks. The price of the strict gate is false rejection of some valid
|
| 48 |
-
Bashkir sentences, especially short or BA/Tatar-like sentences.
|
| 49 |
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
Model version, feature count and export checks are recorded in
|
| 54 |
-
[`META.json`](META.json); the runtime contract is in [`config.json`](config.json).
|
| 55 |
|
| 56 |
-
##
|
| 57 |
|
| 58 |
```bash
|
| 59 |
pip install huggingface_hub onnxruntime numpy
|
|
@@ -68,6 +148,7 @@ model_dir = snapshot_download(
|
|
| 68 |
allow_patterns=[
|
| 69 |
"lid.py", "config.json", "META.json", "model.onnx", "vectorizer.json"
|
| 70 |
],
|
|
|
|
| 71 |
)
|
| 72 |
sys.path.insert(0, model_dir)
|
| 73 |
from lid import LanguageIdentifier
|
|
@@ -79,72 +160,36 @@ print(lid.predict_proba(texts))
|
|
| 79 |
print(lid.classes.tolist()) # probability-column order
|
| 80 |
```
|
| 81 |
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
For a local checkout with model files alongside `lid.py`,
|
| 86 |
-
`LanguageIdentifier()` also works. The source project's `model/` layout
|
| 87 |
-
remains supported.
|
| 88 |
-
|
| 89 |
-
## Related Multilingual Model
|
| 90 |
-
|
| 91 |
-
Use this binary model when the main question is whether a sentence should pass
|
| 92 |
-
a Bashkir corpus gate. If you need to identify the predicted language, use the
|
| 93 |
-
companion [Bashkir Multiclass LID](https://huggingface.co/failed09/bashkir-lid-multiclass),
|
| 94 |
-
which distinguishes `ba`, `tt`, `ru` and `other`.
|
| 95 |
-
|
| 96 |
-
## Training and Evaluation
|
| 97 |
-
|
| 98 |
-
The model uses a dual-channel linear classifier trained for the BA/non-BA
|
| 99 |
-
decision. Training texts are not distributed in this repository.
|
| 100 |
-
|
| 101 |
-
The ONNX export was checked against the original classifier for prediction
|
| 102 |
-
and probability agreement, including Unicode edge cases. These checks
|
| 103 |
-
validate the export, rather than measure language-identification accuracy.
|
| 104 |
-
On the source-project benchmark, the model achieved 93.25% binary accuracy:
|
| 105 |
-
it retained 730/1,000 BA examples at the default 0.50 threshold and rejected
|
| 106 |
-
3,000/3,000 non-BA examples. The separate hand-authored adversarial suite
|
| 107 |
-
passed 26/26, including Tatar, Kazakh, Chuvash, Russian with Bashkir inserts,
|
| 108 |
-
code and noise. An independent 100-example stress set achieved 97/100.
|
| 109 |
-
These evaluations use weak or manually constructed labels and are not an
|
| 110 |
-
expert-reviewed Gold benchmark.
|
| 111 |
-
|
| 112 |
-
## Limitations
|
| 113 |
-
|
| 114 |
-
This is a binary gate, not a general multilingual language detector. It does
|
| 115 |
-
not identify which non-BA language was found. Tatar and other languages,
|
| 116 |
-
short strings, names, OCR artifacts and mixed-language text can be ambiguous.
|
| 117 |
-
Valid Bashkir text does not always contain Bashkir-specific letters. Empty
|
| 118 |
-
strings and technical noise should be handled separately when filtering a
|
| 119 |
-
corpus. For high-recall corpus construction, combine this model with the
|
| 120 |
-
[multiclass LID](https://huggingface.co/failed09/bashkir-lid-multiclass),
|
| 121 |
-
dictionaries, quality rules and human review.
|
| 122 |
|
| 123 |
## License
|
| 124 |
|
| 125 |
-
The code and model export are released under Apache-2.0. Training texts
|
| 126 |
-
|
| 127 |
|
| 128 |
## Citation
|
| 129 |
|
| 130 |
```bibtex
|
| 131 |
-
@software{
|
| 132 |
title = {Bashkir LID},
|
| 133 |
author = {failed09},
|
| 134 |
year = {2026},
|
| 135 |
publisher = {Hugging Face},
|
| 136 |
-
url = {https://huggingface.co/failed09/bashkir-lid}
|
|
|
|
| 137 |
}
|
| 138 |
```
|
| 139 |
|
| 140 |
## Open Bashkir Data and Sources 🐝
|
| 141 |
|
| 142 |
-
This
|
| 143 |
preservation and practical use of the Bashkir language. Other related models,
|
| 144 |
datasets and tools are available on the author's Hugging Face profile.
|
| 145 |
|
| 146 |
The author does not claim ownership or authorship of the source texts or other
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
redistributed in this repository
|
| 150 |
-
|
|
|
|
| 2 |
language:
|
| 3 |
- ba
|
| 4 |
license: apache-2.0
|
| 5 |
+
pretty_name: Bashkir LID (Binary BA vs NON_BA)
|
| 6 |
+
library_name: onnxruntime
|
| 7 |
pipeline_tag: text-classification
|
| 8 |
tags:
|
| 9 |
- bashkir
|
|
|
|
| 16 |
|
| 17 |
# Bashkir LID (Binary BA vs NON_BA)
|
| 18 |
|
| 19 |
+
> Compact ONNX language gate that decides whether text is Bashkir, for corpus
|
| 20 |
+
> filtering and local CPU inference.
|
|
|
|
|
|
|
| 21 |
|
| 22 |
+
## Overview
|
| 23 |
+
|
| 24 |
+
A compact binary language gate for Bashkir (`ba` vs `non_ba`), designed for
|
| 25 |
+
corpus filtering, OCR triage and local applications that need lightweight CPU
|
| 26 |
+
inference. It was trained with hard negatives from sister Turkic languages,
|
| 27 |
+
Russian and technical/noisy text, with special attention to Bashkir/Tatar
|
| 28 |
+
separation.
|
| 29 |
+
|
| 30 |
+
| At a glance | |
|
| 31 |
+
| --- | --- |
|
| 32 |
+
| Task | Binary language identification (`ba` vs `non_ba`) |
|
| 33 |
+
| Default artifact | `model.onnx` + `lid.py` |
|
| 34 |
+
| Source | Bashkir, Turkic, Russian and technical text (not redistributed) |
|
| 35 |
+
| Version / license | v3 / Apache-2.0 |
|
| 36 |
+
|
| 37 |
+
## Contents
|
| 38 |
+
|
| 39 |
+
### Files and Configurations
|
| 40 |
+
|
| 41 |
+
| File | Purpose |
|
| 42 |
+
| --- | --- |
|
| 43 |
+
| `model.onnx` | Exported classifier graph and weights (opset 17) |
|
| 44 |
+
| `vectorizer.json` | Dual-channel character and word feature vocabularies |
|
| 45 |
+
| `lid.py` | Portable ONNX Runtime adapter |
|
| 46 |
+
| `config.json` | Runtime contract |
|
| 47 |
+
| `META.json` | Model passport, validation results and artifact hashes |
|
| 48 |
+
| `__init__.py` | Package marker |
|
| 49 |
+
| `SHA256SUMS` | Release checksums |
|
| 50 |
+
|
| 51 |
+
Training texts and the original `joblib` pipeline are not distributed.
|
| 52 |
+
|
| 53 |
+
### Model Architecture
|
| 54 |
|
| 55 |
| Property | Description |
|
| 56 |
| --- | --- |
|
|
|
|
| 61 |
| Preprocessing | Unicode lowercase and dual-channel sparse features, provided by `lid.py` |
|
| 62 |
| Outputs | Class labels (`ba`, `non_ba`) and probabilities |
|
| 63 |
|
| 64 |
+
The ONNX graph takes sparse feature tensors (`char_ids`, `char_counts`,
|
| 65 |
+
`word_ids`, `word_counts`); the adapter handles preprocessing, batching and
|
| 66 |
+
checksum validation. No custom ONNX operators or Transformers installation are
|
| 67 |
+
required.
|
| 68 |
+
|
| 69 |
+
### Examples
|
| 70 |
+
|
| 71 |
+
| Text | Prediction | `ba` probability |
|
| 72 |
+
| --- | --- | ---: |
|
| 73 |
+
| `Мин башҡорт телен яратам.` | `ba` | 0.998 |
|
| 74 |
+
| `Бары бала бара.` | `ba` | 0.927 |
|
| 75 |
+
| `Сегодня хорошая погода.` | `non_ba` | 0.044 |
|
| 76 |
+
| `SELECT id FROM users WHERE active = TRUE;` | `non_ba` | 0.011 |
|
| 77 |
|
| 78 |
+
Note the second row: valid Bashkir text does not always contain
|
| 79 |
+
Bashkir-specific letters.
|
| 80 |
+
|
| 81 |
+
## Method
|
| 82 |
+
|
| 83 |
+
The model is a dual-channel linear classifier: TF-IDF `char_wb` (2–5) and word
|
| 84 |
+
(1–2) features are fused and passed to an SGD classifier exported to ONNX. It was
|
| 85 |
+
trained with hard negatives from sister Turkic languages, Russian and
|
| 86 |
+
technical/noisy text. Training texts are not distributed in this repository.
|
| 87 |
+
|
| 88 |
+
### Evaluation
|
| 89 |
+
|
| 90 |
+
On a weak-label 4,000-sentence benchmark the model reached **93.25% binary
|
| 91 |
+
accuracy**: at the default 0.50 threshold it retained 730/1,000 Bashkir examples
|
| 92 |
+
and rejected 3,000/3,000 non-Bashkir examples. A hand-authored adversarial suite
|
| 93 |
+
passed 26/26 (Tatar, Kazakh, Chuvash, Russian with Bashkir inserts, code and
|
| 94 |
+
noise), and an independent 100-example stress set reached 97/100. Labels are weak
|
| 95 |
+
or manually constructed, not an expert-reviewed Gold benchmark; exact numbers are
|
| 96 |
+
recorded in [`META.json`](META.json).
|
| 97 |
+
|
| 98 |
+
### Operating thresholds
|
| 99 |
+
|
| 100 |
+
`predict()` selects `ba` when its probability is at least 0.50. Measured
|
| 101 |
+
operating points (from [`META.json`](META.json)):
|
| 102 |
|
| 103 |
| BA probability threshold | BA recall | Non-BA false acceptance |
|
| 104 |
| ---: | ---: | ---: |
|
| 105 |
| 0.50 | 73.0% (730/1,000) | 0/3,000 |
|
| 106 |
+
| 0.40 | 78.3% (783/1,000) | 2/3,000 |
|
| 107 |
+
| 0.35 | 80.8% (808/1,000) | 4/3,000 |
|
| 108 |
+
|
| 109 |
+
Use `0.50` for the strictest rejection of non-Bashkir text. Use `0.35–0.40` when
|
| 110 |
+
preserving more Bashkir text matters, preferably with dictionary and quality
|
| 111 |
+
checks. The price of the strict gate is false rejection of some valid Bashkir
|
| 112 |
+
sentences, especially short or BA/Tatar-like sentences.
|
| 113 |
+
|
| 114 |
+
## Quality and Use
|
| 115 |
+
|
| 116 |
+
This is a binary gate, not a general multilingual language detector. It does not
|
| 117 |
+
identify which non-BA language was found. Tatar and other languages, short
|
| 118 |
+
strings, names, OCR artifacts and mixed-language text can be ambiguous.
|
| 119 |
+
|
| 120 |
+
### Limitations
|
| 121 |
+
|
| 122 |
+
- Binary decision only: `ba` vs `non_ba`, with no language label for rejects.
|
| 123 |
+
- Weak-label benchmark; not an expert-reviewed Gold evaluation.
|
| 124 |
+
- Confidence values are model scores, not calibrated human probabilities.
|
| 125 |
+
- Empty strings, OCR fragments and technical noise should be handled with
|
| 126 |
+
explicit pipeline rules.
|
| 127 |
+
- For high-recall corpus construction, combine with the multiclass LID,
|
| 128 |
+
dictionaries, quality rules and human review.
|
| 129 |
|
| 130 |
+
## Related Resources
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
+
- [Bashkir Multiclass LID](https://huggingface.co/failed09/bashkir-lid-multiclass) —
|
| 133 |
+
distinguishes `ba`, `tt`, `ru` and `other`; use it when you need the language
|
| 134 |
+
label instead of a pass/reject gate.
|
|
|
|
|
|
|
| 135 |
|
| 136 |
+
## Usage
|
| 137 |
|
| 138 |
```bash
|
| 139 |
pip install huggingface_hub onnxruntime numpy
|
|
|
|
| 148 |
allow_patterns=[
|
| 149 |
"lid.py", "config.json", "META.json", "model.onnx", "vectorizer.json"
|
| 150 |
],
|
| 151 |
+
revision="main", # pin a reviewed commit for reproducible deployments
|
| 152 |
)
|
| 153 |
sys.path.insert(0, model_dir)
|
| 154 |
from lid import LanguageIdentifier
|
|
|
|
| 160 |
print(lid.classes.tolist()) # probability-column order
|
| 161 |
```
|
| 162 |
|
| 163 |
+
After downloading, inference runs locally without network access. For a local
|
| 164 |
+
checkout with the model files alongside `lid.py`, `LanguageIdentifier()` also
|
| 165 |
+
works without extra arguments.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
|
| 167 |
## License
|
| 168 |
|
| 169 |
+
The code and model export are released under Apache-2.0. Training texts are not
|
| 170 |
+
included in this repository.
|
| 171 |
|
| 172 |
## Citation
|
| 173 |
|
| 174 |
```bibtex
|
| 175 |
+
@software{failed09_bashkir_lid_2026,
|
| 176 |
title = {Bashkir LID},
|
| 177 |
author = {failed09},
|
| 178 |
year = {2026},
|
| 179 |
publisher = {Hugging Face},
|
| 180 |
+
url = {https://huggingface.co/failed09/bashkir-lid},
|
| 181 |
+
note = {Compact binary Bashkir language gate}
|
| 182 |
}
|
| 183 |
```
|
| 184 |
|
| 185 |
## Open Bashkir Data and Sources 🐝
|
| 186 |
|
| 187 |
+
This release is part of an open-source effort to support the development,
|
| 188 |
preservation and practical use of the Bashkir language. Other related models,
|
| 189 |
datasets and tools are available on the author's Hugging Face profile.
|
| 190 |
|
| 191 |
The author does not claim ownership or authorship of the source texts or other
|
| 192 |
+
materials used to derive this release; rights and licensing remain with the
|
| 193 |
+
original authors, publishers and dataset providers. Source texts are not
|
| 194 |
+
redistributed in this repository, so users should follow the licenses and
|
| 195 |
+
attribution requirements of the relevant upstream resources.
|
SHA256SUMS
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
adbfcad591642c7a70aa39fbd53969502327aa2c5750507390557facae469742 model.onnx
|
| 2 |
b9f3f823ce9fc7d22cca995915134d06dc4a31be61f08950751b94fc7a7b553d vectorizer.json
|
| 3 |
-
a64200ae65f0d6449500ec35d4c74a5ed0c43e10902886add7aa3ef406a5724c lid.py
|
| 4 |
-
839ecbad45a88253e2da2154e75744dd92ada5dbefdc3a68a83c8c1873865d1b config.json
|
| 5 |
-
bdc7e484e09fd4bbad81e6e05d0e11c7fb86a88f7c97a7383f6e9c14d6d66639 README.md
|
| 6 |
-
c06fea4773e32df0861cf38addf9426131480dcb5a45d36fbde8ba9de8fd851f META.json
|
|
|
|
| 1 |
+
c06fea4773e32df0861cf38addf9426131480dcb5a45d36fbde8ba9de8fd851f META.json
|
| 2 |
+
f8e267ef870cc080281cc902f1b772afe36f56a60d984fac21b12b10994790a3 README.md
|
| 3 |
+
02102c5d4fb2fe1b7e7d4da028b5e5c685d30b21b7c3507fc88e15ad4a93fc8e __init__.py
|
| 4 |
+
839ecbad45a88253e2da2154e75744dd92ada5dbefdc3a68a83c8c1873865d1b config.json
|
| 5 |
+
a64200ae65f0d6449500ec35d4c74a5ed0c43e10902886add7aa3ef406a5724c lid.py
|
| 6 |
adbfcad591642c7a70aa39fbd53969502327aa2c5750507390557facae469742 model.onnx
|
| 7 |
b9f3f823ce9fc7d22cca995915134d06dc4a31be61f08950751b94fc7a7b553d vectorizer.json
|
|
|
|
|
|
|
|
|
|
|
|