| --- |
| license: apache-2.0 |
| language: |
| - az |
| base_model: jhu-clsp/mmBERT-base |
| pipeline_tag: text-classification |
| tags: |
| - azerbaijani |
| - text-quality |
| - data-filtering |
| datasets: |
| - LocalDoc/azerbaijani-text-quality-labeled |
| --- |
| |
| # Azerbaijani Text Quality Classifier |
|
|
| Regression model that scores the quality of Azerbaijani web text on a |
| continuous 0-3 scale. Built to filter a raw web corpus (OSCAR-derived) |
| before language-model pretraining. |
|
|
| - **Base model:** jhu-clsp/mmBERT-base |
| - **Task:** regression, single output (~0..3). Higher = cleaner text. |
| - **Max length:** 4096 tokens |
|
|
| ## Score scale |
|
|
| - **3** — clean, coherent Azerbaijani prose |
| - **2** — substantial good prose mixed with junk (menus, footers, ads) |
| - **1** — mostly junk, little recoverable prose |
| - **0** — pure junk: navigation pages, spam, machine translation, non-Azerbaijani text |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("LocalDoc/azerbaijani-text-quality-classifier") |
| model = AutoModelForSequenceClassification.from_pretrained("LocalDoc/azerbaijani-text-quality-classifier") |
| model.eval() |
| |
| text = "..." |
| enc = tok(text, truncation=True, max_length=4096, return_tensors="pt") |
| with torch.no_grad(): |
| score = model(**enc).logits.squeeze().item() |
| print(score) |
| ``` |
|
|
| ## Limitations |
|
|
| Training labels were generated by an LLM (Mistral-Small-24B), not by humans. |
| Reported validation metrics (val-MSE ~0.14, rounded accuracy ~0.83) measure |
| **agreement with the LLM labels**, not agreement with human judgement — |
| the latter has not yet been measured against a human-annotated test set. |
| Use with this caveat in mind. |
