| --- |
| language: |
| - en |
| license: apache-2.0 |
| tags: |
| - text-classification |
| - dialect |
| - english |
| - deberta |
| datasets: |
| - surrey-nlp/BESSTIE-CW-26 |
| metrics: |
| - accuracy |
| - f1 |
| pipeline_tag: text-classification |
| --- |
| |
| # DiaLLM Dialect Classifier |
|
|
| A fine-tuned [DeBERTa-v3-base](https://huggingface.co/microsoft/deberta-v3-base) model for classifying English text into three dialect varieties: **en-AU** (Australian), **en-IN** (Indian), and **en-UK** (British). |
|
|
| Trained as part of the DiaLLM project — a study of dialect-adapted language models using CPT, SFT, DPO, GRPO, and GSPO across Gemma, Llama, and Qwen model families. Used as an independent evaluation metric to assess whether generated text exhibits target-dialect characteristics. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import pipeline |
| |
| classifier = pipeline( |
| "text-classification", |
| model="jordanpainter/diallm-dialect-classifier", |
| ) |
| |
| classifier("I reckon it's a ripper idea, mate.") |
| # [{'label': 'en-AU', 'score': 0.87}] |
| ``` |
|
|
| Labels: `en-AU`, `en-IN`, `en-UK`. |
|
|
| ## Training Data |
|
|
| Fine-tuned on [BESSTIE-CW-26](https://huggingface.co/datasets/surrey-nlp/BESSTIE-CW-26), a dataset of 6,243 naturally occurring English sentences annotated for dialect variety. All splits were pooled and re-split 80/10/10 with stratification to ensure balanced dialect representation in dev and test. |
|
|
| | Split | en-AU | en-IN | en-UK | Total | |
| |-------|-------|-------|-------|-------| |
| | Train | ~1,619 | ~1,973 | ~1,693 | ~5,285 | |
| | Val | ~202 | ~246 | ~211 | ~659 | |
| | Test | 192 | 234 | 201 | 627 | |
|
|
| ## Training Details |
|
|
| | Hyperparameter | Value | |
| |---|---| |
| | Base model | microsoft/deberta-v3-base | |
| | Epochs | 5 (early stopping, patience 2) | |
| | Batch size | 16 | |
| | Learning rate | 2e-5 | |
| | Warmup ratio | 0.1 | |
| | Weight decay | 0.01 | |
| | Max length | 512 | |
| | Hardware | 1× NVIDIA A100 | |
|
|
| ## Evaluation |
|
|
| Test-set results (627 examples, stratified): |
|
|
| | Dialect | Precision | Recall | F1 | |
| |---------|-----------|--------|----| |
| | en-AU | 0.6808 | 0.7552 | 0.7160 | |
| | en-IN | 0.8982 | 0.8675 | 0.8826 | |
| | en-UK | 0.7234 | 0.6766 | 0.6992 | |
| | **macro avg** | **0.7675** | **0.7664** | **0.7660** | |
| | **accuracy** | | | **0.7719** | |
|
|
| Indian English is the most separable class; Australian and British English share substantial lexical overlap, leading to some inter-class confusion between the two. |
|
|
| ## Limitations |
|
|
| - Trained on BESSTIE-CW-26, which contains shorter, naturally occurring sentences — performance may vary on longer generated text. |
| - Confusion between en-AU and en-UK is expected given their shared orthographic conventions. |
| - Not intended for high-stakes dialect identification; best used as a soft signal in aggregate across many examples. |
|
|
