| --- |
| language: |
| - hu |
| license: apache-2.0 |
| base_model: SZTAKI-HLT/hubert-base-cc |
| pipeline_tag: text-classification |
| tags: |
| - hungarian |
| - corruption |
| - news-classification |
| datasets: |
| - K-Monitor/kmdb_classification |
| metrics: |
| - accuracy |
| - f1 |
| --- |
| |
| # kmdb_classification_hubert_v5 |
| |
| Hungarian news article classifier for the |
| [K-Monitor press database](https://adatbazis.k-monitor.hu/): decides whether an |
| article belongs in the database, i.e. whether it describes a concrete case of |
| corruption, public procurement, public spending or transparency. |
| |
| Finetuned from `SZTAKI-HLT/hubert-base-cc` on |
| [`K-Monitor/kmdb_classification`](https://huggingface.co/datasets/K-Monitor/kmdb_classification). |
| |
| ## Input format |
| |
| The model is trained on one specific string. Anything else is out of |
| distribution: |
| |
| ```python |
| import re |
| |
| def clean(text): |
| return re.sub(r"\s+", " ", text or "").strip() |
| |
| def format_article(title, description, url, text=""): |
| domain = ".".join(url.split("/")[2].split(".")[-2:]) |
| # trim_title removes trailing newspaper names such as " - Telex" |
| formatted = f"{trim_title(clean(title))}\n{clean(description)}\n({domain})" |
| if text: |
| formatted += "\n" + clean(text)[:800] |
| return formatted |
| ``` |
| |
| Tokenize with `max_length=320, truncation=True`. `label 1` means the article |
| belongs in the database; the probability of that class is the score. The |
| reference implementation lives in |
| [`sajtoadatbazis-automat`](https://github.com/k-monitor/sajtoadatbazis-automat) |
| (`webapp/auto_kmdb/processors/ClassificationProcessor.py` and |
| `train/article_classification/`). |
|
|
| ## Results |
|
|
| Held-out test split of 9,914 articles, deduplicated by URL and by |
| title+description (40.2% positive): |
|
|
| | model | AUPR | accuracy @0.42 | precision | recall | F1 | |
| | --- | --- | --- | --- | --- | --- | |
| | `kmdb_classification_hubert_v2` | 0.8638 | 0.7119 | 0.590 | 0.926 | 0.7208 | |
| | **`kmdb_classification_hubert_v5`** | **0.9856** | **0.9502** | **0.939** | **0.937** | **0.9379** | |
| |
| On the subset the previous version was never trained on (n=4,473), v2 scores |
| AUPR 0.7600 / accuracy 0.4581 and v5 scores AUPR 0.9683 / accuracy 0.9215. |
| |
| The score is well calibrated around the 0.42 operating point used in |
| production; the precision/recall curve is flat between roughly 0.15 and 0.8. |
| |
| | threshold | accuracy | precision | recall | F1 | |
| | --- | --- | --- | --- | --- | |
| | 0.15 | 0.9450 | 0.9096 | 0.9583 | 0.9333 | |
| | 0.374 | 0.9508 | 0.9367 | 0.9410 | 0.9388 | |
| | 0.42 | 0.9502 | 0.9386 | 0.9372 | 0.9379 | |
| | 0.60 | 0.9502 | 0.9488 | 0.9259 | 0.9372 | |
| | 0.80 | 0.9478 | 0.9611 | 0.9066 | 0.9330 | |
| |
| ## Training |
| |
| Full finetune (not LoRA): 3 epochs, lr 2e-5, linear schedule, 300 warmup steps, |
| batch size 24, max length 320, weight decay 0.01, fp16, on a single RTX 3060. |
| The article body is dropped from 15% of training rows so the model does not |
| read a missing body as a label signal. |
| |
| ## Limitations |
| |
| Positive examples come from the published K-Monitor database and negatives from |
| the autokmdb ingestion pipeline, so source and label are correlated in the |
| training data. Whitespace normalisation and title trimming remove the |
| mechanical traces of this (newline conventions, newspaper suffixes), and the |
| model's output moves by 0.0004 on average when fed unnormalised text — but the |
| correlation cannot be fully ruled out. |
| |
| The companion category model |
| `K-Monitor/kmdb_classification_category_v5` is an SVM fitted on **this** |
| checkpoint's CLS embedding and is not interchangeable with the SVM for earlier |
| versions. |
|
|
