| --- |
| tags: |
| - sentence-transformers |
| - transformers |
| - SetFit |
| - News |
| datasets: KnutJaegersberg/News_topics_IPTC_codes_long |
| pipeline_tag: text-classification |
| --- |
| |
|
|
| # IPTC topic classifier (multilingual) |
|
|
| A SetFit model fit on 166 downlsampled multilingual IPTC Subject labels (concatenated for the lowest hierarchy level into artificial sentences of keywords) to predict the mid level news categories. |
| The purpose of this classifier is to support exploring corpora as weak labeler, since the representations of these descriptions are only approximations of real documents from those topics. |
| The dataset I used to train the model is based on this file: |
| https://huggingface.co/datasets/KnutJaegersberg/News_topics_IPTC_codes_long |
|
|
| Accuracy on highest level labels in eval: |
| 0.9779412 |
| Accuracy/F1/mcc on mid level labels in eval: |
| 0.6992481/0.6666667/0.6992617 |
|
|
| More interestingly, I used the kaggle dataset with headlines from huffington post and manually selected 15 overlapping high level categories to evaluate the performance. |
| https://www.kaggle.com/datasets/rmisra/news-category-dataset |
|
|
| While mcc 0.1968043 on this dataset does not sound as good as before, the mistakes usually could also be seen as a re-interpretation. I.e. news on arrests where categorized as entertainment in the huffington post dataset, the classifier put it into the crime category. |
| My current impression is this system is useful for the aimed for purpose. |
|
|
|
|
|
|
| The numeric categories can be joined with the labels by using this table: |
| https://huggingface.co/datasets/KnutJaegersberg/IPTC-topic-classifier-labels |
|
|
|
|
| Looks like try out api box to the right by huggingface does not yet handle setfit models, can't do anything about that. |
|
|
|
|
| Use like any other SetFit model |
|
|
| from setfit import SetFitModel |
|
|
| # Download from Hub and run inference |
| model = SetFitModel.from_pretrained("KnutJaegersberg/IPTC-classifier-ml") |
| # Run inference |
| preds = model(["Rachel Dolezal Faces Felony Charges For Welfare Fraud", "Elon Musk just got lucky", "The hype on AI is different from the hype on other tech topics"]) |
