library_name: transformers
license: mit
base_model: FacebookAI/xlm-roberta-large
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: xlm_roberta_large_linsearch_classification
results: []
xlm_roberta_large_linsearch_classification
This model is an attempt at the shared task LLMs4Subjects (https://sites.google.com/view/llms4subjects-germeval/home) for Subtask 1, to classify subject domains of German and English documents. It is fine-tuned to classify documents into 28 predefined domains according to the LinSearch domain-specific taxonomy (more about the Fachsystematik LinSearch domains: https://terminology.tib.eu/ts/ontologies/linsearch). The task is a multi-class multi-label multilingual classification task, trained and evaluated on the TIBKAT dataset (more about the TIBKAT dataset TIB Open Data Services: https://www.tib.eu/en/services/open-data). The model has received the 1st place for the subtask, with a macro F1 score of 0.653 on the test set evaluated by the organisers on CodaBench (https://www.codabench.org/competitions/8373/#/results-tab).
This model is a fine-tuned version of FacebookAI/xlm-roberta-large on the TIBKAT dataset provided by organisers of the shared task. It achieves the following results on the evaluation set: (automatically generated from the fine-tuning process)
- Loss: 1.3318
- Accuracy: 0.5931
- F1 Macro: 0.5650
- Precision Macro: 0.5755
- Recall Macro: 0.5602
** The model has better performance on the test set than the eval set during training becuase the model here only takes one gold label instead of multiple for evaluation.
Intended uses & limitations
Training and evaluation data
- Training set: https://huggingface.co/datasets/ubffm/linsearch_train_data
- Development set: https://huggingface.co/datasets/ubffm/linsearch_dev_data
- Preprocessing: Mapping multi-label entries to a 1-to-1 for document-to-label by duplicating entries with multiple labels
- Size: 135k examples (116k training set, 18.7k development set)
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 16
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 10
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Precision Macro | Recall Macro |
|---|---|---|---|---|---|---|---|
| 1.3362 | 0.9999 | 7024 | 1.3026 | 0.5520 | 0.4904 | 0.5266 | 0.4959 |
| 1.2241 | 1.9999 | 14048 | 1.2036 | 0.5773 | 0.5348 | 0.5535 | 0.5395 |
| 1.1337 | 2.9999 | 21072 | 1.1903 | 0.5760 | 0.5303 | 0.5466 | 0.5278 |
| 1.069 | 3.9999 | 28096 | 1.1570 | 0.5876 | 0.5418 | 0.5564 | 0.5439 |
| 0.9832 | 4.9999 | 35120 | 1.1723 | 0.5872 | 0.5461 | 0.5570 | 0.5461 |
| 0.9197 | 5.9999 | 42144 | 1.1752 | 0.5871 | 0.5455 | 0.5456 | 0.5572 |
| 0.8278 | 6.9999 | 49168 | 1.2078 | 0.5928 | 0.5537 | 0.5616 | 0.5589 |
| 0.7568 | 7.9999 | 56192 | 1.2398 | 0.5923 | 0.5563 | 0.5612 | 0.5568 |
| 0.6863 | 8.9999 | 63216 | 1.2840 | 0.5937 | 0.5556 | 0.5723 | 0.5465 |
| 0.63 | 9.9999 | 70240 | 1.3318 | 0.5931 | 0.5650 | 0.5755 | 0.5602 |
Framework versions
- Transformers 4.48.0
- Pytorch 2.5.1+cu124
- Datasets 3.2.0
- Tokenizers 0.21.0
How to use
from transformers import pipeline
classifier = pipeline("text-classification", model="ubffm/xlm_roberta_large_linsearch_classification")
classifier("Your input text here")
with transformers
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("ubffm/xlm_roberta_large_linsearch_classification")
model = AutoModelForSequenceClassification.from_pretrained("ubffm/xlm_roberta_large_linsearch_classification")
inputs = tokenizer("Your input text here", return_tensors="pt")
outputs = model(**inputs)
Contact and Citation
@inproceedings{ho-2025-ubffm,
title = "{UBFFM} at the {G}erm{E}val-2025 {LLM}s4{S}ubjects Task: What if we take ``You are an expert in subject indexing'' seriously?",
author = "Ho, Clara Wan Ching",
editor = "Wartena, Christian and
Heid, Ulrich",
booktitle = "Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops",
month = sep,
year = "2025",
address = "Hannover, Germany",
publisher = "HsH Applied Academics",
url = "https://aclanthology.org/2025.konvens-2.44/",
pages = "471--478"
}
@misc{ubffm/xlm_roberta_large_linsearch_classification,
title={xlm_roberta_large_linsearch_classification},
author={UBFFM},
year={2025},
howpublished={\url{https://huggingface.co/ubffm/xlm_roberta_large_linsearch_classification/}},
}
Contact email: info@linguistik.de