Create Model Card

README.md

@@ -1,3 +1,207 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- SBB/BK-Training-Dataset
+pipeline_tag: text-classification
+tags:
+- annif
+- subject_indexing
+- glam
+- lam
+---
+
+# Model Card for bk-omikuji
+
+An [Annif](https://annif.org/) model, trained on bibliographic metadata for automatic subject indexing tasks. It classifies a given text such as a title or description into one or multiple subjects from the classification system *Nederlandse basisclassificatie* / [Basisklassifikation](http://uri.gbv.de/terminology/bk/) (BK). The model was developed in the research project [Human.Machine.Culture](https://mmk.sbb.berlin/?lang=en) at Staatsbibliothek zu Berlin – Berlin State Library (SBB).
+
+Questions and comments about the model can be directed to Sophie Schneider at Sophie.Schneider@sbb.spk-berlin.de. 
+
+
+#  Table of Contents
+
+- [Model Card for bk-omikuji](#model-card-for-bk-omikuji)
+- [Table of Contents](#table-of-contents)
+- [Model Details](#model-details)
+  - [Model Description](#model-description)
+- [Uses](#uses)
+  - [Direct Use](#direct-use)
+  - [Downstream Use](#downstream-use)
+  - [Out-of-Scope Use](#out-of-scope-use)
+- [Bias, Risks, and Limitations](#bias-risks-and-limitations)
+  - [Recommendations](#recommendations)
+- [Training Details](#training-details)
+  - [Training Data](#training-data)
+  - [Training Procedure](#training-procedure)
+    - [Preprocessing](#preprocessing)
+    - [Speeds, Sizes, Times](#speeds-sizes-times)
+    - [Training hyperparameters](#training-hyperparameters)
+    - [Training results](#training-results)
+- [Evaluation](#evaluation)
+  - [Testing Data and Metrics](#testing-data-and-metrics)
+    - [Testing Data](#testing-data)
+    - [Metrics](#metrics)
+  - [Results](#results)
+- [Model Examination](#model-examination)
+- [Environmental Impact](#environmental-impact)
+- [Technical Specifications](#technical-specifications)
+  - [Model Architecture and Objective](#model-architecture-and-objective)
+    - [Software](#software)
+- [Model Card Authors](#model-card-authors)
+- [Model Card Contact](#model-card-contact)
+- [How to Get Started with the Model](#how-to-get-started-with-the-model)
+
+
+
+# Model Details
+
+## Model Description
+
+An [Annif](https://annif.org/) model, trained on title/catalogue metadata for automatic subject indexing tasks. Subject indexing is a classical library task, aiming at describing the content of a resource. The model is intended to be used to automatically classify titles or texts that have not been classified manually so far. For each input, the model outputs one or multiple subjects from the [BK](http://uri.gbv.de/terminology/bk/) classification system. It is part of a collection of 3 models, created with the help of the Annif toolkit, which address the task of automated subject indexing.
+
+- **Developed by:** [Sophie Schneider](Sophie.Schneider@sbb.spk-berlin.de)
+- **Shared by [Optional]:** [Staatsbibliothek zu Berlin / Berlin State Library](https://huggingface.co/SBB)
+- **Model type:** tree-based
+- **Language(s) (NLP):** multilingual
+- **License:** apache-2.0
+
+# Uses
+
+## Direct Use
+
+This model can directly be used to automatically classify texts with the BK classification scheme. It is intended to be used together with the Annif automated subject indexing toolkit version 1.1.0.
+
+
+
+## Downstream Use
+
+Other/downstream uses outside of the Annif setting described above are not intended but also not excluded.
+ 
+
+
+## Out-of-Scope Use
+
+The task of the model is primarily the subject indexing of bibliographic metadata. Basically, the model can be applied to any kind of text describing publications. However, it might not be suitable for colloquial texts from the internet.
+
+
+
+# Bias, Risks, and Limitations
+
+The BK was introduced in the 1980s and therefore its structure partly represents outdated ways of thinking. In itself, the BK is biased, for some examples see [https://verbundkonferenz.gbv.de/wp-content/uploads/2024/09/2024-08-20_VK_beckmann_Kunst-oder-Krempel-Potenziale-der-Basisklassifikation.pdf](https://verbundkonferenz.gbv.de/wp-content/uploads/2024/09/2024-08-20_VK_beckmann_Kunst-oder-Krempel-Potenziale-der-Basisklassifikation.pdf), slide 17. Hence, typical biases are e.g. the binary understanding of gender roles, outmoded use of geographical designators or of social movements. However, in order to keep it up to date, the classification system is under constant revision. As of now, the classes suggested for an input text might not be suitable for today’s understanding and might not conform to contemporary values.
+
+
+
+## Recommendations
+
+This BK model has been trained solely on the basis of title data. We recommend being aware of this limitation and, if available, to use additional training data such as full texts or tables of contents to enhance the existing model (e.g. by running `annif learn` or combining models trained with different kinds of training data to an ensemble model).
+
+
+
+# Training Details
+
+## Training Data
+
+The **vocabulary file** for the BK was downloaded directly from [https://api.dante.gbv.de/export/download/bk/default/](https://api.dante.gbv.de/export/download/bk/default/) (`bk__default.turtle.ttl` as of 05/2024). 
+
+For the **subject data**, several versions were created for each model with slight adaptations.
+
+A dataset was generated following these steps:
+- downloading the `kxp-subjects.tsv.gz` of Voß, J., & Verbundzentrale des GBV. (2024). Normalized subject indexing data of K10plus library union catalog (2024-02-26) [Data set]. VZG. [https://doi.org/10.5281/zenodo.10933926](https://doi.org/10.5281/zenodo.10933926)
+- cleaning this file to only contain PPNs together with their BK notations
+- querying the corresponding BK titles (Pica3 field [4000](https://swbtools.bsz-bw.de/cgi-bin/k10plushelp.pl?cmd=kat&val=4000), subfields a and d) for all PPNs from previous step via [unapi](http://unapi.k10plus.de/)
+- merge the subject and title data based on PPN, filter out duplicates (identical titles)
+- this led to a dataset of overall 6.114.950 entries, split into 80% train and 10% for test and validation subsets 
+
+The training dataset has been published on [Zenodo](https://doi.org/10.5281/zenodo.15690227).
+
+
+
+## Training Procedure
+
+The training procedure includes loading the BK vocabulary into Annif and training the [Omikuji backend](https://github.com/NatLibFi/Annif/wiki/Backend%3A-Omikuji) with the help of the respective training data. Further aspects on technical specifications can be found in the section [Training hyperparameters](#training-hyperparameters).
+
+
+### Preprocessing
+
+Besides merging and transforming the data as described under [Training Data](#training-data), no further preprocessing of natural language or similar has been performed in the dataset creation. For training the model, we applied the [simple analyzer](https://github.com/NatLibFi/Annif/wiki/Analyzers#simple-analyzer) provided by Annif.
+
+### Speeds, Sizes, Times
+
+Training takes from several minutes to an hour on an 2,8 GHz CPU with 32 cores, depending on the choice of dataset and algorithm as well as hyperparameter settings.
+
+### Training hyperparameters
+
+name=BK Omikuji
+language=de
+backend=omikuji
+analyzer=simple
+vocab=bk
+cluster_balanced=False
+cluster_k=100
+max_depth=3
+
+
+### Training results
+
+documents evaluated: 5000
+
+* Precision (`--limit` 2, `--threshold` 0.05): 0.4664
+* Recall (`--limit` 2, `--threshold` 0.05): 0.5382
+* F1 (`--limit` 2, `--threshold` 0.05): 0.4775
+* NDCG (`--limit` 2, `--threshold` 0.05): 0.5385
+* F1@5: 0.3232
+* NDCG@5: 0.6283
+
+# Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results, or cites relevant papers. -->
+
+
+## Testing Data, Factors and Metrics
+
+### Testing Data
+
+The dataset is described under [Training Data](#training-data). It was split into smaller subsets used for training, testing and validating (80%/10%/10% split). 
+
+
+### Metrics
+
+Model performance has been evaluated based on the following metrics: Precision, Recall, F1 and NDCG. These are standard metrics for machine learning and more specifically for automatic subject indexing tasks and are directly provided in Annif by calling the ```annif eval``` statement. Evaluation parameters (limit = maximum number of results to return; threshold = minimum confidence for a suggestion to be considered) have been optimized before using the test subset and chosen based on the best F1 score reached. They affect the final results accordingly. We also state F1@5 and NDCG@5 scores reached without any evaluation parameters.
+
+# Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** A100.
+- **Hours used:** 0,5-1 hour.
+- **Cloud Provider:** No cloud.
+- **Compute Region:** Germany.
+- **Carbon Emitted:** More information needed.
+
+# Technical Specifications
+
+## Model Architecture and Objective
+
+See the [Annif Wiki on the Omikuji algorithm](https://github.com/NatLibFi/Annif/wiki/Backend%3A-Omikuji).
+
+### Software
+
+To run this model, Annif version 1.1.0 must be installed.
+
+# Model Card Authors
+
+[Sophie Schneider](sophie.schneider@sbb.spk-berlin.de) and [Jörg Lehmann](joerg.lehmann@sbb.spk-berlin.de)
+
+# Model Card Contact
+
+Questions and comments about the model can be directed to Sophie Schneider at sophie.schneider@sbb.spk-berlin.de, questions and comments about the model card can be directed to Jörg Lehmann at joerg.lehmann@sbb.spk-berlin.de
+
+# How to Get Started with the Model
+
+Follow the Annif [Getting Started](https://github.com/NatLibFi/Annif/wiki/Getting-started) page to set up and run Annif. Create a projects.cfg file (see section [Training hyperparameters](#training-hyperparameters) for details on the specific project configuration), load the BK vocabulary via ```annif load-vocab``` command and copy the model folder over to ```data/projects```.
+
+
+Model Card as of October 20th, 2025
+
+