An Annif model, trained on historical titles and additional catalogue metadata for automatic subject indexing tasks. It classifies a given text into one or multiple subjects from the “Alter Realkatalog” (ARK) classification system. The model was developed in the research project Human.Machine.Culture at Staatsbibliothek zu Berlin – Berlin State Library (SBB).
An Annif model, trained on historical titles and additional catalogue metadata for automatic subject indexing tasks. Subject indexing is a classical library task, aiming at describing the content of a resource. The model is intended to be used to automatically classify historical texts with a historical classification system developed in the 19th century to enrich those texts that have not been classified manually so far. For each input text, the model outputs one or multiple subjects from the ARK classification system. It is part of a collection of 5 models, created with the help of the Annif toolkit which addresses this task of automated subject indexing.
This model can directly be used to automatically classify historical texts with the ARK classification scheme. It is intended to be used together with the Annif automated subject indexing toolkit version 0.60.0-1.1.0.
Downstream Use
Other/downstream uses outside of the Annif setting described above are not intended but also not excluded.
Out-of-Scope Use
The model is not intended for use on contemporary texts, as language and concept drifts will probably influence the results negatively and some terms from the vocabulary are not appropriate for more recent publications.
Another limitation when using the ARK as a vocabulary arises from its hierarchical structure: the system contains multiple classes that do not describe the same content (e.g. they have different IDs) but are labeled identical (same name). This is due to the fact that the manual inspection of the whole path to a class, including all the upper level classes leading to it, delivers additional information that allows for contextualization. As duplicate label names seem to be - as expected - a challenge for lexical methods, we decided to focus on statistical rather than lexical algorithms.
Recommendations
Considering that the ARK classification scheme consists of 225.691 classes in total and that there is only limited training material at hand plus an overall unbalanced distribution of classes, we might describe this task as an Extreme Multi-Label Classification (XMC) problem. We recommend being aware of this limitation and, if available, use additional training data to improve the current model’s performance (e.g. by running annif learn, see CLI commands documentation).
Training Details
Training Data
Training data include a selection of metadata fields that were obtained via CBS export:
Lehmann, J., & Schneider, S. (2024). Metadata of the "Alter Realkatalog" (ARK) of Berlin State Library (SBB) (Version 1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.12783813
The following title and content data fields have been extracted and combined from this dataset:
The vocabulary files themselves can be found here:
Schneider, S., & Lehmann, J. (2024). Machine-Readable Vocabulary Files of the "Alter Realkatalog" (ARK) of Berlin State Library (SBB) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.13301020
Besides merging and transforming the data described under Training Data to fit the corpus formats accepted by Annif, no further preprocessing of natural language or similar has been performed.
Speeds, Sizes, Times
Training takes from several minutes to a few hours on a V100, depending on the choice of dataset and algorithm as well as hyperparameter settings.
Training hyperparameters
For some of the ARK Annif models, a slight hyperparameter optimization has been conducted to identify the final hyperparameter settings stated below.
hyperparameter configuration (as it needs to be stated in the Annif projects.cfg file):
The dataset is described under Training Data. It was split into smaller subsets used for training, testing and validating (80%/10%/10% split).
Metrics
Model performance has been evaluated based on the following metrics: Precision, Recall, F1 and NDCG. These are standard metrics for machine learning and more specifically automatic subject indexing tasks and are directly provided in Annif by calling the annif eval statement. Evaluation parameters (--limit = maximum number of results to return; --threshold = minimum confidence for a suggestion to be considered) have been optimized before using the validation dataset and affect the results accordingly. We also state F1@5 and NDCG@5 scores reached without any evaluation parameters.
See Annif and Omikuji repositories on Github. Omikuji is an implementation of Partitioned Label Trees (Prabhu et al., 2018):
Y. Prabhu, A. Kag, S. Harsola, R. Agrawal, and M. Varma, “Parabel: Partitioned Label Trees for Extreme Classification with Application to Dynamic Search Advertising,” in Proceedings of the 2018 World Wide Web Conference, 2018, pp. 993–1002.
Software
To run this model, Annif version 0.60.0 or higher (min. up to 1.1.0) must be installed.
