Web register classification (multilingual model)
A multilingual web register classifier, fine-tuned from
XLM-RoBERTa-large.
The model is trained with the multilingual CORE corpora across five languages (English, Finnish, French, Swedish, Turkish) to classify documents based on the
CORE taxonomy.
It can predict labels for the 100 languages covered by XLM-RoBERTa-large. The model achieves state-of-the-art performance in classifying web registers for the trained languages and has strong transfer performance (see Evaluation below).
It is designed to support the development of open language models and for linguists analyzing register variation.
Model Details
Model Description
- Developed by: TurkuNLP
- Funded by: The Research Council of Finland, Emil Aaltonen Foundation, University of Turku
- Shared by: TurkuNLP
- Model type: Language model
- Language(s) (NLP): English, Finnish, French, Swedish, Turkish
- License: apache-2.0
- Finetuned from model: FacebookAI/xlm-roberta-large
Model Sources
- Repository: Coming soon!
- Paper: Coming soon!
Register labels and their abbreviations
Below is a list of the register labels predicted by the model. Note that some labels are hierarchical; when a sublabel is predicted, its parent label is also predicted.
For a more detailed description of the label scheme, see
here.
The main labels are uppercase. To only include these main labels in the predictions, simply slice the model's output to keep only the uppercase labels.
- MT: Machine translated or generated
- LY: Lyrical
- SP: Spoken
- ID: Interactive discussion
- NA: Narrative
- ne: News report
- sr: Sports report
- nb: Narrative blog
- HI: How-to or instructions
- IN: Informational description
- en: Encyclopedia article
- ra: Research article
- dtp: Description of a thing or person
- fi: Frequently asked questions
- lt: Legal terms and conditions
- OP: Opinion
- rv: Review
- ob: Opinion blog
- rs: Denominational religious blog or sermon
- av: Advice
- IP: Informational persuasion
- ds: Description with intent to sell
- ed: News & opinion blog or editorial
How to Get Started with the Model
Use the code below to get started with the model.
1import torch
2from transformers import AutoModelForSequenceClassification, AutoTokenizer
3
4device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
5
6model_id = "TurkuNLP/multilingual-web-register-classification"
7
8# Load model and tokenizer
9model = AutoModelForSequenceClassification.from_pretrained(model_id).to(device)
10tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
11
12# Text to be categorized
13text = "A text to be categorized"
14
15# Tokenize text
16inputs = tokenizer([text], return_tensors="pt", padding=True, truncation=True, max_length=512).to(device)
17
18with torch.no_grad():
19 outputs = model(**inputs)
20
21# Apply sigmoid to the logits to get probabilities
22probabilities = torch.sigmoid(outputs.logits).squeeze()
23
24# Determine a threshold for predicting labels
25threshold = 0.5
26predicted_label_indices = (probabilities > threshold).nonzero(as_tuple=True)[0]
27
28# Extract readable labels using id2label
29id2label = model.config.id2label
30predicted_labels = [id2label[idx.item()] for idx in predicted_label_indices]
31
32print("Predicted labels:", predicted_labels)
33
Training Details
Training Data
The model was trained using the Multilingual CORE Corpora, which will be published soon.
Training Procedure
Training Hyperparameters
- Batch size: 8
- Epochs: 21
- Learning rate: 0.00005
- Precision: bfloat16 (non-mixed precision)
- TF32: Enabled
- Seed: 42
- Max Size: 512 tokens
Inference time
Average inference time (across 1000 iterations), using a single NVIDIA A100 GPU and a batch size of one is 17 ms for a single example. Wirh bigger batches, inference can be considerably faster.
Evaluation
Micro-averaged F1 scores and optimized prediction thresholds for the five training languages (test set):
| Language | F1 (All labels) | F1 (Main labels) | Threshold |
|---|
| English | 0.72 | 0.75 | 0.40 |
| Finnish | 0.79 | 0.82 | 0.45 |
| French | 0.75 | 0.78 | 0.45 |
| Swedish | 0.81 | 0.82 | 0.45 |
| Turkish | 0.77 | 0.78 | 0.45 |
Micro-averaged F1 scores and optimized prediction thresholds for additional languages (zero-shot):
| Language | F1 (All labels) | F1 (Main labels) | Threshold |
|---|
| Arabic | 0.63 | 0.66 | 0.40 |
| Catalan | 0.62 | 0.63 | 0.50 |
| Spanish | 0.62 | 0.67 | 0.65 |
| Persian | 0.71 | 0.70 | 0.35 |
| Hindi | 0.77 | 0.78 | 0.40 |
| Indonesian | 0.60 | 0.61 | 0.30 |
| Japanese | 0.53 | 0.64 | 0.35 |
| Norwegian | 0.65 | 0.70 | 0.65 |
| Portuguese | 0.67 | 0.68 | 0.40 |
| Urdu | 0.81 | 0.83 | 0.35 |
| Chinese | 0.67 | 0.70 | 0.40 |
Technical Specifications
Compute Infrastructure
- Mahti supercomputer (CSC - IT Center for Science, Finland)
- 1 x NVIDIA A100-SXM4-40GB
Software
- torch 2.2.1
- transformers 4.39.3
Citation
If you use this model, please cite the following publication:
1@misc{henriksson2024untanglingunrestrictedwebautomatic,
2 title={Untangling the Unrestricted Web: Automatic Identification of Multilingual Registers},
3 author={Erik Henriksson and Amanda Myntti and Anni Eskelinen and Selcen Erten-Johansson and Saara Hellström and Veronika Laippala},
4 year={2024},
5 eprint={2406.19892},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2406.19892},
9}
Earlier related work include the following:
1@article{Laippala.etal2022,
2 title = {Register Identification from the Unrestricted Open {{Web}} Using the {{Corpus}} of {{Online Registers}} of {{English}}},
3 author = {Laippala, Veronika and R{\"o}nnqvist, Samuel and Oinonen, Miika and Kyr{\"o}l{\"a}inen, Aki-Juhani and Salmela, Anna and Biber, Douglas and Egbert, Jesse and Pyysalo, Sampo},
4 year = {2022},
5 journal = {Language Resources and Evaluation},
6 issn = {1574-0218},
7 doi = {10.1007/s10579-022-09624-1},
8 url = {https://doi.org/10.1007/s10579-022-09624-1},
9}
10
11@article{Skantsi_Laippala_2023,
12 title = {Analyzing the unrestricted web: The finnish corpus of online registers},
13 doi = {10.1017/S0332586523000021},
14 journal = {Nordic Journal of Linguistics},
15 author = {Skantsi, Valtteri and Laippala, Veronika},
16 year = {2023},
17 pages = {1–31}
18}
Model Card Contact
Erik Henriksson, Hugging Face username: erikhenriksson
