This model aims to simplify German texts into plain German language. It belongs to the experiments done at the work of Stodden (2024, to appear). "Reproduction & Benchmarking of German Text Simplification Systems" In Proceedings of the 1st Workshop on Evaluating Text Difficulty in a Multilingual Context (DeTermIt!), Turin, Italy.
Paper: Stodden (2024, to appear). "Reproduction & Benchmarking of German Text Simplification Systems" In Proceedings of the 1st Workshop on Evaluating Text Difficulty in a Multilingual Context (DeTermIt!), Turin, Italy.
Uses
Direct Use & Downstream Use
mT5-DEplain-APA is intended to be used to simplify German sentences for non-native German speakers.
mt5-DEplain-APA is a fine-tuned version of mT5-base, which is fine-tuned on DEplain-APA-sent, a German text simplification corpus of the news domain. The intended use is sentence simplification of German, where the source language is standard German and the target language is plain German.
Out-of-Scope Use
mT5-DEplain-APA is fine-tuned only on complex-simple pairs of the news domains and for German learners (CEFR level: A2), hence, we assume that the model will not work well for other use cases than text simplification, other languages than German, other domains than news, nor other target groups than non-native speakers.
Bias, Risks, and Limitations
The generated simplifications of the TS model might have some errors, therefore they shouldn’t be shown to a potentially vulnerable target group before manually verifying their quality and possibly fixing them.
The text simplification system could be provided to human translators who might improve and timely reduce their effort in manually simplifying a text.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
How to Get Started with the Model
Use the code below to get started with the model. Please specify the maximum target length of the sequence to 128 to reproduce our results.
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("DEplain/mt5-DEplain-APA")
model = AutoModelForSeq2SeqLM.from_pretrained("DEplain/mt5-DEplain-APA")
prefix = "Simplify to plain German: "
sent = "Ganz vorne im Gespann zieht er die anderen 13 Hunde mit, führt sie über vereiste Seen oder steile Berge und findet den Weg, wenn ihn selbst der Musher nicht mehr kennt."
# EN: "At the front of the team, he pulls the other 13 dogs along, leads them over icy lakes or steep mountains and finds the way when even the musher no longer knows it."
inputs = tokenizer([prefix+sent], return_tensors="pt")
outputs = model.generate(**inputs, max_length=128)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])
# expected output: "Ganz vorne im Gespann zieht er die anderen Hunde mit. Er findet den Weg, wenn ihn selbst der Musher nicht mehr kennt."
Training Details
Training Data
The model is fine-tuned on DEplain-APA. DEplain-APA (Stodden et al., 2023) is a dataset for the training and evaluation of sentence simplification in German. All texts of this dataset are provided by the Austrian Press Agency. The simple-complex sentence pairs are manually aligned.
All models are automatically evaluated against one reference and on the same evaluation metrics, i.e., SARI (Xu et al., 2016), BLEU (Papineni et al., 2002), BS_P (Zhang* et al., 2020), and FRE (Amstad, 1978).
Following the recommendation of Alva-Manchego et al. (2021), we use BS_P as the main evaluation metric, if the score is a high we verify it with other metrics, i.e., SARI, BLEU and FRE.
In addition, as recommended by Tanprasert and Kauchak (2021) and Alva-Manchego et al. (2019), we also report linguistic features to get more insights into the system-generated simplifications, i.e., compression ratio and sentence splits.
For the measurement of the metrics and features, we are using the evaluation framework, i.e., EASSE-DE (Stodden, 2024) a multi-lingual adaptation of the EASSE evaluation framework.
Results
Results of mT5-DEplain-APA and related models evaluated on DEplain-APA. For more results on other test data, please have a look at our paper.
BLEU
SARI
BS_P
FRE
Compr- ratio
Sent. splits
hda_LS
22.3
26.06
0.55
64.60
1.00
1.00
sockeye-APA-LHA
11.84
40.16
0.37
63.70
0.94
0.97
sockeye-DEplain-APA
19.58
44.14
0.53
71.45
0.94
1.09
mBART-DEplain-APA
28.49
38.72
0.6}
65.30
0.99
1.07
mBART-DEplain-APA+web
28.03
33.81
0.64
65.20
0.98
1.05
mT5-DEplain-APA
22.32
39.41
0.61
63.20
0.87
1.04
mt5-SGC
8.12
37.92
0.48
71.65
0.74
1.00
BLOOM-zero
16.14
35.43
0.53
65.10
0.87
1.14
BLOOM-10-random
17.97
35.93
0.57
65.50
0.91
1.00
BLOOM-10-similarity
20.97
41.27
0.57
65.70
0.93
1.07
custom-decoder-ats
1.24
36.42
0.16
53.00
7.41
5.07
Identity baseline
26.89
15.25
0.63
58.75
1.00
1.00
Reference baseline
100.00
100.00
1.00
65.80
1.03
1.20
Truncate baseline
16.11
27.20
0.55
66.10
0.80
1.01
Citation
BibTeX:
@inproceedings{stodden-2024-reproduction,
author = {Regina Stodden},
title = {{Reproduction \& Benchmark of German Text Simplification Systems}},
booktitle = "Proceedings of the 1st Workshop on Evaluating Text Difficulty in a Multilingual Context (DeTermIt!)",
year = {2024 (to appear)},
address = "Turino, Italy"
}
APA:
Regina Stodden. 2024 (to appear). "Reproduction & Benchmarking of German Text Simplification Systems". In Proceedings of the Proceedings of the 1st Workshop on Evaluating Text Difficulty in a Multilingual Context (DeTermIt!), Turin, Italy.
