Aleph-Alpha-GermanWeb-Grammar-Classifier-BERT is a model that was used in the creation of
Aleph-Alpha-GermanWeb, a new German-language dataset that combines heuristic and model-based filtering techniques with synthetic data generation to achieve SOTA performance in German-language benchmarks.
Here we provide one of our quality classification models, based on a BERT backbone, along with inference code. This model is released as part of a
collection of four text quality classification models.
To train Aleph-Alpha-GermanWeb-Grammar-Classifier-BERT, we used
LanguageTool to annotate a random subset of 400,000 German FineWeb2 documents with the DE_AGREEMENT rule, which identifies text passages with grammatical disagreement. To train our classifier, we randomly selected 75,000 documents without identified grammar mistakes as high quality examples. As low quality examples, we took 75,000 random documents containing at least one identified grammar error.
We trained Aleph-Alpha-GermanWeb-Grammar-Classifier-BERT on 95% of the data to classify the high and low quality examples -- and used the remaining 5% for validation, reaching a precision of 67% and recall of 66% on the validation set.
Further details can be found in our
accompanying paper.
1import torch
2from transformers import BertTokenizer, BertForSequenceClassification
3
4device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
5
6model = BertForSequenceClassification.from_pretrained("Aleph-Alpha/Aleph-Alpha-GermanWeb-Grammar-Classifier-BERT", num_labels=2).to(device)
7
8tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
9
10# disclaimer: short text is not in the model distribution
11text = 'Das ist ein Beispieltext, um die Grammatik zu überprüfen.'
12
13target_names = ['Low Quality', 'High Quality']
14
15with torch.no_grad():
16 prediction = torch.argmax(
17 model(**tokenizer(text, return_tensors='pt', truncation=True, padding=True).to(device)).logits
18 ).item()
19print(target_names[prediction])
20
