german-tokenizer-benchmark/

ud-hdt

This datasets hosts a sentence-tokenized version of the Universal Dependencies German-HDT dataset.

The following script can be used to reproduce the creation of the dataset: import json

from flair.datasets import UD_GERMAN_HDT

corpus = UD_GERMAN_HDT()

with open("./train.jsonl", "wt") as f_out: for sentence in corpus.train: current_example = { "text": sentence.to_tokenized_string() }… See the full description on the dataset page: https://huggingface.co/datasets/german-tokenizer-benchmark/ud-hdt.