This is a variant of the original dataset.
It was shuffled (seed=42);
Deduplicated on rows (96,613 rows removed);
Split into train, validation and test sets (the latter have 8192 samples each)
from datasets import load_dataset, Dataset, DatasetDict
ds = load_dataset("UWV/Leesplank_NL_wikipedia_simplifications", split="train")
ds = ds.shuffle(seed=42)
print("original", ds)
df = ds.to_pandas()
df = df.drop_duplicates().reset_index()
ds = Dataset.from_pandas(df)… See the full description on the dataset page:
https://huggingface.co/datasets/BramVanroy/wiki_simplifications_dutch_dedup_split.
