A composite of Chinese books, papers, legal documents, and patents from common crawl.
Data cleaning
Text with more than 2% of non-Latin, non-Chinese characters are removed.
Text with large portions of special characters are removed.
Traditional Chinese is converted to simplified Chinese.
Model filtering
Qwen2.5-32B-Instruct is used to generate language quality annotation (on a scale of 1-5) for 398K Chinese samples and 250K English samples. An XLM-RoBERT-large classifier is trained with… See the full description on the dataset page:
https://huggingface.co/datasets/Geralt-Targaryen/Literature-zh.
