documents containing non-Chinese, non-English text are removed
documents containing more than 30% English text are removed
all text in Traditional Chinese is converted into Simplified Chinese using zhconv
low-quality text (e.g. boilerplates, advertisements) are heuristically removed
Number of samples: 32,485,463.
Size of parquet files: 61G.
Model-filtered version in the filter… See the full description on the dataset page:
https://huggingface.co/datasets/TiWu-Lab/C4-zh.
