Downloaded and processed using code based on another project attempting to recreate the C4 dataset.
The resultant traditional Chinese dataset can be found here.
This dataset contains data processed with CantoneseDetect.
In CantoneseDetect, you can choose whether to include quotes (i.e. categorise the data as Cantonese even if Cantonese appeared only in quotes).
And I found that a lot of entries came from Wikipedia and LIHKG. If you… See the full description on the dataset page: https://huggingface.co/datasets/jed351/Cantonese_Common_Crawl_Filtered.
