eduagarcia/

OSCAR-2301-pt_dedup

OSCAR-2301-PT (deduplicated) is a curation from OSCAR-2301 in the Portuguese language, depuplicated by MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022). This version of the dataset is the portuguese subset from OSCAR-2301. OSCAR-2301 is a multilingual corpus extracted from the November/December 2022 dump of Common Crawl.

We used 5-grams and a signature of size 256, considering two… See the full description on the dataset page: https://huggingface.co/datasets/eduagarcia/OSCAR-2301-pt_dedup.