This gated dataset contains deduplicated image URL text files extracted from Common Crawl.
Source S3 prefix: s3://core-cn-oss.canva.com/usr/xinyang/datasets/cc-full/raw/url_batches_filtered_10m/cyrusli-cc-url-filter-10m-08121819/
URL count: 144,911,156,406
File count: 14,492
URLs per full file: 10,000,000
Final partial file URLs: 1,156,406
Compressed byte count: 8,095,053,791,842
File format: gzip-compressed text, one URL per line
Created from… See the full description on the dataset page:
https://huggingface.co/datasets/xinyangli/cc-full-image-urls.
