brian-learns/

cdx-cc-news

CDXj indexes for Common Crawl News Dataset. See also News Dataset Announcement I could not find indexes to Common Crawl News, so I decided to create some. Then I created a RocksDB Index and a lookup API. Direct url to API: https://brian-learns-cc-news-cdx-server.hf.space/lookup

CLI for querying this index and retrieving archived web pages from CC-NEWS Common Crawl WARC files https://github.com/brian-learns/ccnget

├──… See the full description on the dataset page: https://huggingface.co/datasets/brian-learns/cdx-cc-news.