OdiaGenAIdata/

pre_train_odia_data

The present dataset is compiled by using the following datasets:

Licesnse - ODC-By, CC0, Paper Source - https://huggingface.co/datasets/uonlp/CulturaX/viewer/or? 49M tokens, 2.9M sentences Collection of different versions of Ocsar (Commom Crawl data) and mC4 dataset (Common Crawl's web crawl corpus). mC4 forms 66% of CulturaX dataset.

License - cc-by-4.0 Source - https://huggingface.co/datasets/ai4bharat/IndicQA/viewer/indicqa.or 0.23M tokens, 15K… See the full description on the dataset page: https://huggingface.co/datasets/OdiaGenAIdata/pre_train_odia_data.