This is a preprocessed version of the realnewslike subdirectory of C4
C4 from:
https://huggingface.co/datasets/allenai/c4
Files generated by using Megatron-LM
https://github.com/NVIDIA/Megatron-LM/
python tools/preprocess_data.py
--input 'c4/realnewslike/c4-train.0000[0-9]-of-00512.json'
--partitions 8
--output-prefix preprocessed/c4
--tokenizer-type GPTSentencePieceTokenizer
--tokenizer-model tokenizers/tokenizer.model
--workers 8
