The original slimpajama_Llama2_Tokenizer.tar.gz archive (≈794 GB on Ubuntu) was split into smaller 40 GB chunks for easier upload to Hugging Face. sudo apt install git-lfs pip install -U huggingface_hub # hf version==1.1.4 tar cvf - slimpajama_Llama2_Tokenizer/ | pigz -p 16 > slimpajama_Llama2_Tokenizer.tar.gz split -b 40G -d -a 3 slimpajama_Llama2_Tokenizer.tar.gz slimpajama_Llama2_Tokenizer/slimpajama_Llama2_Tokenizer_part_