The dataset consists of 2084 jsonl files.
You can download the dataset using HuggingFace:
from datasets import load_dataset
ds = load_dataset("togethercomputer/RedPajama-Data-1T")
Or you can directly download the files using the following command:
wget '
https://data.together.xyz/redpajama-data-1T/v1.0.0/urls.txt'
while read line; do
dload_loc=${line#
https://data.together.xyz/redpajama-data-1T/v1.0.0/}
mkdir -p $(dirname $dload_loc)
wget "$line" -O… See the full description on the dataset page:
https://huggingface.co/datasets/TNSA/PT-LMD.
