This repository contains the pre-training dataset for TAIDE.
The different data groups are stored in their respective branches.
You can use the following code to load them:
from datasets import load_dataset
dataset = load_dataset(
'TLLM/TAIDE-CP-Data',
subset='<DATA_GROUP_NAME>',
num_proc=32 # 32 cores for downloading & building dataset
)
- Note that there is overlap between data groups.
