ce-lery/

corpus-ja-11b

Welcome to this repository.This dataset is japanese corpus that includes wiki, wikibooks, wikiversity, and cc100.

If you want to use this, please run as follows.This process takes about 3 hours.
mkdir -p pretrain/input/ cd pretrain/input/

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/ce-lery/corpus-ja-11b.git cd corpus-ja-11b git lfs pull bash merge_train.sh