Tokenized Corpus Statistics
Shard
Total Tokens
Avg Tokens
Median Tokens
train-00000-of-00077
64,973,050
259.89
213.0
train-00001-of-00077
65,024,877
260.10
214.0
train-00002-of-00077
64,976,757
259.91
213.0
train-00003-of-00077
65,035,795
260.14
214.0
train-00004-of-00077
64,962,798
259.85
214.0
train-00005-of-00077
64,945,077
259.78
213.0
train-00006-of-00077
64,885,161
259.54
213.0
train-00007-of-00077
65,060,639
260.24
214.0
train-00008-of-0007764,905… See the full description on the dataset page:
https://huggingface.co/datasets/Ba2han/tokenized-corpus-0603.
