š OVERALL DATASET STATISTICS
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
Total samples: 2,419,111
Total tokens in dataset: 2,052,945,533
Average tokens per sample: 848.64
š TOKEN DISTRIBUTION
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
Min tokens: 21
Max tokens: 142,971
Median tokens: 534.00
Standard deviation: 1047.89
š TOKEN PERCENTILES
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
5th percentile: 83⦠See the full description on the dataset page:
https://huggingface.co/datasets/kshitijthakkar/loggenix-merged-dataset.