Total sequences: 155946
Total tokens: 99999541
Unique tokens used in all positions: 49908
Unique tokens in first position: 2727
Total number of unique sequences: 155893
Mean token count: 641.2446680261099
Token count std: 363.0479530697014
Min sequence length: 0
Max sequence length: 1075
Tokenizer vocabulary size: 50254
Unique tokens in dataset first position: 2727
KL(Model -> Dataset): 1.0498
KL(Dataset -> Model): 0.1143… See the full description on the dataset page:
https://huggingface.co/datasets/seonglae/faithful-pythia1.4b.
