This is Wikidedia passages dataset for ODQA retriever.
Each passages have 256~ tokens splitteed by gpt-4 tokenizer using tiktoken.
Token count
{'128': 1415068, '128256': 1290011,
'256512': 18756476, '5121024': 667,
'10242048': 12, '20484096': 0, '40968192': 0,
'819216384': 0, '1638432768': 0, '3276865536': 0,
'65536128000': 0, '128000': 0}
Text count
{'
512': 1556876,'5121024': 6074975, '1024
2048': 13830329,
'20484096': 49, '4096
8192': 2, '819216384': 3, '16384~32768': 0… See the full description on the dataset page:
https://huggingface.co/datasets/seonglae/wikipedia-256-token.
