BEE-spoke-data/

rp_books-en

Filtering/cleaning on the 'red pajama books' subset of togethercomputer/Long-Data-Collections The default config: Dataset({ features: ['meta', 'text'], num_rows: 26372 })

	token count




	
	default

GPT-4 tiktoken token count: token_count count 2.637200e+04 mean 1.009725e+05 std 1.161315e+05 min 3.811000e+03 25% 3.752750e+04 50% 7.757950e+04 75% 1.294130e+05 max 8.687685e+06

Total count: 2662.85 M… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/rp_books-en.