codelion/

finewiki-1B

This is a sampled subset of HuggingFaceFW/finewiki containing approximately 1,000,000,332 tokens.

	Dataset Details




	
	Source

Original Dataset: HuggingFaceFW/finewiki (English subset, train split) Sampling Method: Reservoir sampling (unbiased random sampling) Target Token Count: 1,000,000,332 tokens Tokenizer: GPT-2 (50,257 vocabulary)

Documents Sampled: 52,721 Average Tokens/Doc:… See the full description on the dataset page: https://huggingface.co/datasets/codelion/finewiki-1B.