You can deploy this model and start earning money today!

eoinf/
pile_tok4096_200m

Views

No views yet

	Dataset Card for eoinf/pile_tok4096_200m




	
	Original dataset

Original dataset: monology/pile-uncopyrighted

Dataset Details

Total Tokens: 200,018,944 Total Sequences: 195,331 Context Length: 1024 tokens Tokenizer: eoinf/pile_tokenizer_4096 Format: Each example contains a single field tokens with a list of 1024 token IDs

Preprocessing

Each document was:

Tokenized using the eoinf/pile_tokenizer_4096 tokenizer Prefixed with a BOS (beginning of sequence)… See the full description on the dataset page: https://huggingface.co/datasets/eoinf/pile_tok4096_200m.