eoinf/

tokenized_dataset_test7

	Dataset Card for eoinf/tokenized_dataset_test7




	
	Original dataset

Original dataset: monology/pile-uncopyrighted

Total Tokens: 10,003,456 Total Sequences: 9,769 Context Length: 1024 tokens Tokenizer: meta-llama/Llama-2-7b-hf Format: Each example contains a single field tokens with a list of 1024 token IDs

Each document was:

Tokenized using the meta-llama/Llama-2-7b-hf tokenizer Prefixed with a BOS (beginning of sequence)… See the full description on the dataset page: https://huggingface.co/datasets/eoinf/tokenized_dataset_test7.