eoinf/

tokenized_dataset_test5

	Dataset Card for eoinf/tokenized_dataset_test5




	
	Original dataset

Original dataset: monology/pile-uncopyrighted

Total Tokens: 51,200 Total Sequences: 50 Context Length: 1024 tokens Tokenizer: meta-llama/Llama-2-7b-hf Format: Each example contains a single field tokens with a list of 1024 token IDs

Each document was:

Tokenized using the meta-llama/Llama-2-7b-hf tokenizer Prefixed with a BOS (beginning of sequence) token… See the full description on the dataset page: https://huggingface.co/datasets/eoinf/tokenized_dataset_test5.