Yuki5177/

owt-ngrams

	Dataset Card for OpenWebText n-grams




	


	Dataset Summary

This dataset contains 246K of the most common token-based (GPT-2/GPT-3) n-grams (n=1 to n=6), in the OpenWebText (OWT) dataset. For convenient searching, it provides full tokens/strings, as well as per-position tokens/strings.

Generally, this dataset allows identifying the most common n-grams in a text corpus. When researching LLMs tokenized similarly to GPT-2/GPT-3, it allows:

Constructing… See the full description on the dataset page: https://huggingface.co/datasets/Yuki5177/owt-ngrams.