Retrieve-Pile (Knowledge-Pile) is a knowledge-related data leveraging Retrieve-from-CC (We also called this method as "Query of CC")，a total of 735GB disk size and 188B tokens (using Llama2 tokenizer).
Just like the figure below, we initially collected seed information in some specific domains, such as keywords, frequently asked questions, and textbooks, to serve as inputs for the Query Bootstrapping stage. Leveraging the great generalization capability of large… See the full description on the dataset page:
https://huggingface.co/datasets/Query-of-CC/Retrieve-Pile.
