castorini/

NanoKnow-Fineweb-Edu-Index

[Paper] [Code] A pre-built Lucene BM25 index over karpathy/fineweb-edu-100b-shuffle—the exact pre-training corpus used by the nanochat family of language models. Built with Anserini. This index is part of the NanoKnow project: github.com/castorini/NanoKnow

	Index Details




	

Property Value

Corpus karpathy/fineweb-edu-100b-shuffle

Documents 97,230,848

Index Size ~325 GB (extracted)

Index Type Lucene (BM25)

Built With… See the full description on the dataset page: https://huggingface.co/datasets/castorini/NanoKnow-Fineweb-Edu-Index.