You can deploy this model and start earning money today!

UIC-AI-lab/
infigram-wikipedia-index

Views

No views yet

Infini-gram Wikipedia Index (v4, enwiki 202311, LLaMA tokenizer)

Pre-built suffix array index over English Wikipedia (November 2023 dump) for use with infini-gram.

	Details

Property Value

Wikipedia version 20231101.en

Articles 6,407,814

Tokenizer LLaMA-2 (vocab=32000, u16)

Index size ~38 GB

Engine version v4 (InfiniGramEngine)

Usage

Download

huggingface-cli download UIC-R2-lab/infigram-wikipedia-index --repo-type dataset… See the full description on the dataset page: https://huggingface.co/datasets/UIC-AI-lab/infigram-wikipedia-index.