This is the chunked version of Wikipedia HF Dataset.
{
"article_hash": "... hashes ...",
"language": 'en',
"chunks": ["chunks1", "chunks2", "chunks3"],
"num_chunks": 3
}
chunking strategy is in the scripts/chunking.py.
detailed description will be provided later. Please stay tuned!
All licence reserved to the original author.
