chenzhe0000/

wikitext_cleaned

	Dataset Card for Cleaned & Chunked WikiText Corpus




	
	📌 Dataset Description

This dataset is a cleaned and chunked version of WikiText-style corpus, designed for language model pretraining and evaluation. The preprocessing pipeline follows a lightweight data curation paradigm:

cleaning → deduplication → normalization → token-aware chunking

Specifically, the dataset includes:

Removal of special and noisy characters
Cleaning of formatting artifacts (e.g., HTML tags… See the full description on the dataset page: https://huggingface.co/datasets/chenzhe0000/wikitext_cleaned.