Generates preprocessed training data for the UKS Energy System from the FineWeb dataset.
This pipeline processes text from HuggingFace's FineWeb dataset and produces two output formats:
pre_tokenized.jsonl.gz: Ready for direct model training
intermediate.parquet: For exploration and custom tokenization workflows
pip install spacy transformers datasets tqdm
python -m spacy download en_core_web_sm
pip install… See the full description on the dataset page:
https://huggingface.co/datasets/SPOOKEXE/LargeCorpusSpaCy.
