M-H-MARUF/

bengali-tokenization-corpus

	Bengali Tokenization Corpus (25k Sentences)




	


	Dataset Description

A balanced 25,000-sentence Bengali corpus designed for tokenization benchmarking.

This dataset is used in the manuscript: Quantifying the Tokenization Tax on Bengali: A Multi-Tokenizer Audit Across 25k Sentences MD. Muktadirul Haque Maruf, Israt Jahan Munny, Moutithi Sarker MouManuscript in preparation

Academic News Literary Colloquial Dialectal… See the full description on the dataset page: https://huggingface.co/datasets/M-H-MARUF/bengali-tokenization-corpus.