A benchmark dataset measuring LLM prompt compression quality across 100 Wikipedia articles in 5 categories, generated using TokenLens.
This dataset contains compression quality measurements for 100 Wikipedia articles compressed at 9 different ratios (0.1 to 0.9) using extractive embedding-based compression. For each article and compression ratio, the dataset records tokens saved, semantic similarity, ROUGE-L… See the full description on the dataset page:
https://huggingface.co/datasets/lavanyaashri/tokenlens-compression-benchmark.
