hkust-nlp/

llm-compression

This is the compression corpora dataset used in the paper "Compression Represents Intelligence Linearly". We find that LLMs’ intelligence – reflected by benchmark scores – almost linearly correlates with their ability to compress external text corpora. We measure intelligence along three key abilities: knowledge and commonsense, coding, and mathematical reasoning, and provide the corresponding compression corpora here respectively named cc, python, and arxiv_math.