Kompress: ModernBERT Token Compressor for LLM Context Windows
Kompress compresses text in LLM context windows so agents can do more with less. It's a drop-in replacement for LLMLingua-2 that's higher quality and 2.3x faster.
Quality scores are judged by Claude Sonnet 4.6: "Can an LLM fully understand and act on the compressed version?" (1-10 scale).
How It Works
Kompress is a dual-head ModernBERT model trained to classify each token as keep or discard:
Token head: Binary classifier (keep/discard per token via argmax)
Span head: 1D CNN that identifies important regions, boosts borderline tokens in critical spans
The model decides how much to compress based on content density — no fixed compression ratio.
Example
ORIGINAL (98 words):
After investigating the memory leak, I traced it to the event listener
registration in the WebSocket handler. Every time a client connects, we
register a new listener on the global event bus, but when the client
disconnects, the cleanup function only removes the WebSocket connection
from the pool — it doesn't unregister the event listener. Over time,
these orphaned listeners accumulate and each one holds a reference to
the connection's closure, which in turn holds the entire request context.
The fix is straightforward: store the listener reference at connection
time and explicitly remove it in the disconnect handler.
COMPRESSED (59 words, 60% kept):
investigating memory leak, traced event listener registration WebSocket
handler. Every time client connects, register new listener global event
bus, client disconnects, cleanup function only removes WebSocket
connection pool — doesn't unregister event listener. Over time, orphaned
listeners accumulate each one holds reference connection's closure, holds
entire request context. fix straightforward: store listener reference
connection time explicitly remove disconnect handler.
An LLM can fully understand and act on the compressed version.
Usage
python
1from kompress.inference.pytorch_runner import KompressRunner
23# Auto-downloads from HuggingFace on first use4runner = KompressRunner()56result = runner.compress("Your long text here...")7print(result.compressed)# Compressed text8print(result.compression_ratio)# e.g., 0.629print(result.tokens_saved)# Number of tokens saved
With Headroom (LLM Proxy)
pip install headroom-ai
Kompress is built into Headroom as the default text compressor. It auto-downloads and runs on every API request that passes through the proxy.
215K extractive compression labels from 8 diverse datasets, labeled by Claude Sonnet 4.6:
Dataset
Count
Type
LMSYS-Chat-1M
57K
LLM conversations
CNN/DailyMail
50K
News articles
WikiHow
50K
How-to guides
MeetingBank
50K
Meeting transcripts
XSum
47K
News articles
GovReport
25K
Government reports
ArXiv
25K
Academic papers
SAMSum
14K
Dialogues
Labeling Approach
Key insight: the labels must be strictly extractive — a subset of original words in original order. Previous versions failed because the labeling LLM rephrased text, causing alignment failures (5-12% keep ratio instead of the intended 40-60%).
The fix: prompt Claude to "select words like highlighting with a marker" rather than "compress this text." This ensures every word in the compressed output exists in the original, and the greedy alignment recovers 95%+ of the intended labels.
