You can deploy this model and start earning money today!

encodr1/
encodr-snrv1-d-base

Name: encodr-snrv1-d-base
Availability: InStock

answerdotai/ModernBERT-base

apache-2.0

Views

No views yet

encodr-snrv1-d-base

Extractive KEEP/DROP compressor for chat prompts. Predicts, per word, whether it should survive compression — the surviving words form a shorter version of the conversation that keeps the information needed to answer follow-up questions, at a fraction of the token cost.

encodr-snrv1-d-base is the ModernBERT-base member of the encodr-SNR ("Signal-to-Noise") family, trained on labels from DeepSeek-V4-Flash. It's a LoRA adapter (r=16) on top of answerdotai/ModernBERT-base (149M params) with a token-classification head, so it is small, fast, and cheap to serve.

Quick start

Requires transformers and peft.

python

import torch
from peft import PeftModel
from transformers import AutoModelForTokenClassification, AutoTokenizer

REPO = "encodr1/encodr-snrv1-d-base"
BASE = "answerdotai/ModernBERT-base"

tokenizer = AutoTokenizer.from_pretrained(REPO)
base_model = AutoModelForTokenClassification.from_pretrained(BASE, num_labels=2)
model = PeftModel.from_pretrained(base_model, REPO).eval()

text = "User: What's the capital of France?\n\nAssistant: The capital of France is Paris."
words = text.split()

enc = tokenizer(words, is_split_into_words=True, truncation=True, max_length=1536, return_tensors="pt")
word_ids = enc.word_ids()

with torch.no_grad():
    logits = model(**enc).logits[0]
p_keep = torch.softmax(logits, dim=-1)[:, 1]  # P(KEEP) per subword token

# average subword scores back up to word level, then threshold at 0.5
sums, counts = [0.0] * len(words), [0] * len(words)
for tok_idx, word_id in enumerate(word_ids):
    if word_id is not None:
        sums[word_id] += p_keep[tok_idx].item()
        counts[word_id] += 1
word_scores = [s / c if c else 1.0 for s, c in zip(sums, counts)]

compressed = " ".join(w for w, s in zip(words, word_scores) if s >= 0.5)
print(compressed)

Word-level scoring matters: the model is trained on whitespace-split words (text.split()), with every subword of a word sharing that word's label, so score at the word level, not the raw-token level, to match training.

Controllable compression ratio

Instead of thresholding at 0.5, rank words by word_scores and keep the top-k for a target compression ratio (LLMLingua-2 style):

python

target_keep_ratio = 0.5
n_keep = max(1, round(len(words) * target_keep_ratio))
keep_idx = set(sorted(range(len(words)), key=lambda i: word_scores[i], reverse=True)[:n_keep])
compressed = " ".join(w for i, w in enumerate(words) if i in keep_idx)

Training details

Data

Trained on English single-turn-flattened chat conversations from UltraChat (train_sft split), extractively compressed by DeepSeek-V4-Flash (via Azure AI Foundry, temperature=0) under a system prompt that only allows word deletion (no rewriting/paraphrasing/summarizing), targeting ~60–85% word retention, with User:/Assistant: turn markers always preserved.

Word-level KEEP/DROP labels come from a diff (difflib.SequenceMatcher) between the original and teacher-compressed text: a word is KEEP if it survives in the same relative position, DROP otherwise. A hard-keep overlay then force-labels structurally important spans as KEEP regardless of what the teacher did — regex for URLs, emails, currency, percentages, decimal numbers, years, dates, filenames, and snake_case/camelCase/PascalCase identifiers, plus GLiNER NER for person/organization/location/date/product entities.

Train: 111,711 examples · Validation: 5,880 examples (95/5 split, seed 42)
Single domain (chat), max sequence length 1,536 tokens

Only chat-domain data has been used so far — out-of-domain behavior (code, long documents, etc.) is not covered by training, though see the MeetingBank generalization numbers below.

Procedure

Base: answerdotai/ModernBERT-base (149M params), AutoModelForTokenClassification head (2 labels), fine-tuned via LoRA (PEFT)
LoRA: r=16, alpha=32, dropout=0.05, target_modules=[Wqkv, Wi, Wo], task_type=TOKEN_CLS, classifier head also trained (modules_to_save=["classifier"])
Trainable params: 3,380,738 / 152,987,140 total (2.2%)
Optimizer: AdamW, lr=2e-4, 1 epoch, effective batch size 32 (16 × grad_accum 2), bf16, no gradient checkpointing
Hardware: 1× RTX 3090 (24GB), ~2h39m wall-clock

Held-out eval (own DeepSeek-labeled test split, threshold 0.5)

Metric	Value
keep precision	0.910
keep recall	0.941
keep F1	0.926
accuracy	0.886
eval loss	0.280

Note: this is not directly comparable to the GPT-5.6-labeled siblings' eval numbers below — the label distributions differ (DeepSeek keeps ~70% of words on average vs. GPT-5.6's ~96%), so the tasks have different class balance.

Intended use

Designed to sit in front of an LLM call (e.g. in an LLM proxy) to shrink conversation context before it's sent to a downstream model, cutting token cost while preserving the information needed to answer questions about that context. Not a summarizer — it only deletes words, never rewrites them, so compressed output is always a subsequence of the original text.

Limitations

Trained on English chat data only (UltraChat); other domains/languages are out-of-distribution.
Extractive by construction — cannot compress via paraphrase, only word deletion, so there's a hard floor on achievable compression for information-dense text.
Word/label alignment assumes whitespace tokenization (text.split()); text without spaces (e.g. some non-Latin scripts) will not align correctly.

License

Apache 2.0. answerdotai/ModernBERT-base is also Apache 2.0.

Framework versions

PEFT 0.19.1
transformers (ModernBERT support required)