Views
No views yet
slim token-saverslim-int8 is a CPU-friendly, int8-quantized ONNX repackaging of
chopratejas/kompress-v2-base,
the extractive prompt-compression model (LLMLingua-2 family) used by
headroom.slim CLI (a token-saver for coding assistants) can run the
semantic-compression path on a laptop CPU with no GPU and no build step — the
model auto-downloads from here on first use.P(keep) ∈ [0,1]; the
caller drops the low scorers. Output is a strict subsequence of the input —
lossy but order-preserving and never hallucinated. Base architecture is
ModernBERT-base + a LoRA adapter + a dual head (token classifier + span-smoothing
1-D CNN). See the base model card for training details (F1 0.918, must-keep
recall 0.974 @ threshold 0.5).| File | Size | Notes |
|---|---|---|
onnx/slim-int8.onnx | ~257 MB | Primary. Dynamic int8 (MatMul-only, so the span-head Conv stays fp32) — loads on stock onnxruntime CPU. |
onnx/slim-fp32.onnx | ~573 MB | Fallback for environments where the int8 op set is unavailable. |
tokenizer.json, tokenizer_config.json, special_tokens_map.json, config.json | small | ModernBERT tokenizer + model config. |
Quantized MatMul-only on purpose: quantizing the span-headConvproduced aConvIntegerop that stock onnxruntime-CPU can't execute. Keeping Conv in fp32 yields a model that loads everywhere while still shrinking the transformer.
1import numpy as np, onnxruntime as ort
2from transformers import AutoTokenizer
3
4sess = ort.InferenceSession("onnx/slim-int8.onnx", providers=["CPUExecutionProvider"])
5tok = AutoTokenizer.from_pretrained(".") # this repo
6
7words = "The authentication handler validates the token by calling verify().".split()
8enc = tok(words, is_split_into_words=True, truncation=True, max_length=512,
9 padding=True, return_tensors="np")
10scores = sess.run(["final_scores"], {
11 "input_ids": enc["input_ids"].astype(np.int64),
12 "attention_mask": enc["attention_mask"].astype(np.int64),
13})[0][0] # P(keep) per subword
14
15# aggregate subword -> word via enc.word_ids(), keep word if mean score >= 0.5input_ids, attention_mask (int64); output
final_scores [batch, seq] float. Threshold is tunable — 0.5 keeps ~81%,
0.7 keeps ~70%.chopratejas/kompress-v2-base (Apache-2.0); this repackaging is
Apache-2.0. All model weights and training credit belong to the original authors.
Built for the slim token-optimization CLI.