Sentinel: Attention Probing of Proxy Models for LLM Context Compression with an Understanding Perspective
Yong Zhang, Yanwen Huang, Ning Cheng, Yang Guo, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao
📄 Paper (Arxiv 2025) | 💻 Code on GitHub

🧠 What is Sentinel?

Sentinel reframes LLM context compression as a lightweight attention-based understanding task. Instead of fine-tuning a full compression model, it:

Extracts decoder attention from a small proxy LLM (e.g., Qwen-2.5-0.5B)
Computes sentence-level attention features
Applies a logistic regression (LR) classifier to select relevant sentences

This approach is efficient, model-agnostic, and highly interpretable.

📦 Files Included

File	Description
`sentinel_lr_model.pkl`	Trained logistic regression classifier
`sentinel_config.json`	Feature extraction configuration

🚀 Usage

Use this classifier on attention-derived feature vectors to predict sentence-level relevance scores:

🛠 Feature extraction code and full pipeline available at: 👉 https://github.com/yzhangchuck/Sentinel

📈 Benchmark Results

📄 Citation

Please cite us if you use this model:

@misc{zhang2025sentinelattentionprobingproxy, title={Sentinel: Attention Probing of Proxy Models for LLM Context Compression with an Understanding Perspective}, author={Yong Zhang and Yanwen Huang and Ning Cheng and Yang Guo and Yun Zhu and Yanmeng Wang and Shaojun Wang and Jing Xiao}, year={2025}, eprint={2505.23277}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2505.23277}, }

📬 Contact

•	📧 zhangyong.chuck@gmail.com
•	🔗 Project: https://github.com/yzhangchuck/Sentinel

🔒 License

Apache License 2.0 — Free for research and commercial use with attribution.

ReRaWo/Sentinel

Sentinel Probing Classifier (Logistic Regression)