Dense retrieval is a crucial task in Information Retrieval (IR), serving as the basis for downstream tasks such as re-ranking and augmenting generation. Recently, large language models (LLMs) have demonstrated impressive semantic understanding capabilities, making them attractive to researchers focusing on dense retrieval. While LLMs, as decoder-style generative models, excel in language generation, they often fall short in modeling global information due to a lack of attention to subsequent tokens. Drawing inspiration from the classical word-based language modeling approach for IR, specifically the query likelihood (QL) model, we aim to leverage the generative strengths of LLMs through QL maximization. Rather than employing QL estimation for document ranking, we propose an auxiliary task of QL maximization to enhance the backbone for subsequent contrastive learning of the retriever. We introduce our model, LLM-QL, which incorporates two key components: Attention Block (AB) and Document Corruption (DC). AB blocks the attention of predictive tokens to the document tokens before the document's ending token, while DC corrupts a document by masking a portion of its tokens during prediction. Evaluations on the in-domain (MS MARCO) and out-of-domain dataset (BEIR) indicate LLM-QL's superiority over other LLM-based retrievers. Furthermore, comprehensive analyses also validate the efficacy of LLM-QL and its components.
Framework
Training data:
MS MARCO, About 500K data.
Model Training
First stage: Query likelihood modeling using Attention Block (AB) and Document Corruption (DC).
Second stage: Constrative Learning using InfNCE Loss
AttentionBlock
Citation
If you find our work helpful, feel free to give us a cite.
@article{zhang2025unleashing,
title={Unleashing the power of llms in dense retrieval with query likelihood modeling},
author={Zhang, Hengran and Bi, Keping and Guo, Jiafeng and Sun, Xiaojie and Liu, Shihao and Shi, Daiting and Yin, Dawei and Cheng, Xueqi},
journal={arXiv preprint arXiv:2504.05216},
year={2025}
}
