You can deploy this model and start earning money today!

lightblue/
lb-reranker-0.5B-v1.0

Name: lb-reranker-0.5B-v1.0
Availability: InStock

Views

No views yet

LB Reranker v1.0

LBR

The LB Reranker has been trained to determine the relatedness of a given query to a piece of text, therefore allowing it to be used as a ranker or reranker in various retrieval-based tasks.

This model is fine-tuned from a Qwen/Qwen2.5-0.5B-Instruct model checkpoint and was trained for roughly 5.5 hours using the 8 x L20 instance (ecs.gn8is-8x.32xlarge) on Alibaba Cloud.

The training data for this model can be found at lightblue/reranker_continuous_filt_max7_train and the code for generating this data as well as running the training of the model can be found on our Github repo.

Trained on data in over 95 languages, this model is applicable to a broad range of use cases.

This model has three main benefits over comparable rerankers.

It has shown slightly higher performance on evaluation benchmarks.
It has been trained on more languages than any previous model.
It is a simple Causal LM model trained to output a string between "1" and "7".

This last point means that this model can be used natively with many widely available inference packages, including vLLM and LMDeploy. This in turns allows our reranker to benefit from improvements to inference as and when these packages release them.

Update: We have also found that this model works pretty well as a code snippet reranker too (P@1 of 96%)! See our Colab for more details.

How to use

The model was trained to expect an input such as:

<<<Query>>>
{your_query_here}

<<<Context>>>
{your_context_here}

And to output a string of a number between 1-7.

In order to make a continuous score that can be used for reranking query-context pairs (i.e. a method with few ties), we calculate the expectation value of the scores.

We include scripts to do this in vLLM, LMDeploy, and OpenAI (hosted for free on Huggingface):

vLLM
Install vLLM using pip install vllm.

Show vLLM code

python
1from vllm import LLM, SamplingParams 2import numpy as np 3 4def make_reranker_input(t, q): 5 return f"<<<Query>>>\n{q}\n\n<<<Context>>>\n{t}" 6 7def make_reranker_inference_conversation(context, question): 8 system_message = "Given a query and a piece of text, output a score of 1-7 based on how related the query is to the text. 1 means least related and 7 is most related." 9 10 return [ 11 {"role": "system", "content": system_message}, 12 {"role": "user", "content": make_reranker_input(context, question)}, 13 ] 14 15def get_prob(logprob_dict, tok_id): 16 return np.exp(logprob_dict[tok_id].logprob) if tok_id in logprob_dict.keys() else 0 17 18llm = LLM("lightblue/lb-reranker-v1.0") 19sampling_params = SamplingParams(temperature=0.0, logprobs=14, max_tokens=1) 20tok = llm.llm_engine.tokenizer.tokenizer 21idx_tokens = [tok.encode(str(i))[0] for i in range(1, 8)] 22 23query_texts = [ 24 ("What is the scientific name of apples?", "An apple is a round, edible fruit produced by an apple tree (Malus spp., among them the domestic or orchard apple; Malus domestica)."), 25 ("What is the Chinese word for 'apple'?", "An apple is a round, edible fruit produced by an apple tree (Malus spp., among them the domestic or orchard apple; Malus domestica)."), 26 ("What is the square root of 999?", "An apple is a round, edible fruit produced by an apple tree (Malus spp., among them the domestic or orchard apple; Malus domestica)."), 27] 28 29chats = [make_reranker_inference_conversation(c, q) for q, c in query_texts] 30responses = llm.chat(chats, sampling_params) 31probs = np.array([[get_prob(r.outputs[0].logprobs[0], y) for y in idx_tokens] for r in responses]) 32 33N = probs.shape[1] 34M = probs.shape[0] 35idxs = np.tile(np.arange(1, N + 1), M).reshape(M, N) 36 37expected_vals = (probs * idxs).sum(axis=1) 38print(expected_vals) 39# [6.66570732 1.86686378 1.01102923]
LMDeploy
Install LMDeploy using pip install lmdeploy.

Show LMDeploy code

python
1# Un-comment this if running in a Jupyter notebook, Colab etc. 2# import nest_asyncio 3# nest_asyncio.apply() 4 5from lmdeploy import GenerationConfig, ChatTemplateConfig, pipeline 6import numpy as np 7 8def make_reranker_input(t, q): 9 return f"<<<Query>>>\n{q}\n\n<<<Context>>>\n{t}" 10 11def make_reranker_inference_conversation(context, question): 12 system_message = "Given a query and a piece of text, output a score of 1-7 based on how related the query is to the text. 1 means least related and 7 is most related." 13 14 return [ 15 {"role": "system", "content": system_message}, 16 {"role": "user", "content": make_reranker_input(context, question)}, 17 ] 18 19def get_prob(logprob_dict, tok_id): 20 return np.exp(logprob_dict[tok_id]) if tok_id in logprob_dict.keys() else 0 21 22pipe = pipeline( 23 "lightblue/lb-reranker-v1.0", 24 chat_template_config=ChatTemplateConfig( 25 model_name='qwen2d5', 26 capability='chat' 27 ) 28) 29tok = pipe.tokenizer.model 30idx_tokens = [tok.encode(str(i))[0] for i in range(1, 8)] 31 32query_texts = [ 33 ("What is the scientific name of apples?", "An apple is a round, edible fruit produced by an apple tree (Malus spp., among them the domestic or orchard apple; Malus domestica)."), 34 ("What is the Chinese word for 'apple'?", "An apple is a round, edible fruit produced by an apple tree (Malus spp., among them the domestic or orchard apple; Malus domestica)."), 35 ("What is the square root of 999?", "An apple is a round, edible fruit produced by an apple tree (Malus spp., among them the domestic or orchard apple; Malus domestica)."), 36] 37 38chats = [make_reranker_inference_conversation(c, q) for q, c in query_texts] 39responses = pipe( 40 chats, 41 gen_config=GenerationConfig(temperature=1.0, logprobs=14, max_new_tokens=1, do_sample=True) 42) 43probs = np.array([[get_prob(r.logprobs[0], y) for y in idx_tokens] for r in responses]) 44 45N = probs.shape[1] 46M = probs.shape[0] 47idxs = np.tile(np.arange(1, N + 1), M).reshape(M, N) 48 49expected_vals = (probs * idxs).sum(axis=1) 50print(expected_vals) 51# [6.66415229 1.84342025 1.01133205]
OpenAI (Hosted on Huggingface)
Install openai using pip install openai.

Show OpenAI + Huggingface Inference code

python
1from openai import OpenAI 2import numpy as np 3from multiprocessing import Pool 4from tqdm.auto import tqdm 5 6client = OpenAI( 7 base_url="https://api-inference.huggingface.co/v1/", 8 api_key="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" # Change this to an access token from https://huggingface.co/settings/tokens 9) 10 11def make_reranker_input(t, q): 12 return f"<<<Query>>>\n{q}\n\n<<<Context>>>\n{t}" 13 14def make_reranker_inference_conversation(context, question): 15 system_message = "Given a query and a piece of text, output a score of 1-7 based on how related the query is to the text. 1 means least related and 7 is most related." 16 17 return [ 18 {"role": "system", "content": system_message}, 19 {"role": "user", "content": make_reranker_input(context, question)}, 20 ] 21 22def get_reranker_score(context_question_tuple): 23 question, context = context_question_tuple 24 25 messages = make_reranker_inference_conversation(context, question) 26 27 completion = client.chat.completions.create( 28 model="lightblue/lb-reranker-0.5B-v1.0", 29 messages=messages, 30 max_tokens=1, 31 temperature=0.0, 32 logprobs=True, 33 top_logprobs=5, # Max allowed by the openai API as top_n_tokens must be >= 0 and <= 5. If this gets changed, fix to > 7. 34 ) 35 36 logprobs = completion.choices[0].logprobs.content[0].top_logprobs 37 38 calculated_score = sum([int(x.token) * np.exp(x.logprob) for x in logprobs]) 39 40 return calculated_score 41 42query_texts = [ 43 ("What is the scientific name of apples?", "An apple is a round, edible fruit produced by an apple tree (Malus spp., among them the domestic or orchard apple; Malus domestica)."), 44 ("What is the Chinese word for 'apple'?", "An apple is a round, edible fruit produced by an apple tree (Malus spp., among them the domestic or orchard apple; Malus domestica)."), 45 ("What is the square root of 999?", "An apple is a round, edible fruit produced by an apple tree (Malus spp., among them the domestic or orchard apple; Malus domestica)."), 46] 47 48with Pool(processes=16) as p: # Allows for parallel processing 49 expected_vals = list(tqdm(p.imap(get_reranker_score, query_texts), total=len(query_texts))) 50 51print(expected_vals) 52# [6.64866580, 1.85144404, 1.010719508]

Evaluation

We perform an evaluation on 9 datasets from the BEIR benchmark that none of the evaluated models have been trained upon (to our knowledge).

Arguana
Dbpedia-entity
Fiqa
NFcorpus
Scidocs
Scifact
Trec-covid-v2
Vihealthqa
Webis-touche2020

We evaluate on a subset of all queries (the first 250) to save evaluation time.

We find that our model performs similarly or better than many of the state-of-the-art reranker models in our evaluation, without compromising on inference speed.

We make our evaluation code and results available on our Github.