The Llama Nemotron Reranking 1B model is optimized for providing a logit score that represents how relevant a document(s) is to a given query. The model was fine-tuned for multilingual, cross-lingual text question-answering retrieval, with support for long documents (up to 8192 tokens). This model was evaluated on 26 languages: English, Arabic, Bengali, Chinese, Czech, Danish, Dutch, Finnish, French, German, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, and Turkish.
This model is a component in a text retrieval system to improve the overall accuracy. A text retrieval system often uses an embedding model (dense) or lexical search (sparse) index to return relevant text passages given the input. A reranking model can be used to rerank the potential candidate into a final order. The reranking model has the question-passage pairs as an input and therefore, can process cross attention between the words. It’s not feasible to apply a Ranking model on all documents in the knowledge base, therefore, ranking models are often deployed in combination with embedding models.
This model is ready for commercial use.
The Llama Nemotron Reranking 1B model is a part of the NeMo Retriever collection of NIM, which provide state-of-the-art, commercially-ready models and microservices, optimized for the lowest latency and highest throughput. It features a production-ready information retrieval pipeline with enterprise support. The models that form the core of this solution have been trained using responsibly selected, auditable data sources. With multiple pre-trained models available as starting points, developers can also readily customize them for their domain-specific use cases, such as information technology, human resource help assistants, and research & development research assistants.
We are excited to announce the open sourcing of this commercial embedding model. For users interested in deploying this model in production environments, it is also available via the model API in NVIDIA Inference Microservices (NIM) at llama-nemotron-rerank-1b-v2.
This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
Intended use
The Llama Nemotron Reranking 1B model is most suitable for users who want to improve their multilingual retrieval tasks by reranking a set of candidates for a given question.
Model Architecture
Architecture Type: Transformer Network Architecture: Fine-tuned ranker model from the meta-llama/Llama-3.2-1B model.
The Llama Nemotron Reranking 1B model is a transformer cross-encoder fine-tuned with contrastive learning. We employ bi-directional attention when fine-tuning for higher accuracy. The last embedding output by the decoder model is used with a mean pooling strategy, and a binary classification head is fine-tuned for the ranking task.
Ranking models for text ranking are typically trained as a cross-encoder for sentence classification. This involves predicting the relevancy of a sentence pair (for example, question and chunked passages). The CrossEntropy loss is used to maximize the likelihood of passages containing information to answer the question and minimize the likelihood for (negative) passages that do not contain information to answer the question.
We trained the model on public datasets described in the Dataset and Training section.
Input
Input Type: Pair of Texts Input Format: List of text pairs Input Parameters: 1D Other Properties Related to Input: The model was trained on question and answering over text documents from multiple languages. It was evaluated to work successfully with up to a sequence length of 8192 tokens. Longer texts are recommended to be either chunked or truncated.
Output
Output Type: Floats Output Format: List of floats Output Parameters: 1D Other Properties Related to Output: Each value corresponds to a raw logit. Users can choose to apply a Sigmoid activation function to the logits to convert them into probabilities during model usage.
Installation
The model requires transformers version 4.44 or above.
pip install transformers>=4.44
Usage
python
1import torch
2from transformers import AutoTokenizer, AutoModelForSequenceClassification
34model_name_or_path ="nvidia/llama-nemotron-rerank-1b-v2"56device ="cuda:0"7max_length =51289queries =[10"how much protein should a female eat?",11]12documents =[13"As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",14"Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments.",15"Calorie intake should not fall below 1,200 a day in women or 1,500 a day in men, except under the supervision of a health professional."16]1718# Create pairs from queries and documents19pairs =[[q, d]for q in queries for d in documents]2021defprompt_template(q, p):22"""Format query and passage with a prompt template."""23returnf"question:{q} \n \n passage:{p}"242526tokenizer = AutoTokenizer.from_pretrained(27 model_name_or_path,28 trust_remote_code=True,29 padding_side="left"30)31if tokenizer.pad_token isNone:32 tokenizer.pad_token = tokenizer.eos_token
3334model_kwargs ={35"trust_remote_code":True,36"torch_dtype": torch.bfloat16,37}3839print(f"Loading model from {model_name_or_path}...")40model = AutoModelForSequenceClassification.from_pretrained(41 model_name_or_path,42**model_kwargs
43).eval()4445if model.config.pad_token_id isNone:46 model.config.pad_token_id = tokenizer.eos_token_id
4748model = model.to(device)495051# Apply prompt template and tokenize as single sequence52texts =[prompt_template(query, doc)for query, doc in pairs]53batch_dict = tokenizer(54 texts,55 padding=True,56 truncation=True,57 return_tensors="pt",58 max_length=max_length,59)6061# Move to device62batch_dict ={k: v.to(device)for k, v in batch_dict.items()}6364with torch.inference_mode():65 logits = model(**batch_dict).logits
66 scores = logits.view(-1).cpu().tolist()6768for i,(pair, score)inenumerate(zip(pairs, scores)):69 query, doc = pair
70print(f" Query: {query}")71print(f" Document: {doc[:100]}{'...'iflen(doc)>100else''}")72print(f" Score: {score:.4f}")7374# Query: how much protein should a female eat?75# Document: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams...76# Score: 20.625077# Query: how much protein should a female eat?78# Document: Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top o...79# Score: -23.125080# Query: how much protein should a female eat?81# Document: Calorie intake should not fall below 1,200 a day in women or 1,500 a day in men, except under the su...82# Score: -0.2617
vLLM Usage
Ensure you are using vllm>=0.14.0.
A score template must be provided via --chat-template to correctly format
query-document pairs. Without it, the question:... passage:... prompt format
is not applied and results will be incorrect.
Create the score template file and start the server:
If you already have a local copy of the model, you can also pass the local
path instead of the HF repo ID.
Optional flags:
--dtype <float32|bfloat16|float16> to force precision (the default is auto, which resolves from model config; this model defaults to BF16).
--data-parallel-size <num_gpus_to_use> for multi-GPU serving.
--port 8000 to set the server port.
Online serving example (/rerank API):
python
1import requests
23query ="What is machine learning?"4documents =[5"Machine learning is a branch of AI that learns patterns from data.",6"Python is a programming language commonly used for data science.",7"Neural networks are one family of machine learning models.",8"Bananas are a good source of potassium.",9]1011response = requests.post(12"http://localhost:8000/rerank",13 json={14"model":"nvidia/llama-nemotron-rerank-1b-v2",15"query": query,16"documents": documents,17"top_n":3,18},19 timeout=30,20)21response.raise_for_status()22for item in response.json()["results"]:23print(item["index"], item["relevance_score"])
Offline inference example (Python API, no server required):
python
1from vllm import LLM
23SCORE_TEMPLATE =(4"question:{{ (messages | selectattr(\"role\", \"eq\", \"query\") | first).content }}"5" \n \n "6"passage:{{ (messages | selectattr(\"role\", \"eq\", \"document\") | first).content }}"7)89llm = LLM(10 model="nvidia/llama-nemotron-rerank-1b-v2",11 runner="pooling",12 trust_remote_code=True,13)1415query ="What is machine learning?"16documents =[17"Machine learning is a branch of AI that learns patterns from data.",18"Python is a programming language commonly used for data science.",19"Neural networks are one family of machine learning models.",20"Bananas are a good source of potassium.",21]2223outputs = llm.score(query, documents, chat_template=SCORE_TEMPLATE)24for document, output inzip(documents, outputs):25print(document[:80], output.outputs.score)
Software Integration
Runtime: Llama Nemotron Reranking 1B NIM Supported Hardware Microarchitecture Compatibility: NVIDIA Ampere, NVIDIA Hopper, NVIDIA Lovelace Supported Operating System(s): Linux
Model Version(s)
Llama Nemotron Reranking 1B
Short Name: llama-nemotron-rerank-1b-v2
Training Dataset & Evaluation
Training Dataset
The development of large-scale public open-QA datasets has enabled tremendous progress in powerful embedding models. However, one popular dataset named MSMARCO restricts commercial licensing, limiting the use of these models in commercial settings. To address this, NVIDIA created its own training dataset blend based on public QA datasets, which each have a license for commercial applications.
Data Collection Method by dataset: Automated, Unknown
Labeling Method by dataset: Automated, Unknown
Properties: This model was trained on 800k samples from public datasets.
Evaluation Results
We evaluate the pipelines on a set of evaluation benchmarks. We applied the ranking model to the candidates retrieved from a retrieval embedding model.
Overall, the pipeline llama-nemotron-embed-1b-v2 + llama-nemotron-rerank-1b-v2 provides high BEIR+TechQA accuracy with multilingual and crosslingual support. The llama-nemotron-embed-1b-v2 ranking model is 3.5x smaller than the nv-rerankqa-mistral-4b-v3 model.
We evaluated the NVIDIA Retrieval QA Embedding Model in comparison to literature open & commercial retriever models on academic benchmarks for question-answering - NQ, HotpotQA and FiQA (Finance Q&A) from BeIR benchmark and TechQA dataset. In this benchmark, the metric used was Recall@5. As described, we need to apply the ranking model on the output of an embedding model.
Open & Commercial Reranker Models
Average Recall@5 on NQ, HotpotQA, FiQA, TechQA dataset
We evaluated the model’s multilingual capabilities on the MIRACL academic benchmark - a multilingual retrieval dataset, across 15 languages, and on an additional 11 languages that were translated from the English and Spanish versions of MIRACL. The reported scores are based on a custom subsampled version by selecting hard negatives for each query to reduce the corpus size.
We evaluated the cross-lingual capabilities on the academic benchmark MLQA based on 7 languages (Arabic, Chinese, English, German, Hindi, Spanish, Vietnamese). We consider only evaluation datasets when the query and documents are in different languages. We calculate the average Recall@5 across the 42 different language pairs.
Open & Commercial Retrieval Models
Average Recall@5 on MLQA dataset with different languages
We evaluated the support of long documents on the academic benchmark Multilingual Long-Document Retrieval (MLDR) built on Wikipedia and mC4, covering 12 typologically diverse languages . The English version has a median length of 2399 tokens and 90th percentile of 7483 tokens using the llama 3.2 tokenizer.
Properties
The evaluation datasets are based on three MTEB/BEIR TextQA datasets, the TechQA dataset, MIRACL, MLDR and MLQA multilingual retrieval datasets, which are all public datasets. The sizes range between 10,000s up to 5M depending on the dataset.
InferenceEngine: TensorRT Test Hardware: H100 PCIe/SXM, A100 PCIe/SXM, L40s, L4, and A10G
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
For more detailed information on ethical considerations for this model, please see the Explainability, Bias, Safety, and Privacy sections.
Please report security vulnerabilities or NVIDIA AI Concerns here.
Participation considerations from adversely impacted groups protected classes in model design and testing
None
Measures taken to mitigate against unwanted bias
None
Explainability
Field
Response
Intended Application & Domain:
Passage ranking for question and answer retrieval
Model Type:
Transformer encoder
Intended User:
Generative AI creators working with conversational AI models - users who want to build a multilingual question and answer application over a large text corpus, leveraging the latest dense retrieval technologies.
Output:
Array of float numbers (Dense Vector Representation for the input text)
Describe how the model works:
Model transforms the tokenized input text into a dense vector representation.
Performance Metrics:
Accuracy, Throughput, and Latency
Potential Known Risks:
This model does not always guarantee to retrieve the correct passage(s) for a given query.
The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.
