NV-QwenOmni-Embed-3B-v1 is a versatile multimodal embedding model capable of encoding content across multiple modalities, including text, image, audio, and video, either individually or in combination, and supports retrieval using queries that can also be multimodal. It is designed to serve as a foundational component in multi-modal Retrieval-Augmented Generation (RAG) systems.
The foundational Qwen Omni model (Qwen/Qwen2.5-Omni-3B) is based on the Thinker-Talker architecture. We only leverage the Thinker component to encode and understand diverse modalities. In this implementation, we do not include the Talker component, as the model focuses on multimodal understanding rather than response generation.
This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
Team
Mengyao Xu
Gabriel Moreira
Radek Osmulski
Ronay Ak
Yauhen Babakhin
Bo Liu
Even Oldridge
Benedikt Schifferer
Citation
@article{xu2025omni,
title={Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video},
author={Xu, Mengyao and Zhou, Wenfei and Babakhin, Yauhen and Moreira, Gabriel and Ak, Ronay and Osmulski, Radek and Liu, Bo and Oldridge, Even and Schifferer, Benedikt},
journal={arXiv preprint arXiv:2510.03458},
year={2025}
}
@misc{moreira2025nvretrieverimprovingtextembedding,
title={NV-Retriever: Improving text embedding models with effective hard-negative mining},
author={Gabriel de Souza P. Moreira and Radek Osmulski and Mengyao Xu and Ronay Ak and Benedikt Schifferer and Even Oldridge},
year={2025},
eprint={2407.15831},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2407.15831},
}
Deployment Geography
Global
Use Case
NV-Omni-Embed is intended for researchers and developers building retrieval-based applications that require understanding and retrieve information across multiple modalities. It is particularly useful in multimodal RAG systems, where queries and documents may include combinations of text, images, audio, and videos. Potential applications include multimedia search engines, cross-modal retrieval systems, and conversational AI with rich input understanding.
NV-QwenOmni-Embed-3B-v1 is a transformer-based multimodal embedding model built on top of the Thinker component from Qwen/Qwen2.5-Omni-3B. Unlike the original Thinker-Talker architecture, this model does not include the Talker module, as it is designed specifically for multimodal understanding and retrieval rather than response generation. Number of model parameters is 4.7B.
The model incorporates a vision encoder, an audio encoder, and a large language model (LLM) from the Qwen architecture to process diverse modalities. Unlike the Omni model, which interleaves audio and video tokens with TMRoPE, our retrieval encoder keeps the two streams separate. Audio and video are encoded independently, preserving their full temporal structure without interleaving. Our experiments show this design improves retrieval performance.
NV-QwenOmni-Embed-3B-v1 is trained using a bi-encoder architecture where queries and candidate inputs are embedded independently. A contrastive learning objective is employed to align relevant query-content pairs while pushing apart unrelated ones in the shared embedding space.
Input
Property
Query
Document
Input Type
Text | Image | Audio | Video | Any combination
Text | Image | Audio | Video | Any combination
Input Format
List of strings, image tensors, audio arrays, or video clips
List of text strings, images, audio, or video clips
Text
Image
Video
Audio
Input Parameter
str, list[str], or pre-tokenized list[list[str]]; encoded to token IDs; per-sample 1D; batched 2D [batch, seq_len]
PIL.Image, np.ndarray, or torch.Tensor; per-sample 3D; batched 4D
np.ndarray, torch.Tensor, list of frames per-sample 4D; batched 5D; or file (like .mp4)
1D waveform (np.ndarray or torch.Tensor) per-sample 1D; batched 2D [batch, num_samples], or file
Other Properties: The model's maximum context length is 32768 tokens.
Output
Output Type: Floats
Output Format: List of float arrays
Output Parameters: A tensor of floats equivalent to [batchsize x 2048]
Other Properties Related to Output: Model outputs embedding vectors of dimension 2048 for each input.
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
Usage
Using Sentence Transformers
Install Sentence Transformers:
pip install sentence_transformers
python
1from sentence_transformers import SentenceTransformer
23model = SentenceTransformer(4"nvidia/omni-embed-nemotron-3b",5 trust_remote_code=True,6 model_kwargs={"attn_implementation":"flash_attention_2"},7)89# Configure Transformer video/audio processing10model[0].processing_kwargs.update({11"video":{12"min_pixels":32*14*14,13"max_pixels":64*28*28,14"do_sample_frames":True,15"fps":2,16},17"audio":{"max_length":2048000},18})1920# In this example we're using text queries and text+video+audio documents, but other combinations are also supported21queries =["Drawing of a cat","Drawing of a guitar","Man walking down the street"]22# One document: text + video + audio (a video of someone drawing a guitar)23documents =[24{25"text":"This is a passage to be embedded",26"video":"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/draw.mp4",27"audio":"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/draw.mp4",28},29]3031query_embeddings = model.encode_query(queries)32document_embeddings = model.encode_document(documents)33print(query_embeddings.shape, document_embeddings.shape)34# (3, 2048) (1, 2048)3536similarities = model.similarity(query_embeddings, document_embeddings)37print(similarities)38# tensor([[0.3532],39# [0.5502],40# [0.2215]])
Note: The model supports text, image, audio, and video inputs. For larger non-text inputs (e.g. long videos), you may need to adjust model[0].processing_kwargs to control resolution and frame sampling.
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Model Version(s)
Nvidia Omni Embed Nemotron 3B
Short name: omni-embed-nemotron-3b-v1
Training and Evaluation Datasets
Training Dataset
Data Modality:
Image
Text
Image Training Data Size: 1 Million to 1 Billion Images
Text Training Data Size: Less than a Billion Tokens
Data Collection Method by dataset: Hybrid: Automated, Human, Synthetic
Labeling Method by dataset: Hybrid: Automated, Human, Synthetic
Properties: 1M samples from public datasets.
Evaluation Dataset
We evaluate our model on multiple benchmarks covering different modalities. For text retrieval, we select some text retrieval datasets from MTEB. For image retrieval, evaluation is conducted on the public ViDoRe V1 dataset. Since no established video retrieval benchmarks exist, we construct two custom evaluation sets based on the LPM dataset and FineVideo. To provide fair comparison with the state-of-the-art text-only baselines, we use the speech-to-text transcripts released with FineVideo and the transcripts from LPM as the input corpus for standard text retrieval models.
Data Collection Method by dataset: Hybrid: Automated, Human, Synthetic
Labeling Method by dataset: Hybrid: Automated, Human, Synthetic
Model performance comparison on Video retrieval datasets (LPM and FineVideo) using NDCG@10 and NDCG@5 metrics:
Model
NDCG@10 LPM
NDCG@10 FineVideo
NDCG@10 Avg
NDCG@5 LPM
NDCG@5 FineVideo
NDCG@5 Avg
Qwen/Qwen3-Embedding-4B
0.8634
0.5405
0.7020
0.8518
0.5264
0.6891
intfloat/multilingual-e5-large-instruct
0.7952
0.4456
0.6204
0.7759
0.4300
0.6030
stella_en_1.5B_v5
0.8522
0.5359
0.6941
0.8404
0.5206
0.6805
nvidia/omni-embed-nemotron-3b
0.8465
0.5662
0.7064
0.8355
0.5486
0.6921
Multimodal retrieval performance across input modalities on LPM and FineVideo using NDCG@10. Baselines support text only; multimodal settings apply to Omni.
Evaluation of embedding models across text retrieval benchmarks. Results are reported using nDCG@10.
Model
Avg.
NQ
FiQA-2018
SciFact
SCIDOCS
ArguAna
NFCorpus
Quora
LegalBench-CorpLobby
CQAdupGaming
CQAdupUnix
Qwen/Qwen3-Embedding-4B
0.6654
0.6313
0.6122
0.7833
0.3144
0.7564
0.4110
0.8806
0.9542
0.7151
0.5960
intfloat/multilingual-e5-large-instruct
0.5900
0.6350
0.4865
0.7162
0.1924
0.5848
0.3634
0.8926
0.9425
0.6396
0.4473
stella_en_1.5B_v5
0.6050
0.7180
0.5996
0.8009
0.2677
0.5706
0.4200
0.9003
0.9468
0.5359
0.2903
nvidia/omni-embed-nemotron-3b
0.6059
0.6808
0.5382
0.7405
0.2163
0.5891
0.3644
0.8347
0.9413
0.6432
0.5102
Evaluation of baseline models and our models on ViDoRe V1 (as of September 30th). Results are presented using nDCG@5 metrics.
Model
Size (M)
Avg.
ArxivQA
DocVQA
InfoVQA
Shift Project
AI
Energy
Gov. Reports
Healthcare
TabFQuad
TAT-DQA
nvidia/llama-nemoretriever-colembed-1b-v1
2418
90.5
87.6
64.5
93.6
92.3
100
96.6
96.7
99.6
94.3
79.8
nvidia/llama-nemoretriever-colembed-3b-v1
4407
91.0
88.4
66.2
94.9
90.7
99.6
96.6
97.8
99.3
95.9
80.6
nomic-ai/colnomic-embed-multimodal-3b
3000
89.9
88.2
61.3
92.8
90.2
96.3
97.3
96.6
98.3
94.5
83.1
vidore/colqwen2.5-v0.2
3000
89.6
89.1
63.5
92.6
88.0
99.6
95.8
96.6
98.0
90.8
82.1
vidore/colqwen2-v1.0
2210
89.2
88.0
61.5
92.5
89.9
99.0
95.9
95.5
98.8
89.0
82.2
vidore/colpali-v1.3
2920
84.7
83.7
58.7
85.7
76.5
96.6
94.6
95.9
97.4
86.7
70.7
vidore/colpali-v1.2
2920
83.4
77.9
56.5
82.4
78.3
97.5
94.4
94.9
95.4
88.4
68.1
nvidia/omni-embed-nemotron-3b
4703
85.7
85.3
59.2
89.2
78.6
98.1
93.5
95.4
95.8
91.0
69.7
Inference:
Acceleration Engine: Not Applicable Test Hardware: A100 40GB, A100 80GB, H100 80GB
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.
Bias
Field
Response
Participation considerations from adversely impacted groups protected classes in model design and testing
None
Measures taken to mitigate against unwanted bias
None
Explainability
Field
Response
Intended Application & Domain:
Multi-modality corpus and query embedding for question and answer retrieval.
Model Type:
Transformer encoder.
Intended User:
Creators of generative AI focused on conversational models, as well as users aiming to develop question-and-answer applications, can benefit from leveraging the dense retrieval technologies. These applications can efficiently handle large, multi-modal corpora, including images, text, videos, and audio.
Output:
Array of float numbers (Dense vector for input content, which may include multi-modal corpora).
Describe how the model works:
Model transforms the input into a dense vector representation.
Performance Metrics:
Accuracy
Potential Known Risks:
This model does not guarantee to always retrieve the correct corpus for a given query.
Multi-modal Corpus Embedding for Retrieval. The model processes input from various modalities—text, image, audio, and video—either independently or in combination.
The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.
