Property	Value
Method	Post-Training Quantization (PTQ)
Format	NVFP4 (W4A4)
Config	`mtq.NVFP4_DEFAULT_CFG` with `audio_tower` and `audio_projector` excluded
Calibration	cnn_dailymail 3.0.0, 128 samples, max_seq_len=512
Base Precision	BF16 (non-quantized layers remain BF16)
GPU Required	NVIDIA Blackwell (Compute Capability ≥ 10.0)
Tool	nvidia-modelopt 0.44.0

What is and isn't quantized

Component	Precision	Why
Language model (Qwen3)	NVFP4 W4A4	vLLM's `Qwen3ForCausalLM` is quantization-aware
Vision tower (Qwen3VL)	NVFP4 W4A4	vLLM's `Qwen3_VisionTransformer` is quantization-aware
Audio tower	BF16 (unquantized)	`transformers.Qwen2_5OmniAudioEncoder` uses plain `nn.Linear`
Audio projector	BF16 (unquantized)	Plain `nn.Linear` in `vllm_qwen3vl_audio.py`

The audio path was explicitly excluded from quantization because the model's vllm_qwen3vl_audio.py constructs its audio components with plain (non-quantization-aware) nn.Linear layers. Quantizing those layers produces NVFP4-packed safetensors that don't fit the BF16 parameter shapes at load time. Keeping audio in BF16 costs ~30 MB on a ~1.1 GB model (roughly 3% size overhead) and lets the model serve out of the box with vLLM's --quantization modelopt_fp4 flag.

Hardware Requirements

Quantized on: NVIDIA Thor (GB10, Blackwell CC 11.0)
Inference requires: Blackwell GPU (B100, B200, GB10, GB200, etc.)
Minimum CUDA: 13.0 / Driver 580.00+

Usage

vLLM (recommended)

bash

vllm serve georgelpreput/jina-embeddings-v5-omni-small-retrieval-NVFP4 \
  --trust-remote-code \
  --runner pooling \
  --quantization modelopt_fp4

Use Jina v5's asymmetric retrieval prefixes for queries and documents:

python

import requests

r = requests.post("http://localhost:8000/v1/embeddings", json={
    "model": "georgelpreput/jina-embeddings-v5-omni-small-retrieval-NVFP4",
    "input": [
        "Query: What is the capital of France?",
        "Document: Paris is the capital city of France.",
    ],
})

Required pre-step: stage `vllm_qwen3vl_audio.py`

The model ships a custom vllm_qwen3vl_audio.py that self-registers Qwen3VLAudioModel with vLLM. In NGC vLLM 26.04-py3 (vLLM 0.19.0+nv26.04) this registration silently fails — even when loading via the HF Hub repo ID — because HuggingFace's trust_remote_code loader copies only the file listed in config.json's auto_map (modeling_qwen3vl_audio.py) into the transformers_modules/<SHA>/ cache namespace. The companion vllm_qwen3vl_audio.py lives in the snapshot dir but never gets copied into the import namespace, so modeling_qwen3vl_audio.py's from .vllm_qwen3vl_audio import _register_vllm raises ModuleNotFoundError. vLLM then falls back to its generic transformers backend with the wrong pooling, producing degenerate embeddings (every text → near-identical vector).

The fix is a one-time pre-stage. Run this once (or wrap vllm serve in a script that runs it on every start):

bash

python3 - <<'PY'
import shutil
from pathlib import Path
from huggingface_hub import snapshot_download

snap = Path(snapshot_download(
    repo_id="georgelpreput/jina-embeddings-v5-omni-small-retrieval-NVFP4",
    allow_patterns=["*.py", "config.json"],
))
sha = snap.name
hf_home = Path.home() / ".cache" / "huggingface"
dst = hf_home / "modules" / "transformers_modules" / sha
dst.mkdir(parents=True, exist_ok=True)
(dst / "__init__.py").touch()
for src in snap.glob("*.py"):
    shutil.copy2(src, dst / src.name)
    print(f"staged {src.name}")
PY

This may be unnecessary on newer upstream vLLM versions (>= 0.20.1 per the Jina model card claims native support); we have not tested those builds.

Limitations

NVFP4 inference is only supported on Blackwell GPUs (not Ampere, Hopper, or Ada)
The CC BY-NC 4.0 license of the base model carries through — commercial use is not permitted
Audio and video modalities were not included in the calibration set (text-only calibration)
Audio tower and audio projector remain in BF16 (see "What is and isn't quantized" above)

License

CC BY-NC 4.0 (inherited from base model — non-commercial use only).

georgelpreput/jina-embeddings-v5-omni-small-retrieval-NVFP4

jina-embeddings-v5-omni-small-retrieval-NVFP4

Quantization Details