All Linear layers (attention + MLP) are quantized to 8-bit affine with group_size=64. Embeddings and the projector stay in bf16 for accuracy. 0.78 GB vs 1.19 GB (~35% smaller) with no measurable quality loss for reranking.

Benchmark (NDCG@10)

Measured against the bf16 original on MTEB reranking tasks (identical candidates across models):

Dataset	bf16	q8	q4
NFCorpus (323 queries)	0.9648	0.9650	0.9654
SciFact + hard negatives (300)	0.9817	0.9820	0.9867
ArguAna + hard negatives (100)	0.9963	0.9963	0.9963

q8 matches bf16 within noise (delta ≤ 0.0003). See BENCHMARK.md for methodology. A 4-bit variant is also available.

Differences from the original MLX repo

	Original	This repo
Backbone Linear layers	bf16	8-bit affine (group_size=64)
Embeddings	bf16	bf16 (unchanged)
Projector	bf16	bf16 (unchanged)
Model size	1.19 GB	0.78 GB

Loading

The quantized weights use the standard MLX quantization format. Loaders that support MLX safetensors (mlx-lm, oMLX) will dequantize on the fly:

python

from mlx_lm import load

model, tokenizer = load("underlotus/jina-reranker-v3.5-mlx-q8")

Note: the standalone rerank.py / MLXReranker helper in the original repo does not support quantized weights (its modeling.py has no dequantization logic). Use mlx-lm or oMLX to load this model.

Usage with oMLX

Download this repo into your oMLX model directory.
Keep a directory name containing reranker (e.g. jina-reranker-v3.5-mlx-q8) so oMLX routes it to its reranker engine.
Load it from the oMLX app. The Jina v3.5 sliding-window patch is applied automatically.

Quantization details

Quantization was performed on Apple Silicon with a small script:

python

import mlx.nn as nn
from mlx.utils import tree_flatten

# only Linear layers, 8-bit affine
nn.quantize(
    model,
    group_size=64,
    bits=8,
    class_predicate=lambda p, m: isinstance(m, nn.Linear),
)

Citation

If you find jina-reranker-v3.5 useful in your research, please cite the original paper:

bibtex

@misc{nasika2026jinarerankerv35efficientlistwisereranker,
      title={jina-reranker-v3.5: An Efficient Listwise Reranker with Hybrid Attention and Self-Distillation},
      author={Christina Nasika and Feng Wang and Antonis Krasakis and Han Xiao},
      year={2026},
      eprint={2607.18152},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2607.18152},
}

License

CC BY-NC 4.0. For commercial usage inquiries, please contact Jina AI.