V-SPLADE is an inference-free sparse retriever for visual document retrieval: document pages (rendered PDFs, slides, scans) are encoded by a ModernVBERT backbone (SigLIP vision tower + pixel-shuffle connector + ModernBERT text encoder) with a SPLADE MLM head into a 50,368-dim vocabulary-space sparse vector, while queries are resolved by a learned Bag-of-Words lookup with no neural encoding at all.

Contents: weights.safetensors (document encoder, bfloat16), query_lookup.npy (inference-free query table, fp32), config.json, plus tokenizer/processor configs for self-contained loading.

Changes from upstream: PyTorch checkpoint converted to MLX safetensors (parameter re-mapping, conv weight transposed to NHWC, cast to bfloat16); the query lookup table softplus(embedding @ projection + bias) is precomputed with special tokens zeroed. No training or fine-tuning was performed.

Quality (see repo REPORT.md for methodology):

Separate fp32 conversion parity vs the PyTorch reference: max |logit delta| 6.4e-04 on real document-page inputs, sparse-vector cosine 1.000000, top-64 term overlap 100%; the query table matches the shipped Sentence Transformers static embedding to 1.2e-07.
ViDoRe docvqa_test_subsampled nDCG@5 (fp32): 0.4331 (torch) -> 0.4331 (MLX), delta +0.0000 (gate: ±0.002).

This repository itself stores bfloat16 document-encoder weights. The fp32 numbers above describe a separate fp32 conversion and must not be attributed to this linked bfloat16 artifact.

Usage

python

from splade_mlx.convert_vsplade import load_vsplade
import mlx.core as mx
from PIL import Image

model, query_encoder, processor = load_vsplade("NomaDamas/v-splade-quality-mlx")

# documents (page images)
enc = processor(text=["User:<image><end_of_utterance>\nAssistant:"],
                images=[[Image.open("page.png")]], return_tensors="np")
d = model.encode(mx.array(enc["input_ids"]), mx.array(enc["attention_mask"]),
                 enc["pixel_values"])   # (1, 50368)

# queries: inference-free lookup, no neural network
q = processor.tokenizer(["total revenue 2023"], return_tensors="np")
qw = query_encoder.encode(q["input_ids"], q["attention_mask"])  # (1, 50368)

score = d @ qw.T

NomaDamas/v-splade-quality-mlx

v-splade-quality-mlx

Usage

License

NomaDamas/
v-splade-quality-mlx