mmarco-mMiniLMv2-L12-H384-v1, in GGUF
cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 in one f16 GGUF file, so
llama-server can serve it as a /v1/rerank endpoint. Nothing about the
model was changed: these are the published weights in a different container.
It exists because Semantic Deterministic Graph runs every model it uses on
llama.cpp, and the published cross-encoder ships safetensors only. If you
are using PyTorch, use the original: this file buys you nothing there.
The file
| |
|---|
| name | mmarco-mminilmv2-l12-f16.gguf |
| sha256 | 1116037dfab848873f20248bb62df2090aec2e7fca03c99c27a5a0c8ce0f23b8 |
| size | 242,628,704 bytes |
| precision | f16 |
| converted with | convert_hf_to_gguf.py, llama.cpp |
Why f16 rather than a quantization
A reranker used with a relevance floor is read for its raw score, not only
for the order it produces: the floor is what turns "nothing here is about
that" into an answer, and it is a number on this model's own scale. A
quantization that preserves ranking can still move a score across a floor.
f16 keeps the file small enough not to matter and the question closed.
Fidelity against the PyTorch model
Ten query/passage pairs scored both ways, GGUF against
sentence_transformers.CrossEncoder on the original weights:
- ranking identical, Spearman 1.0000
- largest score difference 0.0124, on a range of about twenty
Serving it
1llama-server -m mmarco-mminilmv2-l12-f16.gguf \
2 --reranking --pooling rank -c 4096 -ngl 99
--reranking with rank pooling is what turns a sequence-classification
model into the /v1/rerank endpoint.
Two things will bite you if they are not said:
The model reads 512 tokens. A passage longer than that must be cut
before it is sent, or the request comes back as a 500 rather than a score.
Cut by tokens, through the server's own /tokenize, not by characters: the
PyTorch model truncates silently at the same width, and cutting the same
way is what makes the two comparable.
The context is shared across the server's slots. A question and one
passage of a few hundred words take roughly three hundred tokens; -c 4096
leaves four slots a thousand each. Sized for one slot instead, a long
passage is a 500.
What it was measured on
Not a public benchmark. These are 44 graded Italian questions over 182
notes, plus 10 unanswerable ones, from the project this conversion was made
for. Same candidates, same session, the reranker as the only variable.
| reranker | r@3 | r@5 | r@10 | silent on 10 unanswerable | median latency |
|---|
| none (vector + lexical) | 33/44 | 33/44 | 37/44 | 5/10 | 19 ms |
| Qwen3-Reranker-0.6B | 38/44 | 41/44 | 42/44 | 6/10 | 18,340 ms |
| this model (PyTorch) | 42/44 | 42/44 | 43/44 | 7/10 | 364 ms |
Read it for what it is: one corpus, one language, one machine, questions
written by the people who ran the test. It says this model beat the
alternative here on every column, and it does not say what it will do on
your corpus. Retrieval thresholds have never transferred between corpora in
this project's experience, and neither does a recall number.
The base model's own card publishes no benchmark, which is why there is
one here at all.
Attribution
Apache 2.0, unchanged from the model this file was converted from.
