This is a causal-LM-style reranker — not a cross-encoder. You score relevance by reading the logits of two specific tokens (yes=9405, no=2083) on the next position after a structured prompt. Details below.
Heads-up if you're converting this base model yourself
The Qwen3_5ForCausalLM arch supports MTP (multi-token prediction), and the base config carries mtp_num_hidden_layers: 1. But this reranker fine-tune dropped the MTP head — the safetensors file has 320 tensors, zero of them named mtp.*.
If you run convert_hf_to_gguf.py straight on the HF download, the converter sees mtp_num_hidden_layers: 1, sets block_count = num_hidden_layers + 1 = 25, then silently skips the MTP block because there's nothing to write. You end up with a GGUF that has 24 blocks but block_count metadata says 25. llama-server then errors on load with:
The fix: edit config.json to set mtp_num_hidden_layers: 0before converting. Then block_count = 24 and the load succeeds. That's what I did here.
What I tested
Single-query rerank on "Who painted the Mona Lisa?" with 4 candidate documents, scored at temperature=1.0, top_k=-1, top_p=1.0, min_p=0.0, n_probs=50, post_sampling_probs=true (samplers fully disabled so the raw softmax distribution survives in top_probs):
Score
Document
0.83
Leonardo da Vinci painted the Mona Lisa around 1503.
0.38
Mona Lisa is housed in the Louvre Museum in Paris.
0.28
Vincent van Gogh painted Starry Night in 1889.
0.14
The 2024 Super Bowl was won by the Kansas City Chiefs.
Top-vs-bottom gap of 0.69. The matching document wins clearly; related-but-wrong-angle ("Mona Lisa is in the Louvre" — about location, not artist) sits in the middle; off-topic doc loses.
Live retrieval against my actual memory bank (ov find on rei-opus memories with a "Willie wife identity Rei" query) returned top results at 0.89–0.90 — solid signal on real-world recall.
I did not run MTEB or any standard benchmark. This is "works for my RAG setup, here are the numbers I have." Your mileage on other domains may vary.
Hardware
Component
Spec
GPU
RTX 5080 Laptop (Blackwell, sm_120, 16 GB)
Driver
596.36
CPU
Intel Core Ultra 9 275HX (24 cores)
RAM
32 GB
OS
Windows 11 (build 26220)
How to use it (the scoring pattern)
Causal-LM rerankers don't work with llama.cpp's built-in /v1/rerank endpoint (that's for cross-encoders like BGE). You need to do the scoring yourself.
1. Format the prompt (Qwen3.5-Reranker template, matches what infgrad's sentence-transformers wrapper uses internally):
<|im_start|>system
Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>
<|im_start|>user
<Instruct>: {instruct}
<Query>: {query}
<Document>: {doc}<|im_end|>
<|im_start|>assistant
<think>
</think>
2. POST to llama-server /completion with sampling disabled so you get the raw distribution:
python
1import httpx
2body ={3"prompt": prompt,4"n_predict":1,5"temperature":1.0,6"top_k":-1,"top_p":1.0,"min_p":0.0,7"n_probs":50,8"post_sampling_probs":True,9}10r = httpx.post("http://127.0.0.1:8011/completion", json=body, timeout=60).json()11top = r["completion_probabilities"][0]["top_probs"]12p_yes =next((float(t["prob"])for t in top ifint(t["id"])==9405),0.0)13p_no =next((float(t["prob"])for t in top ifint(t["id"])==2083),0.0)14score = p_yes /(p_yes + p_no)if(p_yes + p_no)>0else0.5
If you want a drop-in OpenAI-compatible /v1/rerank endpoint to plug into RAG frameworks that expect one (Cohere/Voyage shape), you'll want to wrap the above in a small FastAPI shim. I run mine on port 8001 forwarding to llama-server on 8011.
Conversion details
Converter: convert_hf_to_gguf.py from llama.cpp (Esmaeel Nabil's fork, build dated 2026-05-22, which adds the Qwen3_5ForCausalLM registration with _Qwen35MtpMixin + _LinearAttentionVReorderBase mixins)
Quant time: ~5.7 seconds wall-clock on the hardware above
320 tensors → 24 blocks (linear-attention layers have 14 tensors each, full-attention layers have 11, hybrid pattern with full_attention_interval: 4)
Architecture footnote
This is a hybrid Mamba+Attention model — most layers are gated linear attention (Mamba-style SSM with ssm_a, ssm_alpha, ssm_beta, ssm_conv1d, ssm_norm, ssm_out tensors), with every 4th layer being full attention. Plus mrope (multi-axis RoPE). It loads and runs fine on Blackwell sm_120 with both Esmaeel-fork and upstream b9284 builds — I tested both.
If you're on older llama.cpp builds that don't have Qwen3_5ForConditionalGeneration / Qwen3_5ForCausalLM registered, the load will fail at architecture parsing. You need a build from approximately May 2026 onward.
