# Build llama.cpp (https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)
cmake -B build && cmake --build build --config Release -j

# Start the reranking server
./build/bin/llama-server \
    -m llama-nemotron-rerank-1b-v2-q8_0.gguf \
    --reranking \
    --port 8080 \
    -ngl 99

Query the /rerank endpoint

python

import requests

response = requests.post("http://localhost:8080/rerank", json={
    "query": "What is machine learning?",
    "documents": [
        "Machine learning is a branch of AI that learns patterns from data.",
        "Python is a programming language commonly used for data science.",
        "Bananas are a good source of potassium.",
    ],
    "top_n": 3,
})

for item in response.json()["results"]:
    print(f"Doc {item['index']}: score={item['relevance_score']:.4f}")

Accuracy

Scores from the Q8_0 quantization match the HuggingFace reference (BF16) within 0.25 across all tested documents, with identical ranking order. The F16 quantization matches within 0.17.

Model details

See the original model card at nvidia/llama-nemotron-rerank-1b-v2 for full details including:

Architecture: LlamaBidirectionalForSequenceClassification (LLaMA 3.2 1B backbone, bidirectional attention, mean pooling, linear score head)
Languages: 26 languages (English, Arabic, Bengali, Chinese, Czech, Danish, Dutch, Finnish, French, German, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish)
Max sequence length: 8192 tokens
License: NVIDIA Open Model License + Llama 3.2 Community License

kread/llama-nemotron-rerank-1b-v2-GGUF

llama-nemotron-rerank-1b-v2 — GGUF

Available files

Usage with llama.cpp server

Query the /rerank endpoint

Accuracy

Model details

kread/
llama-nemotron-rerank-1b-v2-GGUF