You can deploy this model and start earning money today!

memoryco-ai/
llama-nemotron-rerank-1b-v2-GGUF

Name: llama-nemotron-rerank-1b-v2-GGUF
Availability: InStock

gguf

endpoints_compatible

feature-extraction

Views

No views yet

llama-nemotron-rerank-1b-v2 GGUF

GGUF quantizations of nvidia/llama-nemotron-rerank-1b-v2 for use with llama.cpp.

Model Details

Base model: nvidia/llama-nemotron-rerank-1b-v2
Architecture: Llama 3.2 1B, fine-tuned cross-encoder with bidirectional attention
Task: Reranking / sequence classification
Languages: 26 (English, Arabic, Bengali, Chinese, Czech, Danish, Dutch, Finnish, French, German, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish)
License: NVIDIA Open Model License + Llama 3.2 Community License

Requirements

These GGUFs require a patched llama.cpp that supports reranking via the llama-embed architecture. Specifically:

LLM_TENSOR_CLS and LLM_TENSOR_CLS_OUT in the LLM_ARCH_LLAMA_EMBED tensor allowlist
causal_attn and pooling_type hparam reads in the LLM_ARCH_LLAMA_EMBED case

The memoryco/llama.cpp fork (branch pinned-for-rs) includes both patches.

GGUF Fixes Applied

The initial convert_hf_to_gguf.py conversion produced a GGUF that wouldn't load correctly for reranking. These post-conversion fixes were applied:

Tensor rename: cls.weight → cls.output.weight (maps to LLM_TENSOR_CLS_OUT prefix "cls.output")
Tensor reshape: [2048] (1D) → {2048, 1} (2D classification head)
Added metadata: llama-embed.classifier.output_labels: ["relevance"] (sets n_cls_out=1)
Added metadata: tokenizer.ggml.add_eos_token: true (required for non-zero rerank scores)

Files

File	Quant	Size
llama-nemotron-rerank-1b-v2.Q8_0.gguf	Q8_0	~1.25 GB
llama-nemotron-rerank-1b-v2.Q4_K_M.gguf	Q4_K_M	~770 MB

Usage with llama.cpp

bash

# Using the patched memoryco/llama.cpp fork
./build/bin/llama-embedding \
  -m llama-nemotron-rerank-1b-v2.Q8_0.gguf \
  --pooling rank \
  -p 'passage: Paris is the capital of France. query: what is the capital of france'

Input format: passage: {document_text}. query: {query_text} as a single string. The model outputs a single float score (higher = more relevant).

Part of the MemoryCo Nemotron Stack

Generation: memoryco-ai/Nemotron-3-Nano-4B-GGUF
Embeddings: memoryco-ai/llama-nemotron-embed-1b-v2-GGUF
Reranking: This repo

Converted by memoryco-ai using llama.cpp's convert_hf_to_gguf.py with custom architecture support and post-conversion GGUF fixes.

memoryco-ai/llama-nemotron-rerank-1b-v2-GGUF