F16: Full 16-bit floating point - highest quality, largest size
Q8_0: 8-bit quantization - high quality, good balance
Q4_K_M: 4-bit quantization with medium quality - smaller size, faster inference

Usage

Base Model: <a href="https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">cross-encoder/ms-marco-MiniLM-L-6-v2
Quantization: Q4_K_M
Format: GGUF (GPT-Generated Unified Format)
Converted with: llama.cpp

This model can be used with llama.cpp and other GGUF-compatible inference engines.

bash

# Example using llama.cpp
./llama-rerank -m ms-marco-MiniLM-L6-v2-Q4_K_M.gguf

Model Files

Quantization	Use Case
F16	Maximum quality, largest size
Q8_0	High quality, good balance of size/performance
Q4_K_M	Good quality, smallest size, fastest inference

If you use this model, please cite the original model:

# See original model card for citation information

This model inherits the license from the original model. Please refer to the original model card for license details.