The purpose of this repository is to provide fast, easy-to-use local inference files for llama.cpp, Ollama, LM Studio, Jan, Open WebUI, and llama-cpp-python users.

Model Details

Base model: nvidia/LocateAnything-3B
Architecture: Transformer
Format: GGUF
Source license: other
Conversion tool: convert_hf_to_gguf.py from llama.cpp
Quantization tool: llama-quantize
Recommended file: LocateAnything-3B-Q4_K_M.gguf

Quantized Files

Quant	Filename	Size	SHA256	Notes
FP16	`LocateAnything-3B-FP16.gguf`	~6.34 GiB	`a4c0b3ebe23f...`	Full precision converted GGUF baseline
Q2_K	`LocateAnything-3B-Q2_K.gguf`	~1.28 GiB	`6569e7b98cd3...`	Smallest, lowest quality
Q3_K_M	`LocateAnything-3B-Q3_K_M.gguf`	~1.61 GiB	`bdfbe9fe0b80...`	Small balanced version
Q4_0	`LocateAnything-3B-Q4_0.gguf`	~1.86 GiB	`d42738b1cb53...`	Simple 4-bit quantization
Q4_K_M	`LocateAnything-3B-Q4_K_M.gguf`	~1.96 GiB	`d06f4263b54f...`	Recommended default for most users
Q5_K_M	`LocateAnything-3B-Q5_K_M.gguf`	~2.27 GiB	`7177002d8228...`	Better quality with moderate size
Q6_K	`LocateAnything-3B-Q6_K.gguf`	~2.60 GiB	`bd0775eb3c08...`	High quality
Q8_0	`LocateAnything-3B-Q8_0.gguf`	~3.37 GiB	`11d4ab968014...`	Near FP16 quality

Validation

Each file was tested with llama-cli for basic load + generation.

Quant	Filename	Status
FP16	`LocateAnything-3B-FP16.gguf`	✅ passed
Q2_K	`LocateAnything-3B-Q2_K.gguf`	✅ passed
Q3_K_M	`LocateAnything-3B-Q3_K_M.gguf`	✅ passed
Q4_0	`LocateAnything-3B-Q4_0.gguf`	✅ passed
Q4_K_M	`LocateAnything-3B-Q4_K_M.gguf`	✅ passed
Q5_K_M	`LocateAnything-3B-Q5_K_M.gguf`	✅ passed
Q6_K	`LocateAnything-3B-Q6_K.gguf`	✅ passed
Q8_0	`LocateAnything-3B-Q8_0.gguf`	✅ passed

Usage

llama.cpp

llama-cli -m LocateAnything-3B-Q4_K_M.gguf -p "Hello! Introduce yourself briefly."

Older builds may use:

./main -m LocateAnything-3B-Q4_K_M.gguf -p "Hello! Introduce yourself briefly."

llama.cpp directly from Hugging Face

llama-cli -hf ShahzebKhoso/LocateAnything-3B-GGUF:Q4_K_M -p "Hello! Introduce yourself briefly."

llama-cpp-python

python

from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="ShahzebKhoso/LocateAnything-3B-GGUF",
    filename="LocateAnything-3B-Q4_K_M.gguf",
)

llm = Llama(model_path=model_path)

out = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello! Introduce yourself briefly."},
    ],
    max_tokens=128,
)

print(out["choices"][0]["message"]["content"])

Which file should I use?

Use Q4_K_M for the best default balance.
Use Q5_K_M for better quality.
Use Q8_0 if you want near-original quality and have more memory.
Use Q2_K or Q3_K_M only when memory is very limited.

Provenance

This repository is a quantized derivative of:

nvidia/LocateAnything-3B

Base model metadata:

text

revision: 7a81d810571dc5f244b2f0b6868128f24b1cbd85
pipeline_tag: image-text-to-text
tags: transformers, safetensors, locateanything, feature-extraction, nvidia, eagle, vision, object-detection, grounding, arxiv:2605.27365, image-text-to-text, conversational, custom_code, en, arxiv:2504.07491, arxiv:2109.10852, arxiv:2510.12798, arxiv:2303.05499, arxiv:1405.0312, arxiv:1908.03195