GGUF conversions of NVIDIA's LocateAnything-3B (MoonViT-SO-400M vision tower + Eagle MLP + Qwen2.5-3B) for llama.cpp and its multimodal (mtmd) stack. The model is a vision-language grounding model: given an image and a natural-language query, it emits a label plus normalized (0–1000) bounding-box or point coordinates.
License: NVIDIA License — non-commercial / research use only (commercial use is reserved to NVIDIA and its affiliates). A complete copy of the upstream LICENSE is bundled in this repo and must be retained on redistribution, along with attribution to NVIDIA. See the original model card below.
About GGUF
GGUF is the model file format used by llama.cpp. Each conversion has two halves and both are required:
the LLM half (LocateAnything-3B-{quant}.gguf), and
the mmproj vision projector (mmproj-LocateAnything-3B-BF16.gguf).
You pair one LLM quant with the mmproj.
Provided files
File
Quant
Size
Accuracy vs BF16
Notes
LocateAnything-3B-BF16.gguf
BF16
6.4 GB
baseline
reference
LocateAnything-3B-Q8_0.gguf
Q8_0
3.6 GB
≤0.5 norm-units
minimal quant noise
LocateAnything-3B-Q6_K.gguf
Q6_K
2.8 GB
sub-pixel
near-BF16 quality
LocateAnything-3B-Q5_K_M.gguf
Q5_K_M
2.4 GB
sub-pixel
safety margin
LocateAnything-3B-Q4_K_M.gguf
Q4_K_M
2.1 GB
sub-pixel
recommended default
mmproj-LocateAnything-3B-BF16.gguf
BF16
0.87 GB
—
vision projector (required)
Accuracy figures are the mean absolute coordinate delta vs BF16 across a 5-landmark macOS-screenshot grounding suite (0–1000 normalized space), greedy decoding. Q8_0 ≤ 0.5; Q6_K / Q5_K_M / Q4_K_M all sub-pixel.
Low-bit IQ2_M / IQ3_M are intentionally not shipped: a good calibration (imatrix) for a vision-grounding model needs image input, which the current llama.cpp imatrix tool doesn't support, and uncalibrated IQ2 regresses noticeably.
How to run
Requires the fork build. The LocateAnything mtmd integration is not yet in upstream llama.cpp. Build from github.com/yuuko-eth/llama.cpp @ mtmd-grounders — stock llama.cpp will not load these GGUFs.
Important — llama-server needs --special. Grounding output is emitted as control tokens (<ref>, </ref>, <box>, and coordinate tokens <0>…<999>). llama-server drops control tokens from responses unless launched with --special; without it you only get the plain-text label (e.g. "Apple logo") with no coordinates. llama-mtmd-cli shows them by default.
Then call /v1/chat/completions (OpenAI-style, image as a data: URL) or /completion. Output matches the CLI.
Tip: cost is dominated by image encoding, which scales with resolution. Capping the longer image side to ~1280 px keeps grounding quality (coordinates are normalized) while cutting a multi-second encode to well under a second.
Prompt templates (from upstream)
Task
Prompt
Output
Box grounding
Locate the {phrase}.
<box><x1><y1><x2><y2></box>
Region / GUI box
Locate the region that matches the following description: {phrase}.
bbox
Point / GUI point
Point to: {phrase}.
<box><x><y></box>
Multi-instance
Locate all the instances that match the following description: {phrase}.
many bboxes
Text grounding
Please locate the text referred as {phrase}.
bbox
Parsing the output
python
1import re
23PAT = re.compile(r"<ref>(.*?)</ref><box>((?:<\d+>)+)</box>")45defparse(text):6 results =[]7for label, coords in PAT.findall(text):8 nums =[int(n)for n in re.findall(r"<(\d+)>", coords)]# normalized 0-10009 results.append({"label": label,"coords": nums})# 4 = bbox, 2 = point10return results
Known issues
llama-server requires --special (see above) — the single most common gotcha.
Vision head_dim=72: some CUDA flash-attention backends assert on this head dim. llama.cpp auto-falls back, but if your backend asserts, add --no-flash-attn.
Tested single-image / single-turn. Multi-image batching in llama-server is not validated.
LocateAnything is a vision-language model for fast and high-quality visual grounding, enabling precise object localization, dense detection, and point-based localization across diverse domains in both Enterprise Intelligence and Physical AI. The model adopts a generalist design, supporting tasks such as referring expression grounding, multi-object detection, GUI element grounding, and text localization, with strong performance in complex and cluttered scenes.
Its core innovation, Parallel Box Decoding (PBD), predicts complete bounding box coordinates in a single parallel step rather than autoregressive token-by-token decoding, improving efficiency while preserving geometric consistency. This enables up to 2.5× higher throughput compared to prior approaches.
The model is trained on a large-scale multi-domain dataset (12M images, 138M+ queries, 785M bounding boxes) spanning natural scenes, robotics, driving, GUI interaction, and document understanding. It serves as a foundation for generalist multimodal perception and has been integrated into NVIDIA’s frontier production-grade vision-language models, such as Nemotron 3 Nano Omni, supporting grounding, GUI understanding, and multimodal agentic capabilities.
LocateAnything is developed as part of the Eagle VLM model family. This model is for research and development only.
Demo Videos
License/Terms of Use:
This model is released under the NVIDIA License for non-commercial use, which permits use, reproduction, and modification for academic and non-profit research purposes only. Commercial use is not permitted, except by NVIDIA and its affiliates. Redistribution must retain the license and all applicable copyright and attribution notices. The model is provided “as is” without warranty of any kind, and users assume all associated risks.
This model is built using components from third-party models with their respective licenses:
LocateAnything-3B is intended for developers and researchers building vision-language models and applications that require fast and precise visual localization from natural language instructions.
Supported use cases include:
Open-set, common, and long-tail object detection
Dense multi-object detection in cluttered scenes
Phrase and referring-expression grounding
Automated dataset labeling and annotation (e.g., detection, grounding, pointing)
GUI element grounding for interactive and agentic systems
Robotics and autonomous driving perception
Document understanding, layout grounding, and OCR localization
Industrial inspection, surveillance, and remote sensing applications
Point-based localization and fine-grained spatial reasoning
Release Date [Insert the expected release date below]:
Architecture Type: Transformer-based vision-language model (VLM).
Network Architecture: Native-resolution VLM with the following components:
Vision encoder: MoonViT
Language model: Qwen2.5-3B-Instruct
Multimodal projector: MLP projector
Output formulation: Block-based structure for visual grounding
Number of model parameters: 3B.
LocateAnything extends a vision-language model with Parallel Box Decoding (PBD), a block-wise multi-token prediction framework for efficient visual grounding. Instead of autoregressive coordinate generation, the model predicts complete bounding boxes and points in parallel structured units, improving decoding efficiency while preserving geometric consistency. The architecture jointly optimizes next-token prediction and multi-token prediction to balance reasoning ability and parallel inference. Training follows a four-stage pipeline: initial multimodal knowledge adaptation using captioning, VQA, OCR, and related data, followed by grounding and dense-scene localization fine-tuning.
Input(s):
Input Type(s): Image and Text.
Input Format(s):
Image: RGB image input with original source resolution.
Text: Natural-language prompt or task template, such as object categories, referring expressions, GUI instructions, OCR/layout requests, or pointing queries.
Input Parameters:
Image: Two-Dimensional (2D)
Text: One-Dimensional (1D)
Other Properties Related to Input:
Production image resolution supports up to 2.5K.
Prompt length supports up to 24K tokens.
Training detection and grounding stages use a maximum sequence length of 25,600 tokens.
Inference supports up to 8,192 newly generated tokens.
Output(s):
Output Type(s): Text.
Output Format(s):
Text: Model-generated token sequence containing semantic labels and structured coordinate tokens, such as bounding boxes (<box> x1, y1, x2, y2 </box>) and points (<box> x, y </box>).
Outputs are organized into fixed-length blocks (length 6), including Semantic, Box, Negative, and End blocks.
A Box block encodes quantized spatial coordinates with structural tokens; unused positions are padded with <null>.
Fast Mode predicts box-aligned blocks in parallel; Slow Mode uses autoregressive decoding; Hybrid Mode defaults to parallel decoding with fallback to autoregressive decoding for format irregularity or spatial ambiguity.
Our AI models are designed and optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves improved training and inference performance compared to CPU-only solutions.
Software Integration:
Runtime Engine(s):
Transformers. The inference setup uses standard VLM generation with BF16 precision and KV cache. TensorRT, TensorRT-LLM, and Triton are not yet supported.
Deployment on embedded platforms such as NVIDIA Thor is possible with additional model optimization, including quantization, compression, or distillation. Other architectures may be supported depending on available memory, precision support, and software configuration.
Supported Operating System(s):
Linux
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Model Version(s):
LocateAnything-3B: 3B-parameter research model variant evaluated in Hybrid Mode by default. Fast, Hybrid, and Slow inference modes are supported by the same model formulation.
LocateAnything-3B can be integrated into systems that require spatial grounding from natural language, such as GUI agents, robotics/embodied agents, document-understanding pipelines, OCR/text localization, and open-world detection workflows.
Training, Testing, and Evaluation Datasets:
Data Modality:
Image and Text.
Image
Text
Training Data Size:
Image Training Data Size:
1 Million to 1 Billion Images - 12M unique images.
Text Training Data Size:
1 Billion to 10 Trillion Tokens - Derived from approximately 140M natural-language queries.
Data Collection Method by dataset:
Hybrid: Human, Automated
Data is collected from human-curated and open-source datasets, as well as automated ingestion of publicly available data sources.
Labeling Method by dataset:
Hybrid: Human, Synthetic, Automated
Labeling includes original human or open-source annotations, along with model-assisted and synthetic annotation generation using Qwen3-VL, Molmo, SAM 3, and Rex-Omni, with automated post-verification.
Properties: The training data consists of supervised fine-tuning (SFT) datasets with multimodal inputs, primarily image-text pairs and structured annotations such as bounding boxes, points, and negative samples.
The data spans multiple domains, including grounding, open-world grounding, general and dense object detection, scene text detection, GUI understanding and grounding, document layout understanding, and OCR.
Modalities include visual inputs (images) and natural-language queries or instructions. The dataset is derived from a mixture of publicly available academic datasets, along with model-assisted and synthetic annotations. It may include publicly available and potentially copyrighted content; users are responsible for ensuring compliance with applicable usage rights.
The linguistic content primarily consists of short, task-oriented natural-language expressions, such as object categories, referring expressions, GUI instructions, OCR queries, and grounding prompts, typically in English.
Evaluation Dataset:
Data Collection Method by dataset:
Hybrid: Human, Automated
Labeling Method by dataset:
Hybrid: Human, Synthetic, Automated
Properties: The evaluation datasets consist of publicly available benchmarks spanning visual grounding, object detection, document understanding, scene text detection, and GUI-related tasks. Modalities include image inputs paired with natural-language queries and structured annotations such as bounding boxes and points.
The evaluation suite covers both box-level and point-level grounding tasks, with approximately 48K images for box evaluation and 35K images for point evaluation across multiple datasets. These datasets span diverse domains including natural scenes, documents, aerial imagery, and human-centric interactions, enabling comprehensive assessment of localization accuracy and robustness.
Evaluation queries are typically short, task-oriented natural-language expressions such as referring phrases, object categories, and grounding prompts.
Performance is measured using box-based F1 at IoU thresholds of 0.5 and 0.95, as well as mean IoU for detection, layout, and OCR tasks. Point-based localization is evaluated based on whether predicted points fall within ground-truth segmentation masks or bounding boxes. Inference efficiency is reported in boxes per second (BPS) on a single NVIDIA H100 GPU with batch size 1.
Quantitative Evaluation Benchmarks
General Object Detection
Dense Object Detection
GUI Understanding
Layout Grounding and OCR
Referring Expression Grounding
Pointing
Inference:
Test Hardware: H100
We suggest using max_new_tokens=8192 and generation_mode="hybrid" to avoid truncated response and balance speed with accuracy.
If MagiAttention is installed, the model will automatically use it for efficient MTP block-diffusion attention. If not installed, it will fall back to PyTorch SDPA — fully functional but slower for MTP decoding.
Worker (recommended)
Below is a self-contained worker that loads the model once and serves perception queries via a unified predict() plus task-specific convenience methods. You can drop this class into any FastAPI / gRPC / Triton serving framework.
python
1import re
2import torch
3from PIL import Image
4from transformers import AutoModel, AutoTokenizer, AutoProcessor
567classLocateAnythingWorker:8"""Stateful worker that loads the model once and serves perception queries."""910def__init__(self, model_path:str, device:str="cuda", dtype=torch.bfloat16):11 self.device = device
12 self.dtype = dtype
1314 self.tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)15 self.processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)16 self.model = AutoModel.from_pretrained(17 model_path,18 torch_dtype=dtype,19 trust_remote_code=True,20).to(device).eval()2122@torch.no_grad()23defpredict(24 self,25 image: Image.Image,26 question:str,27 generation_mode:str="hybrid",# "fast" (MTP) | "slow" (NTP/AR) | "hybrid"28 max_new_tokens:int=2048,29 temperature:float=0.7,30 verbose:bool=True,31)->dict:32 messages =[33{"role":"user","content":[34{"type":"image","image": image},35{"type":"text","text": question},36]}37]3839 text = self.processor.py_apply_chat_template(40 messages, tokenize=False, add_generation_prompt=True41)42 images, videos = self.processor.process_vision_info(messages)43 inputs = self.processor(44 text=[text], images=images, videos=videos, return_tensors="pt"45).to(self.device)4647 pixel_values = inputs["pixel_values"].to(self.dtype)48 input_ids = inputs["input_ids"]49 image_grid_hws = inputs.get("image_grid_hws",None)5051 response = self.model.generate(52 pixel_values=pixel_values,53 input_ids=input_ids,54 attention_mask=inputs["attention_mask"],55 image_grid_hws=image_grid_hws,56 tokenizer=self.tokenizer,57 max_new_tokens=max_new_tokens,58 use_cache=True,59 generation_mode=generation_mode,60 temperature=temperature,61 do_sample=True,62 top_p=0.9,63 repetition_penalty=1.1,64 verbose=verbose,65)6667 result ={"answer": response[0]ifisinstance(response,tuple)else response}68ifisinstance(response,tuple)andlen(response)>=3:69 result["history"]= response[1]70 result["stats"]= response[2]71return result
7273# ---- Convenience methods for each task ----7475defdetect(self, image: Image.Image, categories:list[str],**kwargs)->dict:76"""Object detection / document layout analysis."""77 cats ="</c>".join(categories)78 prompt =f"Locate all the instances that matches the following description: {cats}."79return self.predict(image, prompt,**kwargs)8081defground_single(self, image: Image.Image, phrase:str,**kwargs)->dict:82"""Phrase grounding — single instance."""83 prompt =f"Locate a single instance that matches the following description: {phrase}."84return self.predict(image, prompt,**kwargs)8586defground_multi(self, image: Image.Image, phrase:str,**kwargs)->dict:87"""Phrase grounding — multiple instances."""88 prompt =f"Locate all the instances that match the following description: {phrase}."89return self.predict(image, prompt,**kwargs)9091defground_text(self, image: Image.Image, phrase:str,**kwargs)->dict:92"""Text grounding."""93 prompt =f"Please locate the text referred as {phrase}."94return self.predict(image, prompt,**kwargs)9596defdetect_text(self, image: Image.Image,**kwargs)->dict:97"""Scene text detection."""98 prompt ="Detect all the text in box format."99return self.predict(image, prompt,**kwargs)100101defground_gui(self, image: Image.Image, phrase:str, output_type:str="box",**kwargs)->dict:102"""GUI grounding (box or point)."""103if output_type =="point":104 prompt =f"Point to: {phrase}."105else:106 prompt =f"Locate the region that matches the following description: {phrase}."107return self.predict(image, prompt,**kwargs)108109defpoint(self, image: Image.Image, phrase:str,**kwargs)->dict:110"""Pointing."""111 prompt =f"Point to: {phrase}."112return self.predict(image, prompt,**kwargs)113114# ---- Utility: parse model output ----115116@staticmethod117defparse_boxes(answer:str, image_width:int, image_height:int)->list[dict]:118"""Parse model output into pixel-coordinate bounding boxes.
119120 Coordinates in model output are normalized integers in [0, 1000].
121 """122 boxes =[]123for m in re.finditer(r"<box><(\d+)><(\d+)><(\d+)><(\d+)></box>", answer):124 x1, y1, x2, y2 =[int(g)for g in m.groups()]125 boxes.append({126"x1": x1 /1000* image_width,127"y1": y1 /1000* image_height,128"x2": x2 /1000* image_width,129"y2": y2 /1000* image_height,130})131return boxes
132133@staticmethod134defparse_points(answer:str, image_width:int, image_height:int)->list[dict]:135"""Parse model output into pixel-coordinate points."""136 points =[]137for m in re.finditer(r"<box><(\d+)><(\d+)></box>", answer):138 x, y =int(m.group(1)),int(m.group(2))139 points.append({140"x": x /1000* image_width,141"y": y /1000* image_height,142})143return points
[PHRASE] is a free-form natural-language description; [CATEGORIES] is a comma-separated list (multiple categories may also be joined with </c>).
Generation Modes
Mode
Description
Speed
Accuracy
fast
MTP only, never falls back to AR
Fastest
Good for simple scenes
slow
Pure auto-regressive decoding
Slowest
Most robust
hybrid (default)
MTP first, falls back to AR on uncertain boxes, switches back after box boundary
Balanced
Best overall
Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.
