I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.
In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the --tensor-type option in llama.cpp to manually "bump" important layers to higher precision. You can see the implementation here:
👉 Layer bumping with llama.cpp
While this does increase model file size, it significantly improves precision for a given quantization level.
I'd love your feedback—have you tried this? How does it perform for you?
LocateAnything is a vision-language model for fast and high-quality visual grounding, enabling precise object localization, dense detection, and point-based localization across diverse domains in both Enterprise Intelligence and Physical AI. The model adopts a generalist design, supporting tasks such as referring expression grounding, multi-object detection, GUI element grounding, and text localization, with strong performance in complex and cluttered scenes.
Its core innovation, Parallel Box Decoding (PBD), predicts complete bounding box coordinates in a single parallel step rather than autoregressive token-by-token decoding, improving efficiency while preserving geometric consistency. This enables up to 2.5× higher throughput compared to prior approaches.
The model is trained on a large-scale multi-domain dataset (12M images, 138M+ queries, 785M bounding boxes) spanning natural scenes, robotics, driving, GUI interaction, and document understanding. It serves as a foundation for generalist multimodal perception and has been integrated into NVIDIA’s frontier production-grade vision-language models, such as Nemotron 3 Nano Omni, supporting grounding, GUI understanding, and multimodal agentic capabilities.
LocateAnything is developed as part of the Eagle VLM model family. This model is for research and development only.
Demo Videos
License/Terms of Use:
This model is released under the NVIDIA License for non-commercial use, which permits use, reproduction, and modification for academic and non-profit research purposes only. Commercial use is not permitted, except by NVIDIA and its affiliates. Redistribution must retain the license and all applicable copyright and attribution notices. The model is provided “as is” without warranty of any kind, and users assume all associated risks.
This model is built using components from third-party models with their respective licenses:
LocateAnything-3B is intended for developers and researchers building vision-language models and applications that require fast and precise visual localization from natural language instructions.
Supported use cases include:
Open-set, common, and long-tail object detection
Dense multi-object detection in cluttered scenes
Phrase and referring-expression grounding
Automated dataset labeling and annotation (e.g., detection, grounding, pointing)
GUI element grounding for interactive and agentic systems
Robotics and autonomous driving perception
Document understanding, layout grounding, and OCR localization
Industrial inspection, surveillance, and remote sensing applications
Point-based localization and fine-grained spatial reasoning
Release Date [Insert the expected release date below]:
Architecture Type: Transformer-based vision-language model (VLM).
Network Architecture: Native-resolution VLM with the following components:
Vision encoder: MoonViT
Language model: Qwen2.5-3B-Instruct
Multimodal projector: MLP projector
Output formulation: Block-based structure for visual grounding
Number of model parameters: 3B.
LocateAnything extends a vision-language model with Parallel Box Decoding (PBD), a block-wise multi-token prediction framework for efficient visual grounding. Instead of autoregressive coordinate generation, the model predicts complete bounding boxes and points in parallel structured units, improving decoding efficiency while preserving geometric consistency. The architecture jointly optimizes next-token prediction and multi-token prediction to balance reasoning ability and parallel inference. Training follows a four-stage pipeline: initial multimodal knowledge adaptation using captioning, VQA, OCR, and related data, followed by grounding and dense-scene localization fine-tuning.
Input(s):
Input Type(s): Image and Text.
Input Format(s):
Image: RGB image input with original source resolution.
Text: Natural-language prompt or task template, such as object categories, referring expressions, GUI instructions, OCR/layout requests, or pointing queries.
Input Parameters:
Image: Two-Dimensional (2D)
Text: One-Dimensional (1D)
Other Properties Related to Input:
Production image resolution supports up to 2.5K.
Prompt length supports up to 24K tokens.
Training detection and grounding stages use a maximum sequence length of 25,600 tokens.
Inference supports up to 8,192 newly generated tokens.
Output(s):
Output Type(s): Text.
Output Format(s):
Text: Model-generated token sequence containing semantic labels and structured coordinate tokens, such as bounding boxes (<box> x1, y1, x2, y2 </box>) and points (<box> x, y </box>).
Outputs are organized into fixed-length blocks (length 6), including Semantic, Box, Negative, and End blocks.
A Box block encodes quantized spatial coordinates with structural tokens; unused positions are padded with <null>.
Fast Mode predicts box-aligned blocks in parallel; Slow Mode uses autoregressive decoding; Hybrid Mode defaults to parallel decoding with fallback to autoregressive decoding for format irregularity or spatial ambiguity.
Our AI models are designed and optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves improved training and inference performance compared to CPU-only solutions.
Software Integration:
Runtime Engine(s):
Transformers. The inference setup uses standard VLM generation with BF16 precision and KV cache. TensorRT, TensorRT-LLM, and Triton are not yet supported.
Deployment on embedded platforms such as NVIDIA Thor is possible with additional model optimization, including quantization, compression, or distillation. Other architectures may be supported depending on available memory, precision support, and software configuration.
Supported Operating System(s):
Linux
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Model Version(s):
LocateAnything-3B: 3B-parameter research model variant evaluated in Hybrid Mode by default. Fast, Hybrid, and Slow inference modes are supported by the same model formulation.
LocateAnything-3B can be integrated into systems that require spatial grounding from natural language, such as GUI agents, robotics/embodied agents, document-understanding pipelines, OCR/text localization, and open-world detection workflows.
Training, Testing, and Evaluation Datasets:
Data Modality:
Image and Text.
Image
Text
Training Data Size:
Image Training Data Size:
1 Million to 1 Billion Images - 12M unique images.
Text Training Data Size:
1 Billion to 10 Trillion Tokens - Derived from approximately 140M natural-language queries.
Data Collection Method by dataset:
Hybrid: Human, Automated
Data is collected from human-curated and open-source datasets, as well as automated ingestion of publicly available data sources.
Labeling Method by dataset:
Hybrid: Human, Synthetic, Automated
Labeling includes original human or open-source annotations, along with model-assisted and synthetic annotation generation using Qwen3-VL, Molmo, SAM 3, and Rex-Omni, with automated post-verification.
Properties: The training data consists of supervised fine-tuning (SFT) datasets with multimodal inputs, primarily image-text pairs and structured annotations such as bounding boxes, points, and negative samples.
The data spans multiple domains, including grounding, open-world grounding, general and dense object detection, scene text detection, GUI understanding and grounding, document layout understanding, and OCR.
Modalities include visual inputs (images) and natural-language queries or instructions. The dataset is derived from a mixture of publicly available academic datasets, along with model-assisted and synthetic annotations. It may include publicly available and potentially copyrighted content; users are responsible for ensuring compliance with applicable usage rights.
The linguistic content primarily consists of short, task-oriented natural-language expressions, such as object categories, referring expressions, GUI instructions, OCR queries, and grounding prompts, typically in English.
Evaluation Dataset:
Data Collection Method by dataset:
Hybrid: Human, Automated
Labeling Method by dataset:
Hybrid: Human, Synthetic, Automated
Properties: The evaluation datasets consist of publicly available benchmarks spanning visual grounding, object detection, document understanding, scene text detection, and GUI-related tasks. Modalities include image inputs paired with natural-language queries and structured annotations such as bounding boxes and points.
The evaluation suite covers both box-level and point-level grounding tasks, with approximately 48K images for box evaluation and 35K images for point evaluation across multiple datasets. These datasets span diverse domains including natural scenes, documents, aerial imagery, and human-centric interactions, enabling comprehensive assessment of localization accuracy and robustness.
Evaluation queries are typically short, task-oriented natural-language expressions such as referring phrases, object categories, and grounding prompts.
Performance is measured using box-based F1 at IoU thresholds of 0.5 and 0.95, as well as mean IoU for detection, layout, and OCR tasks. Point-based localization is evaluated based on whether predicted points fall within ground-truth segmentation masks or bounding boxes. Inference efficiency is reported in boxes per second (BPS) on a single NVIDIA H100 GPU with batch size 1.
Quantitative Evaluation Benchmarks
General Object Detection
Dense Object Detection
GUI Understanding
Layout Grounding and OCR
Referring Expression Grounding
Pointing
Inference:
Test Hardware: H100
We suggest using max_new_tokens=8192 and generation_mode="hybrid" to avoid truncated response and balance speed with accuracy.
If MagiAttention is installed, the model will automatically use it for efficient MTP block-diffusion attention. If not installed, it will fall back to PyTorch SDPA — fully functional but slower for MTP decoding.
Worker (recommended)
Below is a self-contained worker that loads the model once and serves perception queries via a unified predict() plus task-specific convenience methods. You can drop this class into any FastAPI / gRPC / Triton serving framework.
python
1import re
2import torch
3from PIL import Image
4from transformers import AutoModel, AutoTokenizer, AutoProcessor
567classLocateAnythingWorker:8"""Stateful worker that loads the model once and serves perception queries."""910def__init__(self, model_path:str, device:str="cuda", dtype=torch.bfloat16):11 self.device = device
12 self.dtype = dtype
1314 self.tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)15 self.processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)16 self.model = AutoModel.from_pretrained(17 model_path,18 torch_dtype=dtype,19 trust_remote_code=True,20).to(device).eval()2122@torch.no_grad()23defpredict(24 self,25 image: Image.Image,26 question:str,27 generation_mode:str="hybrid",# "fast" (MTP) | "slow" (NTP/AR) | "hybrid"28 max_new_tokens:int=2048,29 temperature:float=0.7,30 verbose:bool=True,31)->dict:32 messages =[33{"role":"user","content":[34{"type":"image","image": image},35{"type":"text","text": question},36]}37]3839 text = self.processor.py_apply_chat_template(40 messages, tokenize=False, add_generation_prompt=True41)42 images, videos = self.processor.process_vision_info(messages)43 inputs = self.processor(44 text=[text], images=images, videos=videos, return_tensors="pt"45).to(self.device)4647 pixel_values = inputs["pixel_values"].to(self.dtype)48 input_ids = inputs["input_ids"]49 image_grid_hws = inputs.get("image_grid_hws",None)5051 response = self.model.generate(52 pixel_values=pixel_values,53 input_ids=input_ids,54 attention_mask=inputs["attention_mask"],55 image_grid_hws=image_grid_hws,56 tokenizer=self.tokenizer,57 max_new_tokens=max_new_tokens,58 use_cache=True,59 generation_mode=generation_mode,60 temperature=temperature,61 do_sample=True,62 top_p=0.9,63 repetition_penalty=1.1,64 verbose=verbose,65)6667 result ={"answer": response[0]ifisinstance(response,tuple)else response}68ifisinstance(response,tuple)andlen(response)>=3:69 result["history"]= response[1]70 result["stats"]= response[2]71return result
7273# ---- Convenience methods for each task ----7475defdetect(self, image: Image.Image, categories:list[str],**kwargs)->dict:76"""Object detection / document layout analysis."""77 cats ="</c>".join(categories)78 prompt =f"Locate all the instances that matches the following description: {cats}."79return self.predict(image, prompt,**kwargs)8081defground_single(self, image: Image.Image, phrase:str,**kwargs)->dict:82"""Phrase grounding — single instance."""83 prompt =f"Locate a single instance that matches the following description: {phrase}."84return self.predict(image, prompt,**kwargs)8586defground_multi(self, image: Image.Image, phrase:str,**kwargs)->dict:87"""Phrase grounding — multiple instances."""88 prompt =f"Locate all the instances that match the following description: {phrase}."89return self.predict(image, prompt,**kwargs)9091defground_text(self, image: Image.Image, phrase:str,**kwargs)->dict:92"""Text grounding."""93 prompt =f"Please locate the text referred as {phrase}."94return self.predict(image, prompt,**kwargs)9596defdetect_text(self, image: Image.Image,**kwargs)->dict:97"""Scene text detection."""98 prompt ="Detect all the text in box format."99return self.predict(image, prompt,**kwargs)100101defground_gui(self, image: Image.Image, phrase:str, output_type:str="box",**kwargs)->dict:102"""GUI grounding (box or point)."""103if output_type =="point":104 prompt =f"Point to: {phrase}."105else:106 prompt =f"Locate the region that matches the following description: {phrase}."107return self.predict(image, prompt,**kwargs)108109defpoint(self, image: Image.Image, phrase:str,**kwargs)->dict:110"""Pointing."""111 prompt =f"Point to: {phrase}."112return self.predict(image, prompt,**kwargs)113114# ---- Utility: parse model output ----115116@staticmethod117defparse_boxes(answer:str, image_width:int, image_height:int)->list[dict]:118"""Parse model output into pixel-coordinate bounding boxes.
119120 Coordinates in model output are normalized integers in [0, 1000].
121 """122 boxes =[]123for m in re.finditer(r"<box><(\d+)><(\d+)><(\d+)><(\d+)></box>", answer):124 x1, y1, x2, y2 =[int(g)for g in m.groups()]125 boxes.append({126"x1": x1 /1000* image_width,127"y1": y1 /1000* image_height,128"x2": x2 /1000* image_width,129"y2": y2 /1000* image_height,130})131return boxes
132133@staticmethod134defparse_points(answer:str, image_width:int, image_height:int)->list[dict]:135"""Parse model output into pixel-coordinate points."""136 points =[]137for m in re.finditer(r"<box><(\d+)><(\d+)></box>", answer):138 x, y =int(m.group(1)),int(m.group(2))139 points.append({140"x": x /1000* image_width,141"y": y /1000* image_height,142})143return points
[PHRASE] is a free-form natural-language description; [CATEGORIES] is a comma-separated list (multiple categories may also be joined with </c>).
Generation Modes
Mode
Description
Speed
Accuracy
fast
MTP only, never falls back to AR
Fastest
Good for simple scenes
slow
Pure auto-regressive decoding
Slowest
Most robust
hybrid (default)
MTP first, falls back to AR on uncertain boxes, switches back after box boundary
Balanced
Best overall
Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.
🚀 If you find these models useful
Help me test my AI-Powered Quantum Network Monitor Assistant with quantum-ready security checks:
The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : Source Code Quantum Network Monitor. You will also find the code I use to quantize the models if you want to do it yourself GGUFModelBuilder
💬 How to test:
Choose an AI assistant type:
TurboLLM (GPT-4.1-mini)
HugLLM (Hugginface Open-source models)
TestLLM (Experimental CPU-only)
What I’m Testing
I’m pushing the limits of small open-source models for AI network monitoring, specifically:
Function calling against live network services
How small can a model go while still handling:
Automated Nmap security scans
Quantum-readiness checks
Network Monitoring tasks
🟡 TestLLM – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):
✅ Zero-configuration setup
⏳ 30s load time (slow inference but no API costs) . No token limited as the cost is low.
🔧 Help wanted! If you’re into edge-device AI, let’s collaborate!
Other Assistants
🟢 TurboLLM – Uses gpt-4.1-mini :
**It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited.
Create custom cmd processors to run .net code on Quantum Network Monitor Agents
Real-time network diagnostics and monitoring
Security Audits
Penetration testing (Nmap/Metasploit)
🔵 HugLLM – Latest Open-source models:
🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.
💡 Example commands you could test:
"Give me info on my websites SSL certificate"
"Check if my server is using quantum safe encyption for communication"
"Run a comprehensive security audit on my server"
'"Create a cmd processor to .. (what ever you want)" Note you need to install a Quantum Network Monitor Agent to run the .net code on. This is a very flexible and powerful feature. Use with caution!
Final Word
I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is open source. Feel free to use whatever you find helpful.
If you appreciate the work, please consider buying me a coffee ☕. Your support helps cover service costs and allows me to raise token limits for everyone.
I'm also open to job opportunities or sponsorship.
