This repository contains the official models from the paper "Mitigating Hallucinations in Object Detection Models: A Revisit to Out-of-Distribution Detection". It provides a collection of YOLOv10 and Faster R-CNN models, including both standard ("vanilla") baselines and our novel fine-tuned models designed to significantly reduce false positive detections on out-of-distribution (OoD) data.
Our work first identifies critical flaws in existing OoD benchmarks and then introduces a novel fine-tuning strategy that teaches detectors to be less overconfident. By training on specially curated "proximal OoD" samples, our models learn to suppress objectness scores for unfamiliar objects, leading to an 88% reduction in overall hallucination error on the BDD-100K benchmark when combined with an OoD filter.
🎯 Key Features
Novel Fine-Tuning for Hallucination Mitigation: Models specifically fine-tuned to reduce false positives on novel objects, enhancing safety and reliability.
Vanilla vs. Fine-tuned Comparison: Provides both baseline and improved models to clearly demonstrate the effectiveness of our approach.
Dual Architecture Support: Includes both the real-time YOLOv10 and the high-accuracy Faster R-CNN.
Multi-Dataset Scope: Models trained on BDD 100K for autonomous driving and Pascal VOC for general object detection.
Corrected Benchmark Datasets: Accompanied by new, carefully curated OoD test sets (Near-OoD, Far-OoD) that address the flaws in previous benchmarks.
🔬 The M-Hood Approach: How It Works
Object detectors often "hallucinate" and produce high-confidence false predictions when faced with novel objects not seen during training. Our research tackles this in two ways:
Benchmarking Revisited: We found existing OoD test sets were flawed, with ~13% of supposedly "OoD" images containing in-distribution (ID) objects. This leads to inaccurate performance evaluation. We created new, clean OoD benchmarks for reliable assessment.
Proximal OoD Fine-Tuning: Our core contribution is a fine-tuning strategy that makes the detector itself more robust.
We create a dataset of "proximal OoD" samples—objects that are semantically similar to the training classes but are not part of them (e.g., using 'deer' as a proximal OoD sample for a model trained on 'horse' and 'cow').
We fine-tune the original models on a combined dataset of original ID data and these new proximal OoD samples.
During this process, the proximal OoD samples are treated as background, effectively teaching the model to suppress its predictions for these and other similar novel objects.
The result is a model that is inherently more conservative and less likely to hallucinate, without significantly compromising its performance on the original task.
📊 Performance Highlights
Our fine-tuning method drastically reduces the number of hallucinated detections on OoD datasets. The tables below show the number of false predictions (lower is better) on our Near-OoD benchmark.
YOLOv10 on BDD-100K (Near-OoD Hallucination Counts)
Model Configuration
Hallucination Count
Reduction
Original (Vanilla)
708
-
Ours (Fine-tuned)
145
-79.5%
Original + KNN Filter
297
-58.1%
Ours + KNN Filter
78
-89.0%
Faster R-CNN on BDD-100K (Near-OoD Hallucination Counts)
(Note: The KITTI models mentioned in the old README are not explicitly detailed in the provided paper text, so I have focused on BDD and VOC, which are central to the experiments shown.)
🚀 Quick Start
YOLOv10 Usage
python
1from ultralytics import YOLO
23# Load our robust, fine-tuned YOLOv10 model4model = YOLO('yolov10-bdd-finetune.pt')56# Run inference7results = model('path/to/your/image.jpg')89# Process results10for result in results:11 boxes = result.boxes.xyxy # bounding boxes12 scores = result.boxes.conf # confidence scores13 classes = result.boxes.cls # class predictions
Faster R-CNN Usage
python
1import torch
2import torchvision
3from torchvision.models.detection import fasterrcnn_resnet50_fpn
45# NOTE: The provided .pth files are state_dicts.6# You need to load them into a model instance.7# Example for a vanilla VOC model:8num_classes =21# 20 classes + background9model = fasterrcnn_resnet50_fpn(weights=None, num_classes=num_classes)10model.load_state_dict(torch.load('faster-rcnn-voc-vanilla.pth'))11model.eval()1213# Run inference on a pre-processed image tensor14with torch.no_grad():15 predictions = model(image_tensor)1617# Process results18boxes = predictions[0]['boxes']19scores = predictions[0]['scores']20labels = predictions[0]['labels']
📄 Citation
If you use our models, datasets, or methodology in your research, please cite our papers.
For the IROS 2025 conference version, which primarily focuses on YOLO models and represents an earlier conference publication, please cite:
@inproceedings{he2025mitigating,
title={Mitigating Hallucinations in YOLO-based Object Detection Models: A Revisit to Out-of-Distribution Detection},
author={Weicheng He and Changshun Wu and Chih-Hong Cheng and Xiaowei Huang and Saddek Bensalem},
booktitle={Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year={2025},
note={Accepted to IROS 2025},
eprint={2503.07330v2},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2503.07330v2}
}
For the journal version, which expands the methodology to Faster-RCNN and RT-DETR, includes an automated data curation pipeline, and provides an in-depth analysis of the approach, please cite:
@inproceedings{wu2025revisiting,
title={Revisiting Out-of-Distribution Detection in Real-time Object Detection: From Benchmark Pitfalls to a New Mitigation Paradigm},
author={Changshun Wu and Weicheng He and Chih-Hong Cheng and Xiaowei Huang and Saddek Bensalem},
year={2025},
eprint={2503.07330v3},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2503.07330v3},
}
Please also consider citing the original works for the model architectures and datasets used.
