You can deploy this model and start earning money today!

hnamt/
RT-DisDINOv3-ViT-Base

Name: RT-DisDINOv3-ViT-Base
Availability: InStock

Views

No views yet

license: apache-2.0 language: en library_name: pytorch pipeline_tag: object-detection tags:

rtdetr
object-detection
knowledge-distillation
taco-dataset
dinov3
vit

RT-DisDINOv3-ViT: A Distilled RT-DETR-L Model

This model is an RT-DETR-L whose backbone and encoder have been pre-trained using knowledge distillation from a powerful DINOv3 ViT-Base teacher model. The distillation process was performed on feature maps from the TACO (Trash Annotations in Context) dataset using the lightly-train framework.

This pre-trained checkpoint contains the "distilled knowledge" and is intended to be used as a starting point for fine-tuning on downstream object detection tasks.

This work is part of the RT-DisDINOv3 project. For full details on the training pipeline, baseline comparisons, and analysis, please visit the main GitHub repository.

How to Use

You can load these distilled weights and apply them to the original RT-DETR-L model's backbone and encoder before fine-tuning.

python

import torch
from torch.hub import load_state_dict_from_url

# 1. Load the original RT-DETR-L model architecture
# Make sure you have the 'rtdetr' repository cloned locally or installed
rtdetr_l = torch.hub.load('lyuwenyu/RT-DETR', 'rtdetrv2_l', pretrained=True)
model = rtdetr_l.model

# 2. Load the distilled weights from this Hugging Face Hub repository
MODEL_URL = "https://huggingface.co/hnamt/RT-DisDINOv3-ViT-Base/resolve/main/distilled_rtdetr_vit_teacher_BEST.pth"
distilled_state_dict = load_state_dict_from_url(MODEL_URL, map_location='cpu')['model']

# 3. Load the weights into the model's backbone and encoder
# The `strict=False` flag ensures that only matching keys (backbone + encoder) are loaded.
model.load_state_dict(distilled_state_dict, strict=False)

print("Successfully loaded and applied distilled knowledge from ViT teacher!")

# Now the 'model' is ready for fine-tuning on your own dataset.
# For example:
# optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
# model.train()
# ... your fine-tuning loop ...

Training Details

Student Model: RT-DETR-L (rtdetrv2_l from lyuwenyu/RT-DETR).
Teacher Model: DINOv3 ViT-Base (dinov3/vitb16 via Lightly).
Dataset for Distillation: TACO dataset images.
Distillation Procedure: The student model's backbone and encoder were trained to minimize the Mean Squared Error (MSE) between their output feature maps and those of the teacher model, orchestrated by the lightly-train library.

Evaluation Results

After the distillation pre-training, the model was fine-tuned on the TACO dataset. In our experiments, this particular teacher did not yield an improvement over the baseline.

Model	mAP@50-95	mAP@50	Speed (ms)	Notes
RT-DETR-L (Baseline)	2.80%	4.60%	50.05	Fine-tuned from COCO pre-trained.
RT-DisDINOv3 (w/ ViT)	2.80%	4.20%	49.80	No performance improvement observed.

License

The weights in this repository are released under the Apache 2.0 License. Please be aware that the models used for training (RT-DETR, DINOv3) have their own licenses.