You can deploy this model and start earning money today!

kkk5/
RPI-CLIP-Qwen2.5-VL-7B

Name: RPI-CLIP-Qwen2.5-VL-7B
Availability: InStock

safetensors

qwen2_5_vl

Views

No views yet

Re3Cap-CLIP-Qwen2.5-VL-7B

This model is fine-tuned from Qwen2.5-VL-7B-Instruct using GRPO with the CLIP image-text similarity (CLIP) reward and the Re³Cap retrieval-guided refinement strategy, as described in our EMNLP 2026 Main paper.

arXiv

Overview

Re³Cap (Retrieval-Guided Refinement for Image Captioning) is a retrieval-guided reasoning strategy that improves image captioning without requiring additional annotations. It is instantiated by Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA), which together identify hallucinations and omissions in generated captions, leading to more accurate and detailed descriptions.

This checkpoint applies Re³Cap on top of the CLIP reward, which uses the CLIP image-text similarity score between the image and the generated caption as the reward signal (Cho et al., 2022).

Usage

python

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2_5_VLForConditionalGeneration.from_pretrained("kkk5/Re3Cap-CLIP-Qwen2.5-VL-7B", torch_dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained("kkk5/Re3Cap-CLIP-Qwen2.5-VL-7B")

messages = [{"role": "user", "content": [
    {"type": "image", "image": "your_image.jpg"},
    {"type": "text", "text": "Caption this image as accurately as possible, without speculation. Describe what you see."},
]}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt").to(model.device)

output_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
print(output_text)

Citation

bibtex

@inproceedings{jia2026re3cap,
    title     = {Re$^3$Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning},
    author    = {Jia, Haonan and Dong, Shichao and Sun, Zenghui and Zheng, Jiawen and Miao, Ziqi and Shi, Gege and Zhao, Qiuyu and Lan, Jinsong and Zhu, Xiaoyong and Zheng, Bo},
    booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
    year      = {2026}
}

kkk5/RPI-CLIP-Qwen2.5-VL-7B

Re3Cap-CLIP-Qwen2.5-VL-7B

Overview

Usage

Citation

kkk5/
RPI-CLIP-Qwen2.5-VL-7B