You can deploy this model and start earning money today!

Luxuriant16/
Med-RwR

Name: Med-RwR
Rating: 2.5 (28 reviews)

safetensors

qwen2_5_vl

visual-question-answering

2510.18303

Qwen/Qwen2.5-VL-7B-Instruct

finetune

mit

Views

No views yet

Med-RwR

| [ArXiv] | [Github] |

Overview

Med-RwR is the first Multimodal Medical Reasoning-with-Retrieval framework, which proactively retrieves external knowledge by querying observed symptoms or domain-specific medical concepts during reasoning. This approach encourages the model to ground its diagnostic analysis in verifiable external information retrieved after analyzing both visual and textual inputs.

Quick Start

Environment Setup

bash

git clone https://github.com/xmed-lab/Med-RwR.git
conda create -n medrwr python==3.10
conda activate medrwr
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
pip install -e .

Start the Retriever

python retrieve/retrieve.py

Inference

python

import os
import torch

from swift.llm.infer.infer_engine.pt_engine_retrieve_infer import PtEngine
from swift.llm.infer.protocol import RequestConfig
from swift.llm.template.template_inputs import InferRequest

SYSTEM_PROMPT = """You are an experienced expert in medicine. You are given a question, an image and a list of choices. You are required to select the correct answer from the choices.
First observe the image, think about the question and each choice within <think> </think> tags. During thinking, if needed, retrieve medical knowledge using <query> </query> tags. Only one query is allowed. An external agent will retrieve information and return it within <retrieve> </retrieve> tags. 
You can use the retrieved information to continue thinking and further query if more information is needed. When you can reach a conclusion, output your answer within <answer> </answer> tags.
The output should be in the following format:
1. If you need more information, output <think> ... </think>\n<query> ... </query>\n<retrieve> ... </retrieve>\n (Multiple think-query-retrieve cycles may occur)
2. If you can directly reach a conclusion without query, output <think> ... </think>\n<answer> ... </answer>"""


def run_demo(model, messages, max_new_tokens=512, temperature=0.6, top_p=None, top_k=None, repetition_penalty=None, attn_impl="flash_attn", device_map="cuda"):
    engine = PtEngine(
        model_id_or_path=model,
        attn_impl=attn_impl,
        device_map=device_map,
        max_batch_size=1,
    )
    infer_request = InferRequest(messages=messages)
    request_cfg = RequestConfig(
        max_tokens=max_new_tokens,
        temperature=temperature,
        top_p=top_p,
        top_k=top_k,
        repetition_penalty=repetition_penalty,
    )
    outputs = engine.infer([infer_request], request_cfg, template=engine.default_template, use_tqdm=False)
    first = outputs[0]
    text = first.choices[0].message.content
    print(text)

if __name__ == "__main__":
    model_path = "Luxuriant16/MedRwR"
    image_path = "Image Path Here"
    question = "User Input Here"
    messages = [
            {
                "role": "system",
                "content": SYSTEM_PROMPT
            },
            {
                "role": "user",
                "content": [
                    {"type": "image", "image": image_path},
                    {"type": "text", "text": question},
                ],
            }
        ]
    run_demo(model=model_path, messages=messages)

Luxuriant16/Med-RwR