Views
No views yet
1git clone https://github.com/OpenBMB/VisRAG.git
2conda create --name EVisRAG python==3.10
3conda activate EVisRAG
4cd EVisRAG
5pip install -r EVisRAG_requirements.txt1git clone https://github.com/hiyouga/LLaMA-Factory.git
2bash evisrag_scripts/full_sft.shbash evisrag_scripts/run_rsgrpo.shEVisRAG-Train, which is referenced at the beginning of this page.LLaMA-Factory and update the model path in the full_sft.sh script. In the second stage, we built our customized algorithm RS-GRPO based on Easy-R1, specifically designed for EVisRAG, whose implementation can be found in src/RS-GRPO.1bash evisrag_scripts/predict.sh
2bash evisrag_scripts/eval.sh EVisRAG-Test-xxx, as referenced at the beginning of this page.predict.sh script. The model outputs will be saved in the preds directory. Then, use the eval.sh script to evaluate the predictions. The metrics EM, Accuracy, and F1 will be reported directly.1from transformers import AutoProcessor
2from vllm import LLM, SamplingParams
3from qwen_vl_utils import process_vision_info
4
5def evidence_promot_grpo(query):
6 return f"""You are an AI Visual QA assistant. I will provide you with a question and several images. Please follow the four steps below:
7
8Step 1: Observe the Images
9First, analyze the question and consider what types of images may contain relevant information. Then, examine each image one by one, paying special attention to aspects related to the question. Identify whether each image contains any potentially relevant information.
10Wrap your observations within <observe></observe> tags.
11
12Step 2: Record Evidences from Images
13After reviewing all images, record the evidence you find for each image within <evidence></evidence> tags.
14If you are certain that an image contains no relevant information, record it as: [i]: no relevant information(where i denotes the index of the image).
15If an image contains relevant evidence, record it as: [j]: [the evidence you find for the question](where j is the index of the image).
16
17Step 3: Reason Based on the Question and Evidences
18Based on the recorded evidences, reason about the answer to the question.
19Include your step-by-step reasoning within <think></think> tags.
20
21Step 4: Answer the Question
22Provide your final answer based only on the evidences you found in the images.
23Wrap your answer within <answer></answer> tags.
24Avoid adding unnecessary contents in your final answer, like if the question is a yes/no question, simply answer "yes" or "no".
25If none of the images contain sufficient information to answer the question, respond with <answer>insufficient to answer</answer>.
26
27Formatting Requirements:
28Use the exact tags <observe>, <evidence>, <think>, and <answer> for structured output.
29It is possible that none, one, or several images contain relevant evidence.
30If you find no evidence or few evidences, and insufficient to help you answer the question, follow the instruction above for insufficient information.
31
32Question and images are provided below. Please follow the steps as instructed.
33Question: {query}
34"""
35
36model_path = "xxx"
37processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True, padding_side='left')
38
39imgs, query = ["imgpath1", "imgpath2", ..., "imgpathX"], "What xxx?"
40input_prompt = evidence_promot_grpo(query)
41
42content = [{"type": "text", "text": input_prompt}]
43for imgP in imgs:
44 content.append({
45 "type": "image",
46 "image": imgP
47 })
48msg = [{
49 "role": "user",
50 "content": content,
51 }]
52
53llm = LLM(
54 model=model_path,
55 tensor_parallel_size=1,
56 dtype="bfloat16",
57 limit_mm_per_prompt={"image":5, "video":0},
58)
59
60sampling_params = SamplingParams(
61 temperature=0.1,
62 repetition_penalty=1.05,
63 max_tokens=2048,
64)
65
66prompt = processor.apply_chat_template(
67 msg,
68 tokenize=False,
69 add_generation_prompt=True,
70)
71
72image_inputs, _ = process_vision_info(msg)
73
74msg_input = [{
75 "prompt": prompt,
76 "multi_modal_data": {"image": image_inputs},
77}]
78
79output_texts = llm.generate(msg_input,
80 sampling_params=sampling_params,
81)
82
83print(output_texts[0].outputs[0].text)