Views
No views yet
1import requests
2from PIL import Image
3from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
4quantized_model_path="OPEA/QVQ-72B-Preview-int4-sym-inc"
5model = Qwen2VLForConditionalGeneration.from_pretrained(
6 quantized_model_path,
7 torch_dtype="auto",
8 device_map="auto",
9 ##revision="118625f" ##AutoGPTQ format
10)
11processor = AutoProcessor.from_pretrained(quantized_model_path)
12image_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/QVQ/demo.png"
13messages = [
14 {
15 "role": "system",
16 "content": [
17 {"type": "text", "text": "You are a helpful and harmless assistant. You are Qwen developed by Alibaba. You should think step-by-step."}
18 ],
19 },
20 {
21 "role": "user",
22 "content": [
23 {
24 "type": "image",
25 "image": image_url,
26 },
27 {"type": "text", "text": "What value should be filled in the blank space?"},
28 ],
29 }
30]
31
32# Preparation for inference
33text = processor.apply_chat_template(
34 messages, tokenize=False, add_generation_prompt=True
35)
36image_inputs = Image.open(requests.get(image_url, stream=True).raw)
37inputs = processor(
38 text=[text],
39 images=image_inputs,
40 padding=True,
41 return_tensors="pt",
42)
43inputs = inputs.to(model.device)
44
45generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
46generated_ids_trimmed = [
47 out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
48]
49output_text = processor.batch_decode(
50 generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
51)
52print(output_text[0])
53
54##INT4:
55# So I've got this puzzle here with emojis representing numbers, and I need to figure out what goes in the blank space. Let's see, there are hearts, bows, and a dog emoji. Each probably stands for a certain number, and I have to solve for them based on the equations given.
56# First, there's an equation with four hearts added up to make 24. So, 4 hearts = 24. That seems straightforward. If I divide both sides by 4, I get one heart = 6. Okay, that makes sense.
57# Next, there's an equation with one heart minus one bow equals
58
59##BF16:
60# So I've got this puzzle here with emojis representing numbers, and I need to figure out what goes in the blank space. Let's see, there are four equations, and the last one has a blank box where the result should be. The emojis used are hearts, bows, and dogs. I need to assign numbers to these emojis based on the equations provided.
61# First, let's look at the first equation:
62# Heart + Heart + Heart + Heart + Heart = 24
63# So, there are five hearts added together equaling 24. Let's call the heart value "h". So:5h = 24
64
65
66
67image_url = "http://images.cocodataset.org/train2017/000000411975.jpg"
68messages = [
69 {
70 "role": "system",
71 "content": [
72 {"type": "text", "text": "You are a helpful and harmless assistant. You are Qwen developed by Alibaba. You should think step-by-step."}
73 ],
74 },
75 {
76 "role": "user",
77 "content": [
78 {
79 "type": "image",
80 "image": image_url,
81 },
82 {"type": "text", "text": "图片中的棒球场上有多少人？"},
83 ],
84 }
85]
86##INT4:
87# So I've got this image of a baseball field, and there are a few people on it. Let me try to count how many people there are. First, I see three people near the infield grass. One of them is bending over, maybe picking something up, and another is standing next to them, also bent over. Then there's a third person standing nearby, wearing a striped shirt and khaki shorts. He seems to be observing or waiting.
88# Wait, actually, looking closer, it seems like two of them are bending over towards the ground, perhaps picking up baseballs or something similar. The person in the striped shirt is
89
90
91##BF16:
92## So I've got this image of a baseball field, and there are a few people on it. Let me try to describe what I see.
93# First off, the field itself has a mix of grass and dirt. The infield is dirt, and the outfield is grass, which is pretty standard for a baseball field. There are three main people in the scene.
94# Starting from the left, there's a person bending over, picking something up from the ground. They're wearing a light blue shirt and dark blue pants. Next to them, another person is also bent over, but they're wearing a white shirt and light-colored pants. Both of
95
96image_url = "https://intelcorp.scene7.com/is/image/intelcorp/processor-overview-framed-badge:1920-1080?wid=480&hei=270"
97messages = [
98 {
99 "role": "system",
100 "content": [
101 {"type": "text", "text": "You are a helpful and harmless assistant. You are Qwen developed by Alibaba. You should think step-by-step."}
102 ],
103 },
104 {
105 "role": "user",
106 "content": [
107 {
108 "type": "image",
109 "image": image_url,
110 },
111 {"type": "text", "text": "这张图片代表哪家公司？"},
112 ],
113 }
114]
115##INT4:
116## 这张图片代表了Intel公司，是一家美国的半导体公司。
117
118##BF16:
119## 这张图片代表的是英特尔公司（Intel Corporation）。图中的标志是英特尔著名的“intel inside”标识，它由两个单词组成：“intel”和“inside”，其中“intel”是公司名称，“inside”表示英特尔的处理器或芯片被内置于各种电子设备中，尤其是计算机和笔记本电脑中。这个标志通常被贴在使用英特尔处理器的设备上，以表明其内部搭载了英特尔的芯片。英特尔是一家总部位于美国加州圣克拉拉的跨国公
120# 司，是全球最大的半导体芯片制造商之一，也是x86架构微处理器的开创者。
1211pip install auto-round
2auto-round-mllm
3--model Qwen/QVQ-72B-Preview \
4--device 0 \
5--group_size 128 \
6--bits 4 \
7--iters 1000 \
8--nsample 512 \
9--low_gpu_mem_usage \
10--seqlen 2048 \
11--model_dtype "float16" \
12--format 'auto_gptq,auto_round' \
13--output_dir "./tmp_autoround"