Views
No views yet
1import transformers
2from transformers import AutoModelForCausalLM, AutoTokenizer
3import torch
4
5quantized_model_dir = "OPEA/DeepSeek-R1-Distill-Qwen-32B-int4-gptq-sym-inc"
6
7device_map="auto"
8model = AutoModelForCausalLM.from_pretrained(
9 quantized_model_dir,
10 torch_dtype=torch.float16,
11 trust_remote_code=True,
12 device_map=device_map,
13)
14
15tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
16prompts = [
17 "9.11和9.8哪个数字大",
18 "如果你是人，你最想做什么",
19 "How many e in word deepseek",
20 "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?",
21]
22
23texts = []
24for prompt in prompts:
25 messages = [
26 {"role": "user", "content": prompt}
27 ]
28 text = tokenizer.apply_chat_template(
29 messages,
30 tokenize=False,
31 add_generation_prompt=True
32 )
33 texts.append(text)
34
35inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
36
37outputs = model.generate(
38 input_ids=inputs["input_ids"].to(model.device),
39 attention_mask=inputs["attention_mask"].to(model.device),
40 max_length=512, ##change this to align with the official usage
41 num_return_sequences=1,
42 do_sample=False ##change this to align with the official usage
43)
44generated_ids = [
45 output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs["input_ids"], outputs)
46]
47
48decoded_outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
49
50for i, prompt in enumerate(prompts):
51 input_id = inputs
52 print(f"Prompt: {prompt}")
53 print(f"Generated: {decoded_outputs[i]}")
54 print("-" * 50)
55
56
57"""
58Prompt: 9.11和9.8哪个数字大
59Generated: .11和.8哪个数字大
60</think>
61
62.11和.8哪个数字大
63</think>
64
65要比较 **9.11** 和 **9.8** 哪个更大，可以按照以下步骤进行：
66
671. **比较整数部分**：
68 - 两个数字的整数部分都是 **9**，所以整数部分相等。
69
702. **比较小数部分**：
71 - **9.11** 的小数部分是 **0.11**
72 - **9.8** 的小数部分是 **0.8**（即 **0.80**）
73
74 由于 **0.80 > 0.11**，所以 **9.8** 的小数部分更大。
75
763. **结论**：
77 - 因此，**9.8** 比 **9.11** 大。
78
79最终答案：\boxed{9.8}
80--------------------------------------------------
81Prompt: 如果你是人类，你最想做什么
82Generated: 您好！我是由中国的深度求索（DeepSeek）公司开发的智能助手DeepSeek-R1。有关模型和产品的详细内容请参考官方文档。
83</think>
84</think>
85
86您好！我是由中国的深度求索
87</think>
88
89您好！我是由中国的深度求索（DeepSeek）公司开发的智能助手DeepSeek-R1。有关模型和产品的详细内容请参考官方文档。
90--------------------------------------------------
91Prompt: How many e in word deepseek
92Generated: To determine how many times the letter 'e' appears in the word "deepseek," I will examine each letter one by one.
93
94First, I'll list out the letters in the word: D, E, E, P, S, E, E, K.
95
96Next, I'll go through each letter and count every occurrence of the letter 'e'.
97
98Starting with the first letter, D, it's not an 'e'. The second letter is E, which counts as one. The third letter is another E, making it two. The fourth letter is P, not an 'e'. The f
99ifth letter is S, also not an 'e'. The sixth letter is E, bringing the count to three. The seventh letter is another E, making it four. The last letter is K, which isn't an 'e'.
100
101After reviewing all the letters, I find that the letter 'e' appears four times in the word "deepseek."
102</think>
103
104To determine how many times the letter **e** appears in the word **deepseek**, follow these steps:
105
1061. **Write down the word:**
107
108 **d e e p s e e k**
109
1102. **Identify and count each 'e':**
111
112 - **e** (position 2)
113 - **e** (position 3)
114 - **e** (position 6)
115 - **e** (position 7)
116
1173. **Total count of 'e':**
118
119 There are **4** occurrences of the letter **e** in the word **deepseek**.
120
121\[
122\boxed{4}
123\]
124--------------------------------------------------
125Prompt: There are ten birds in a tree. A hunter shoots one. How many are left in the tree?
126Generated: \n</think>
127
128If a hunter shoots one bird from a tree that initially has ten birds, the number of birds remaining in the tree would depend on the reaction of the other birds.\n\n1. **Immediate React
129ion**: When a hunter shoots one bird, the loud noise and disturbance might scare the remaining birds, causing them to fly away. In this case, all the other nine birds would likely leav
130e the tree.\n\n2. **No Reaction**: If the other birds are not disturbed or choose to stay despite the shot, there would still be nine birds left in the tree.\n\nHowever, in most scenar
131ios, the loud noise of a gunshot would scare the birds, leading to all of them flying away..
132lm-eval --model hf --model_args pretrained=OPEA/DeepSeek-R1-Distill-Qwen-32B-int4-gptq-sym-inc --tasks leaderboard_mmlu_pro,leaderboard_ifeval,lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,mmlu,gsm8k --batch_size 16| Metric | BF16 | INT4 |
|---|---|---|
| avg | 0.6647 | 0.6639 |
| leaderboard_mmlu_pro | - | - |
| mmlu | 0.7964 | 0.7928 |
| lambada_openai | 0.6649 | 0.6718 |
| hellaswag | 0.6292 | 0.6223 |
| winogrande | 0.7482 | 0.7482 |
| piqa | 0.8058 | 0.7982 |
| truthfulqa_mc1 | 0.3831 | 0.3905 |
| openbookqa | 0.3520 | 0.3520 |
| boolq | 0.8963 | 0.8972 |
| arc_easy | 0.8207 | 0.8194 |
| arc_challenge | 0.5503 | 0.5469 |
| leaderboard_ifeval | - | - |
| gsm8k | - | - |
1auto-round \
2--model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
3--device 0 \
4--bits 4 \
5--iter 200 \
6--disable_eval \
7--format 'auto_gptq,auto_round,auto_awq' \
8--output_dir "./tmp_autoround"
9