Views
No views yet
e_score_correction_bias is stored in BF16 because, when loaded in Transformers, its dtype is automatically converted to BF16. As a result, it is difficult for us to preserve it in FP32 within our tools.
Please use it with causion1from transformers import AutoModelForCausalLM, AutoTokenizer
2import transformers
3import torch
4quantized_model_dir = "Intel/DeepSeek-V3.1-Terminus-int4-mixed-AutoRound"
5
6model = AutoModelForCausalLM.from_pretrained(
7 quantized_model_dir,
8 torch_dtype=torch.bfloat16,
9 device_map="auto",
10)
11tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
12prompts = [
13 "9.11和9.8哪个数字大",
14 "strawberry中有几个r?",
15 "There is a girl who likes adventure,",
16 "Please give a brief introduction of DeepSeek company.",
17 ]
18
19texts=[]
20for prompt in prompts:
21 messages = [
22 {"role": "system", "content": "You are a helpful assistant."},
23 {"role": "user", "content": prompt}
24 ]
25 text = tokenizer.apply_chat_template(
26 messages,
27 tokenize=False,
28 add_generation_prompt=True
29 )
30 texts.append(text)
31inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
32
33outputs = model.generate(
34 input_ids=inputs["input_ids"].to(model.device),
35 attention_mask=inputs["attention_mask"].to(model.device),
36 max_length=200, ##change this to align with the official usage
37 num_return_sequences=1,
38 do_sample=False ##change this to align with the official usage
39 )
40generated_ids = [
41 output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs["input_ids"], outputs)
42 ]
43decoded_outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
44
45for i, prompt in enumerate(prompts):
46 input_id = inputs
47 print(f"Prompt: {prompt}")
48 print(f"Generated: {decoded_outputs[i]}")
49 print("-"*50)
50
51
52"""
53Prompt: 9.11和9.8哪个数字大
54Generated: 9.11 比 9.8 大。
55
56比较两个小数时，先比较整数部分（都是 9），然后比较小数部分：
57- 9.11 的小数部分是 0.11
58- 9.8 的小数部分是 0.8
59由于 0.11 小于 0.8，但这里需要对齐小数位比较：
609.11 = 9.11
619.8 = 9.80
62比较 0.11 和 0.80，0.11 < 0.80，所以 9.11 < 9.8？
63**不对，我纠正一下**：
64实际上 9.11 的十分位是 1，而 9.8 的十分位是 8，因为 1 < 8，所以 9.
65--------------------------------------------------
66Prompt: strawberry中有几个r?
67Generated: 我们来数一下单词 **strawberry** 中的字母 **r** 的数量。
68
69单词：s t r a w b e r r y
70
71逐个字母看：
72- 第 3 个字母：r
73- 第 8 个字母：r
74- 第 9 个字母：r
75
76一共有 **3** 个字母 **r**。
77
78**答案：3**
79--------------------------------------------------
80Prompt: There is a girl who likes adventure,
81Generated: That's a wonderful start to a story. A girl who likes adventure is a character full of potential.
82
83What would you like to do with this idea?
84
85* **Create a character profile?** We could give her a name, a backstory, and define what *kind* of adventure she seeks.
86 * **Name:** Elara, Maya, Kaelen, Juniper?
87 * **Type of Adventure:** Is she an explorer of ancient ruins, a solver of mysteries in her town, a traveler to fantastical worlds, or a protector of nature?
88
89* **Start a story?** We can begin a narrative. Where is she, and what is the call to adventure?
90 * *Example:* "Elara traced the faded lines on the old map she'd found tucked inside a library book. It led to a part of the forest everyone
91--------------------------------------------------
92Prompt: Please give a brief introduction of DeepSeek company.
93Generated: Of course! Here is a brief introduction to DeepSeek.
94
95**DeepSeek** is a leading Chinese artificial intelligence research company, widely recognized for developing advanced large language models (LLMs).
96
97Here are the key points about the company:
98
99* **Core Focus:** Their primary mission is to achieve Artificial General Intelligence (AGI). They are best known for their series of "DeepSeek" models, which are among the most powerful and capable open-source LLMs in the world, competing with models from major global AI labs.
100
101* **Key Products & Models:**
102 * **DeepSeek-V2:** A state-of-the-art mixture-of-experts (MoE) model that delivers high performance at a significantly lower cost for inference compared to similar-sized models.
103 * **DeepSeek Coder:** A family of models specifically designed for code generation and
104--------------------------------------------------
105
106"""1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3from auto_round import AutoRound
4from auto_round.utils import llm_load_model
5
6model_name = "deepseek-ai/DeepSeek-V3.1-Terminus"
7
8model, tokenizer,_=llm_load_model(model_name,trust_remote_code=False,device="cpu")
9layer_config = {}
10for n, m in model.named_modules():
11 if isinstance(m, torch.nn.Linear):
12 if "expert" in n and "shared_experts" not in n:
13 layer_config[n] = {"bits": 4}
14 print(n, 4)
15 elif n != "lm_head":
16 layer_config[n] = {"bits": 8}
17 print(n, 8)
18
19ar = AutoRound(model, tokenizer=tokenizer, iters=0, layer_config=layer_config)
20ar.quantize_and_save(format="auto_round", output_dir="tmp_autoround")
21