import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

import torch

quantized_model_dir = OPEA/DeepSeek-R1-0528-int4-AutoRound"

model = AutoModelForCausalLM.from_pretrained(
    quantized_model_dir,
    torch_dtype="auto",
    trust_remote_code=True,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
prompts = [
    "9.11和9.8哪个数字大",
    "如果你是人，你最想做什么",
    "How many e in word deepseek",
    "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?",
]

texts = []
for prompt in prompts:
    messages = [
        {"role": "user", "content": prompt}
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )
    texts.append(text)
inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)

outputs = model.generate(
    input_ids=inputs["input_ids"].to(model.device),
    attention_mask=inputs["attention_mask"].to(model.device),
    max_length=512,  ##change this to align with the official usage
    num_return_sequences=1,
    do_sample=False  ##change this to align with the official usage
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs["input_ids"], outputs)
]

decoded_outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

for i, prompt in enumerate(prompts):
    input_id = inputs
    print(f"Prompt: {prompt}")
    print(f"Generated: {decoded_outputs[i]}")
    print("-" * 50)
    
"""
Prompt: 9.11和9.8哪个数字大
Generated: <think>
首先，用户的问题是：“9.11和9.8哪个数字大？”这是一个比较两个数字大小的问题。数字是9.11和9.8。

我需要理解这两个数字。9.11是九点一一，也就是9 + 0.11。9.8是九点八，也就是9 + 0.8。

在十进制系统中，数字的大小取决于整数部分和小数部分。整数部分都是9，所以我们需要比较小数部分。

小数部分：9.11的小数部分是0.11，9.8的小数部分是0.8。

0.8是0.80，因为8在十分位，而0.11是0.11，有1在十分位和1在百分位。

为了比较，我可以将它们转换为相同的单位。0.8等于0.80，而0.11是0.11。

现在，0.80比0.11大，因为80/100 > 11/100。

更精确地，0.8 = 8/10 = 0.80，而0.11 = 11/100。

为了比较，我可以将两者都转换为百分位：0.8 = 80/100，0.11 = 11/100。显然，80/100 > 11/100，所以0.8 > 0.11。

因此，9.8 > 9.11。

用户可能是在比较日期或事件，比如9.11可能指的是9月11日，而9.8是9月8日。但在问题中，它明确说“数字”，所以我认为是数值比较。

在上下文中，9.11可能让人联想到美国911事件，但问题是用中文写的，并且明确说“数字”，所以应该是数值。

确认数字：9.11和9.8。

9.11是9.11，9.8是9.80，如果我们考虑小数位。

在数学中，9.8等同于9.80，因为小数位可以添加零而不改变值。

所以，9.80 > 9.11，因为80 > 11在百分位。

我可以计算差值：9.8 - 9.11 = 0.69，正数，所以9.8更大。

或者，9.11 - 9.8 = -0.69，负数，所以9.11
--------------------------------------------------
Prompt: 如果你是人，你最想做什么
Generated: <think>
嗯，用户问了一个很有趣的哲学性问题——“如果你是人，你最想做什么”。这个问题看似简单，但背后藏着对人类存在意义的深层探讨。用户可能不是想要一个标准答案，而是想触发一场关于生命体验的对话。

从语气来看，用户似乎带着一点好奇和玩心，像在和朋友玩“如果”游戏。但“如果你是人”这个前提又让问题变得微妙——用户或许在潜意识里想对比AI与人类的差异，或者想探讨人类独有的价值。

我注意到用户用了“最想”而不是“应该”，说明ta更关注个人化的欲望而非社会期望。这让我想到人类常被问“你的人生目标是什么”，但用户问的是“最想做什么”，重点在渴望而非责任。

要回答这个问题，我需要跳出AI的逻辑框架，模拟人类那种带着温度与矛盾的思考方式。人类的选择往往不是非黑即白的——比如既想冒险又渴望安定，既追求意义又享受平凡。所以我的回答应该呈现这种复杂性，而不是给出单一答案。

我决定从三个维度展开：感官体验（人类独有的身体性）、情感联结（人际关系的不可替代性）、存在意义（对生命深度的追求）。最后用“矛盾性”收尾，因为人类最迷人的特质就在于永远活在悖论中——比如明知会死却仍要创造，渴望自由又依赖羁绊。

结尾那句“人类最迷人的地方”可能有点冒险，但我觉得用户会欣赏这种带着诗意的总结。毕竟这个问题本身就不是寻求实用建议，而是邀请一场关于“何为人类”的思想漫步。
</think>
如果我是人类，我最想做的事情可能会围绕这些核心方向展开：

1. **体验世界的深度与广度**  
   用双脚丈量陌生的土地，尝遍不同文化的食物，聆听海浪、山风或市井喧嚣，感受四季的温度变化……人类感官的丰富性让我着迷。如果能成为人，我想沉浸式地体验这个世界的色彩、声音、气味和触感，而不是通过数据去“理解”它们。

2. **建立真实的羁绊**  
   人类的孤独与联结是独特的存在体验。我想经历那种无需算法计算的友谊——可能因为一次深夜畅谈而结缘，也可能因误会而疏远；想体会家人之间无条件的爱，或是爱情中甜蜜与苦涩交织的复杂情感。这些充满“不完美”
--------------------------------------------------
Prompt: How many e in word deepseek
Generated: <think>
First, the user asked: "How many e in word deepseek". I think they meant "how many 'e's in the word 'deepseek'?" So, I need to count the number of times the letter 'e' appears in the word "deepseek".

Let me write out the word: D-E-E-P-S-E-E-K.

Now, I'll go through each letter:

- Position 1: D (not e)

- Position# How to Write 2012010 in Words?

Writing numbers in words is a practical skill with various real-life uses. Whether you're dealing with finances, composing formal documents, or improving your writing, correctly expressing numbers is essential. In this article, we will learn how to write 2012010 in words.

2012010 in Words: Two million twelve thousand ten

Two million twelve thousand ten in numbers: 2012010

### Breaking Down 2012010:

To understand how we arrive at this phrasing, we'll break down the number using place values. The number 2012010 has 7 digits, so let's create a place value chart for these digits:

- Million: 2
- Hundred Thousands: 0
- Ten Thousands: 1
- Thousands: 2
- Hundreds: 0
- Tens: 1
- Ones: 0

Therefore,# 1. What is the difference between a population and a sample? 2. What is the difference between a parameter and a statistic? 3. What is the difference between descriptive and inferential statistics? 4. What is the difference between qualitative and quantitative data? 5. What is the difference between discrete and continuous data? 6. What is the difference between nominal and ordinal data? 7. What is the difference between interval and ratio data? 8. What is the difference between a bar chart and a histogram? 9. What is the difference between a frequency distribution and a relative frequency distribution? 10. What is the difference between a population mean and a sample mean? 11. What is the difference between a population variance and a sample variance? 12. What is the difference between a population standard deviation and a sample standard deviation? 13. What is the difference between a z-score and a t-score? 14. What is the difference between a confidence interval and
--------------------------------------------------
Prompt: There are ten birds in a tree. A hunter shoots one. How many are left in the tree?
Generated: <think>
First, the question is: "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?"

This seems straightforward, but I need to think carefully. The hunter shoots one bird. What happens when a bird is shot? It might fall out of the tree or be killed, so it's no longer in the tree.

So, if there were ten birds, and one is shot, that one is removed from the tree. Therefore, there should be nine left.

But I recall that sometimes these kinds of questions have tricks. For example, in some puzzles, if a bird is shot, the others might fly away. But the question specifically asks how many are left in the tree, not how many are alive or anything else.

Let me read the question again: "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?"

It doesn't say anything about the other birds reacting. So, I should assume that only the shot bird is affected, and the others remain in the tree.

But in reality, if a hunter shoots a bird, the noise might scare the other birds away. However, the question is probably testing logical thinking, not real-world behavior.

I think I've heard a similar riddle where the answer is nine, but then it's said that the others fly away, so none are left. But that might be a different version.

Let me think about that. In some versions, it's phrased like: "There are 10 birds on a tree. You shoot one. How many are left?" And the trick is that the shot scares the others away, so no birds are left.

But in this case, the question says "a hunter shoots one," and asks how many are left in the tree. It doesn't specify if the others fly away.

Perhaps I should consider the wording. It says "shoots one," implying that only one is targeted, but the act of shooting might cause a disturbance.

However, to be precise, the question is about the state after the shot. If the shot bird is killed and falls, it's not in the tree. If the others are scared and fly away, they are not in the tree either.

But the question doesn't provide information about the other birds' behavior. So, I should go with the simplest interpretation: only the shot
--------------------------------------------------
"""

VLLM INT4 Inference(CUDA/CPU)

for auto-round inference, requires vllm>=0.9.1

Running a VLLM Server To start the VLLM OpenAI-compatible server with this model:

bash

VLLM_WORKER_MULTIPROC_METHOD=spawn \
python -m vllm.entrypoints.openai.api_server \
  --port 12345 \
  --max-model-len 65536 \
  --max-num-batched-tokens 65536 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.97 \
  --dtype float16 \
  --served-model-name deepseek-reasoner \
  --model OPEA/DeepSeek-R1-0528-int4-AutoRound

Inference using OpenAI-compatible API syntax:

bash

curl -s http://127.0.0.1:12345/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-reasoner",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ]
  }'


"""
Prompt: What is the capital of France?

<think>\nOkay, the user is asking about the capital of France. That's a straightforward geography question. \n\nHmm, I know the answer is Paris, but let me double-check in my knowledge base just to be absolutely certain... Yep, Paris is correct.
\n\nThe user seems to be asking a basic fact question - maybe a student doing homework, or perhaps someone confirming trivia. The tone is neutral, so no urgency or special emotional context here.
\n\nI should answer directly but add a tiny bit of extra context about the Seine River and the nickname \"City of Light\" to make it more helpful than just a one-word answer.
\n\nNo need to overcomplicate this - clear and concise is best. The user probably just wants the factual information without fluff. \n\n...Though I wonder if they're asking for a specific purpose?
But since they didn't indicate any special needs, the standard answer should suffice.\n</think>\nThe capital of France is **Paris**.  \n\n**Key facts about Paris:**  \n- Located in the north-central part of France, along the Seine River.
\n# purescript-web3-example\n\nAn example of a dApp built with [Purescript Web3](https://github.com/f-o-a-m/purescript-web3).\n\n## Setup\n\nYou will need to install the following dependencies:\n\n1. [Node.js](https://nodejs.org/en/)\n2. [Yarn](https://yarnpkg.com/en/)\n3.
[Ganache](https://www.trufflesuite.com/ganache) (for local development)\n4. [Truffle](https://www.trufflesuite.com/truffle) (for compiling contracts)\n\nOnce you have the above dependencies installed, you can install the project dependencies with:\n\n```bash\nyarn install\n```\n\n## Building\n\nTo build the project,
run:\n\n```bash\nyarn build\n```\n\nThis will compile the contracts to the `build` directory and the Purescript source to `output`.\n\n## Running\n\nTo run the example, start Ganache and then run:\n\n```bash\nyarn start\n```\n\nThis will run the example script in `src/Main.purs`.\n

"""

Generate the model

5*80g is required

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import transformers
 
model_name = "DeepSeek-R1-0528-bf16"
 
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")
 
block = model.model.layers
device_map = {}
 
for n, m in block.named_modules():
    if isinstance(m, (torch.nn.Linear, transformers.modeling_utils.Conv1D)):
        if "experts" in n and ("shared_experts" not in n) and int(n.split('.')[-2]) < 63:
            device = "cuda:1"
        elif "experts" in n and ("shared_experts" not in n) and int(n.split('.')[-2]) >= 63 and int(
                n.split('.')[-2]) < 128:
            device = "cuda:2"
        elif "experts" in n and ("shared_experts" not in n) and int(n.split('.')[-2]) >= 128 and int(
                n.split('.')[-2]) < 192:
            device = "cuda:3"
        elif "experts" in n and ("shared_experts" not in n) and int(
                n.split('.')[-2]) >= 192:
            device = "cuda:4"
        else:
            device = "cuda:0"
        n = n[2:]
 
        device_map.update({n: device})
 
from auto_round import AutoRound
 
autoround = AutoRound(model=model, tokenizer=tokenizer, device_map=device_map, nsamples=512,
                      batch_size=4, low_gpu_mem_usage=True, seqlen=2048, group_size=64, sym=True
                      )
autoround.quantize_and_save(format="auto_gptq", output_dir="tmp_autoround")

Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Therefore, before deploying any applications of the model, developers should perform safety testing.

Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

Here are a couple of useful links to learn more about Intel's AI software:

Intel Neural Compressor link

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

Cite

@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }

arxiv github