from auto_round import AutoRoundConfig ##must import for autoround format
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
# Set `torch_dtype=torch.float16` to load model in float16, otherwise it will be loaded as float32 and cause OOM Error.
quantized_model_dir = “OPEA/internlm2_5-7b-chat-int4-sym-inc”
model = AutoModelForCausalLM.from_pretrained(quantized_model_dir, 
                                             device_map="auto",
                                             torch_dtype=torch.float16, 
                                             trust_remote_code=True,
                                             ##revision="fe1db26", ##AutoGPTQ format
                                             )
model = model.eval()
prompt = "9.11和9.8哪个数字大"

response, history = model.chat(tokenizer, 
                               prompt, 
                               history=[],
                               do_sample=False ## remove this to align with official usage
                              )
print(response)

prompt = "9.11和9.8哪个数字大"
## INT4
"""9.11大于9.8。

在比较两个小数时，我们首先比较它们的整数部分。如果整数部分相同，我们再看小数部分。在9.11和9.8中，整数部分都是9，但9.11的小数部分是0.11，而9.8的小数部分是0.8。因为0.11大于0.8，所以9.11大于9.8。"""

## BF16
"""在比较数字9.11和9.8时，我们可以直接比较它们的小数部分。

9.11的小数部分是0.11，而9.8的小数部分是0.8。

由于0.11大于0.8，因此9.11大于9.8。

所以，9.11比9.8大。"""

prompt = "strawberry中有多少个r"
## INT4
"""在单词 "strawberry" 中，字母 "r" 出现了 4 次。具体来说，"strawberry" 可以分解为 "straw" 和 "berry" 两个部分，其中 "straw" 中有一个 "r"，"berry" 中有一个 "r"，所以总共是 4 个 "r"。"""
## BF16
"""在单词 "strawberry" 中，字母 "r" 出现了两次。这个单词由8个字母组成，其中 "r" 出现了两次，分别在单词的第二个和第六个位置。
"""


prompt = "There is a girl who likes adventure,"
## INT4
"""and she is always seeking new experiences and pushing her boundaries. She might be drawn to activities like hiking, rock climbing, or even skydiving. She is not afraid to take risks and is always looking for the next adventure to embark on. Her sense of curiosity and excitement for life makes her a true adventurer at heart.
"""
## BF16
"""听起来这个女孩很有探索精神。冒险往往能带来新的体验和成长。她可能会喜欢户外活动，比如徒步旅行、攀岩或者探险。她也可能对历史和文化的探索感兴趣，比如考古、旅行到不同的地方了解当地文化。

冒险精神还可能体现在她对未知的探索上，比如尝试新的事物，学习新的技能，或者挑战自己的极限。这样的性格让她充满活力，总是对世界充满好奇。

当然，冒险并不总是意味着身体上的挑战，有时候，它也可以是对自我认知的探索，比如通过阅读、写作、艺术创作等方式，深入了解自己的内心世界。

这样的女孩，无论她选择什么样的冒险方式，都值得我们尊重和欣赏。因为冒险精神本身就是一种积极向上、勇于探索的生活态度"""

prompt = "Once upon a time,
## INT4
"""Once upon a time, in a faraway land, there was a kingdom ruled by a wise and just king. The kingdom was known for its prosperity, peace, and the beauty of its landscapes. The people of the kingdom lived in harmony with nature, and their lives were filled with joy and contentment.

In the heart of the kingdom, there was a magnificent castle where the king and his family resided. The castle was surrounded by lush gardens, and its walls were adorned with intricate carvings and vibrant murals. The king's wisdom and kindness were reflected in every aspect of the kingdom, from the way the land was cared for to the way the people were treated.

The kingdom was also known for its festivals and celebrations. The most famous of these was the Annual Festival of Lights, which took place every year on the night of the full moon. On this night, the entire kingdom would come together to light up the sky with thousands of lanterns, each carrying a wish or a story from its bearer.

One of the most beloved traditions of the festival was the story-telling contest. Young and old alike would gather in the castle courtyard, eager to listen to the tales of bravery, love, and adventure that were shared by the storytellers. These stories were not just entertainment; they were a way to pass down the wisdom and values of the kingdom from one generation to the next.

As the years went by, the kingdom continued to thrive under the wise rule of the king. The people's lives were filled with happiness and purpose, and the kingdom became a beacon of hope and inspiration for all who knew of its existence.

And so, the story of the kingdom of light and joy continued to be told, not just in the hearts of its people, but in the legends and folklore that were passed down through the ages. It was a tale of a kingdom that had found the secret to a life of peace, prosperity, and happiness, a secret that many had sought but few had found.

And so, the story of the kingdom of light and joy lives on, a reminder of the power of wisdom, kindness, and community to create a world where every heart is filled with joy and every life is filled with purpose.
"""

##BF16
"""Once upon a time, in a far-off land, there was a kingdom ruled by a wise and just king. The kingdom was known for its prosperity, beauty, and the harmony between its people. The king was beloved by all, and his court was filled with wise advisors and loyal knights.

In this kingdom, there was a young girl named Elara. She was kind-hearted, intelligent, and had a curious spirit. Elara spent her days exploring the vast forests and ancient ruins that surrounded her home, learning about the history and magic that filled the land.

One day, while wandering through the forest, Elara stumbled upon an old, enchanted tree. Its branches reached high into the sky, and its leaves shimmered with a soft, ethereal glow. As she approached, a voice spoke to her, belonging to an ancient spirit that had been trapped within the tree for centuries.

The spirit revealed to Elara that the kingdom was in danger. A dark force, known as the Shadow of Eternity, was rising, threatening to consume the land and all its inhabitants. The only way to stop it was to find the three ancient artifacts hidden within the kingdom: the Crystal of Light, the Stone of Time, and the Heart of the Earth.

With the spirit's guidance, Elara set out on a journey to find these artifacts. Along the way, she encountered many challenges and allies. She met a brave knight who had lost his way, a wise old wizard who had knowledge of ancient magic, and a group of mischievous fairies who offered their help in exchange for stories.

Together, they faced treacherous landscapes, battled fierce creatures, and solved complex puzzles. Each artifact they found brought them closer to their goal, but they also learned valuable lessons about courage, friendship, and the true meaning of power.

Finally, after many trials and tribulations, Elara and her companions reached the final resting place of the Shadow of Eternity. With the combined power of the three artifacts, they were able to defeat the dark force and restore peace to the kingdom.

The people of the kingdom celebrated Elara's bravery and the return of their prosperity. The kingdom flourished once again, and Elara became a legend, inspiring generations to come with her tale of adventure and hope.

And so, the story of Elara and her quest to save her kingdom became a cherished legend, passed down through the ages, reminding all who heard it of the power of courage, friendship, and the light that shines even in the darkest of times."""

Evaluate the model

pip3 install lm-eval==0.4.5

auto-round --model "OPEA/internlm2_5-7b-chat-int4-sym-inc" --eval --eval_bs 16  --tasks leaderboard_ifeval,leaderboard_mmlu_pro,gsm8k,lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,cmmlu,ceval-valid

Metric	BF16	INT4
Avg	0.6339	0.6294
leaderboard_mmlu_pro 5 shots	0.3757	0.3768
leaderboard_ifeval strict_acc	0.55905=(0.6079+0.5102)/2	0.5477=(0.6019+0.4935)/2
cmmlu	0.7873	0.7778
ceval-valid	0.7719	0.7645
gsm8k 5 shots	0.6429	0.6103
lambada_openai	0.6798	0.6746
hellaswag	0.6154	0.6060
winogrande	0.7901	0.7916
piqa	0.8014	0.8025
truthfulqa_mc1	0.3880	0.3794
openbookqa	0.3300	0.3400
boolq	0.8869	0.8844
arc_easy	0.7782	0.7875
arc_challenge	0.4650	0.4684

Generate the model

Here is the sample command to generate the model.

bash

auto-round \
--model internlm/internlm2_5-7b-chat \
--device 0 \
--group_size 128 \
--nsamples 512 \
--bits 4 \
--iter 1000 \
--disable_eval \
--model_dtype "fp16" \
--format 'auto_awq,auto_gptq,auto_round' \
--output_dir "./tmp_autoround" 

Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Therefore, before deploying any applications of the model, developers should perform safety testing.

Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

Here are a couple of useful links to learn more about Intel's AI software:

Intel Neural Compressor link

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

Cite

@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }

arxiv github