You can deploy this model and start earning money today!

aisensiy/
Qwen-72B-Chat-GGUF

Name: Qwen-72B-Chat-GGUF
Availability: InStock

template

mit

Views

No views yet

How to convert

First, you need git clone llama.cpp and make it.

Then follow the instrution to generate gguf files.

# convert Qwen HF models to gguf fp16 format
python convert-hf-to-gguf.py --outfile qwen7b-chat-f16.gguf --outtype f16 Qwen-7B-Chat

# quantize the model to 4-bits (using q4_0 method)
./quantize qwen7b-chat-f16.gguf qwen7b-chat-q4_0.gguf q4_0

# chat with Qwen models
./main -m qwen7b-chat-q4_0.gguf -n 512 --color -i -cml -f prompts/chat-with-qwen.txt

Files are split and require joining

Note: HF does not support uploading files larger than 50GB but upload a 41GB file is too hard for me. Therefore I have uploaded the Q4_0 by splitting it of 5GB per file.

To join the files, do the following:

Linux and macOS:

cat qwen72b-chat-q4_0.gguf-split-* >qwen72b-chat-q4_0.gguf && rm qwen72b-chat-q4_0.gguf-split-*

Windows:

copy /B qwen72b-chat-q4_0.gguf-split-aa + qwen72b-chat-q4_0.gguf-split-ab + qwen72b-chat-q4_0.gguf-split-ac + qwen72b-chat-q4_0.gguf-split-ad + qwen72b-chat-q4_0.gguf-split-ae + qwen72b-chat-q4_0.gguf-split-af + qwen72b-chat-q4_0.gguf-split-ag + qwen72b-chat-q4_0.gguf-split-ah qwen72b-chat-q4_0.gguf

aisensiy/Qwen-72B-Chat-GGUF

How to convert

Files are split and require joining

aisensiy/
Qwen-72B-Chat-GGUF