This model was obtained by quantizing activations and weights of
QwQ-32B to FP8 data type.
This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x).
Weight quantization also reduces disk size requirements by approximately 50%.
Only weights and activations of the linear operators within transformers blocks are quantized.
Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme.
The
llm-compressor library is used for quantization.
This model can be deployed efficiently using the
vLLM backend, as shown in the example below.
1from vllm import LLM, SamplingParams
2from transformers import AutoTokenizer
3
4model_id = "duydq12/QwQ-32B-FP8-dynamic"
5number_gpus = 1
6sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, min_p=0, max_tokens=256)
7
8messages = [
9 {"role": "user", "content": prompt}
10]
11
12tokenizer = AutoTokenizer.from_pretrained(model_id)
13
14messages = [{"role": "user", "content": "Give me a short introduction to large language model."}]
15
16prompts = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
17
18llm = LLM(model=model_id, tensor_parallel_size=number_gpus)
19
20outputs = llm.generate(prompts, sampling_params)
21
22generated_text = outputs[0].outputs[0].text
23print(generated_text)
vLLM aslo supports OpenAI-compatible serving. See the
documentation for more details.
