You can deploy this model and start earning money today!

drawais/
Qwen2-0.5B-Instruct-AWQ-INT4

Name: Qwen2-0.5B-Instruct-AWQ-INT4
Availability: InStock

Qwen/Qwen2-0.5B-Instruct

quantized

apache-2.0

text-generation-inference

endpoints_compatible

compressed-tensors

Views

No views yet

Qwen2-0.5B-Instruct-AWQ-INT4

INT4 weight-only quantization of Qwen/Qwen2-0.5B-Instruct.

Qwen 2 0.5B-Instruct in INT4. About 0.5 GB on disk. Runs on any GPU. Even on a phone.

Property	Value
Base model	Qwen/Qwen2-0.5B-Instruct
Quantization	INT4 weight-only
Approx. on-disk size	~0.5 GB
License	Apache License, Version 2.0
Languages	English

Load (vLLM)

bash

vllm serve drawais/Qwen2-0.5B-Instruct-AWQ-INT4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.94

python

from vllm import LLM, SamplingParams
llm = LLM(model="drawais/Qwen2-0.5B-Instruct-AWQ-INT4", max_model_len=32768)
print(llm.generate(["Hello!"], SamplingParams(max_tokens=128))[0].outputs[0].text)

Footprint

~0.5 GB on disk. Recommended VRAM: enough headroom for KV cache.

License & attribution

This artifact is a derivative work of Qwen/Qwen2-0.5B-Instruct, released by its original authors under the Apache License, Version 2.0.

This artifact is distributed under the same license. The full license text is included in LICENSE, and required attribution is in NOTICE.

License text: https://www.apache.org/licenses/LICENSE-2.0 Source model: https://huggingface.co/Qwen/Qwen2-0.5B-Instruct