Base Model: Qwen3-4B
Quantization: Q4_K_M (4-bit quantization, medium quality)
Format: GGUF
Library: Compatible with llama.cpp and other GGUF-compatible inference engines

Usage

With llama.cpp

bash

# Install llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# Run inference
./main -m Qwen3-4B-Q4_K_M.gguf -p "Your prompt here" -n 128

With Python (llama-cpp-python)

python

from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="Qwen3-4B-Q4_K_M.gguf",
    n_ctx=2048,
    n_threads=8
)

# Generate text
output = llm("Your prompt here", max_tokens=128)
print(output['choices'][0]['text'])

With ctransformers

python

from ctransformers import AutoModelForCausalLM

# Load model
llm = AutoModelForCausalLM.from_pretrained(
    "Pinsara/Qwen3-4B-Q4_K_M-GGUF",
    model_file="Qwen3-4B-Q4_K_M.gguf",
    model_type="qwen2"
)

# Generate text
print(llm("Your prompt here"))

Quantization Details

This model uses Q4_K_M quantization:

4-bit quantization for most weights
Medium quality/size tradeoff
Good balance between model size and performance
Recommended for most use cases

Model Size

The quantized model is approximately 2.5-3GB (vs ~8GB for the original FP16 model).

License

This model inherits the Apache 2.0 license from the base Qwen model.