The models were converted from the original Hugging Face checkpoint into GGUF format and quantized using llama.cpp, making them compatible with local inference engines such as llama.cpp, llama-cpp-python, Ollama, LM Studio, Jan, and Open WebUI.

Original Model

Model: Qwen/Qwen3-1.7B
Developer: Qwen Team
Original Repository: https://huggingface.co/Qwen/Qwen3-1.7B
License: Apache-2.0

This repository contains only GGUF conversions of the original model.

Available Quantizations

Model	Size	Recommended RAM	Description
Qwen3-1.7B-Q3_K_M.gguf	1.07 GB	4–6 GB	Smallest practical quantization
Qwen3-1.7B-Q4_K_M.gguf	1.28 GB	6–8 GB	Recommended for most users
Qwen3-1.7B-Q5_K_M.gguf	1.47 GB	8–10 GB	Better response quality
Qwen3-1.7B-Q6_K.gguf	1.67 GB	10–12 GB	High quality
Qwen3-1.7B-Q8_0.gguf	2.17 GB	12–16 GB	Closest to the original FP16 model

Which Model Should I Download?

Your System	Recommended Model
4–6 GB RAM	Q3_K_M
6–8 GB RAM	⭐ Q4_K_M
8–10 GB RAM	Q5_K_M
10–12 GB RAM	Q6_K
12 GB+ RAM	Q8_0

If you're unsure, download Qwen3-1.7B-Q4_K_M.gguf. It provides the best balance between quality, memory usage, and inference speed.

Download

Download the required .gguf file from the Files and versions section of this repository.

Usage with llama.cpp

bash

llama-cli \
  -m Qwen3-1.7B-Q4_K_M.gguf \
  -p "Explain quantum computing in simple words."

Usage with Python (llama-cpp-python)

Install

pip install llama-cpp-python

Example

python

from llama_cpp import Llama

llm = Llama(
    model_path="Qwen3-1.7B-Q4_K_M.gguf",
    n_ctx=4096,
    n_threads=8,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": "Explain quantum computing in simple words."
        }
    ],
    temperature=0.7,
    max_tokens=256,
)

print(response["choices"][0]["message"]["content"])

Usage with Ollama

Create a file named Modelfile

FROM ./Qwen3-1.7B-Q4_K_M.gguf

Create the model

ollama create qwen3-1.7b-q4 -f Modelfile

Run it

ollama run qwen3-1.7b-q4

Compatible Software

These GGUF models can be used with:

llama.cpp
llama-cpp-python
Ollama
LM Studio
Jan
Open WebUI

Note: GGUF models are not compatible with the Hugging Face transformers library. For Transformers, download the original model from the Qwen repository.

Quantization Workflow

text

Original Hugging Face Model (.safetensors)
                │
                ▼
convert_hf_to_gguf.py
                │
                ▼
FP16 GGUF
                │
                ▼
llama-quantize
                │
                ▼
Q3_K_M
Q4_K_M
Q5_K_M
Q6_K
Q8_0

Credits

Original Model: Qwen Team
Original Repository: https://huggingface.co/Qwen/Qwen3-1.7B
GGUF Conversion & Quantization: Sayujaya Sharma
Quantization Tool: llama.cpp

License

This repository distributes quantized versions of the original Qwen3-1.7B model.

The original model is licensed under Apache-2.0, and this repository follows the same license.

Please refer to the original model repository for complete licensing details.

⭐ If you find this repository useful, consider giving it a ⭐ on Hugging Face.

Sayujaya14/Qwen3-1.7B-GGUF

Qwen3-1.7B GGUF