You can deploy this model and start earning money today!

kelexine/
Qwen3.5-0.8B-GGUF

Name: Qwen3.5-0.8B-GGUF
Availability: InStock

Views

No views yet

Qwen3.5-0.8B-GGUF

This repository contains the quantized 8-bit GGUF version of the Qwen/Qwen3.5-0.8B model, compiled using llama.cpp.

Qwen3.5-0.8B is an ultra-compact, multi-lingual foundation model. It features a hybrid architecture combining Gated Delta Networks and standard attention, offering a native context window of 262K tokens and support for 201 languages.

📦 Provided GGUF File

File Name	Size (MB)	Precision	Type	Description
`qwen3.5-0.8b-q8_0.gguf`	~890 MB	Q8_0 (8-bit)	Text LLM	Standard quantized 8-bit text GGUF model.

🚀 Quick Start Guide

Setup & Compilation

To compile llama.cpp locally, run:

bash

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

Run Inference

To download and run inference with this model directly using llama.cpp:

bash

./build/bin/llama-cli \
  -hf Kelexine/Qwen3.5-0.8B-GGUF \
  -p "You are a helpful assistant. Write a short poem about coding."

🐍 Python Inference (llama-cpp-python)

For integration into Python scripts, install the bindings:

pip install llama-cpp-python

python

from llama_cpp import Llama

llm = Llama(
    model_path="qwen3.5-0.8b-q8_0.gguf",
    n_ctx=2048
)

response = llm.create_chat_completion(
    messages=[
        {"role": "user", "content": "Explain quantizing neural networks in simple terms."}
    ]
)
print(response["choices"][0]["message"]["content"])

⚖️ License & Limitations

This model is released under the Apache 2.0 License in accordance with the base weights. Quantization can lead to minor accuracy loss, particularly in highly compressed weights.

kelexine/Qwen3.5-0.8B-GGUF