Qwen3.5-9B GGUF
Community-made GGUF conversions and quantizations of Qwen3.5-9B, intended for local inference with GGUF-compatible software.
The original model was converted from its Hugging Face / SafeTensors distribution to GGUF using tools provided by
llama.cpp.
This is an unofficial community conversion.
The Qwen model, architecture, and original weights were developed and released by the Qwen Team at Alibaba. This repository provides converted and quantized GGUF files derived from the original model.
Original Model
- Model:
Qwen/Qwen3.5-9B
- Developer: Qwen Team / Alibaba
- Original format: SafeTensors
- License: Apache License 2.0
- Original model: https://huggingface.co/Qwen/Qwen3.5-9B
Refer to the original model repository for the authoritative model card, capabilities, limitations, usage information, and license terms.
Available GGUF Quantizations
This repository provides the following quantizations:
| Quantization | Description |
|---|
Q4_K_M | Lower storage and memory requirements. Suitable as a general-purpose local inference option. |
Q5_K_M | Balanced option with additional weight precision compared with Q4_K_M. |
Q6_K | Higher-precision quantization for systems with more available memory. |
Q8_0 | High-precision quantization with substantially larger memory and storage requirements. |
Actual memory consumption is higher than the GGUF file size and depends on factors such as context length, KV cache configuration, inference backend, and runtime settings.
Environment
The conversion workflow and Jupyter Notebook were developed and tested using the Jupyter Notebook environment provided by Vast.ai.
The notebook is not inherently limited to Vast.ai. It is open source and can be adapted to other Jupyter-compatible environments, local Linux systems, cloud instances, or other platforms capable of running the required Python dependencies and compiling llama.cpp.
Environment-specific paths, storage configuration, package installation, and hardware settings may require adjustment when running outside Vast.ai.
The tested Vast.ai workflow should therefore be treated as the reference environment, not as a platform requirement.
Conversion Pipeline
The files in this repository were produced using a reproducible workflow based on llama.cpp:
1Qwen3.5-9B
2 │
3 │ SafeTensors
4 ▼
5convert_hf_to_gguf.py
6 │
7 ▼
8Intermediate GGUF
9 │
10 │ llama-quantize
11 ▼
12┌────────┬────────┬───────┬──────┐
13│Q4_K_M │Q5_K_M │ Q6_K │ Q8_0 │
14└────────┴────────┴───────┴──────┘
No additional training or fine-tuning is performed as part of this conversion process.
Quantization changes the numerical representation of the model weights to reduce storage and memory requirements and may affect model quality.
Conversion Workflow & Jupyter Notebook
The Jupyter Notebook used to produce these GGUF files is maintained separately on GitHub:
The repository contains the documented workflow required to reproduce the SafeTensors → GGUF → quantization process.
The notebook was tested on Vast.ai but can be modified for other compatible environments. Platform-specific configuration may need to be adjusted accordingly.
The .ipynb is maintained on GitHub rather than bundled with the model weights in this Hugging Face repository.
Usage
These GGUF files are intended for applications and inference engines with compatible GGUF support, particularly llama.cpp and software built around it.
Example with llama.cpp:
1llama-cli \
2 -m Qwen3.5-9B-Q5_K_M.gguf \
3 -p "Explain how GGUF quantization works."
Runtime parameters should be adjusted according to your hardware, available memory, and desired context length.
Compatibility
GGUF compatibility depends on the version of llama.cpp and its support for the underlying model architecture.
Because both llama.cpp and GGUF continue to evolve, older inference engines may not correctly load files produced by newer versions.
The conversion notebook may also require modifications when used with model architectures other than the one documented and tested by this repository.
If you encounter GGUF compatibility problems, first test with a recent version of llama.cpp or your preferred GGUF runtime.
Reproducibility
The public conversion workflow documents relevant information such as:
- source model;
- Python environment;
- conversion procedure;
llama.cpp revision/commit;- intermediate GGUF generation;
- quantization procedure;
- output quantizations.
See the linked GitHub repository for the conversion implementation and notebook documentation.
Credits
Qwen / Alibaba
The original Qwen3.5-9B model, architecture, and model weights were developed and released by the Qwen Team at Alibaba.
This repository would not exist without their work and release of the original model.
All credit for the original model belongs to its respective authors and contributors.
llama.cpp
GGUF conversion and quantization are performed using tools from the open-source llama.cpp project.
This workflow relies on tooling including:
convert_hf_to_gguf.pyllama-quantize- GGUF infrastructure provided by the project
Project:
Credit belongs to the llama.cpp maintainers and contributors for the conversion, quantization, GGUF, and local inference tooling used by this workflow.
License
The original Qwen3.5-9B model is distributed under the Apache License 2.0.
These files are converted and quantized derivatives of the original model weights and retain the applicable licensing terms of the original model.
The separately maintained conversion notebook and its source code may have its own repository license. Refer to the linked GitHub repository for the licensing terms applicable to that code.
Please review the original Qwen3.5-9B repository and its license before using or redistributing these files.
Disclaimer
This repository is an unofficial community conversion and is not affiliated with, endorsed by, or maintained by Alibaba, the Qwen Team, Vast.ai, or the llama.cpp project.
Vast.ai was used as the environment in which the conversion workflow was tested. Its use does not imply affiliation, endorsement, or a technical requirement to use Vast.ai.
The purpose of this repository is to provide GGUF variants of the original openly released model for local inference while documenting and crediting the upstream projects used to create them.
