This repository contains GGUF exports of a Qwen 3.5 9B finetune based on unsloth/Qwen3.5-9B, prepared for local inference with llama.cpp-compatible runtimes such as llama.cpp, LM Studio, Jan, Open WebUI backends, and similar tools.

The naming is intentional:

qwen35-9b = base model family
opus46 = primary training source was nohurry/Opus-4.6-Reasoning-3000x-filtered
mix = extra training data was blended in beyond the primary source
i1 = imatrix was used during GGUF quantization

Model Summary

Base model: unsloth/Qwen3.5-9B
Format: GGUF
Intended runtimes: llama.cpp and compatible local UIs
Quantizations in this repo:
- Q4_K_M
- Q8_0
Main goal: stronger local reasoning behavior and more structured assistant outputs than a plain stock GGUF export

Training / Build Notes

This release came from a full train/export pipeline rather than a direct one-step conversion.

Workflow:

LoRA SFT with ms-swift
Base model: unsloth/Qwen3.5-9B
Primary dataset: nohurry/Opus-4.6-Reasoning-3000x-filtered
Extra mixed datasets:
- Salesforce/xlam-function-calling-60k
- OpenAssistant/oasst2
Export path: LoRA merge, GGUF conversion via llama.cpp, imatrix-assisted quantization for the imatrix-tagged output set

This is still a Qwen 3.5 model family export. The added training data changes behavior, not the underlying GGUF architecture family or tokenizer family.

What To Expect

Relative to a plain base-model GGUF, this model was aimed more toward:

stronger reasoning-style responses
better structured assistant behavior
better tool / function-call shaped outputs
stronger instruction following

That does not guarantee it will outperform stock Qwen3.5-9B for every prompt. Treat this as a targeted finetune, not a universal replacement.

Files

This repository contains:

*Q4_K_M.gguf
*Q8_0.gguf

Q4_K_M is the smaller and more practical default for many local setups.

Q8_0 is larger, but may preserve more quality if you have the RAM / VRAM budget.

Benchmark Snapshot

I ran both speed and an initial quality check on a local RTX 4090 machine.

Hardware / runtime:

GPU: NVIDIA GeForce RTX 4090
CPU: AMD Ryzen 9 7900X
llama.cpp build commit: 6729d49
GPU offload: -ngl 99
llama-bench batch settings: n_batch=2048, n_ubatch=512

Quant	Prompt tok/s (512 prompt, 0 gen)	Prompt tok/s (1024 prompt, 0 gen)	Gen tok/s (0 prompt, 128 gen)
`Q4_K_M`	`9838.32`	`9748.59`	`137.57`
`Q8_0`	`9974.62`	`9954.67`	`92.40`

Interpretation:

Q4_K_M is much faster for generation on this hardware.
Q8_0 is slower on decode, but may still be worth trying if you want the larger quant.
These numbers were produced from the actual released GGUF files, not a different local build.

Initial quality snapshot:

Task: gsm8k
Setup: lm-eval-harness using local-completions against llama-server
Quant: Q4_K_M
Tokenizer reference: Qwen/Qwen3-8B
Server context: 8192
Concurrency: 4
Result:
- flexible-extract exact_match = 0.8415
- strict-match exact_match = 0.8400

This is a real full-run GSM8K result, not a limited smoke test. I have not posted a broader multi-task quality table yet.

Prompting

This is still a Qwen-family chat model, so standard chat prompting is appropriate.

Simple example:

text

You are a helpful assistant.

User: Explain what GGUF is and when to use Q4_K_M versus Q8_0.
Assistant:

If your runtime supports a native Qwen chat template, prefer that.

llama.cpp Example

bash

./llama-cli \
  -m ./qwen35-9b-opus46-mix-i1-Q4_K_M.gguf \
  -c 4096 \
  -ngl 99 \
  -p "You are a helpful assistant.\n\nUser: Explain what imatrix means in a GGUF release.\nAssistant:"

Notes

GGUF files are for inference, not further fine-tuning.
Quantization level, context size, prompt format, and runtime all affect quality.
The i1 tag indicates imatrix usage in the quantization flow. It is a quantization/build detail, not a separate base model family.

Limitations

Quantized models can lose quality relative to higher-precision checkpoints.
This model can still hallucinate facts, tools, or arguments.
Function-calling style does not guarantee correctness of produced calls.
Verify important outputs before using them in real workflows.

License

This release follows the license and usage constraints of the base model and the training sources used in the workflow. Review the upstream model and datasets before downstream use:

unsloth/Qwen3.5-9B
nohurry/Opus-4.6-Reasoning-3000x-filtered
Salesforce/xlam-function-calling-60k
OpenAssistant/oasst2

Credits

Base model: Qwen / Unsloth
Training stack: ms-swift
GGUF conversion and quantization: llama.cpp

slyfox1186/qwen35-9b-opus46-mix-i1-GGUF

qwen35-9b-opus46-mix-i1-GGUF