Views
No views yet
| File | Type | Size | Notes |
|---|---|---|---|
Qwen3.5-text-4B-Q8_0.gguf | Q8_0 | ~53% of f16 | near-lossless — for high-quality inference |
Qwen3.5-text-4B-Q6_K.gguf | Q6_K | ~41% of f16 | excellent quality, good balance with f16 |
Qwen3.5-text-4B-Q5_K_M.gguf | Q5_K_M | ~37% of f16 | very good quality, smaller than Q6 |
Qwen3.5-text-4B-Q4_K_M.gguf | Q4_K_M | ~31% of f16 | ✅ recommended — best size/quality balance |
Qwen3.5-text-4B-Q4_K_S.gguf | Q4_K_S | ~30% of f16 | optional — slightly smaller, slightly lower quality |
| Model | Type | Base model |
|---|---|---|
| Qwen/Qwen3.5-4B | f16 · VLM · source | — |
| techwithsergiu/Qwen3.5-4B-bnb-4bit | BNB NF4 · VLM | Qwen/Qwen3.5-4B |
| techwithsergiu/Qwen3.5-text-4B | bf16 · text-only | Qwen/Qwen3.5-4B |
| techwithsergiu/Qwen3.5-text-4B-bnb-4bit | BNB NF4 · text-only | Qwen3.5-text-4B |
| techwithsergiu/Qwen3.5-text-4B-GGUF | GGUF quants | Qwen3.5-text-4B |
base_model points to the f16 text variant to keep the VLM and text lineages
distinct on the Hub.1./llama.cpp/build/bin/llama-cli \
2 -m Qwen3.5-text-4B-Q4_K_M.gguf \
3 -p "What is the capital of Romania?" \
4 -n 256<think> block before the answer.
Thinking is enabled by default in llama.cpp.--chat-template-kwargs '{"enable_thinking":...}' is deprecated — do not use.
Known issue: --reasoning off is accepted but does not actually disable thinking.
Workaround: use --reasoning-budget 0 — this reliably disables the <think> block.
Track the bug at llama.cpp issues.1# Thinking OFF — direct answer (workaround: --reasoning-budget 0)
2./llama.cpp/build/bin/llama-cli \
3 -m Qwen3.5-text-4B-Q4_K_M.gguf \
4 --reasoning-budget 0 \
5 -p "What is the capital of Romania?" \
6 -n 256
7
8# Thinking ON — default, no flag needed
9./llama.cpp/build/bin/llama-cli \
10 -m Qwen3.5-text-4B-Q4_K_M.gguf \
11 -p "What is 17 × 34?" \
12 -n 1024
1# 1. Merge adapter into f16 weights
2qlora-merge \
3 --base Qwen/Qwen3.5-4B \
4 --adapter adapters/<run_name> \
5 --output merged/qwen35-text-4B-sft-f16
6
7# 2. Convert merged model to GGUF (requires llama.cpp)
8python llama.cpp/convert_hf_to_gguf.py merged/qwen35-text-4B-sft-f16 \
9 --outtype f16 \
10 --outfile merged/qwen35-text-4B-sft-F16.gguf
11
12# 3. Quantize
13./llama.cpp/build/bin/llama-quantize \
14 merged/qwen35-text-4B-sft-F16.gguf \
15 merged/qwen35-text-4B-sft-Q4_K_M.gguf \
16 Q4_K_M1@misc{qwen3.5,
2 title = {{Qwen3.5}: Towards Native Multimodal Agents},
3 author = {{Qwen Team}},
4 month = {February},
5 year = {2026},
6 url = {https://qwen.ai/blog?id=qwen3.5}
7}