
Original Model	typhoon-ai/typhoon2.5-qwen3-4b
Architecture	Qwen3 (decoder-only)
Parameters	4B
Context Length	256K tokens
License	Apache 2.0
Tokenizer	Qwen-style (`<\|im_start\|>`, `<\|im_end\|>`, `<tool_call>`, `<tool_response>`)

Available Quantizations

Converter: convert_hf_to_gguf.py from <a href="https://github.com/ggerganov/llama.cpp" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">llama.cpp
Quantizer: llama-quantize
FP16 size: 7.5 GB (not included in this repo)
Source commit: llama.cpp build 9843 (86b94708f)

Quant	Size	BPW	Notes
Q2_K	~1.6 GB	2.96	Smallest, lower quality
Q3_K_S	~1.8 GB	3.41
Q3_K_M	~1.9 GB	3.74
Q3_K_L	~2.1 GB	4.03
Q4_0	~2.2 GB	4.34	Legacy
Q4_1	~2.4 GB	4.78	Legacy
Q4_K_S	~2.2 GB	4.37
Q4_K_M	~2.3 GB	4.58	⭐ Recommended (balanced)
Q5_0	~2.6 GB	5.21	Legacy
Q5_1	~2.8 GB	5.65	Legacy
Q5_K_S	~2.6 GB	5.21
Q5_K_M	~2.7 GB	5.33
Q6_K	~3.1 GB	6.14	Near-perfect quality
Q8_0	~4.0 GB	8.50	Near-FP16 quality
IQ3_S	~1.8 GB	3.44
IQ3_M	~1.8 GB	3.66
IQ4_NL	~2.2 GB	4.50	Non-linear
IQ4_XS	~1.7 GB	4.25	Non-linear

Usage

./llama-cli -m typhoon2.5-qwen3-4b-Q4_K_M.gguf -p "สวัสดีครับ" -n 256

bash

# Create Modelfile:
FROM ./typhoon2.5-qwen3-4b-Q4_K_M.gguf

# Import:
ollama create typhoon2.5-4b:q4_k_m -f Modelfile
ollama run typhoon2.5-4b:q4_k_m