Modification notice (Apache-2.0): these files are a GGUF quantization of the original model — no weights were retrained or fine-tuned. Original model © SCB 10X, released under Apache-2.0. This repository preserves that license.

Files

File	Quant	Size	Notes
`typhoon2-qwen2.5-7b-instruct-Q4_K_M.gguf`	Q4_K_M	~4.4 GB	recommended — best size/quality for CPU & 16 GB RAM
`typhoon2-qwen2.5-7b-instruct-Q5_K_M.gguf`	Q5_K_M	~5.1 GB	a bit higher quality
`typhoon2-qwen2.5-7b-instruct-Q8_0.gguf`	Q8_0	~7.6 GB	near-lossless

K-quants are used (best for CPU inference). For a 7B on ~16 GB RAM, Q4_K_M is the sweet spot.

Usage

Ollama

ollama run hf.co/CaldarianX/typhoon2-qwen2.5-7b-instruct-GGUF:Q4_K_M

llama.cpp

bash

# chat
./llama-cli -m typhoon2-qwen2.5-7b-instruct-Q4_K_M.gguf -cnv -c 8192
# OpenAI-compatible server (use with Aider, Open WebUI, scripts, etc.)
./llama-server -m typhoon2-qwen2.5-7b-instruct-Q4_K_M.gguf -c 8192

LM Studio

Search this repo, download the Q4_K_M file, load, chat.

Prompt format

Qwen2.5 ChatML:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
สวัสดีครับ<|im_end|>
<|im_start|>assistant

Credits & License

Paper: Typhoon 2 (arXiv:2412.13702)
License: Apache-2.0 (inherited from the base model)
Quantization: llama.cpp · uploaded by CaldarianX