What is this model?
This model is intended to strip all other unused languaged and only support Thai, English.
By using generated iMatrix data, I have managed to keep good part of this model and only using 2-4 bit range.
The quant used are mostly IQ quants, this will compress better with provided IMatrix.
Who is it for?
By design the model is is around 8GB to 10GB that will fit A GPU size of 12GB much better.
Smallest Quant IQ2_M with 64K Context at Q4_1 will comsume around 11.6GB of VRAM.
What was calibrated using iMatrix?
- Tool, Math, Code and Language
- Language calibration: Thai, English
What quantization provided:
| Quant | Size | Mixture | Recommendation |
|---|
| IQ3_XS | 10.517 GiB (3.307 BPW) | Q8_0,Q6_K,IQ4_XS,IQ3_S,IQ3_S,IQ2_S | Solid embd/output, Good Math/Tool Use |
| IQ3_XXS_F | 10.08 GiB (3.17 BPW) | IQ4_XS,IQ3_S,IQ3_XXS,IQ2_S,IQ2_XS | Fast, avg Math/Tool Use |
| IQ3_XXS | 9.488 GiB (2.983 BPW) | Q8_0,Q6_K,IQ4_XS,IQ3_XXS,IQ2_XS | Solid embd/output, Reduced ffn*,attn* |
| IQ2_M | 8.564 GiB (2.693 BPW) | IQ4_XS,IQ3_XXS,IQ2_XXS | Fast, Slightly inaccurate serious use |
Side Models provided:
MTP is quantized generic from
gglm.
So Speed will gain mostly on non-Thai Prompts.
MMPROJ is Quantized from Official Qwen3.8-27B, this will save and speed up your use.
| Type | Quant | Size |
|---|
| MTP | IQ4_M | 1607.38 MiB (4.54 BPW) |
| MMRPOJ | Q8_0 | 601 MiB |
| IMATRIX | - | 14 MiB |
Recommendation:
IQ2_M/IQ3_XS_F Will gain the best speed when using ik_llama --repack
GGUF-Bench
bfcl: tool and agentic use capability
bigcodebench: coding capability
musr: reasoning
| Quant | bfcl | bigcodebench | musr |
|---|
| UD-Q4_K_M | 0.708 | 0.604 | 0.72 |
| UD-IQ3_XXS | 0.687 | 0.599 | 0.724 |
| IQ3_XS | 0.693 | 0.567 | 0.716 |
| IQ3_XXS_F | 0.669 | 0.588 | 0.668 |
| IQ3_XXS | 0.681 | 0.537 | 0.644 |
| IQ2_M | 0.615 | 0.408 | 0.672 |
Example Runs:
Use Temp settings same as unsloth's.
Use MTP with Ngram-Mod to speed up yout TG and PP.
If you build llama with CUDA you can mix differnt KV quants using -DGGML_CUDA_FA_ALL_QUANTS.
llama-server \
-t 4 -dev CUDA0,CUDA1 \
--reasoning off --reasoning-budget 0 --reasoning-format deepseek \
--temp 0.7 --top-p 0.8 --top-k 20 --presence-penalty 1.5 --min-p 0.0 \
-b 4096 -ub 2048 \
--jinja --alias Qwen/Qwen3.8-27B -sm tensor \
-m Qwen3.8-27B-padthai-IQ2_M.gguf \
-mm mmproj-Q8_0.gguf \
-md mtp-Qwen3.8-27B-padthai-IQ4_XS.gguf \
--spec-type draft-mtp,ngram-mod --spec-draft-n-max 2 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 0 --spec-ngram-mod-n-max 128 \
-c 64000 -ctk q8_0 -ctv q5_1 -ngl 99 -ngld 99 \
--cache-ram 1000 \
-ts 1,1 -fa on -fit off \
--host 0.0.0.0 --timeout 3600
