This model has the token_emd.weight tensor bumped up to q8_0, which increases its download size but makes no difference in VRAM. By default, the token_emd.weight tensor is loaded entirely into system RAM. Upgrading it to q8_0 is therefore, in practice, a free upgrade to the model's quality, for example in interpreting non-English languages.

This repository contains GGUF format weights for the Qwen3.8-27B model, quantized using the ik_llama.cpp project.

This model was specifically created to run on consumer GPUs with 16GB VRAM. By utilizing q4_0 KV cache quantization, it allows pushing the context length up to 110k tokens.

Note: This model is designed exclusively for nVidia GPUs and is based on the advanced KS and KT quants developed by ikawrakow from the ik_llama.cpp repository.

Quantization Details & Imatrix File

Quantization Base: IQ_KS,IQ_KT Quants (ik_llama.cpp).
Imatrix File Used: Mix of mradermacher and ubergarm imatrix files.
Other Tested Imatrix Files:
- bartowski – yielded significantly worse results.
- ubergarm – yielded comparable results.
- If you find or generate a better Imatrix file, please let me know in the Community tab!

GPU Only full 16GB VRAM the configuration below will give you a 105k context (110k is the max) (q4_0/q4_0). Decode speed starts at 45 t/s and falls to 25 t/s at the end, with a prefill speed above 1300 t/s (RTX 5070 Ti).

bash

llama-server" \
        -m "$MODEL_PATH" \
        -a Qwen3.8-27B \
        --ctx-size 105000 \
        --n-gpu-layers 99 \
        --cache-type-k q4_0 \
        --cache-type-v q4_0 \
        --spec-type ngram-mod:n_max=2 \
        --batch-size 512 \
        --ubatch-size 128 \
        --flash-attn on \
        --host 0.0.0.0 \
        --port 8081 \
        --reasoning on \
        --reasoning-format deepseek \
        --reasoning-budget 32000 \
        -t 8 \
        -tb 8 \
        --parallel 1 \
        --metrics \
        --merge-qkv \
        -khad \
        -vhad \
        --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \
        --defrag-thold 0.4 \
        --jinja \
        --cont-batching \
        --temp 1.0 \
        --top-k 20 \
        --min-p 0.00 \
        --top-p 0.95 \
        --presence-penalty 0.0 \
        --repeat-last-n 512 \
        --repeat-penalty 1.0

Perplexity Comparison: mradermacher vs. cHunter789

I compared the model by mradermacher with the model by cHunter789 using the War and Peace text file as a dataset.

Test dataset download:

wget [https://www.gutenberg.org/files/2600/2600-0.txt](https://www.gutenberg.org/files/2600/2600-0.txt) -O pg19.txt

1. Model: IQ4_XS (mradermacher)

Source: mradermacher/Qwen3.8-27B-i1-GGUF

Command

bash

./llama-perplexity \
    -m Qwen3.8-27B.i1-IQ4_XS.gguf \
    -f pg19.txt \
    -c 65536 \
    --chunks 32 \
    -ngl 99 \
    -ctk q4_0 \
    -ctv q4_0 \
    -khad \
    -vhad \
    --merge-qkv \
    -fa 1 \
    -b 512 \
    -ub 128 \
    -t 8

Output

text

perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 77.21 seconds per pass - ETA 15.43 minutes
[1]7.7001,[2]8.0855,[3]8.2723,[4]8.3528,[5]8.5027,[6]8.3485,[7]8.3948,[8]8.3789,[9]8.4068,[10]8.4465,[11]8.4932,[12]8.2955,
Final estimate: PPL over 12 chunks for n_ctx=65536 = 8.2955 +/- 0.03221

llama_print_timings:        load time =    2522.03 ms
llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  750310.64 ms / 786432 tokens (    0.95 ms per token,  1048.14 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time = 1360215.26 ms / 786433 tokens

2. Model: IQ4_KT (cHunter789)

Source: cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF

Command

bash

./llama-perplexity \
    -m Qwen3.8-27B.i1-IQ4_KT-attn_qkv-IQ4_KS.gguf \
    -f pg19.txt \
    -c 65536 \
    --chunks 32 \
    -ngl 99 \
    -ctk q4_0 \
    -ctv q4_0 \
    -khad \
    -vhad \
    --merge-qkv \
    -fa 1 \
    -b 512 \
    -ub 128 \
    -t 8

Output

text

perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 84.14 seconds per pass - ETA 16.82 minutes
[1]7.6895,[2]8.0682,[3]8.2542,[4]8.3333,[5]8.4853,[6]8.3416,[7]8.3890,[8]8.3734,[9]8.3999,[10]8.4386,[11]8.4857,[12]8.2890,
Final estimate: PPL over 12 chunks for n_ctx=65536 = 8.2890 +/- 0.03224

llama_print_timings:        load time =    9369.56 ms
llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  783243.61 ms / 786432 tokens (    1.00 ms per token,  1004.07 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time = 2323646.53 ms / 786433 tokens

1. Reference Model Test ctx-cliff.py Results (Click to expand)

Test with ctx-cliff.py (RTX5070Ti)

bash

llama-server \
        -m "$MODEL_PATH" \
        -a Qwen3.6-27B \
        --ctx-size 110000 \
        --n-gpu-layers 99 \
        --cache-type-k q4_0 \
        --cache-type-v q4_0 \
        --batch-size 512 \
        --ubatch-size 128 \
        --flash-attn on \
        --host 0.0.0.0 \
        --port 8081 \
        --reasoning on \
        --reasoning-format none \
        --reasoning-budget 32000 \
        -t 8 \
        -tb 8 \
        --parallel 1 \
        --metrics \
        --merge-qkv \
        -khad \
        -vhad \
        --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \
        --defrag-thold 0.1 \
        --jinja \
        --cont-batching \
        --temp 1.0 \
        --top-k 20 \
        --min-p 0.00 \
        --top-p 0.95 \
        --presence-penalty 0.0 \
        --repeat-last-n 512 \
        --repeat-penalty 1.00

text

python3 ctx-cliff.py --file tests/code_4M.py --start 2000 --end 109000 --step 2000 --n-predict 512

   ctx | prefill| decode|    MTP|  wall|  status
-------------------------------------------------
| 1021.1 |  46.72|    0/0| 11.4s|      OK
| 1320.9 |  46.06|    0/0| 12.6s|      OK
| 1261.5 |  45.12|    0/0| 13.0s|      OK
| 1293.4 |  44.20|    0/0| 13.2s|      OK
| 1191.9 |  43.36|    0/0| 13.6s|      OK
| 1184.3 |  42.39|    0/0| 14.0s|      OK
| 1228.9 |  41.66|    0/0| 15.9s|      OK
| 1258.8 |  41.16|    0/0| 17.6s|      OK
| 1237.8 |  40.13|    0/0| 20.8s|      OK
| 1058.4 |  39.67|    0/0| 15.0s|      OK
| 1091.4 |  39.22|    0/0| 15.8s|STOP@463
| 1082.3 |  38.15|    0/0| 16.3s|      OK
| 1060.3 |  37.79|    0/0| 18.0s|      OK
| 1058.8 |  37.16|    0/0| 20.7s|      OK
| 1059.6 |  36.81|    0/0| 22.1s|      OK
| 1048.1 |  36.17|    0/0| 24.5s|      OK
| 1046.3 |  35.75|    0/0| 26.2s|      OK
| 1037.3 |  35.22|    0/0| 28.7s|      OK
| 1027.6 |  34.94|    0/0| 30.7s|      OK
| 1014.6 |  34.14|    0/0| 33.8s|      OK
| 1010.3 |  34.00|    0/0| 35.6s|      OK
| 1002.9 |  33.27|    0/0| 32.1s|STOP@316
|  994.5 |  32.53|    0/0| 44.2s|      OK
|  997.5 |  32.83|    0/0| 47.7s|      OK
|  996.4 |  32.06|    0/0| 53.1s|      OK
|  826.8 |  31.97|    0/0| 18.4s|      OK
|  803.1 |  31.43|    0/0| 18.9s|      OK
|  775.9 |  31.16|    0/0| 18.6s|      OK
|  847.3 |  30.54|    0/0| 24.2s|      OK
|  795.5 |  30.38|    0/0| 21.1s|      OK
|  825.7 |  29.95|    0/0| 26.7s|      OK
|  780.5 |  29.67|    0/0| 23.2s|      OK
|  775.2 |  29.08|    0/0| 25.5s|      OK
|  746.5 |  28.85|    0/0| 22.2s|      OK
|  757.5 |  28.73|    0/0| 24.3s|      OK
|  753.8 |  28.45|    0/0| 27.3s|      OK
|  712.6 |  28.20|    0/0| 22.2s|      OK
|  723.3 |  27.72|    0/0| 25.3s|      OK
|  725.8 |  27.43|    0/0| 28.1s|      OK
|  723.9 |  27.09|    0/0| 31.4s|      OK
|  678.9 |  26.81|    0/0| 23.4s|      OK
|  755.7 |  26.48|    0/0| 36.9s|      OK
|  712.9 |  26.14|    0/0| 41.0s|      OK
|  684.5 |  25.90|    0/0| 31.5s|      OK
|  728.9 |  25.66|    0/0| 44.9s|      OK
|  706.3 |  25.30|    0/0| 50.0s|      OK
|  696.8 |  25.12|    0/0| 53.6s|      OK
|  654.1 |  24.84|    0/0| 34.1s|      OK
|  614.2 |  24.56|    0/0| 25.3s|      OK
|  680.1 |  24.27|    0/0| 40.2s|      OK
101183 |  629.1 |  24.14|    0/0| 31.9s|      OK
103237 |  668.2 |  23.83|    0/0| 46.8s|      OK
105209 |  624.7 |  23.64|    0/0| 38.8s|      OK
107265 |  655.6 |  23.37|    0/0| 53.9s|      OK