Component	Detail
GPU	NVIDIA RTX 5090 (Blackwell)
OS	Windows
Runtime	`llama-server` (main branch)
Conversion tooling	`llama.cpp` PR branch `pr-21095`
Context window	262,144 tokens
Generation speed	~75-100+ tokens/sec

Conversion Workflow

1. Checkout the required `llama.cpp` branch

bash

cd .\llama.cpp
git fetch origin pull/21095/head:pr-21095
git checkout pr-21095
python -m pip install --no-cache-dir -r requirements\requirements-convert_hf_to_gguf.txt

2. Patch the tokenizer configuration

The upstream tokenizer configuration required a compatibility patch before conversion:

powershell

(Get-Content <model_dir>\tokenizer_config.json) `
  -replace '"tokenizer_class": "TokenizersBackend"', '"tokenizer_class": "Qwen2Tokenizer"' `
  | Set-Content <model_dir>\tokenizer_config.json

3. Convert the model

python convert_hf_to_gguf.py <model_dir> --outfile <output.gguf> --outtype bf16 --verbose

4. Transplant MTP blocks

As the MTP blocks were missing after conversion, used this gist to transplant MTP blocks from stevelikesrhino/Qwen3.6-27B-Q8-NVFP4-MTP-GGUF.

`llama-server` Settings

bash

./llama-server ^
    --model ../models/llm/jdermody/Qwen3.6-27B-Text-NVFP4-MTP.gguf ^
    --alias "jdermody/Qwen3.6-27B-Text-NVFP4-MTP" ^
    --ctx-size 262144 ^
    --temp 0.6 ^
    --top-p 0.95 ^
    --min-p 0.00 ^
    --top-k 20 ^
    --port 8001 ^
    --presence-penalty 0.0 ^
    --repeat-penalty 1.0 ^
    --flash-attn on ^
    --jinja ^
    --batch-size 1024 ^
    --ubatch-size 512 ^
    --parallel 1 ^
    --cache-type-k q8_0 ^
    --cache-type-v q8_0 ^
    --spec-type draft-mtp ^
    --spec-draft-n-max 4 ^
    --spec-draft-type-k q4_0 ^
    --spec-draft-type-v q4_0

jdermody/Qwen3.6-27B-Text-NVFP4-MTP

Use Cases

Compatibility

Conversion Workflow

1. Checkout the required llama.cpp branch

2. Patch the tokenizer configuration

3. Convert the model

4. Transplant MTP blocks

llama-server Settings

jdermody/
Qwen3.6-27B-Text-NVFP4-MTP

1. Checkout the required `llama.cpp` branch

`llama-server` Settings