[!WARNING] Experimental community conversion

This repository contains an experimental GGUF conversion of unsloth/Qwen3.6-27B-NVFP4.

Upstream llama.cpp added compressed-tensors NVFP4 conversion support in PR #21095. However, this source checkpoint uses a mixed-precision configuration containing multiple quantization groups, including NVFP4-packed weights, FP8 tensors, and unquantized MTP tensors.

This specific GGUF was produced with a local experimental extension for that mixed layout. It has been tested only with the hardware and llama.cpp builds listed below. Compatibility, numerical behavior, and performance may differ across converter revisions, backends, devices, and future llama.cpp versions. Verify output quality before relying on it.

Current local testing does not show a clear generation-speed advantage over Q4_K_M. Treat this repository primarily as an experimental compatibility and research conversion rather than a performance-oriented replacement for mature K-quant GGUF files.

This is a community GGUF conversion for use with llama.cpp.

It is not a fully NVFP4 model. The resulting GGUF uses a mixed layout:

compatible MLP tensors are preserved as native GGUF NVFP4
source FP8 tensors are dequantized and stored as Q8_0
embedded MTP tensors remain in higher precision
multimodal and vision tensors are excluded from the text-model GGUF

Model Details

Source checkpoint: unsloth/Qwen3.6-27B-NVFP4
Base model: Qwen/Qwen3.6-27B
Format: GGUF
Architecture: Qwen3.6 27B dense model
Quantization layout: experimental mixed NVFP4 + Q8_0 + BF16
Runtime: llama.cpp
Purpose: experimental local inference, mixed NVFP4 GGUF compatibility, Blackwell kernel testing, and Qwen3.6 MTP speculative decoding

The source checkpoint uses mixed-precision compressed-tensors groups:

NVFP4-packed MLP weights
FP8 attention, linear-attention projections, selected MLP tensors, and lm_head
unquantized MTP tensors

For the GGUF uploaded to this repository, the historical local converter extension handled the stored layout as follows:

compatible NVFP4-packed tensors were repacked into GGUF NVFP4 blocks
supported FP8 tensors were dequantized with their scales and converted to Q8_0 with --fp8-as-q8
tensors without compressed scales were exported using their source floating-point precision

Upstream llama.cpp Support

Two separate llama.cpp changes are relevant:

PR #21095 adds compressed-tensors NVFP4 conversion support. It normalizes compressed-tensors metadata and scale conventions into llama.cpp's existing NVFP4 conversion path.
PR #22196 adds native Blackwell NVFP4 CUDA execution support for applicable matrix operations.

These changes solve different parts of the pipeline:

text

compressed-tensors NVFP4 checkpoint
                |
                | PR #21095: conversion support
                v
             NVFP4 GGUF
                |
                | PR #22196: Blackwell CUDA kernels
                v
            llama.cpp runtime

Support for loading or executing NVFP4 does not guarantee that every operator, tensor shape, or token-generation path uses the fastest available FP4 Tensor Core kernel. End-to-end performance still depends on tensor type, operator shape, kernel dispatch, batch size, device placement, and the llama.cpp revision.

Current Status

The model loads and runs with a recent native sm_120a llama.cpp CUDA build.

On the tested RTX 5070 Ti + RTX 5060 Ti dual-GPU system, the conversion is functional and performs within a few percent of the tested NVIDIA Qwen3.6-27B-NVFP4-GGUF.

The current mixed NVFP4 GGUF does not provide a clear generation-speed advantage over the mature Q4_K_M path.

Latest mixed-task results:

NVIDIA NVFP4 was about 3% faster than this experimental conversion
Q4_K_M was about 5% faster than NVIDIA NVFP4
Q4_K_M was about 8% faster than this experimental conversion in average generation throughput

This conversion is currently most useful for:

NVFP4 compatibility testing
Blackwell CUDA-kernel experimentation
mixed compressed-tensors conversion research
future llama.cpp NVFP4 regression and improvement comparisons

Important Limitations

This conversion is experimental and is not an official Unsloth, Qwen, NVIDIA, or llama.cpp release.

Known limitations:

the uploaded GGUF was created using a historical local extension targeting the specific mixed FP8 + NVFP4 layout used by this checkpoint
upstream compressed-tensors NVFP4 support does not necessarily imply that every mixed multi-group checkpoint is converted identically
the GGUF is not bit-identical to the source checkpoint
FP8 tensors are converted to Q8_0, not preserved as native FP8
embedded MTP tensors remain in source floating-point precision
output quality and performance may differ from vLLM, SGLang, TensorRT-LLM, or other runtimes
successful loading does not guarantee identical numerical behavior
native sm_120a code does not prove that every NVFP4 operation uses an optimal FP4 Tensor Core kernel
batch-1 token generation may not benefit as much as prompt processing or larger matrix workloads
future converter, kernel, and dispatch changes may alter compatibility and performance

Compatibility

A recent llama.cpp build with support for the following is required:

Qwen3.6
GGUF NVFP4 tensors
native MTP speculative decoding
external MTP draft models

Tested with:

Windows 11
NVIDIA GeForce RTX 5070 Ti 16 GB
NVIDIA GeForce RTX 5060 Ti 16 GB
llama.cpp b9967-4f37f5197
locally compiled static CUDA build
CUDA architecture: native sm_120a
CUDA 13.0 toolchain
dual-GPU layer split
--tensor-split 14,12
external Q4_K_M MTP-only drafter on CUDA1
--spec-draft-n-max 4
--spec-draft-p-min 0.60

The static CUDA build was inspected with cuobjdump and contained native sm_120a cubins. This confirms that the benchmark did not rely only on older architecture cubins or PTX fallback.

Native sm_120a code targets the correct GPU architecture, but it does not prove that every NVFP4 operation uses an optimal FP4 Tensor Core kernel. Runtime performance still depends on tensor type, operator shape, kernel dispatch, device placement, and llama.cpp implementation details.

Recommended Usage

The embedded MTP tensors are usable, but on the tested dual-GPU system an external Q4_K_M MTP-only drafter was substantially faster while preserving almost the same aggregate acceptance rate.

Recommended: external Q4_K_M MTP drafter

bat

llama-server.exe ^
  -m "unsloth-Qwen3.6-27B-NVFP4.gguf" ^
  -md "Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf" ^
  --device-draft CUDA1 ^
  --host 127.0.0.1 ^
  --port 18010 ^
  -np 1 ^
  --threads 12 ^
  --threads-batch 16 ^
  --split-mode layer ^
  --tensor-split 14,12 ^
  --ctx-size 81920 ^
  --no-mmap ^
  -ngl -1 ^
  --flash-attn on ^
  --jinja ^
  --ubatch-size 512 ^
  --batch-size 2048 ^
  --no-warmup ^
  --temp 0.6 ^
  --top-k 20 ^
  --top-p 0.80 ^
  --min-p 0.0 ^
  --presence-penalty 1.5 ^
  --repeat-penalty 1.0 ^
  --fit off ^
  --reasoning off ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 4 ^
  --spec-draft-p-min 0.60

Embedded native MTP

bat

llama-server.exe ^
  -m "unsloth-Qwen3.6-27B-NVFP4.gguf" ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 4 ^
  --spec-draft-p-min 0.60 ^
  --flash-attn on ^
  -ngl -1

The embedded path may consume more bandwidth because the MTP tensors remain at higher precision. Performance also depends on how llama.cpp places the embedded MTP tensors across multiple GPUs.

Conversion

At the time this GGUF was created, the upstream llama.cpp converter stopped on this checkpoint with:

NotImplementedError: Can't handle multiple config groups for compressed-tensors yet

Upstream llama.cpp later added compressed-tensors NVFP4 conversion support in PR #21095. That support normalizes compressed-tensors NVFP4 metadata and scales into the existing ModelOpt-compatible conversion path.

This checkpoint additionally contains a mixed FP8 + NVFP4 configuration and unquantized MTP tensors. The historical local extension used for the uploaded file specifically handled that mixed layout:

compatible nvfp4-pack-quantized tensors were exported as native GGUF NVFP4
supported FP8 tensors were dequantized and stored as Q8_0 with --fp8-as-q8
tensors without compressed scales, including embedded MTP tensors, retained their source floating-point precision

Users converting the model again should use a recent upstream llama.cpp revision and verify the resulting tensor layout. Exact output may differ from the historical local conversion used to produce this repository.

Example conversion command:

bash

python convert_hf_to_gguf.py \
  /path/to/unsloth-Qwen3.6-27B-NVFP4 \
  --outfile unsloth-Qwen3.6-27B-NVFP4.gguf \
  --outtype auto \
  --verbose

Expected mixed layout:

text

NVFP4-packed compatible tensors -> native GGUF NVFP4
FP8 tensors                      -> dequantized and stored as Q8_0
embedded MTP tensors             -> BF16 / source floating-point precision

No neural-network weights were retrained.

Verifying the GGUF

Inspect the result with a recent llama-gguf build:

.\llama-gguf.exe .\unsloth-Qwen3.6-27B-NVFP4.gguf r

Expected tensor examples:

text

tensor: name = blk.0.ffn_down.weight,       type = nvfp4
tensor: name = blk.0.ffn_down.scale,        type = f32
tensor: name = blk.0.ffn_down.input_scale,  type = f32

The file should also contain Q8_0 tensors corresponding to source FP8 weights and floating-point tensors corresponding to the embedded MTP layers.

MTP Server Benchmark

The earlier server-based mtp-bench workload primarily compared embedded and external MTP execution on this experimental GGUF.

Aggregate Results

Main model	Drafter	Draft precision	Acceptance	Wall time	Aggregate throughput
Unsloth experimental GGUF	embedded MTP	BF16 / source precision	87.37%	57.46 s	32.77 tok/s
Unsloth experimental GGUF	external MTP-only	Q4_K_M	86.88%	42.12 s	44.71 tok/s
NVIDIA NVFP4 GGUF comparison	native MTP	source precision	76.55%	46.00 s	40.37 tok/s

Aggregate throughput is total predicted tokens divided by total wall time.

The NVIDIA row is only a local comparison. Different conversion layouts, quantization choices, embedded MTP weights, and runtime behavior prevent it from being a controlled model-quality comparison.

External Q4_K_M MTP Results

text

code_python        pred= 192 draft= 161 acc= 142 rate=0.882 tok/s=56.7
code_cpp           pred=  53 draft=  47 acc=  41 rate=0.872 tok/s=59.8
explain_concept    pred= 192 draft= 134 acc= 103 rate=0.769 tok/s=34.0
summarize          pred=  48 draft=  42 acc=  33 rate=0.786 tok/s=47.3
qa_factual         pred= 192 draft= 154 acc= 127 rate=0.825 tok/s=44.4
translation        pred=  17 draft=  17 acc=  13 rate=0.765 tok/s=40.3
creative_short     pred=  37 draft=  19 acc=  18 rate=0.947 tok/s=29.0
stepwise_math      pred= 192 draft= 158 acc= 140 rate=0.886 tok/s=54.5
json_output        pred= 192 draft= 155 acc= 149 rate=0.961 tok/s=67.8
long_reasoning     pred= 192 draft= 155 acc= 132 rate=0.852 tok/s=47.9
repeat_pattern     pred= 192 draft= 152 acc= 152 rate=1.000 tok/s=73.8
code_completion    pred= 192 draft= 166 acc= 145 rate=0.874 tok/s=62.4
long_code_review   pred= 192 draft= 141 acc= 109 rate=0.773 tok/s=35.7

Aggregate:
  requests:            13
  predicted tokens:    1883
  draft tokens:        1501
  accepted tokens:     1304
  acceptance rate:     0.8688
  total wall time:     42.12 s

Embedded MTP Results

text

code_python        pred= 192 draft= 159 acc= 140 rate=0.880 tok/s=37.9
code_cpp           pred=  53 draft=  47 acc=  41 rate=0.872 tok/s=39.8
explain_concept    pred= 192 draft= 126 acc= 100 rate=0.794 tok/s=26.5
summarize          pred=  48 draft=  41 acc=  33 rate=0.805 tok/s=34.0
qa_factual         pred= 192 draft= 157 acc= 130 rate=0.828 tok/s=32.7
translation        pred=  17 draft=  18 acc=  13 rate=0.722 tok/s=28.1
creative_short     pred=  37 draft=  20 acc=  19 rate=0.950 tok/s=25.0
stepwise_math      pred= 192 draft= 157 acc= 140 rate=0.892 tok/s=37.9
json_output        pred= 192 draft= 155 acc= 149 rate=0.961 tok/s=45.5
long_reasoning     pred= 192 draft= 158 acc= 132 rate=0.835 tok/s=34.4
repeat_pattern     pred= 192 draft= 155 acc= 152 rate=0.981 tok/s=48.5
code_completion    pred= 192 draft= 159 acc= 149 rate=0.937 tok/s=44.0
long_code_review   pred= 192 draft= 145 acc= 110 rate=0.759 tok/s=28.0

Aggregate:
  requests:            13
  predicted tokens:    1883
  draft tokens:        1497
  accepted tokens:     1308
  acceptance rate:     0.8737
  total wall time:     57.46 s

Native `sm_120a` Mixed-Task Comparison

A second benchmark used a locally compiled native sm_120a build:

text

build: b9967-4f37f5197
main GPU: NVIDIA GeForce RTX 5070 Ti 16 GB
secondary GPU: NVIDIA GeForce RTX 5060 Ti 16 GB
split mode: layer
tensor split: 14,12
external draft model: Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf
draft device: CUDA1
speculative decoding: draft-mtp
draft max: 4
draft probability minimum: 0.60
temperature: 0
context size: 2048
maximum generated tokens: 1024

The workload covered translation, tagging, styled generation, Objective-C, LiveKit, tool-use prompting, KL divergence, Swift concurrency, release-note translation, and knowledge-graph planning.

Average Performance

Main model	Prompt throughput	Generation throughput
Q4_K_M heretic comparison	241.85 tok/s	40.94 tok/s
NVIDIA Qwen3.6-27B NVFP4 GGUF	227.58 tok/s	38.85 tok/s
Unsloth experimental mixed NVFP4 GGUF	220.06 tok/s	37.71 tok/s

Relative results:

NVIDIA NVFP4 was approximately 3.4% faster than the experimental Unsloth conversion in prompt processing
NVIDIA NVFP4 was approximately 3.0% faster in generation
Q4_K_M was approximately 5.9% faster than NVIDIA NVFP4 in prompt processing
Q4_K_M was approximately 5.4% faster than NVIDIA NVFP4 in generation
Q4_K_M was approximately 9.9% faster than the experimental conversion in prompt processing
Q4_K_M was approximately 8.6% faster than the experimental conversion in generation

These results apply only to this dual-GPU configuration and should not be generalized to other GPUs, single-GPU execution, larger batches, or different context lengths.

Per-Prompt Generation Throughput

Prompt category	Q4_K_M	NVIDIA NVFP4	Unsloth mixed NVFP4
Japanese translation	28.0	24.6	25.2
Danbooru tags	42.6	46.8	47.7
Styled weather sentence	32.8	30.1	27.3
Objective-C	47.2	47.2	45.6
LiveKit explanation	35.5	32.2	30.5
Tool-use prompt	40.1	36.3	36.6
KL divergence	43.0	37.2	36.6
Swift concurrency	50.2	48.1	42.2
Release note	49.9	47.7	48.0
Knowledge-graph planning	40.1	38.3	37.4

Values are generation tokens per second.

Q4_K_M was faster in most tasks, while the NVFP4 models occasionally matched or exceeded it on individual prompts. Short single-prompt results are sensitive to output length, token distribution, speculative-decoding acceptance, and runtime scheduling.

Performance Interpretation

The native sm_120a results show that missing native Blackwell cubins were not the only cause of the earlier NVFP4 performance.

Even with verified native sm_120a code:

the experimental mixed NVFP4 model did not outperform Q4_K_M
the NVIDIA NVFP4 comparison also remained slightly slower than Q4_K_M
the NVIDIA version was only modestly faster than this community conversion

Likely contributing factors include:

only part of this GGUF uses native NVFP4 tensors
source FP8 tensors are represented as Q8_0
some tensors remain BF16 or another floating-point type
mixed tensor formats require different kernels
native Blackwell NVFP4 acceleration may benefit prompt processing and larger matrix workloads more than batch-1 token generation
token-by-token decode may be limited by memory bandwidth, kernel-launch overhead, and small matrix shapes rather than peak Tensor Core FLOPS
dual-GPU layer splitting introduces synchronization and peer-transfer costs
the external draft model shares GPU resources with part of the main model
Q4_K_M CUDA kernels and layouts are more mature in llama.cpp
NVFP4 kernel dispatch and operator coverage are still evolving

Native sm_120a support improves compatibility and targets the correct architecture, but it does not guarantee an end-to-end NVFP4 speedup.

Quality Notes

The mixed-task outputs did not show a clear overall quality advantage for any NVFP4 GGUF.

The NVIDIA and experimental Unsloth NVFP4 outputs were generally in the same quality range:

some NVIDIA answers were technically better
some introduced unsupported factual details
both produced plausible-looking but incorrect framework-specific code in some tests
translation and short-form writing quality were broadly similar

The Q4_K_M comparison model also produced both strong answers and confident technical or factual errors.

This is not a controlled quantization-quality comparison because the tested files do not all use identical model weights:

the Q4_K_M comparison uses an uncensored-heretic-v2 variant
the experimental NVFP4 GGUF originates from the Unsloth checkpoint
the NVIDIA NVFP4 GGUF may use a different conversion and quantization path

A valid quality comparison would require:

the same original checkpoint
identical tokenizer and chat template
identical prompts and sampling parameters
deterministic seeds
reference logits from the unquantized model
KL divergence, cross-entropy, perplexity, or token-agreement measurements
repeated runs for non-deterministic decoding

Treat this benchmark primarily as a local runtime and practical-output comparison, not proof that one quantization format has better model quality.

Benchmark Observations

Embedded versus external MTP

On the tested system:

the external Q4_K_M drafter reduced wall time from 57.46 to 42.12 seconds
wall time was approximately 26.7% lower
aggregate throughput rose from about 32.77 to 44.71 tok/s
acceptance changed only slightly, from 87.37% to 86.88%
predictable tasks such as JSON, repeated patterns, code completion, and stepwise reasoning benefited the most

The results suggest that an external low-precision MTP-only draft model can be faster than embedded higher-precision MTP tensors on this dual-GPU setup.

They do not prove that precision alone is responsible. Device placement, synchronization, graph construction, memory bandwidth, and backend behavior may also contribute.

NVFP4 versus Q4_K_M

In the native sm_120a mixed-task test:

experimental mixed NVFP4 averaged 37.71 generation tok/s
NVIDIA NVFP4 averaged 38.85 generation tok/s
Q4_K_M averaged 40.94 generation tok/s

The NVIDIA NVFP4 GGUF was modestly faster than this conversion, indicating that the experimental conversion is reasonably close to the comparison implementation.

Neither NVFP4 model provided a clear end-to-end generation advantage over Q4_K_M on this setup. At the tested llama.cpp revision, Q4_K_M remains the more practical choice when maximum interactive generation speed is the priority.

Reproducibility Notes

Performance is highly dependent on:

llama.cpp commit and CUDA build
GPU model and PCIe topology
tensor split
context size
KV-cache type
flash-attention support
sampling parameters
draft length and probability threshold
embedded versus external draft model
draft-device placement

The MTP benchmark does not include a non-speculative baseline and should not be interpreted as a universal MTP speedup ratio.

The ten-prompt CLI comparison was run once per prompt. It is useful for practical A/B testing but is not statistically rigorous.

Short responses are particularly sensitive to:

CUDA graph warm-up
model loading and initialization
GPU clock state
output length
end-of-sequence timing
speculative-decoding acceptance
Windows background activity

For regression testing, run each prompt at least three times and compare median generation throughput. Prompt throughput, generation throughput, and end-to-end output throughput should be treated as separate metrics.

Conclusion

This conversion demonstrates that the mixed FP8 + NVFP4 compressed-tensors checkpoint can be exported into a loadable GGUF while preserving compatible tensors as native NVFP4.

Upstream llama.cpp now supports compressed-tensors NVFP4 conversion through PR #21095, while PR #22196 provides native Blackwell CUDA execution for applicable NVFP4 matrix operations. Those additions do not make this uploaded mixed-layout GGUF fully NVFP4, nor do they guarantee that batch-1 generation uses the fastest possible FP4 path for every operation.

The model is functional and performs within a few percent of the tested NVIDIA NVFP4 GGUF comparison.

Current native sm_120a llama.cpp testing does not show a practical generation-speed advantage over Q4_K_M. On the tested dual-GPU system, Q4_K_M remained approximately 5-9% faster in average generation throughput.

The most effective optimization observed was not the main weight format, but replacing the embedded higher-precision MTP tensors with an external Q4_K_M MTP-only draft model.

Future llama.cpp improvements to small-batch and token-generation NVFP4 kernels, operator coverage, dispatch behavior, converter support, and multi-GPU execution may change these results.

Credits

Qwen Team / Alibaba Cloud — Qwen3.6-27B
Unsloth — mixed-precision NVFP4 source checkpoint
ggml-org — llama.cpp, GGUF, compressed-tensors conversion, NVFP4 inference, and MTP support
NVIDIA — NVFP4 format and related quantization work

License

The source model is distributed under the Apache License 2.0.

Review the upstream unsloth/Qwen3.6-27B-NVFP4 and Qwen/Qwen3.6-27B model cards before redistribution or commercial use.

This repository is an unofficial, experimental community conversion and is not endorsed by Unsloth, Qwen, NVIDIA, or the llama.cpp project.

williamliao/Qwen3.6-27B-Unsloth-NVFP4-Experimental-GGUF

Qwen3.6-27B Unsloth NVFP4 GGUF

Model Details

Upstream llama.cpp Support

Current Status

Important Limitations

Compatibility

Recommended Usage

Recommended: external Q4_K_M MTP drafter

Embedded native MTP

Conversion

Verifying the GGUF

MTP Server Benchmark

Aggregate Results

External Q4_K_M MTP Results

Embedded MTP Results

Native sm_120a Mixed-Task Comparison

Average Performance

Per-Prompt Generation Throughput

Performance Interpretation

Quality Notes

Benchmark Observations

Embedded versus external MTP

NVFP4 versus Q4_K_M

Reproducibility Notes

Conclusion

Credits

License

williamliao/
Qwen3.6-27B-Unsloth-NVFP4-Experimental-GGUF

Native `sm_120a` Mixed-Task Comparison