This repository contains a llama.cpp-ready text-generation GGUF. The source model is a native vision-language model; this GGUF export is text-only and does not include a multimodal projector.

Files

File	Size	Description
`Qwen3.8-27B-Samantha-NVFP4-MTP-BF16.gguf`	19.69 GB	Native NVFP4 main model with the one-layer MTP/NextN head retained in BF16

SHA256:

71744136f18bc053e720e021d29bd3411f2d949a52e4f5d0485e54ca976c4a85

How this GGUF was made

This is a post-training quantization export only: no additional fine-tuning or merge was performed for this repository.

Starting checkpoint: the merged BF16 safetensors from Lathly/Qwen3.8-27B-Samantha.
PTQ: NVIDIA Model Optimizer, using its native nvfp4 configuration.
Calibration: Lathly/samantha-nvfp4-calibration, 512 samples, sequence length 2048, batch size 1.
MTP: the Qwen3.5-family MTP/NextN head was intentionally excluded from NVFP4 quantization and retained in BF16 for speculative-decoding accuracy.
GGUF conversion: NVFP4 tensors were repacked into GGML's native NVFP4 layout, while non-quantized tensors remain BF16/F32 as appropriate.

The completed file contains 400 NVFP4 tensors, 106 BF16 tensors, and 65 total transformer blocks. GGUF metadata records qwen35.nextn_predict_layers = 1.

Running with llama.cpp

Use a recent CUDA build of llama.cpp with native NVFP4 and Qwen3.5/NextN (MTP) support:

bash

llama-server \
  -m Qwen3.8-27B-Samantha-NVFP4-MTP-BF16.gguf \
  -ngl 999 \
  --host 0.0.0.0 --port 8080

NVFP4 acceleration requires NVIDIA Blackwell-class hardware. Other backends may not support this quantization type or may fall back to a slower path.

Validation

The finished GGUF was loaded with a fresh CUDA build of llama.cpp on an NVIDIA RTX PRO 6000 Blackwell GPU. llama.cpp detected ftype: NVFP4, loaded all layers to GPU, and completed a short Norwegian generation smoke test at approximately 75 tokens/s. This is a functional smoke test, not a formal quality or benchmark evaluation.

Original fine-tune

The source is a full merged Samantha persona fine-tune: the LoRA adapter from Lathly/Qwen3.8-27B-Samantha_Uncensored_1.1_LoRA was trained with Unsloth (QLoRA) and merged into Qwen3.8-27B. For training hyperparameters, source usage, and the original model description, see Lathly/Qwen3.8-27B-Samantha.

License, attribution, and disclaimer

Released under Apache-2.0, inheriting from the source model and its parents:

Qwen/Qwen3.8-27B — Apache-2.0, Alibaba Cloud / Qwen Team
Lathly/Qwen3.8-27B-Samantha — merged Samantha fine-tune
digitalpipelines/samantha-1.1-uncensored — source training dataset, Apache-2.0

This is an "uncensored" persona fine-tune intended for open conversational use. It may generate objectionable content. Use responsibly and in accordance with applicable laws and upstream licenses.

Lathly/Qwen3.8-27B-Samantha-NVFP4-GGUF

Qwen3.8-27B-Samantha - NVFP4 GGUF with BF16 MTP