NVFP4 · Experimental

Smallest build, but lower fidelity and a custom CUDA backend are required. Not for stock llama.cpp.

171 MiB
Download NVFP4

This repository contains LoRA adapters only. Download a compatible Qwen3.6-27B base model separately. Testing used Ternary-Bonsai-27B-Q2_0.gguf.

Quick start

bash

hf download prism-ml/Ternary-Bonsai-27B-gguf \
  Ternary-Bonsai-27B-Q2_0.gguf --local-dir .

hf download ajh-code/Ternary-Bonsai-27B-Grug-LoRA-GGUF \
  Grug-LoRA-r64-Q8_0.gguf --local-dir .

llama-server \
  -m Ternary-Bonsai-27B-Q2_0.gguf \
  --lora-scaled Grug-LoRA-r64-Q8_0.gguf:1.5 \
  -ngl 99 -c 32768

Scale 1.5 is recommended for Ternary Bonsai. It compensates for the changed ternary base and produced the best completion reliability and token efficiency in our calibration. Scale 1.0 matches the source-adapter baseline, but was too weak on this base.

Measured on RTX 5080

Adapter	Single stream	DSpark, high acceptance	Batch 32 aggregate	Peak VRAM
F16	65.1 tok/s	158.8 tok/s	299.7 tok/s	8,218 MiB
Q8_0	73.9 tok/s	152.4 tok/s	290.4 tok/s	7,868 MiB
NVFP4	71.6 tok/s	154.6 tok/s	285.3 tok/s	7,716 MiB

These are adapter comparisons, not universal speed claims. DSpark used a separate RTX 5060 Ti drafter; acceptance rate strongly affects its speed.

Compatibility and limits

F16 and Q8_0 use standard GGUF LoRA tensor formats.
NVFP4 is a research artifact for a custom Blackwell CUDA path and is not compatible with stock llama.cpp. It is included for research and archival comparison while the companion backend remains unpublished.
Short quality checks passed in all three formats. Q8_0 is the recommended release; NVFP4 showed a long-form completion failure and remains experimental.
Apache-2.0. This conversion is derived from the linked Grug projects and is tested with, but not created by or affiliated with, Prism ML.

What was tested?

Single-stream decoding, split-GPU DSpark, real concurrent batches through llama-server, deterministic behavior checks, and two preallocated 134,144-token KVarN K4/V4 slots. Full methods and raw results are maintained in the companion research project.