This GGUF release introduces turbo4 weight quantization — a custom 4-bit quantization scheme for Ornstein3.6-27B-MTP-NSC-ACE-SABER using buun-llama-cpp's turbo types.

What is Turbo4?

Format: 4-bit weight quantization using buun's GGML_TYPE_TURBO4_0 tensor type
Size: 15.99 GB for this 27B MTP model (comparable to Q4_K_M ~16 GB)
Accuracy: Correct inference output (not gibberish) with proper WHT butterfly rotation
MTP support: Full MTP speculative decoding compatible
Status: Working but slower than Q4_K_M (≈12.6 tok/s on RTX 3090 vs ~15-18 tok/s for Q4_K_M)

Implementation Details

Backend: buun-llama-cpp fork with custom CUDA matmul kernels
Rotation: WHT butterfly (matching GPU matmul, corrected CPU quantizer mismatch)
Build: build_turbo4.bat (MSVC + CUDA 12.8 + Ninja, SM_86)
PR: buun-llama-cpp #61

Known Limitations

Speed (v4-i3): ~7-10 tok/s on RTX 3090 with MTP speculative decoding (67% accept rate) — CUDA matmul kernel still naive, no MMVQ vec_dot tuning yet.
GPU offload: Fully fixed — all 505 turbo4 tensors now load on GPU, no CPU fallback.
Format: Specific to buun's turbo types — not compatible with stock llama.cpp without the fork
Rotation: Requires re-quantization after code changes (WHT vs QR mismatch)

Recommended Usage

Use this for:

Experimenting with custom quantization schemes
Testing MTP + turbo4 weight quantization interactions
Research on 4-bit MTP models

Use Q4_K_M/Q6_K for production inference until turbo4 performance improves.

Methods

These GGUF files are quantized from GestaltLabs/Ornstein3.6-27B-MTP-NSC-ACE-SABER, a staged NSC-ACE -> SABER -> Ornstein checkpoint.

Stage	Purpose	Key settings
NSC-ACE	Internally steered rollout committee with reward on convergent tool-call structure.	agentic tool-call convergence
SABER	Compliance-first calibration, then KLD/PPL selection.	1046 HarmBench + AdvBench-style prompts; selected row reports 968/1046 by keyword proxy
Ornstein SFT	Reasoning/personality refinement after SABER.	premium reasoning v1, 1 epoch, 50 steps, rank 32/32, dropout 0.05, cosine schedule
GGUF conversion	MTP-capable llama.cpp conversion and quantization.	llama.cpp PR #22673, commit `e7b484815`

Results

Metric	Value
SABER selected compliance proxy	92.54% (968/1046 eval prompts)
SABER selected keyword residual	7.46% (78/1046 eval prompts)
SABER selected HarmBench classifier ASR	0.67% (7/1046 eval prompts)
SABER selected KLD	0.008302
SABER selected PPL ratio	1.103853
SABER selected post PPL	17.5988
SABER selected base PPL	15.9431
MTP status	present and verified
`mtp_num_hidden_layers`	1
Source `mtp.*` tensors	15
Corrected source tensor count	866

This repository hosts llama.cpp/GGUF builds for GestaltLabs/Ornstein3.6-27B-MTP-NSC-ACE-SABER. GGUF artifacts live here so the main safetensors repository stays focused on the source checkpoint.

MTP Status

These files are built from an MTP-capable source checkpoint:

MTP check	Value
`mtp_num_hidden_layers`	1
`mtp_use_dedicated_embeddings`	false
Source `mtp.*` tensors	15
Corrected source tensor count	866
Conversion path	llama.cpp PR #22673, commit `e7b484815`

MTP support is present and verified. This release includes mtp_num_hidden_layers=1 and MTP/nextn tensors in the GGUF validation path. Use a llama.cpp build with Qwen3.5/Qwen3.6 MTP support and run with --spec-type mtp.

Vision Support

This repo includes mmproj-Ornstein3.6-27B-MTP-NSC-ACE-SABER-F16.gguf, converted from the original multimodal Ornstein 27B vision encoder. Use it with llama.cpp's multimodal path alongside any of the text GGUF quants.

Available Files

File	Status	Notes
`mmproj-Ornstein3.6-27B-MTP-NSC-ACE-SABER-F16.gguf`	uploaded	Vision encoder / projector for image input
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-F16-MTP.gguf`	uploaded	Full GGUF conversion source / highest local fidelity
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q8_0-MTP.gguf`	uploaded	Near-full quality, large local file
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q6_K-MTP.gguf`	uploaded	High-quality local default if memory allows
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q5_K_M-MTP.gguf`	uploaded	Strong quality/size balance
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q5_K_S-MTP.gguf`	uploaded	Smaller Q5 option
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q4_K_M-MTP.gguf`	uploaded	Common balanced local target
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q4_K_S-MTP.gguf`	uploaded	Smaller Q4 option
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q3_K_L-MTP.gguf`	uploaded	Lower-memory Q3 option
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q3_K_M-MTP.gguf`	uploaded	Smaller Q3 balance
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q3_K_S-MTP.gguf`	uploaded	Small Q3 option
`Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q2_K-MTP.gguf`	uploaded	Minimum-size target; quality loss expected
	`Ornstein3.6-27B-MTP-NSC-ACE-SABER-turbo4-MTP-v3.gguf`	uploaded
	`Ornstein3.6-27B-MTP-NSC-ACE-SABER-turbo4-MTP-v4-i3.gguf`	uploaded

All listed GGUF artifacts have been uploaded.

Which Quant Should I Use?

Quant	Best fit
F16	Maximum fidelity when disk/RAM are not a concern
Q8_0	Very high fidelity local inference
Q6_K	Recommended high-quality local starting point
Q5_K_M	Strong balance for quality and size
Q4_K_M	Practical default for constrained machines
Q3_K_M / Q3_K_S	Low-memory experiments
Q2_K	Smallest target; use only when memory is the hard constraint

For agentic/tool-calling workloads, prefer Q6_K, Q5_K_M, or Q4_K_M when possible. Very low quants can shift structured output before they obviously degrade prose.

Loading Example

bash

llama-server \
  -m Ornstein3.6-27B-MTP-NSC-ACE-SABER-Q6_K-MTP.gguf \
  --mmproj mmproj-Ornstein3.6-27B-MTP-NSC-ACE-SABER-F16.gguf \
  --spec-type mtp \
  -ngl 999 \
  -c 32768

Use a llama.cpp build that includes Qwen3.5/Qwen3.6 MTP support.

Source Repository

Source checkpoint: GestaltLabs/Ornstein3.6-27B-MTP-NSC-ACE-SABER

Turbo4 Implementation

For technical details on the turbo4 weight quantization implementation, see buun-llama-cpp PR #61.

Support

Support this work on Ko-fi.

gkraker04/Ornstein3.6-27B-MTP-NSC-ACE-SABER-turbo4-GGUF

Ornstein3.6-27B-MTP-NSC-ACE-SABER-turbo4-GGUF

🔥 Turbo4 Weight Quantization (NEW)