It is engineered specifically for 24GB consumer GPUs (NVIDIA GeForce RTX 4090 / RTX 3090) to run speculative decoding alongside Qwen3.8-27B dense models without spilling into system RAM.

🚀 Benchmark Discovery: Zero Quality Loss at 2-Bit

By quantizing the DFlash 2 block-diffusion drafter down from the official Q4_K_M (~1.1 GB) to Q2_K (~700 MB), we save ~400–450 MB of VRAM with zero degradation in speculative draft accuracy.

Strict A/B Performance on Single RTX 4090 (24GB VRAM):

Evaluated on Qwen3.8-27B-UD-Q4_K_XL using llama-server (PR #27342) with --spec-draft-n-max 3 across a 28k prompt baseline.

Drafter Quant	File Size	Mean Accepted Length	Acceptance Rate	Decode Speed	Max Usable Context (24GB VRAM)
Official Q4_K_M	~1.10 GB	2.80 / 3.0 tokens	60.00%	75.89 t/s	150,000 tokens (Q4 KV)
Custom Q2_K (This Repo)	~700 MB	2.81 / 3.0 tokens	60.22%	75.93 t/s	170,000 tokens (+20k)

Key Takeaways:

Identical Acceptance Rate: The target 27B model accepts 2.81 out of 3.0 drafted tokens on average $(93.7% \text{ theoretical efficiency})$, matching the 4-bit drafter identically.
Context Expansion: The ~400 MB saved VRAM directly unlocks:
- 170,000 Context on Q4_0 KV (~76 t/s @ 23.89 GB)
- 100,000 Context on high-precision Q8_0 KV (~75 t/s @ 23.88 GB)

⚙️ Why Q2_K is the True Physical Floor for DFlash 2

DFlash 2 is a cross-layer block-diffusion drafter that extracts hidden states from layers [6, 20, 34, 48, 62] of the base model. Because it is not a standalone causal LM, standalone llama-imatrix calibration fails (dflash requires ctx_other to be set).

This means importance-matrix quants (IQ1_S, IQ2_XXS) cannot be computed, making Q2_K the absolute lowest non-degrading quantization floor for DFlash 2.

🛠️ Quickstart with `llama.cpp`

Requires llama.cpp built from PR #27342.

1. Build `llama.cpp`:

bash

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342

# Build with CUDA
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

analogalok/Qwen3.8-27B-DFlash2-Q2_K-GGUF

Qwen3.8-27B-DFlash2-Q2_K (2-Bit Zero-Loss Speculative Drafter)

🚀 Benchmark Discovery: Zero Quality Loss at 2-Bit

Strict A/B Performance on Single RTX 4090 (24GB VRAM):

Key Takeaways:

⚙️ Why Q2_K is the True Physical Floor for DFlash 2

🛠️ Quickstart with llama.cpp

1. Build llama.cpp:

analogalok/
Qwen3.8-27B-DFlash2-Q2_K-GGUF

🛠️ Quickstart with `llama.cpp`

1. Build `llama.cpp`: