Views
No views yet
Q4_0_ROCMFP4_FAST is GGUF file type 103, which
exists only in the fork linked below. Upstream llama.cpp fails with
failed to load model before it reaches the GPU. Get a prebuilt binary here:failed to create context, which is expected. It must be passed with -md
alongside a target model.1llama-server \
2 -m Qwen3.8-27B-ROCmFP4-FAST.gguf \
3 -md Qwen3.8-27B-DFlash2-Q4_0_ROCMFP4_FAST.gguf \
4 --spec-type draft-dflash --spec-draft-adaptive \
5 --spec-draft-n-min 3 --spec-draft-n-max 7 --spec-draft-ngl 99 \
6 -ngl 999 -fa on -b 2048 -ub 512 -c 32768--spec-draft-adaptive sizes the draft from measured acceptance instead of a fixed n. It is
what makes n-max 7 safe here: at a fixed n=7 acceptance collapses to 18 % and throughput
drops to 20.2 t/s, while adaptive holds 96 % acceptance and reaches 65.6.| Qwen3.8-27B, FP4 target + this sidecar, greedy, 300 tokens | structured output | prose |
|---|---|---|
| bare decode | 14.0 t/s | 14.1 t/s |
| fixed draft n=3 | 41.6 | 25.4 |
| fixed draft n=7 | 20.2 | 24.8 |
adaptive, n_max 7 n_min 3 | 65.6 t/s - 4.7x | 26.1 |
Q8_0_ROCMFPX expecting parity with Q8_0. At identical
bits per weight, ours scored 53.5 % acceptance against z-lab's 60.2 %. The cause is the block
scale, not the codes: Q8_0 stores an fp16 scale, ROCmFPx stores a UE4M3 byte. At 8 bits per
weight the coarse scale is what costs you. FP4 is a different trade and is the one measured here.