or a wall of NRNRNRNR or ? 2? 4? 2?. There's a whole issue about it (#23953). CPU and CUDA are fine, so people kept blaming the server, batching, the chat template, flash-attn... none of it.

what's actually going on

Talkie has one cursed channel. Layer 14, the ffn_down input, spikes to about 139,000. fp16 tops out at 65,504. llama.cpp's Vulkan matmul shaders stash the activation operand in fp16 tiles, so that number turns into inf, then NaN, then everything downstream is garbage. CPU/CUDA keep it in fp32 so they never notice.

This is why nothing people tried worked: it's the activations blowing up, not the weights, so no quant level saves you. And -ub 1, the workaround in the issue thread, didn't even do anything on my Radeon 890M — just gave me a different flavor of garbage.

the fix

You can't fix an activation overflow by being clever about quantization, but you can just make the activation smaller. That one channel feeds through two linear layers, so:

mlp_linear[ch] /= 32      # shrink it going in
mlp_resid[:, ch] *= 32     # blow it back up coming out

The two cancel. Pick a power of two and it's bit-exact in bf16 — literally the same model, the number just never crosses 65,504 mid-flight anymore. Two channels needed it (×32 and ×4), everything else is untouched. 441 of 443 tensors are byte-for-byte identical to the original.

does it work though

Same prompt, greedy, same Q4_K_M, on a real AMD Radeon 890M over Vulkan:

before: * * * * * * * *
after: "The history of the steam locomotive may be said to begin in the year 1800, when Richard Trevithick first constructed a locomotive engine at Trewithan, in Cornwall..."

And to be sure I didn't actually change the model: rescaled vs original ffn_down output matches to 0.000e+00 in fp32. Same weights, just exponent-shifted on two channels.

use it

llama-server -m talkie-1930-13b-it.Q4_K_M.gguf -ngl 999 -c 2048 --jinja

Vulkan / AMD / Intel / CUDA / CPU, all fine. 2048 context.

quant	size	notes
Q2_K	~5 GB	smallest, rough
Q3_K_S/M/L	~5.6–7 GB
Q4_K_S/M	~7.5–8.6 GB	Q4_K_M is the sweet spot
Q5_K_S/M	~9–9.4 GB
Q6_K	~10.8 GB
Q8_0	~14 GB	basically lossless

Converted from lewtun/talkie-1930-13b-it-hf (the HF port the llama.cpp converter expects). The proper long-term fix is bf16 Vulkan shaders upstream — this just sidesteps the whole thing at the weights so you don't have to wait for it.

zakarth/talkie-1930-13b-it-vulkan-fixed-GGUF

talkie-1930-13b-it — GGUF that actually works on AMD/Vulkan

what's actually going on

the fix

does it work though

use it

zakarth/
talkie-1930-13b-it-vulkan-fixed-GGUF