Requant of
laguna-s-2.1-Q4_K_M.gguf
Problem: The upstream model replaced their Q4_K_M version with a one using
"mixed-precision Q4_K_Mx40-BF16x8 (BF16 sensitive layers)", increasing model size from ~68 GB to 96 GB and making it no longer possible to run on 96 GB VRAM systems and a tight fit on many 128 GB variants.
Solution: This version quantizes all BF16 layers to Q8 while leaving all other layers untouched.
Result: Model size reduced to 77 GB, though at the cost of some "sensitive layers" precision.
A minor edit was also made to the
jinja template (line 93), changing
</think> to
<think>\n. This does not guarantee the model will reason — it only makes it slightly more likely based on testing. The model may still close the tag immediately and continue without it.
Reasoning remains largely dependent on the first assistant message in a session: if it contains reasoning, the model will often continue; if not, it won't.
For frameworks that avoid context overflow by generating a summary and then starting a new session with it as a primer this can be problematic. If the summary is in the form of an assistant message it must have a thinking block otherwise reasoning is likely disabled for the entire session. The safest approach is to probably to prefill a thinking tag opener.
If the edited template causes issues, the default can be loaded by launching llama-server with the --chat-template-file flag.
PPL: Final estimate: PPL = 8.1036 ± 0.05985
PPL could not be run for comparison on other sizes due to NaN errors, possibly related to
PR 21652 and
PR 22571. The error also appears to be partially system related, one system completed the test fine while another found NaNs.
As a side note the default -b and -ub values in llama.cpp seems sub-par for this model. I recommend trying 1024 for both and then adjust both symmetrically upwards to find the sweet spot.
