The hybrid quant employs different quantization levels on a per layer basis to enable
both high performance and small file size at the same time. This particular quant
was optimized for high performance across a set of test prompts with ~IQ4_XS size. The
quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. For
this file the layer quants are as follows:
This is a RL trained thinking model. The layer quants for this model were optimized for high success rate on a curated set of test/eval prompts. This
model exhibits overthinking when solving and this problem was found to be unsolvable by adjusting quant layers. The overthinking is baked into the model by
its RL training. The model was clearly trained not to just pull a formula out of its latent space, apply it and call it done, but to reason its way through solutions to
form a logical framework for its final presented solution. The quant was evaluated with greedy sampling and found to be stable on all test set problems but
often ran out of tokens while clearly on its way to a correct solution due to overthinking with ~8k context. To solve moderate difficulty problems a minimum of 8k and
recommended 16k context size should be configured. 8b KV vs F16 KV is recommended help increase context size. The model can be fully offloaded to two 4070s over RPC
with ~25t/s gen rate when using a downstream speculator with llama.cpp server. Qwen3 0.6B is the recommended speculator for the model. At llama.cpp b6045 universal
speculator support was added to upstream llama.cpp but the upstream speculation functionality was not evaluated in tests.
