The hybrid quant employs different quantization levels on a per layer basis to enable
both high performance and small file size at the same time. The quants employed are all K to avoid
slow CPU or older GPU processing of IQ quants. For this file the Q4_E_H layer quants using new extended
K layer definitions are as follows:
The layer quants were optimized for very strong performance over a small set of curated reasoning prompts and sized to
slightly larger than Q4_K_M bit efficiency. It solves the majority of the set of eval prompts efficiently
and correctly without using think block.
Comparison:
Quant
size
PPL
Comment
Q4_K_M
7.38e9
14.4
modified PPL, see discussion below.
Q4_E_H
7.43e9
13.6
modified PPL, 0.05B larger than Q4_K_M, Q6_K embed/out
Usage:
gemma 4 12B it is a vision capable dense RL model. It can be used together with its multimedia projector layers to process audio and images and text inputs
and generate text outputs. The mmproj file is made available in this repository.
Thinking:
By default the model will not create a RL reasoning block and just outputs
<|channel>thought
<channel|>
at the start of gen. To get it to fill in the think block use a system prompt with:
<|think|>
as the first token. This is a special token in the model vocab and must be tokenized as such to work. No other text in the system prompt
besides the think token is needed to get it to fill in the RL block though other text can be added if desired.
The model was found to be highly capable on reasoning tasks when skipping think block.
Running:
The model can be speculated with gemma-3-270m-it-256k. To use this speculator the inference platform must support dynamic vocab translation between
draft and target. Google is creating optimized speculators for gemma 4 series which might work also but have not been tested.
The model can be run fully offloaded into a 12G VRAM GPU. Due to its SWA attention scheme very large context is available.
On a 9900k + 4070 setup approx performance for the Q4_E_H quant using fixed speculation block size ND with a downstream llama.cpp speculator:
CONFIG (no multimodal tower)
QKV
NKV
gen tps
pp tps (batch 128)
ND=0 (no draft)
F16
~204k
48
~1400
ND=2
""
~150k
69
""
ND=3
""
""
63
""
ND=0 (no draft)
Q8_0
256k
47
~1100
ND=2
~200k
64
""
The Q4_E_H model passed two long context tests showing very fast prompt processing speed for 100k+ token prompts. This fast speed is due to combination
of SWA + decimated full attention layers (only 1 of 6 layers is full attention). The attention scheme seems to work well. The quant handles a 106k token prompt https://thireus.com/REDDIT/Qwen3_Runescape_Massive_Prompt.txt
(greedy sampling used for gen) with a self correction:
lm Qwen3_Runescape_Massive_Prompt.txt
<|channel>thought
<channel|>Based on the "Skills/Experience Table" provided in the text, you reach the halfway point of the experience required for the maximum level (level 99) at **level 70**.
**Calculation:**
* The experience required to reach the maximum level (99) is **13,034,431**.
* Half of that amount is **6,517,215.5**.
* According to the table, the experience required to reach **level 70** is **737,627**, and the experience required to reach **level 71** is **814,445**.
Wait, looking closer at the table provided, the "Experience needed to reach level" column represents the total cumulative experience for that level. To find the level where you have half of the 13,034,431 required for level 99:
* Level 70: 737,627
* Level 80: 1,986,068
* Level 90: 5,346,332
* Level 91: 5,902,831
* Level 92: 6,517,253
* Level 93: 7,195,629
The value **6,517,253** (which is the closest whole number to the 6,517,215.5 midpoint) is the experience required to reach **level 92**.
This prompt and answer are proven baked into gemma 4 training set by prior experiments with gemma 26/31. What is notable on this 12B response
is the self correction showing the RL/introspection training kicking in to direct the model through differnt parts of its latent space to
increase the accuracy of generating the (pretrained) correct answer.
Vision:
As of llama.cpp b9518 vision mode will not work without applying the patch https://github.com/ggml-org/llama.cpp/compare/master...chippydip:llama.cpp:gemma4uv-vision-fix.diff
from issue https://github.com/ggml-org/llama.cpp/issues/24146 . With the patch vision works but still exhibits the same poor performance found
in 26B/31B, failing to identify tough bird ID images which entire Qwen3.5 series aces. The root cause of the poor performance can now more confidently be inferred to
be low quality vision training since this model using a completely different mechanism to handle multimodal compared to 26B/31B and still shows similar poor performance.
Audio:
A quick test on audio transcription showed the model to be fairly accurate on Oppenheimer bhagavad gita:
The new world would not be the same. A few people laughed, a few people cried, most people were silent. I remembered the line from the Hindu scripture, the Bhagavad Gita. Vishnu is trying to persuade the prince that he should do his duty and to impress him takes on his multi-armed form and says now i am become death the destroyer of worlds. I suppose we all thought that one way or another.
Code:
The model was tested across a small set of code gen promtps and did OK, nothing spectacular but was able to generate working
programs from short functional descriptions.
Llama.cpp inference/isssues:
multimodal vision functionality is borked as of llama.cpp b9518 but vision can be patched as described above.
The model cannot compute valid perplexity due to the instruct tune forcing it to generate
<|channel>thought
as assitant gen independent of previous prompt contents. To work around this problem a modifed perplexity is
computed by overwriting the beginning of the perplexity chunk contents with the forced assistent gen as follows:
# chunk is a string of text to eval perplexity on
injects='model\n<|channel>thought\n<channel|>'
chunk="${injects}${chunk:${#injects}}"
