The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. The quants employed are all legacy or K to avoid slow CPU or older GPU processing of IQ quants. For this file the Q4_E_H layer quants using extended layer definitions (E quant nomenclature) are as follows:

  LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_0","5","Q5_0","6","Q5_1","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_666"],[1 ,"Q4_K_656"],[2 ,"Q4_K_545"],[3 ,"Q4_K_554"],[4 ,"Q4_K_554"],[5 ,"Q4_K_555"],[6 ,"Q4_K_544"],[7 ,"Q4_K_554"],
   [8 ,"Q4_K_544"],[9 ,"Q4_K_544"],[10,"Q4_K_544"],[11,"Q4_K_555"],[12,"Q4_K_544"],[13,"Q4_K_544"],[14,"Q4_K_544"],[15,"Q4_K_544"],
   [16,"Q4_K_544"],[17,"Q4_K_555"],[18,"Q4_K_544"],[19,"Q4_K_544"],[20,"Q4_K_544"],[21,"Q4_K_544"],[22,"Q4_K_544"],[23,"Q4_K_555"],
   [24,"Q4_K_644"],[25,"Q4_K_644"],[26,"Q4_K_644"],[27,"Q4_K_644"],[28,"Q4_K_644"],[29,"Q4_K_655"],[30,"Q4_K_644"],[31,"Q4_K_644"],
   [32,"Q4_K_654"],[33,"Q4_K_654"],[34,"Q4_K_654"],[35,"Q4_K_655"],[36,"Q4_K_654"],[37,"Q4_K_654"],[38,"Q4_K_654"],[39,"Q4_K_654"],
   [40,"Q4_K_665"],[41,"Q4_K_666"],[42,"Q4_K_665"],[43,"Q4_K_665"],[44,"Q5_K_665"],[45,"Q5_K_666"],[46,"Q5_K_668"],[47,"Q6_K_886"]
   ]'
   FLAGS="--token-embedding-type Q5_1 --output-tensor-type Q5_1 --layer-types-high"

The layer quants were optimized for very strong performance over a small set of curated reasoning prompts and sized to slightly larger than google q4_0 quant bit efficiency. It solves the majority of the set of eval prompts efficiently and correctly without using think block. The weights for this model were backpropped with q4_0 quantization on the model parameters in the loop with a goal to train the weights optimally in the discrete quantization space of q4_0, while the model itself was left in F16 so parameter residuals outside q4_0 space are available to downstream quantizations. This E quant used an empirical heuristic based on use of legacy quants for VOD and K quants other layers using the same profile optimized for non QAT version of the model. The heuristic was optimized based on testing the model across the set of reasoning prompts and selecting the method with best performance.

Comparison:

Quant	size	PPL	Comment
Q4_0	6.98e9	-	Not tested
Q4_E_H	7.32e9	12.8	modified PPL, 0.34B larger than Q4_0, Q5_1 embed

Usage:

gemma 4 12B it qat q4_0 unquantized is a text, vision, and audio capable dense RL model. It can be used together with its multimedia projector layers to process audio and images and text inputs and generate text outputs. The mmproj file is made available in this repository.

Thinking:

By default the model will not create a RL reasoning block and just outputs

<|channel>thought
<channel|>

at the start of gen. To get it to fill in the think block use a system prompt with:

<|think|>

as the first token. This is a special token in the model vocab and must be tokenized as such to work. No other text in the system prompt besides the think token is needed to get it to fill in the RL block though other text can be added if desired.

The model was found to be highly capable on reasoning tasks when skipping think block.

Running:

The model can be speculated with gemma-3-270m-it-256k. To use this speculator the inference platform must support dynamic vocab translation between draft and target. Google is creating optimized speculators for gemma 4 series which might work also but have not been tested.

The model can be run fully offloaded into a 12G VRAM GPU. Due to its SWA attention scheme very large context is available.

On a 9900k + 4070 setup approx performance for the Q4_E_H quant using fixed speculation block size ND with a downstream llama.cpp speculator:

CONFIG (no multimodal tower)	QKV	NKV	gen tps	pp tps (batch 128)
ND=0 (no draft)	F16	~204k	53	~1400
ND=2	""	~150k	78	""
ND=3	""	""	75	""
ND=0 (no draft)	Q8_0	256k	51	~1320
ND=2		~200k	75	""

The Q4_E_H model passed two long context tests showing very fast prompt processing speed for 100k+ token prompts. This fast speed is due to combination of SWA + decimated full attention layers (only 1 of 6 layers is full attention). The attention scheme seems to work well. The quant handles a zero-shot 106k token prompt https://thireus.com/REDDIT/Qwen3_Runescape_Massive_Prompt.txt (greedy sampling used for gen) with a one-off error:

lm Qwen3_Runescape_Massive_Prompt.txt 
<|channel>thought
<channel|>Based on the "Skills/Experience Table" provided in the book, here is the step-by-step calculation to find the level at which you reach half of the experience required for max level:

1.  **Identify the Max Level Experience:**
    The table shows that the experience needed to reach level 99 is **13,034,431**.

2.  **Calculate Half of Max Level Experience:**
    $13,034,431 \div 2 = 6,517,215.5$

3.  **Find the corresponding level in the table:**
    Looking at the "Experience needed to reach level" column:
    *   Level 92 requires **6,517,253**
    *   Level 91 requires **5,902,831**

Since 6,517,215.5 is just slightly less than the experience required for level 92, you are at **level 91** (having just completed the experience for level 91 and being very close to the threshold for level 92).

With no thinking the model is one off in its answer (91 vs 92). Think mode version:

SYSTEM="<|think|>" SPECIAL=1 lm Qwen3_Runescape_Massive_Prompt.txt 
<|channel>thought
*   Question: In RuneScape, what level are you at when you get to half the xp of max level?
    *   Constraint: Use the provided book as knowledge.

    *   I need to find the "Skills/Experience Table" section.
    *   I need to find the experience required for level 99 (max level).
    *   I need to find the level where the cumulative experience is half of that value.

    *   *Experience needed to reach level 99:* 13,034,431.
    *   *Half of that experience:* 13,034,431 / 2 = 6,517,215.5.

    *   Level 1: 0
    *   Level 2: 83
    *   Level 3: 174
    *   Level 4: 276
    *   Level 5: 388
    *   Level 6: 512
    *   Level 7: 650
    *   Level 8: 801
    *   Level 9: 969
    *   Level 10: 1,154
    *   Level 11: 1,358
    *   Level 12: 1,584
    *   Level 13: 1,833
    *   Level 14: 2,107
    *   Level 15: 2,411
    *   Level 16: 2,746
    *   Level 17: 3,115
    *   Level 18: 3,523
    *   Level 19: 3,973
    *   Level 20: 4,470
    *   Level 21: 5,018
    *   Level 22: 5,624
    *   Level 23: 6,291
    *   Level 24: 7,028
    *   Level 25: 7,842
    *   Level 26: 8,740
    *   Level 27: 9,730
    *   Level 28: 10,824
    *   Level 29: 12,031
    *   Level 30: 13,363
    *   Level 31: 14,833
    *   Level 32: 16,456
    *   Level 33: 18,247
    *   Level 34: 20,224
    *   Level 35: 22,406
    *   Level 36: 24,815
    *   Level 37: 27,473
    *   Level 38: 30,408
    *   Level 39: 33,648
    *   Level 40: 37,224
    *   Level 41: 41,171
    *   Level 42: 45,529
    *   Level 43: 50,339
    *   Level 44: 55,649
    *   Level 45: 61,512
    *   Level 46: 67,983
    *   Level 47: 75,127
    *   Level 48: 83,014
    *   Level 49: 91,721
    *   Level 50: 101,333
    *   Level 51: 111,945
    *   Level 52: 123,660
    *   Level 53: 136,594
    *   Level 54: 150,872
    *   Level 55: 166,636
    *   Level 56: 184,040
    *   Level 57: 203,254
    *   Level 58: 224,466
    *   Level 59: 247,886
    *   Level 60: 273,742
    *   Level 61: 302,288
    *   Level 62: 333,804
    *   Level 63: 368,599
    *   Level 64: 407,015
    *   Level 65: 449,428
    *   Level 66: 496,254
    *   Level 67: 547,953
    *   Level 68: 605,032
    *   Level 69: 668,051
    *   Level 70: 737,627
    *   Level 71: 814,445
    *   Level 72: 899,257
    *   Level 73: 992,895
    *   Level 74: 1,096,278
    *   Level 75: 1,210,421
    *   Level 76: 1,336,443
    *   Level 77: 1,475,581
    *   Level 78: 1,629,200
    *   Level 79: 1,798,808
    *   Level 80: 1,986,068
    *   Level 81: 2,192,818
    *   Level 82: 2,421,087
    *   Level 83: 2,673,114
    *   Level 84: 2,951,373
    *   Level 85: 3,258,594
    *   Level 86: 3,597,792
    *   Level 87: 3,972,294
    *   Level 88: 4,385,776
    *   Level 89: 4,842,295
    *   Level 90: 5,346,332
    *   Level 91: 5,902,831
    *   Level 92: 6,517,253
    *   Level 93: 7,195,629
    *   Level 94: 7,944,614
    *   Level 95: 8,771,558
    *   Level 96: 9,684,577
    *   Level 97: 10,692,629
    *   Level 98: 11,805,606
    *   Level 99: 13,034,431

    *   Max level (99) experience: 13,034,431.
    *   Half of max level experience: 13,034,431 / 2 = 6,517,215.5.
    *   Looking at the table:
        *   Level 91: 5,902,831
        *   Level 92: 6,517,253
    *   Since 6,517,253 is the first value that exceeds 6,517,215.5, the player reaches half the experience of max level at **level 92**.

    *   State the max experience for level 99.
    *   Calculate half of that.
    *   Identify the level corresponding to that value.<channel|>Based on the "Skills/Experience Table" provided in the book:

1.  The experience required to reach the maximum level (**level 99**) is **13,034,431**.
2.  Half of that experience is **6,517,215.5**.
3.  Looking at the table, the experience required for **level 92** is **6,517,253**.

Therefore, you reach half the experience of the max level at **level 92**.

With thinking turned on the model kicks into extended reasoning and derives the correct answer. Note greedy determininistic sampling is used on all tests/evals of the model,

Vision:

As of llama.cpp b9518 vision mode will not work without applying the patch https://github.com/ggml-org/llama.cpp/compare/master...chippydip:llama.cpp:gemma4uv-vision-fix.diff from issue https://github.com/ggml-org/llama.cpp/issues/24146 . With the patch vision works but still exhibits the same poor performance found in 26B/31B, failing to identify tough bird ID images which entire Qwen3.5 series aces. The root cause of the poor performance can now more confidently be inferred to be low quality vision training since this model using a completely different mechanism to handle multimodal compared to 26B/31B and still shows similar poor performance.

Audio:

A quick test on audio transcription showed the model to be fairly accurate on Oppenheimer bhagavad gita:

The new world would not be the same. A few people laughed. A few people cried. Most people were silent. I remembered the line from the Hindu scripture, the Bhagavad Gita. Vishnu is trying to persuade the prince that he should do his duty and to impress him takes on his multi-armed form and says now I am become death the destroyer of worlds I suppose we all thought that one way or another

Code:

The model was tested across a small set of code gen promtps and did quite well, subjectively working much better than non qat model suggesting less entropy loss against training in the quantized model as desired by the QAT method.

Llama.cpp inference/isssues:

multimodal vision functionality is borked as of llama.cpp b9518 but vision can be patched as described above.

The model cannot compute valid perplexity due to the instruct tune forcing it to generate

<|channel>thought

as assitant gen independent of previous prompt contents. To work around this problem a modifed perplexity is computed by overwriting the beginning of the perplexity chunk contents with the forced assistent gen as follows:

      # chunk is a string of text to eval perplexity on
      injects='model\n<|channel>thought\n<channel|>'
      chunk="${injects}${chunk:${#injects}}"

logprobs are skipped over the beginning part of the perplexity prompt using a modified llama.cpp downstream server to compute perplexity. Discussion at: https://github.com/ggml-org/llama.cpp/issues/21388#issuecomment-4185889930

Benchmarks:

Benchmarks for the model (audio, vision, and math) will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
gemma-4-12B-it-qat-q4_0-unquantized.Q4_E_H.gguf	Q4_E_H	7.32e9 B	0.34B bigger than google q4_0 gguf
gemma-4-12B-it-qat-q4_0-unquantized..mmproj.gguf	F16	0.12e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040

steampunque/gemma-4-12B-it-qat-q4_0-unquantized-MP-GGUF

Mixed Precision GGUF layer quantization of gemma-4-12B-it-qat-q4_0-unquantized by Google

Download the file from below:

steampunque/
gemma-4-12B-it-qat-q4_0-unquantized-MP-GGUF