The models were quantized using various schemes provided by the auto-round tool. For better compatibility and smaller size, we provide unified multimodal projector (mmproj) files in F16, BF16, and F32 formats.

Files and Sizes

File Name	Quant Type	Size	Description
`gemma-4-31B-it-Q2_K_S.gguf`	Q2_K_S	11 GB	Extremely high compression, significant quality loss.
`gemma-4-31B-it-Q2_K_MIXED.gguf`	Q2_K_MIXED	17 GB	Recommended high-compression option. Uses Q4 for KV cache with good quality.
`gemma-4-31B-it-Q3_K_S.gguf`	Q3_K_S	13 GB	Very high compression, notable quality loss.
`gemma-4-31B-it-Q3_K_M.gguf`	Q3_K_M	13 GB	Balanced 3-bit quantization.
`gemma-4-31B-it-Q3_K_L.gguf`	Q3_K_L	13 GB	High quality 3-bit quantization.
`gemma-4-31B-it-Q4_0.gguf`	Q4_0	17 GB	Standard 4-bit quantization, good balance.
`gemma-4-31B-it-Q4_1.gguf`	Q4_1	19 GB	Higher quality 4-bit quantization than Q4_0.
`gemma-4-31B-it-Q4_K_S.gguf`	Q4_K_S	17 GB	Small 4-bit K-quant, good efficiency.
`gemma-4-31B-it-Q4_K_M.gguf`	Q4_K_M	17 GB	Recommended 4-bit K-quant, excellent balance.
`gemma-4-31B-it-Q5_0.gguf`	Q5_0	20 GB	Standard 5-bit quantization, very high quality.
`gemma-4-31B-it-Q5_1.gguf`	Q5_1	22 GB	Higher quality 5-bit quantization than Q5_0.
`gemma-4-31B-it-Q5_K_S.gguf`	Q5_K_S	20 GB	Small 5-bit K-quant, very high quality.
`gemma-4-31B-it-Q5_K_M.gguf`	Q5_K_M	20 GB	Recommended 5-bit K-quant, near-lossless.
`gemma-4-31B-it-Q6_K.gguf`	Q6_K	24 GB	6-bit K-quant, virtually indistinguishable from F16.
`gemma-4-31B-it-Q8_0.gguf`	Q8_0	31 GB	8-bit quantization, near-lossless.
`mmproj-model-f16.gguf`	F16	1.2 GB	Unified Projector in Float16 format.
`mmproj-model-bf16.gguf`	BF16	1.2 GB	Unified Projector in BFloat16 format.
`mmproj-model-f32.gguf`	F32	2.2 GB	Unified Projector in Float32 format.

Generate the Model

The models were generated using Intel's AutoRound with the following command:

auto-round --model google/gemma-4-31B-it --output_dir ./quantized/ --scheme <SCHEME> --iters 0

Note: To reproduce this quantization, you need the following Pull Request:

Intel AutoRound PR #1655
And the specific fix mentioned in this comment.

Usage with llama.cpp

These models can be used with llama.cpp. For multimodal usage, you must specify the projector file:

./llama-cli -m gemma-4-31B-it-Q4_K_M.gguf --mmproj mmproj-model-f16.gguf --image your_image.jpg -p "Describe this image."

About AutoRound

AutoRound is an advanced quantization technique from Intel that aims to minimize accuracy loss through automated rounding optimization.

sphaela/gemma-4-31B-it-AutoRound-GGUF

Gemma-4-31B-it GGUF (AutoRound Quantized)

Quantization Details

Files and Sizes

Generate the Model

Usage with llama.cpp

About AutoRound

sphaela/
gemma-4-31B-it-AutoRound-GGUF