Filename	Quant	Bits	Size (est.)
`Holo-3.1-35B-A3B-Q2_K.gguf`	Q2_K	2	~12 GB
`Holo-3.1-35B-A3B-Q3_K_S.gguf`	Q3_K_S	3	~15 GB
`Holo-3.1-35B-A3B-Q3_K_M.gguf`	Q3_K_M	3	~17 GB
`Holo-3.1-35B-A3B-Q3_K_L.gguf`	Q3_K_L	3	~18 GB
`Holo-3.1-35B-A3B-IQ3_XXS.gguf`	IQ3_XXS	3	~14 GB
`Holo-3.1-35B-A3B-IQ3_S.gguf`	IQ3_S	3	~15 GB
`Holo-3.1-35B-A3B-Q4_K_S.gguf`	Q4_K_S	4	~20 GB
`Holo-3.1-35B-A3B-Q4_K_M.gguf`	Q4_K_M	4	~22 GB
`Holo-3.1-35B-A3B-Q5_K_S.gguf`	Q5_K_S	5	~24 GB
`Holo-3.1-35B-A3B-Q5_K_M.gguf`	Q5_K_M	5	~26 GB
`Holo-3.1-35B-A3B-Q6_K.gguf`	Q6_K	6	~30 GB
`Holo-3.1-35B-A3B-Q8_0.gguf`	Q8_0	8	~39 GB
`mmproj-Holo-3.1-35B-A3B-F16.gguf`	F16	16	~0.9 GB

BF16 GGUF (69.4 GB) available at source repo.

Usage

bash

# Text + Vision inference
huggingface-cli download barozp/Holo-3.1-35B-A3B-GGUF Holo-3.1-35B-A3B-Q4_K_M.gguf --local-dir ./
huggingface-cli download barozp/Holo-3.1-35B-A3B-GGUF mmproj-Holo-3.1-35B-A3B-F16.gguf --local-dir ./

llama-cli \
  -m Holo-3.1-35B-A3B-Q4_K_M.gguf \
  --mmproj mmproj-Holo-3.1-35B-A3B-F16.gguf \
  --jinja \
  -ngl 99 \
  -c 32768 \
  --image screenshot.png

Quantization Details

BF16 GGUF source: Hcompany/Holo-3.1-35B-A3B-GGUF
imatrix: Pre-computed, sourced from BF16 repo (192 MB)
Tool: llama.cpp llama-quantize
Quantized by: barozp