GGUF conversion of issai/Qolda-AVL-5B, a 5B audio-vision-language model for Kazakh, Russian, and English built on Qwen3-VL-4B-Thinking with an audio branch based on a fine-tuned Whisper-large-v3-turbo encoder and DeepStack audio injection.

Each precision lives in its own subfolder; all of them share the multimodal projectors in mmproj/.

Files

File	Precision	Size	Notes
`BF16/Qolda-AVL-5B-BF16.gguf`	BF16	8.1 GB	lossless reference conversion
`Q8_0/Qolda-AVL-5B-Q8_0.gguf`	Q8_0	4.3 GB	near-lossless
`Q6_K/Qolda-AVL-5B-Q6_K.gguf`	Q6_K	3.3 GB	near-lossless
`Q5_K_M/Qolda-AVL-5B-Q5_K_M.gguf`	Q5_K_M	2.9 GB	imatrix-guided
`Q4_K_M/Qolda-AVL-5B-Q4_K_M.gguf`	Q4_K_M	2.5 GB	imatrix-guided, recommended minimum
`mmproj/mmproj-Qolda-AVL-5B-F16.gguf`	F16	2.2 GB	audio + vision projector (patched llama.cpp)
`mmproj/mmproj-Qolda-AVL-5B-vision-only-F16.gguf`	F16	0.8 GB	vision projector (stock llama.cpp)
`imatrix/imatrix-qolda-avl-5b-trilingual.gguf`	—	4 MB	importance matrix (kk/ru/en Wikipedia)
`llama.cpp-patch/qwen3avl-support.patch`	—	—	adds the `qwen3avl` audio projector to llama.cpp

The Q5_K_M and Q4_K_M quants use an importance matrix computed on trilingual (kk 50% / ru 25% / en 25%) Wikipedia text, so precision is spent where the model's target languages need it. The LLM quantization affects all modalities equally; the encoders always stay in F16.

Modality support

Modality	stock llama.cpp	patched llama.cpp
Text (kk/ru/en)	✅	✅
Image	✅ (`mmproj-...-vision-only-F16.gguf`)	✅
Audio	❌	✅ (`mmproj-...-F16.gguf`)

Patch description

The audio branch of Qolda-AVL is architecturally new to llama.cpp: a custom projection head (LayerNorm→Linear→GELU→Linear) plus audio DeepStack — features tapped from Whisper encoder layers 8/16/24 are injected into the first three LLM layers, mirroring Qwen3-VL's visual DeepStack. No existing llama.cpp audio projector computes this, and approximating it (e.g. dropping DeepStack, as llama.cpp does for Qwen3-Omni) would degrade audio quality. The patch reproduces the HF computation exactly, including running the encoder on the full padded 30 s mel window while emitting only the valid-prefix tokens.

Text and vision use the established qwen3vl architecture completely unchanged — those GGUFs work with any recent stock llama.cpp.

Usage

Text + Vision (stock llama.cpp)

bash

llama-server -m Qolda-AVL-5B-Q8_0.gguf \
    --mmproj mmproj-Qolda-AVL-5B-vision-only-F16.gguf -ngl 99 -c 16384

Text + Audio + Vision (patched llama.cpp)

Build the patched llama.cpp once:

bash

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout ea63b4d
git apply qwen3avl-support.patch   # from llama.cpp-patch/ in this repo
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-mtmd-cli

Then:

bash

# audio transcription / audio QA
llama-mtmd-cli -m Qolda-AVL-5B-Q8_0.gguf --mmproj mmproj-Qolda-AVL-5B-F16.gguf \
    --audio speech.wav -p "Аудиодағы сөйлеуді сөзбе-сөз транскрипциялап жаз." \
    -c 16384 -ngl 99

# image QA
llama-mtmd-cli -m Qolda-AVL-5B-Q8_0.gguf --mmproj mmproj-Qolda-AVL-5B-F16.gguf \
    --image photo.jpg -p "Суретті егжей-тегжейлі сипаттап бер." -c 16384 -ngl 99

# OpenAI-compatible server (image + audio in chat completions)
llama-server -m Qolda-AVL-5B-Q8_0.gguf --mmproj mmproj-Qolda-AVL-5B-F16.gguf -ngl 99 -c 16384

Notes:

Suggested sampling: --temp 0.7 --top-p 0.95 --top-k 20.
Audio is processed in 30 s windows; audio longer than 30 s is split into consecutive windows.

Conversion details

LLM: Qwen3AVLForConditionalGeneration → qwen3vl architecture (36 layers, interleaved M-RoPE [24,20,20], rope θ = 5M, visual DeepStack = 3 layers). Converted losslessly to BF16, then quantized with llama-quantize (imatrix for Q5_K_M / Q4_K_M).
Vision mmproj: standard Qwen3-VL ViT (24 layers, DeepStack taps at 5/11/17) — unchanged upstream path.
Audio mmproj: new qwen3avl projector — fine-tuned Whisper-large-v3-turbo encoder (32 layers, 128 mel bins, learned positional embeddings from the fine-tune), main projection LN→FC→GELU→FC into the LLM embedding space, and three DeepStack mergers whose outputs are stacked along the feature dimension exactly as the LLM's qwen3vl DeepStack mechanism expects. Audio tokens: ceil(ceil(samples/160)/2) ≈ 50 tokens/second, marked with <|audio_start|>/<|audio_end|>.
Conversion used llama.cpp @ ea63b4d + the included patch; the tokenizer round-trips identically to HF on kk/ru/en probes (exact token-ID match).

Quality checks

Perplexity and KL-divergence vs the BF16 GGUF on held-out trilingual (kk/ru/en) Wikipedia text (~600 KB, 122×2048-token chunks):

Quant	Perplexity	PPL ratio vs BF16	Mean KLD	Median KLD	Same top-1 token
BF16	6.179	—	—	—	—
Q8_0	6.178	1.0008	0.0126	0.0007	97.6 %
Q6_K	6.208	1.0056	0.0297	0.0031	95.3 %
Q5_K_M	6.181	1.0013	0.0518	0.0058	93.9 %
Q4_K_M	6.064	0.9823	0.1079	0.0152	90.3 %

License

Apache-2.0, same as the original model. This repository only provides GGUF conversions and the llama.cpp audio-support patch.

issai/Qolda-AVL-5B-GGUF

Qolda-AVL-5B-GGUF

Files

Modality support

Usage

Text + Vision (stock llama.cpp)

Text + Audio + Vision (patched llama.cpp)

Conversion details

Quality checks

License

issai/
Qolda-AVL-5B-GGUF