Text decoder extracted from Qwen/Qwen2.5-Omni-7B — all vision, audio, talker, and token2wav components removed.
This is a pure text LLM (7.62B params) that runs standalone in llama.cpp without any multimodal dependencies.
1# Step 1: Extract decoder to F16
2python convert_hf_to_gguf.py Qwen/Qwen2.5-Omni-7B \
3 --outfile Qwen2.5-Omni-7B-decoder-F16.gguf --outtype f16
4
5# Step 2: Quantize to Q8_0
6llama-quantize Qwen2.5-Omni-7B-decoder-F16.gguf \
7 Qwen2.5-Omni-7B-decoder-Q8_0.gguf Q8_0
1# Benchmark
2./llama-bench -m Qwen2.5-Omni-7B-decoder-Q8_0.gguf -t 6 -p 512 -n 128 -fa 1
3
4# Text generation
5./llama-cli -m Qwen2.5-Omni-7B-decoder-Q8_0.gguf -p "Hello" -n 200
6
7# Further quantize locally
8llama-quantize Qwen2.5-Omni-7B-decoder-Q8_0.gguf \
9 Qwen2.5-Omni-7B-decoder-Q4_0.gguf Q4_0
