Experimental low-memory GGUF quantizations of funasr/paraformer-zh. The vocabulary and CMVN tensors are embedded in each file.

The original exporter only supports F32, F16, and Q8_0. These files were exported directly from the official FP32 checkpoint with an extended exporter; they were not requantized from Q8.

Measurements used the persistent CPU Paraformer GGUF runtime on an Intel Core i7-14700KF and a 30-second Mandarin/English mixed-language clip.

Across 59 clips from the same recording, Q5_0 and Q4_0 each matched the Q8_0 token sequence exactly on 18 clips. Their aggregate token edit distance from Q8_0 was 8.8% and 10.1%, respectively. This is output drift relative to Q8, not a labeled word-error-rate measurement. Validate the quantization with your own language and acoustic conditions before replacing Q8 in quality-sensitive deployments.

These files use the custom paraformer GGUF architecture implemented by the FunASR runtime/llama.cpp/paraformer runtime. They are not language models and cannot be loaded by general-purpose llama.cpp applications.

The compatible runtime is FunASR's runtime/llama.cpp/paraformer implementation. The extended export script used for these files is included in this repository.

The export input was the official model.pt file:

Exported files:

The base model and the FunASR exporter/runtime are provided under Apache-2.0. See LICENSE and the base model repository for details.