GGUF weights for moss-transcribe.cpp, a from-scratch C++/ggml inference port of OpenMOSS MOSS-Transcribe-Diarize. Joint long-form transcription, speaker diarization, and timestamps in one pass, on CPU (and on GPU through ggml backends as those land), with no Python, PyTorch, or CUDA toolkit at inference. Each file is fully self-contained (weights, tokenizer, mel filterbank, and all config live inside the GGUF).
Every file was verified end-to-end against the reference on the JFK sample (CPU, greedy, 8 threads). "Transcript" is versus the original PyTorch model; "speed" is total wall time on an 11 s clip on a 20-core x86 CPU at 8 threads (the whole model, load included; the autoregressive decode is memory-bandwidth bound, so smaller weights run faster).
file
size
vs f32
wall (11 s)
speed vs f32
transcript vs reference
moss-transcribe-f16.gguf
1.8 GB
50%
4.96 s
1.6x
byte-identical
moss-transcribe-q8_0.gguf
942 MB
27%
3.97 s
2.0x
byte-identical
moss-transcribe-q6_k.gguf
733 MB
21%
4.16 s
1.9x
byte-identical
moss-transcribe-q5_k.gguf
619 MB
18%
4.47 s
1.8x
byte-identical
moss-transcribe-q5_0.gguf
619 MB
18%
3.81 s
2.1x
byte-identical
moss-transcribe-q4_k.gguf
511 MB
15%
3.81 s
2.1x
word-identical (one timestamp off 0.02 s)
moss-transcribe-q4_0.gguf
511 MB
15%
3.57 s
2.2x
word-identical (one timestamp off 0.07 s)
Which to pick: q5_k or q5_0 for the best size and accuracy (byte-identical to the reference at about one sixth the size); q4_k/q4_0 for the smallest and fastest (word-identical); q8_0 for the largest fidelity margin; f16 if you want the near-lossless full-precision equivalent. The F32 GGUF (3.4 GB, the parity reference) is not published here; produce it with the converter if you need it.
Only the large ggml_mul_mat-fed weights (Qwen3 and Whisper attention/FFN projections, the adaptor linears, and the token embedding, 343 tensors) are quantized; norms, biases, the conv stem, positional embeddings, and the mel filterbank stay F32.
Benchmarks
Same audio, same F32 weights, same threads, byte-identical transcript. moss-transcribe.cpp (ggml, CPU) stays under real time where PyTorch does not, and the gap holds as clips get longer:
RTF vs audio length: moss-transcribe.cpp vs PyTorch on CPU
Quantization makes the model both smaller and faster (the decode is memory-bandwidth bound), with the transcript byte-identical through q5:
Quantization ladder: size and speed by dtype
Full methodology and the reproducible harness are in the benchmarks.
Output is the compact [start][Sxx]text[end] transcript with inline speaker tags and timestamps, for example:
[0.28][S01] And so, my fellow Americans, ask not what your country can do for you, ask what you can do for your country.[10.59]
Set MTD_THREADS to tune CPU threads (8 is a good default on a 20-core box; the decode is bandwidth bound, so fewer busy threads often beat more).
For production serving
Use LocalAI for an OpenAI-compatible /v1/audio/transcriptions endpoint, a model gallery, concurrency, auth, and metrics.
Model
The MOSS-Transcribe-Diarize model is by the OpenMOSS / MOSI.AI team (arXiv:2601.01554), released under Apache-2.0. moss-transcribe.cpp is MIT-licensed; these GGUF weights keep the model's Apache-2.0 license.
Citation
bibtex
1@software{moss_transcribe_cpp,
2 title = {moss-transcribe.cpp: a C++/ggml inference engine for MOSS-Transcribe-Diarize},
3 author = {Di Giacinto, Ettore},
4 url = {https://github.com/mudler/moss-transcribe.cpp},
5 year = {2026}
6}
Built by the LocalAI team. If you want to run speech transcription and diarization (and LLMs, vision, voice, image, and video models) locally on any hardware with an OpenAI-compatible API, give LocalAI a star.
