Compact mixed Q8/BF16 GigaChat Audio model for Apple Silicon. It uses less
disk and memory and runs faster than the BF16 MLX model while keeping the audio
encoder and projector in BF16.
This is an independent port, not an official release by the upstream authors.
* Time to process a 5 min 17 sec audio file on Apple M4 Pro. Lower is better.
Choose Q8/BF16 when size, memory, and speed matter; choose BF16 for the best
measured answer quality. Both variants use the same
gigachat-audio-mlx runtime.
39% smaller, 35% less memory, and 34% less processing time than MLX BF16 —
with factual errors in 3 more cases in our blinded review.
This is a compact 13.75 GB mixed Q8/BF16 derivative of the
GigaChat Audio MLX BF16
artifact. Choose it when storage, memory, and speed matter; choose BF16 for the
best measured answer quality.
What do I need to run it?
Model
Model size
Free disk to set aside
Measured whole-process memory
Processing 5 min 17 sec*
Q8/BF16
13.75 GB
about 16 GB
14.80 GiB
6.90 s · 46.0× realtime
* Time to process a 5 min 17 sec audio file on Apple M4 Pro. Lower is better.
The free-disk figure is a rounded practical reserve; the CLI dry run reports
the exact value. Memory is measured use, not a minimum-memory guarantee. This
release is verified only on a 48 GB Mac; 16/24/32 GB support is not yet claimed.
Why MLX matters on this Mac
Same Apple M4 Pro / 48 GB. Original is a stopped memory-safety smoke, not a
completed speed benchmark; MLX values are medians of five clean runs.
Runtime
Audio
Outcome
Whole-process footprint
Original PyTorch BF16 on MPS
9.84 s
No answer after 277 s; stopped
at least 51.92 GiB
MLX BF16
317.27 s
Completed in 10.41 s
22.84 GiB
MLX Q8/BF16
317.27 s
Completed in 6.90 s
14.80 GiB
The pinned upstream Transformers path is demonstrated for CUDA. On MPS it
exceeded this Mac's safe memory budget and nearly filled system swap during the
short smoke. The 5-minute Original run was not started, so no speedup ratio is
claimed. MLX is what makes the model usable on the tested Mac.
What changes from MLX BF16
Lower is better in every row.
Metric
MLX BF16
MLX Q8/BF16
Change
Model size, decimal GB
22.54
13.75
−39.0%
Whole-process memory, 5-minute profile
22.84 GiB
14.80 GiB
−35.2%
End-to-end time, 5-minute profile
10.41 s
6.90 s
−33.7%
Model size is the file size, not RAM. The 5-minute profile contains 317.27 seconds
of audio and reports the median of five clean processes on the verified M4 Pro
/ 48 GB machine.
Measured quality difference
In an independent blinded review of 64 paired timed-description and summary
items:
Model
Responses with a factual error, lower is better
MLX BF16
26/64 (40.625%)
MLX Q8/BF16
29/64 (45.3125%)
That is 3 additional reviewed cases with a factual error (+4.6875 percentage points). Most automatic ASR, translation, emotion, and temporal-localization
comparisons were close.
Quantization
decoder default: weight-only affine Q8, group size 64;
decoder BF16 whole-layer overrides: 0, 17, 21, and 23;
Conformer, audio subsampler, and audio projector: BF16;
embeddings, LM head, router/gates, norms, and incompatible tensors follow the
audited tensor manifest;
no activation quantization, custom Metal kernels, PyTorch fallback, or remote
executable code.
Review the download plan before adding --yes. The main package quick start
uses BF16; select q8-bf16 when you prefer the smaller, faster option.
One reviewed difference
Prompt paired with English audio:
Translate the speech into Russian. Return only the translation.
The source discusses Deity Yoga in Tibetan meditation. BF16 rendered the key
concept as «йога Визуализации», while Q8/BF16 changed it to «дзогчен». This is a
semantic substitution, not a harmless wording difference.
Resource evidence
On the verified M4 Pro / 48 GB machine, Q8/BF16 reduced whole-process memory by
35.2% and end-to-end time by 33.7% in the 5-minute profile. This release makes
no 90-minute support claim. The post-publication Original PyTorch/MPS check did
not complete a 9.84-second smoke: after 277 seconds its process footprint had
reached at least 51.92 GiB and system swap was nearly full. It was stopped
before the 5-minute run, so no Original Mac latency is inferred.
Limitations and responsible use
All BF16 limitations apply. Quantization can change factual details, names,
numbers, translations, timestamps, and summaries, so important outputs require
verification against the original audio. Other chips, memory sizes, and macOS
versions are not supported claims.
Do not use this model as the sole basis for medical, legal, employment, safety,
or surveillance decisions. Users are responsible for consent, privacy,
copyright, and dataset terms for audio they process.
