An experimental MLX affine mixed-precision quantization of google/gemma-4-12B-it-qat-q4_0-unquantized. This release contains the extracted language-model path only: the upstream checkpoint is multimodal, but 11 multimodal tensor groups were intentionally omitted. Image, audio, video, and any-to-any inputs are not supported by this repository.
This is a personal experimental research preview. It is a compact release candidate, not a claim of lossless compression or equivalent quality to bf16. Feedback and reports from people trying it on Apple Silicon are welcome through the maintainer account maglun.
The release benchmark used fixed per-item seeds, temperature=1.0, top_k=64, top_p=0.95, repetition penalty 1.08 with context 4096, an 8K generation cap, and suppression of the upstream modality boundary IDs 258883 and 258882. The public command above uses the same intended-use sampling values; exact seed control is benchmark-harness-specific.
The selected root chat_template.jinja is the current official 12B/QAT-compatible template pinned for this release. The original historical template is retained only as a labelled reference at reference/chat_template.original.jinja.reference and is not a loader-recognized template file.
Serialized safetensors bytes: 5,502,187,879 bytes across two shards.
Parameter denominator used for bpw: 11,907,350,320.
Scope: text generation only; no multimodal processors or modality tensors are included.
QAT identifies the upstream source lineage. GPTQ describes a rounding/build stage. This published artifact is not GGUF, AutoGPTQ, or a uniform q4_0 model, and it does not use AWQ.
Evaluation
The release comparison was run with thinking enabled on the candidate and the direct QAT-unquantized bf16 text-path anchor, using identical fixed seeds, prompts, decoding settings, scorers, and suppression contract. Errors and capped outputs remain failures in the full denominators.
Task
Candidate
Direct QAT-bf16
Candidate − bf16
IFEval strict prompt
486/541 (89.83%)
496/541 (91.68%)
−1.85 pp
MMLU-Pro
258/400 (64.50%)
291/400 (72.75%)
−8.25 pp
HumanEval+
145/164 (88.41%)
146/164 (89.02%)
−0.61 pp
GSM8K
372/400 (93.00%)
376/400 (94.00%)
−1.00 pp
Unweighted mean of the four rows
83.94%
86.86%
−2.93 pp
This is one fixed-seed sampled comparison, not a claim of universal ranking. The candidate's lower scores are part of the release evidence. The paired 95% bootstrap intervals for candidate-minus-bf16 were −3.55 to +0.19 pp (IFEval), −10.70 to −1.13 pp (MMLU-Pro), −6.17 to +4.94 pp (HumanEval+), and −2.60 to +2.08 pp (GSM8K).
Wikitext, corrected chat-framed teacher forcing
Wikitext was evaluated separately with native Gemma chat framing, the instruction Continue the following text., explicit thinking disabled, and continuation-only scoring. The pinned English heldout fixture contains 41 documents; Swedish contains 80.
Corpus
Candidate PPL
Direct QAT-bf16 PPL
Candidate / bf16
English heldout
16.6255
15.0995
1.1011×
Swedish heldout
14.2315
12.5226
1.1365×
The candidate and bf16 Wikitext runs used identical document IDs, term counts, and block manifests. The initial bare-BOS measurements are retained in the project ledger as invalid diagnostic evidence and are not the release metrics.
Performance reference
Measured on an Apple Mac mini with Apple M4 Pro, 64 GB unified memory, macOS 26.5, using MLX 0.32.0 and MLX-LM 0.31.3. These are machine-specific measurements, not guarantees.
Measurement
Result
Cold load
1.08–1.09 s
Prompt prefill, 44 tokens
154.42–161.63 tok/s
Generation/decode, 256 tokens
38.42–38.52 tok/s
MLX peak after decode
5.374 GiB
8,192-token prefill MLX peak
10.316 GiB
8,192-token prefill process RSS peak
6.248 GiB
The measurements used fresh processes and sampling with thinking enabled. Do not infer decode speed by dividing generated tokens by total prompt-plus-generation wall time.
Limitations and safety
Text-only: image, audio, video, and any-to-any inputs are unsupported.
The model is substantially degraded relative to the direct QAT-bf16 anchor on the measured capability suite, especially MMLU-Pro.
Evaluation coverage is narrow and uses one fixed-seed sampled draw; it is not a comprehensive leaderboard evaluation.
Memory and speed depend on the MLX-LM and macOS versions, context length, prompt, and concurrent load. A 64 GB machine was not suitable for loading the bf16 comparison anchor.
The result depends on the QAT source lineage and the selected official chat template.
No additional safety tuning was performed. Do not use this experimental artifact for high-stakes decisions.
This is not a company-supported Precisit product, hosted service, or SLA.
