4-bit MLX quantization of stepfun-ai/GOT-OCR2_0, the 560M-parameter OCR-2.0 model. Converted with mlx-vlm 0.6.14 (mlx 0.32.0) for inference on Apple Silicon.

Fidelity was measured against the unquantized bf16 source and against the reference PyTorch implementation, not assumed. Every number below comes from a run recorded in this card.

Requires mlx-vlm with GOT-OCR 2.0 support, which is currently open as Blaizzy/mlx-vlm#1908. The PR is not merged yet, so a released mlx-vlm cannot load this repo.

bash

python -m mlx_vlm generate \
    --model mlx-community/GOT-OCR2_0-4bit \
    --image document.png \
    --prompt "OCR: " \
    --max-tokens 1024

GOT is not a chat model. It takes two instructions: OCR: for plain text and OCR with format: for structured output (tables, formulas, sheet music). Any other prompt is out of distribution.

Quantization


Bits	4
Group size	64
Mode	affine
Effective bits/weight	6.522
Size on disk	457 MB
Quantized tensors	169 language-model tensors
Vision tower	bf16, unquantized (0 `.scales` under `vision_tower`)
Projector	bf16, unquantized

The effective figure sits well above the nominal 4 because mlx-vlm quantizes only the language model. The vision tower and the projector are 96.7M of the 560.5M parameters, 17%, and stay bf16 by design. Packing checks out exactly: U32 x (32/4) = 463,785,984, the quantized parameter count.

Tied embeddings are 155.5M of those 463.8M, 34%, much higher than in a typical conversion where the transformer body dominates. They are the worst-hit tensor in both quantizations, which is what the per-tensor numbers below show.

Fidelity

Weight level, against the bf16 source


Weight SNR	20.53 dB
Tensors compared	169
Parameters compared	463,785,984
Worst tensor	`embed_tokens` (19.10 dB)

Exact, prompt-free and hardware-independent: the quantized weights are dequantized and compared element-wise against the source.

Against the reference PyTorch implementation

One forward pass, same image and same prompt (the MPT conversation the model was trained with), comparing the final-position logits over all 151,860 classes.

Comparison	max abs error	cosine	argmax
MLX bf16 vs torch fp32	0.47469	0.99979109	same, top-5 identical
MLX fp32 vs torch fp32	0.000130	1.0000000000	same

The second row is the one that matters: the source weights are bf16, so running the MLX port in fp32 loses nothing, and the residual is porting error alone. The first row's larger figure is bf16 rounding, not a defect.

Task level, against ground truth

Six documents rendered locally with exactly known text (invoice, lab report, shipping label, receipt, spec table, rotated receipt), transcribed with OCR: and scored directly. This sidesteps using bf16 as the reference at all.

Variant	field	content	numeric	CER vs bf16	tok/s	peak GB
bf16	0.8684	0.9605	0.9720	0 (ref)	138.3	2.50
8-bit	0.8684	0.9605	0.9720	0.0000	210.8	2.06
4-bit	0.8947	0.9474	0.9623	0.0116	272.9	1.83

field counts required strings present exactly, content ignores markup and whitespace, numeric counts ground-truth numbers recovered.

The 8-bit output is byte-identical to bf16 on all six documents, down to the same fields missed.

Two caveats a reader should have:

The 4-bit field score is higher than bf16 and that is not an improvement. The denominator is 38 strings, so one string is 0.026. 4-bit happens to get the invoice fully right and then loses 12.00 on the receipt. content and numeric, which are less format-sensitive, both move the other way. Treat the three variants as separated by one or two strings, not by field.
shipping_label loses the same three fields in every variant, bf16 included (1Z-994-AX-77310582, Halberd Components, 3011AB). That is a limit of the source model on that document, not a quantization effect.

Throughput measured on an M-series Mac during the same run, single image, generation only.

What was not measured

Only the plain 1024x1024 single-crop path. GOT's fine-grained mode (region by box or by colour), its multi-crop path for dense pages, and its multi-page mode are not exercised by anything above.
OCR with format: was not scored. The accuracy table uses OCR: only. Format mode was smoke-tested and runs, but no ground-truth scoring was done for tables, formulas or sheet music.
No perplexity, top-1 agreement or KL. GOT only emits transcriptions, so generic passages give meaningless perplexity. The ground-truth OCR scoring above replaces that layer rather than supplementing it.
Six synthetic documents, one language. They are rendered, not photographed, and English only. Real scans, handwriting and the multilingual claims of the source are untested here.
No accuracy measurement at long context. Everything ran well under the 32k window.

Source model: stepfun-ai/GOT-OCR2_0 (paper).

mlx-community/GOT-OCR2_0-4bit