8-bit MLX quantization of nvidia/NVIDIA-Nemotron-Parse-v1.2, a 936M-parameter encoder-decoder OCR model (C-RADIO vision encoder + mBART-style decoder) for document parsing with spatial grounding: formatted text, bounding boxes and semantic classes, ordered by reading flow. Converted with mlx-vlm (mlx 0.32.0) for inference on Apple Silicon.

The model is image-to-text with a task prompt: the prompt tokens seed the decoder (matching the Hugging Face reference), and the prompt selects the output mode (<predict_bbox>, <predict_classes>, <output_markdown>, <predict_no_text_in_pic>). Without a prompt the model underperforms badly — measured 0.855 field accuracy prompt-less vs 1.0 with the task prompt — so always pass the task prompt, as in the usage below.

Fidelity was measured against the source, not assumed. The numbers below and the method that produced them are in the tables that follow.

Loading requires the nemotron_parse architecture port, new in mlx-vlm (PR #1866). Until it merges, install the port branch: pip install git+https://github.com/axiom-of-choice/mlx-vlm@feat/nemotron-parse

Usage

bash

pip install mlx-vlm
python -m mlx_vlm generate \
    --model mlx-community/Nemotron-Parse-v1.2-8bit \
    --image document.png \
    --prompt "</s><s><predict_bbox><predict_classes><output_markdown><predict_no_text_in_pic>" \
    --max-tokens 1024

The output is the model's native format: text with <x_..><y_..> coordinate tokens and <class_..> labels plus markdown tables, ordered by reading flow.

Quantization


Bits	8
Group size	64
Mode	affine
Effective bits/weight	13.798
Size on disk	1.50 GiB
Quantized tensors	102 (decoder, group 64)
Unquantized tensors	766 (vision tower + neck, bf16)

Effective bits per weight exceeds the nominal 8 because mlx-vlm quantizes only the language side and leaves the C-RADIO vision encoder in bf16 by design. The vision tower is ~72% of all parameters, which is why the effective figure is much higher than on a decoder-only model. Verified, not assumed: of 766 unquantized tensors, zero carry a .scales entry.

Measured fidelity

Weight-level metrics compare dequantized weights against the source tensor by tensor, streamed from disk (the source ships fp32; the 8-bit SNR band for a bf16-comparable baseline is 42.6–42.8 dB). No prompts, no sampling, hardware-independent.

Variant	bpw	SNR	Relative L2	Cosine	Decode tok/s	Peak RAM
4-bit	12.623	20.86 dB	9.06%	0.995902	262.2	3.10 GB
8-bit	13.798	42.71 dB	0.73%	0.999973	231.8	3.23 GB
bf16 (source)	16.001	n/a	n/a	n/a	73.5	3.49 GB

Decode rates are pure decode after the first token; the first token takes ~18 s because the 2048×1664 encoder pass runs once per image. Measured on an M2 Pro / 32 GB, greedy decoding, repetition penalty 1.1.

OCR accuracy vs ground truth

Because OCR has ground truth, transcription can be scored directly rather than inferred from token distributions. Six documents were rendered locally with exactly known content (invoice, clinical lab report, shipping label, receipt, motor-controller datasheet, rotated receipt), transcribed with the NVIDIA task prompt (greedy, repetition penalty 1.1) and scored per field.

Variant	Field	Content	Numeric	CER vs truth
bf16	1.0	1.0	1.0	1.897
8-bit	1.0	1.0	1.0	1.897
4-bit	0.9868	0.9868	0.9902	1.897

8-bit transcribes byte-identically to bf16 on all six documents.
The single 4-bit miss is one field on the receipt (11.45); everything else is perfect, including the datasheet table that degrades on prompt-less runs.
CER vs truth is high because the task-prompt output carries coordinate and class tokens (<x_..><y_..>, <class_..>) that are not part of the ground truth; field/content/numeric are the correctness metrics.

What was not measured

No standard task benchmarks (MMLU, GSM8K) were run — the model is not a general-purpose chat model. Table-reconstruction fidelity beyond the six rendered documents, and long multi-page documents, were not evaluated here. The quality of the parsed output on your own documents depends on their layout similarity to the training distribution; measure on your own data.

Notes

The untied output head of v1.x checkpoints (lm_head.weight, byte-identical to the shared embedding in this checkpoint) is mapped explicitly by the port; the 2.0 checkpoints reconstruct it from the tied embedding.
See the original model card for capabilities, intended use and limitations. All credit for the model belongs to its authors.