Nemotron-3-Embed-1B-oQ5-MLX is an MLX-formatted checkpoint optimized for Apple silicon, published by groxaxo.
It is intended for open-source evaluation, reproducible experimentation, and compatible local or
hosted inference workflows. The wording below is deliberately limited to what can be verified
from this repository's metadata and artifacts.
Use an up-to-date MLX-compatible runtime on Apple silicon and point it at this repository:
mlx_lm.generate --model groxaxo/Nemotron-3-Embed-1B-oQ5-MLX --prompt "Write a concise technical summary."
Embedding and audio repositories may require the task-specific MLX package documented by the
upstream project.
Compatibility and responsible use
Use a runtime that explicitly supports this format, architecture, and modality.
Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
Review the source model card and license before redistribution or deployment.
Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.
Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.
Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for
testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.
This checkpoint was quantized locally on a 24 GiB Apple-silicon Mac using
jundot/omlx. OMLX measured all 16 encoder
layers with 128 calibration samples × 256 tokens, selected a base 5-bit affine
plan with group size 64, and promoted 15 sensitive projections to 6 or 8 bits.
The model preserves the source embedding contract:
12 passages read directly from tracked documentation and package.json
one labelled relevant passage per query
identical query: and passage: role formatting
AWQ evaluated through its live vLLM endpoint
oQ5 evaluated directly through MLX
Metric
AWQ W4A16 / vLLM
OMLX oQ5 / MLX
Top-1 accuracy
90%
90%
Mean reciprocal rank
0.933333
0.933333
Mean relevant-score margin
0.191203
0.191718
Output dimensions
2,048
2,048
Cross-format agreement:
retrieval-score Pearson correlation: 0.999085
mean absolute score delta: 0.005904
mean same-input AWQ↔oQ5 vector cosine: 0.994057
minimum same-input vector cosine: 0.992844
identical top-three ordering for all ten queries
Both models made the same one miss: for a query asking which package script
builds the shared types, both ranked two prose passages about shared
architecture/build flow above the shorter package.json passage containing
the exact script. This points to query/corpus ambiguity, not a
quantization-specific regression.
Important benchmark boundary
This small, domain-specific test shows that the two quantizations behave almost
identically on the Chrome MCP repository. It does not prove universal
quality equivalence.
The AWQ repository includes the broader held-out English PAQ and Spanish
MIRACL-ES evaluations. This oQ5 release has currently been validated with the
Chrome MCP comparison above plus direct BF16 fidelity checks; it has not yet
been rerun on the AWQ repository's 512-pair English and Spanish suites.
BF16 fidelity check
Three role-prefixed inputs—a query, relevant passage, and unrelated
passage—were embedded with both the official BF16 source and this oQ5
checkpoint.
Input
BF16↔oQ5 vector cosine
Query
0.995496
Relevant passage
0.995863
Unrelated passage
0.994637
Semantic scores:
Comparison
BF16
oQ5
Query ↔ relevant passage
0.680575
0.687804
Query ↔ unrelated passage
-0.024595
-0.012204
All outputs were 2,048-dimensional and L2-normalized.
Quick start on Apple Silicon
Create an environment with MLX-LM and download the checkpoint:
1python ./Nemotron-3-Embed-1B-oQ5-MLX/embed_mlx.py \2 --model ./Nemotron-3-Embed-1B-oQ5-MLX \3 --query "Which city is known as the City of Sails?"\4 --passage "Auckland is widely known as the City of Sails."
The script prints the vector dimensions, L2 norms, and query-to-passage cosine
similarity.
Why the included encoder path matters
This is a bidirectional embedding checkpoint, not a causal language model.
Calling the normal mlx-lm generation model interface would construct a causal
mask and would not reproduce the intended embeddings. embed_mlx.py loads the
bare Ministral3 language-model backbone, runs each block without a causal
mask, mean-pools token states, and L2-normalizes the output.
OMLX support for correctly calibrating this checkpoint was contributed in
jundot/omlx#2410. Until that PR is
merged, the tested implementation is also available on the
groxaxo/omlx feature branch.
