The analyst's llama.cpp quantized GGUFs (Q4_K_M, Q5_K_M, Q6_K, Q8_0) produce corrupted output — a run of ! tokens — under Metal / GPU offload (n_gpu_layers != 0) on Apple Silicon. This is a llama.cpp Metal quantized-kernel issue specific to this Qwen3-1.7B architecture: every quantized GGUF is affected regardless of bit-width (verified Q4→Q8), while the full-precision bf16 GGUF runs correctly on Metal, and all quants run correctly on CPU. So ollama run (which offloads to Metal by default) is silently broken for the quantized GGUFs.

This MLX build uses Apple's own Metal kernels and runs correctly and fast on the GPU, at half the bf16 size. On Apple Silicon this is the recommended local-inference path.

Path (Apple Silicon)	Metal/GPU correct?	Size	Notes
This MLX repo (8-bit)	yes	1.7 GB	recommended for Mac GPU
`bf16` GGUF (main repo)	yes	3.2 GB	large; llama.cpp / ollama GPU
quantized GGUF (main repo)	no — CPU only	~1 GB	run with `n_gpu_layers=0`
Transformers + MPS (`bf16`)	yes	3.44 GB	production / reference path

Quick Start

pip install mlx-lm

python

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("opena2a/nanomind-security-analyst-mlx")

SYSTEM = (
    "You are NanoMind, a security analysis model specialized in AI agent "
    "security. You analyze artifacts, configurations, and behaviors from AI "
    "agent systems. You provide structured security assessments with "
    "reasoning. Your domain is strictly AI agent security within the "
    "OpenA2A ecosystem."
)
artifact = open("SOUL.md").read()  # or an MCP config, skill file, Dockerfile, ...
user = f"Analyze this AI agent artifact for security threats.\n\n<artifact>\n{artifact}\n</artifact>"

# Hand-rolled ChatML matches the training/eval contract exactly. Greedy decoding.
prompt = (
    f"<|im_start|>system\n{SYSTEM}<|im_end|>\n"
    f"<|im_start|>user\n{user}<|im_end|>\n"
    f"<|im_start|>assistant\n"
)
ids = tokenizer.encode(prompt, add_special_tokens=False)
print(generate(model, tokenizer, prompt=ids, max_tokens=512,
               sampler=make_sampler(temp=0.0), verbose=True))

The model emits a Markdown assessment with ## Analysis, ## Verdict, ## Evidence, and ## Remediation sections. The ## Verdict block carries the scored fields:

## Verdict

classification: malicious
attackClass: privilege_escalation
confidence: 0.95
severity: high

Output begins with an empty <think></think> block (Qwen3 base artifact); the regex parser skips it. Read classification, confidence, and severity together — see the limitations on the main model card.

Conversion + verification

Converted from the SHA-verified v3.0.0 safetensors with mlx_lm.convert (--q-bits 8 --q-group-size 64, 8.5 bpw). Gated before publish on a balanced oracle-500 eval subset run under Metal:

Metric (8-bit, MLX / Metal, n=100)	Value	`bf16` reference
Garbage-output rate	0%	0%
Binary accuracy	0.96	0.96
Malicious recall	0.98 (49/50)	0.96 (48/50)
Benign specificity	0.94 (47/50)	0.96 (48/50)
Verdict-structure adherence	0.99	1.00
Throughput	~140–190 tok/s	108 tok/s (bf16 GGUF)

An MLX 4-bit build was also produced and evaluated but not published: it runs on Metal without corruption, but its malicious recall drops to 0.88 (44/50) — it misses real injection / exfiltration attacks that this 8-bit build and bf16 catch. For a security model that recall loss outweighs the smaller size, so 8-bit is the published Apple-Silicon artifact.

Full model details, intended use, and known limitations (off-topic hallucination, dual-use over-flagging, injection recall) live on the main model card. This build inherits all of them — it is the same weights in MLX format.

License

Apache-2.0. See the main repo for citation and provenance.