MLX-format quantization of kai-os/Carnice-V2-27b — a Hermes-style SFT of Qwen3.6-27B for agentic workloads — converted for fast Apple Silicon inference.
This is the recommended default of three published variants: smallest, fastest, and quality on par with naive 4-bit on agent tasks.
Quantization
Recipe
mixed_3_6 mixed-bit (critical layers at 6-bit, others at 3-bit)
~51% faster wall-clock than the GGUF Q5_K_M on the same hardware. Per-token throughput ~1.9× the llama.cpp baseline. Quality matches or exceeds naive 4-bit on agent tasks (more complete tool-selection responses, correct severity classification on triage, well-formed JSON).
Benchmarks were also re-run with conciseness preserved — the chat template's enable_thinking: false flag must be propagated through the request (see Usage). Without it, output token counts approximately double and the wall-clock advantage is lost.
Quality (wikitext-2 perplexity)
Variant
seq 256
seq 1024
naive 4-bit
4.949 ± 0.092
3.985 ± 0.036
mixed_3_6 (this)
5.147 ± 0.097
4.073 ± 0.038
6-bit
4.881 ± 0.091
(not measured)
Evaluated with mlx_lm.perplexity --num-samples 64 --batch-size 1 --sequence-length {256,1024}. The 6-bit variant could not be measured at sequence length 1024 on M4 Pro 48 GB — its larger memory footprint plus the 1024-token KV cache exceeds the available unified memory. Numbers are comparable across rows within each column. Do not compare to externally-reported wikitext-2 perplexities without matching settings.
mixed_3_6's slightly higher perplexity than naive 4-bit at both context lengths is the expected tradeoff for its lower bits/weight (3.97 vs 4.50). The gap is preserved at longer context, indicating the lower-bit recipe does not introduce hidden long-range degradation. On the 7-prompt agent benchmark, mixed_3_6 produced more complete responses on tool selection and equivalent-or-better severity classification, so the perplexity gap did not translate into observable agent-task degradation on the prompts tested.
LM Studio's MLX runtime should load this directly via the search-and-download flow.
Hermes Agent / other agent harnesses
If you're driving this model from an agent harness, make sure the harness propagates chat_template_kwargs.enable_thinking: false to mlx_lm.server. Without it the model emits a hidden <think>...</think> block on every turn — roughly ~200 tokens of latency that's invisible to the caller.
Known mismatch with Hermes Agent's custom provider: it sends a top-level think: false field instead of the chat_template_kwargs form, and mlx_lm.server does not interpret it. The simplest workaround is a tiny HTTP proxy that rewrites the field between the agent and the server. Open a discussion if you'd like a reference implementation.
Example output
System: You find bugs. Reply with: BUG: <one-line description>, then FIX: <one-line patch description>. No code fences, no extra prose.
BUG: The function awaits an array of promises instead of awaiting all promises concurrently.
FIX: Replace `await results` with `await Promise.all(results)`.
Quality tier — closer to BF16 fidelity, ~40% slower
Limitations & out-of-scope use
This is a third-party MLX-format quantization of kai-os/Carnice-V2-27b. It is not maintained by kai-os or the upstream Carnice/Qwen teams. It inherits whatever biases, factual limitations, and safety properties the upstream model has — no additional alignment, safety tuning, or behavioral evaluation was performed during conversion.
Apple Silicon only. MLX is Apple's framework; these weights run on M-series Macs. For other hardware use the upstream BF16 weights (kai-os/Carnice-V2-27b) or a GGUF conversion.
Text-only. The upstream Carnice model is multimodal (image-text-to-text); the mlx_lm.convert pipeline used here drops the vision encoder. This release supports text input only. For image input, use the upstream BF16 weights with transformers.
Quantization artifacts. The mixed_3_6 recipe (3.97 bpw — predominantly 3-bit groups with critical layers preserved at 6-bit) is the lowest-bit variant of this release. It introduces more representation error than the 4-bit and 6-bit variants, but the 7-prompt agent benchmark did not surface degradation. Workloads with long context, complex chains-of-thought, or precision-sensitive numerical reasoning may prefer the higher-bit variants.
Issue scope. Issues specific to this MLX conversion (loading errors, quantization fidelity, file integrity) belong on this repo. Issues with model behavior (instruction following, factuality, refusal calibration, training-data concerns) are upstream concerns and should be raised on kai-os/Carnice-V2-27b.
Attribution & license
Original model: kai-os/Carnice-V2-27b — Hermes-style SFT of Qwen3.6-27B by kai-os. Apache 2.0.
This conversion: Apache 2.0, no additional restrictions. Please credit kai-os as the upstream source when discussing or comparing this model.
Citation
If you use this model, please cite the upstream Carnice release:
