GGUF quantizations of CohereLabs/North-Mini-Code-1.0,
a 30.5B-total / ~2.9B-active sparse MoE code model by Cohere (cohere2moe
architecture: Command-R7B-style hybrid SWA/full attention with NoPE on global
layers, parallel residual blocks, 128 fine-grained experts with sigmoid top-8
routing, reasoning-by-default chat format). Trained with SFT followed by RL
with verifiable rewards, aimed at agentic coding and terminal/tool-use work —
see the release blog post.
Status / requirements:cohere2moe support is merged into llama.cpp
(PR #24260 plus the
#24601 tokenizer fix) — these
files run on a current stock llama.cpp, no custom branch needed. Validated on
upstream master: a fresh BF16 conversion is tensor-identical to these weights
(all 442 tensors, sha256), and the quants load and generate correctly end to end.
Weights are released under Apache 2.0, and these files inherit that license.
Quants
All quality numbers are measured against the bf16 model as ground truth.
The headline table uses wikitext-2 (test) — the only evaluation set that is
fully held out from the imatrix calibration data — plus HumanEval/HumanEval+
(pass@1, greedy, thinking on, 6k token budget).
file
size
PPL
mean KLD
top-1 %
HumanEval
HumanEval+
BF16 (2 shards)
61.0 GB
7.7126
—
—
Q8_0
32.4 GB
7.7356
0.007010
96.458
92.07
89.02
Q6_K
25.1 GB
7.7558
0.015611
94.602
93.29
88.41
Q5_K_M
21.7 GB
7.8333
0.020963
93.811
95.73
92.68
Q4_K_M
18.6 GB
7.9468
0.041855
91.342
93.29
90.24
IQ4_XS
16.4 GB
7.9794
0.049137
90.705
92.68
88.41
IQ3_M
13.6 GB
8.2776
0.112035
85.919
90.85
87.20
IQ2_M
10.3 GB
9.9756
0.283656
77.616
84.15
79.88
IQ2_XS
9.2 GB
11.0666
0.426120
73.339
79.88
77.44
IQ2_XXS
8.3 GB
12.6780
0.549859
69.743
59.15
59.15
HumanEval is pass@1 over 164 problems, so single-token greedy flips on a handful
of problems move the score by a few points - read it as a sanity check, not a
fine-grained ranking. The Q4-through-Q8 quants are statistically interchangeable
on it (the spread is noise); mean KLD and top-1 % are the reliable quality
ordering. The slope only becomes clear lower down: IQ3_M holds up, the IQ2 tier
degrades visibly, and IQ2_XXS falls off a cliff (identical HumanEval/HumanEval+
is the giveaway - it produces enough malformed code that the extra tests prune
almost nothing further).
Recommendations: Q5_K_M if you have the memory (effectively lossless),
IQ4_XS for the best size/quality ratio (matches Q4_K_M at -2.2 GB),
IQ3_M as the smallest quant still reasonable for code. The IQ2 tier exists
for memory-constrained setups and degrades noticeably - use with expectations
set accordingly. Embeddings are tied (also the output head) and kept at q6_K
on Q4_K_M and below.
Per-domain breakdown
The three sets below are also part of the imatrix calibration corpus, so their
numbers carry a mild in-distribution bias - read them as domain comparisons
rather than held-out scores. All corpora are included in
eval-corpora.tar.zst for reproduction.
General / multilingual (calibration_datav3)
bartowski's calibration_datav3:
the de-facto community calibration mix - short English prose, multilingual
snippets, code fragments, technical text and deliberate noise sections
(~275 kB).
file
PPL
mean KLD
top-1 %
BF16
9.0079
—
—
Q8_0
9.0261
0.008424
96.788
Q6_K
9.0351
0.014500
95.286
Q5_K_M
9.0491
0.019470
94.506
Q4_K_M
9.1607
0.036786
92.031
IQ4_XS
9.1125
0.039540
91.882
IQ3_M
9.4710
0.087992
87.714
IQ2_M
10.2735
0.208782
80.580
IQ2_XS
11.1268
0.319906
76.376
IQ2_XXS
12.3083
0.427367
72.173
Code
A seeded random sample of real source files from the
llama.cpp tree (MIT): C/C++ core and
ggml, Python conversion tooling, shell scripts; capped at 25 kB per file,
~400 kB total. Note how confident the model is on code (PPL ~2.4) - and that
top-1 agreement holds up better here than on prose at every quant level.
file
PPL
mean KLD
top-1 %
BF16
2.4043
—
—
Q8_0
2.4108
0.005231
98.512
Q6_K
2.4123
0.008321
97.731
Q5_K_M
2.4155
0.012198
97.145
Q4_K_M
2.4314
0.025947
95.898
IQ4_XS
2.4452
0.030205
95.472
IQ3_M
2.4996
0.072891
92.991
IQ2_M
2.7561
0.186894
88.646
IQ2_XS
3.0247
0.290555
85.260
IQ2_XXS
3.2342
0.368478
83.263
Chat (model-native format)
Hand-written for this release: 13 short programming conversations
(Python/SQL/C/Rust/git topics, two in German), each with a thinking block,
plus one complete tool-call round trip - rendered in the model's raw turn-token
dialect (<|START_OF_TURN_TOKEN|>, <|START_THINKING|>, <|START_ACTION|>,
...). This exercises the control-token and expert-routing paths that real chat
traffic hits and plain text never does. Small set (~7 chunks) - treat the
numbers as indicative.
file
PPL
mean KLD
top-1 %
BF16
1.9660
—
—
Q8_0
1.9866
0.022651
98.431
Q6_K
1.9906
0.031189
98.170
Q5_K_M
1.9820
0.025972
97.778
Q4_K_M
1.9641
0.070232
96.993
IQ4_XS
1.9866
0.058722
96.601
IQ3_M
2.0809
0.081966
94.902
IQ2_M
2.1412
0.173477
92.288
IQ2_XS
2.1742
0.251918
89.412
IQ2_XXS
2.2247
0.297151
87.974
Reasoning / chat template
These GGUFs embed Cohere's current canonicalchat_template.jinja (also in
this repo) with one additive tweak: the standard enable_thinking convention is
aliased onto Cohere's native reasoning / reasoning_effort toggle, so
llama.cpp's thinking flag works alongside the native one. Cohere's own
reasoning / reasoning_content handling is used as-is, so both conventions work
and native invocations render identically.
Why this matters: the model ships two templates — the canonical
chat_template.jinja and a legacy one inside tokenizer_config.json. A naive
stock conversion currently embeds the legacy one (old "your name is Command"
preamble, no thinking block), which makes the model spill its reasoning into the
answer. These files use the correctchat_template.jinja.
tool calling works through the OpenAI-compatible API (parallel calls included)
Known llama.cpp issue (unrelated to these files): passing --special can abort
while parsing the response (Failed to parse … <|END_OF_TURN_TOKEN|>) — a parser
bug with the model's turn format; just don't pass --special.
imatrix
North-Mini-Code-1.0.imatrix (included) was computed on the bf16 model over
the v3 + code + chat mix described above (326x512-token chunks), reaching full
coverage of all 128 experts in every layer.
Validation
Runs on stock upstream llama.cpp (post #24260 + #24601): a fresh conversion of
the released checkpoint on master is tensor-identical to these GGUFs (all 442
tensors, sha256), and load + generation were verified on a master CPU build.
f32 logit-level parity vs HF transformers on a truncated-expert variant of
the checkpoint (full-vocab comparison at every position): top-1 agreement
26/27, mean |dlogprob| 0.012 - the only disagreement a 0.013 near-tie.
Tool calling, parallel calls, multi-turn with reasoning passback, and a live
agentic tool-execution loop verified end to end via llama-server.
The official model card states 256K input / 64K output context; the config's
max_position_embeddings is 500k. KV cache at long context stays small
thanks to iSWA (only 13 of 49 layers are global; ~13.6 GB KV at 500k).
