GGUF quantizations of NbAiLab/borealis2-26b-a4b-preview, the Norwegian-centric preview
model from the AI Lab at the National Library of Norway (Nasjonalbiblioteket).
This is a format conversion, not a new model. Nothing was retrained, fine-tuned or merged. All
behaviour, quality and limitations come from the original NbAiLab release.
Unlike the MLX builds, these run anywhere llama.cpp runs — Linux, Windows, macOS, CPU or GPU.
Speeds are generation tok/s measured on an Apple M5 Max (128 GB) with Metal, llama-server -ngl 99 -fa on -c 8192, sampling at temp 1.0 / top-k 64 / top-p 0.95. Your numbers will differ.
Norwegian-calibrated importance matrix
The K-quants here (Q6_K, Q4_K_M) were built with an importance matrix calibrated on
Norwegian text, not the usual English-only calibration. This model is Norwegian-specialized, so
an English imatrix would weight the wrong activations.
The calibration corpus was 2.04 MB of mixed text — 45% Bokmål, 35% Nynorsk, 20% English — drawn from
Norwegian Bokmål Wikipedia, Nynorsk Wikipedia and English WikiText-2, with documents shuffled so
each imatrix chunk sees a mix of languages. It was computed over 1194 chunks at
n_ctx=512 against the BF16 model.
Q8_0 does not use an importance matrix — it doesn't need one.
Running it
LM Studio
Search for BobTheShoplifter/borealis2-26b-a4b-preview-GGUF in the model browser, or drop a .gguf into
~/.lmstudio/models/BobTheShoplifter/borealis2-26b-a4b-preview-GGUF/. Verified loading in LM Studio's llama.cpp runtime.
The --jinja flag matters: it makes llama.cpp use the embedded Gemma 4 chat template rather than a
built-in guess.
Sampling settings
temperature = 1.0 top_k = 64 top_p = 0.95
From the model's own generation_config.json. In testing, plain temperature sampling without
top_k/top_p produced repetition loops on longer generations. If you see looping, check your
sampler first.
Chat template and stop tokens
The upstream repo keeps its chat template in a separate chat_template.jinja (transformers 5.x
style) rather than inside tokenizer_config.json. It converted correctly — convert_hf_to_gguf.py
reads that file and embeds it as tokenizer.chat_template in every GGUF here (18681 characters,
matching the source). No gguf_set_metadata fix or --chat-template-file workaround was needed.
Just remember --jinja.
Gemma 4 uses <|turn> … <turn|> turn markers and a <|channel>thought … <channel|> reasoning
channel, not Gemma 2/3's <start_of_turn>. llama.cpp recognises <eos>, <turn|> and
<|tool_response> as end-of-generation tokens for the gemma4 architecture, so generation stops
cleanly without extra configuration.
Note for the MLX builds: LM Studio's MLX runtime reads eos_token from tokenizer_config.json
and would leave a trailing <turn|> visible. That is fixed in the MLX repos by declaring
<turn|> as eos_token. The GGUF path never had this problem.
Reasoning channel
On harder prompts the model often writes a reasoning block — frequently in English — before its
answer, even though the template's default enable_thinking=false tries to suppress it. The real
answer follows the closing <channel|>. This is upstream behaviour, not a quantization artefact.
Vision / OCR (optional)
The upstream repo ships a gemma4_vision tower but no preprocessor_config.json, and the
instruction tuning was text-only — so it is easy to assume vision is unusable here. It isn't.
mmproj-borealis2-26b-a4b-preview-BF16.gguf (1.19 GB) in this repo is the vision projector, built
from this model's own vision weights. Pair it with any of the GGUFs above:
The .gguf weights work fine without the mmproj if you only want text — it is a separate,
optional download.
Two things worth knowing:
No preprocessing parameters were guessed. This vision tower standardizes internally (the
vision_config sets standardize: true and the weights carry std_bias / std_scale), so image
mean/std are the identity [0,0,0] / [1,1,1]. That matches Google's own published mmproj for
gemma-4-26B-A4B-it byte for byte on every clip.* metadata key.
The vision tower is unchanged from the Gemma 4 base. NbAiLab's continued pre-training and SFT
were text-only, and the vision tensors are bit-identical to the base model's. So image
understanding is stock Gemma 4 — what is new is that the language side answering about the image
is Norwegian-specialized.
Tested on rendered Nynorsk newspaper text: transcription was accurate including æ ø å, and it
summarized the content in correct Nynorsk. Bear in mind the SFT never included image-text pairs, so
treat vision as a capability inherited from the base rather than one NbAiLab tuned for.
⚠️ Preview quality and safety — please read
Carried forward from the original model card, and not boilerplate:
This is a preview experiment, not a production model. Outputs may be unstable and may hallucinate.
It has not been fully safety-aligned. It may produce harmful, biased or offensive content, and
may follow harmful instructions.
Do not use it for safety-critical or high-stakes applications. Add your own safety mitigations
before deploying anything on top of it.
It is an early SFT checkpoint, not a final release.
In testing, on open-ended "write a long article" prompts the model is unreliable across random
seeds — it sometimes stops after only a title, sometimes emits degenerate text. This was verified
against unquantized BF16 and the behaviour is identical, so it belongs to the preview checkpoint
rather than to these quantizations. Short and medium-length instructions behaved well.
License — read before redistributing
This model is not Apache 2.0. It is released under the NB-License 1.0, an adaptation of
Apache 2.0 with additional use-based restrictions. On top of the usual Apache terms, you must not:
intentionally use the model to recreate data it was trained on; or
use the model or its output to power end-user services whose primary purpose is giving access to
licensed press publications contained in the training data.
Redistribution of derivatives is permitted, but the license travels with the model. The full
LICENSE and LICENSE_FAQ.pdf are included here, copied verbatim from the source repository.
Two gotchas worth knowing
llama-server can return HTTP 500 on truncated output. llama.cpp parses this architecture's
replies with a strict peg-gemma4 grammar. If a generation is cut off mid-structure (e.g. you set
max_tokens low enough that the model is still inside its reasoning channel), the server rejects
the whole response with "The model produced output that does not match the expected peg-gemma4
format". Give it enough tokens to finish, or use the raw /completion endpoint. LM Studio uses its
own reasoning parser and is not affected.
MoE expert coverage in the imatrix. With 128 experts and top-8 routing, a handful of experts
never activate during calibration. Across layers, coverage was 96–99% (1–5 of 128 experts per layer
had no data); llama.cpp falls back to default quantization for those. This is normal for wide MoE
models and the quants test clean.
Reasoning — read this before choosing a build
This model can reason, but it is off by default, and which runtime you use decides whether you
can turn it on at all.
Upstream's chat_template.jinja sets enable_thinking | default(false), which pre-closes the
thought channel with <|channel>thought\n<channel|> so the model answers immediately. Setting
enable_thinking=true injects <|turn>system\n<|think|>\n<turn|> and the model reasons first.
What actually works — measured, same prompt and settings on one machine
Runtime
Reasoning
Notes
llama-server / llama-cli with --jinja --reasoning on
✅
reasoning in reasoning_content, answer in content, no leaked markers
LM Studio + GGUF
✅
works out of the box, no configuration — measured 426 reasoning tokens
--chat-template-config / --chat-template-args / per-request chat_template_kwargs had no effect in mlx-lm 0.31.3 — use the Python API
LM Studio + MLX cannot reason
llama.cpp implements Gemma 4's reasoning channel at the architecture level — the same code that
registers <eos>, <turn|> and <|tool_response|> as end-of-generation for gemma4. LM Studio's
llama.cpp runtime inherits that and folds reasoning into a collapsible block automatically.
LM Studio's MLX runtime has no equivalent, so it never enables thinking and never parses the
channel. Identical prompt and sampling, same machine:
Build
reasoning_tokens
GGUF Q6_K
426
MLX 8-bit
0
This affects every gemma4 MLX model in LM Studio, not just this one. It is a runtime gap, not
something a quantization or a model.yaml can fix — setting enable_thinking in the template only
makes the raw <channel|> markers leak into the reply, because nothing parses them.
If you want reasoning — for agents especially — use the GGUF builds. The MLX builds remain the
better choice for everything else on Apple Silicon: faster (94–110 vs 75–109 tok/s) and more accurate
on Norwegian OCR.
bash
1# recommended for agents2llama-server -m borealis2-26b-a4b-preview-Q6_K.gguf \3 --jinja --reasoning on -ngl 99 -fa on -c 8192\4 --temp 1.0 --top-k 64 --top-p 0.95
Budget tokens generously — the reasoning block is often longer than the answer, and a truncated one
leaves content empty. The model tends to reason in English even when answering in Norwegian;
that is upstream behaviour.
llama-server parses replies with a strict peg-gemma4 grammar and returns HTTP 500 if a
generation is cut off mid-structure. If you see "output that does not match the expected
peg-gemma4 format", raise max_tokens rather than assuming the quant is broken.
All builds compared
Same model, same prompts, same machine (Apple M5 Max, 128 GB). Generation tok/s; quality is mean
NLL on held-out Bokmål vs an unquantized bf16 MLX reference (lower is better).
Build
Format
Size
tok/s
Bokmål NLL
Vision
Notes
MLX-8bit-vision
MLX
28.0 GB
95
1.635
✓
Best overall on Apple Silicon
MLX-8bit
MLX
26.8 GB
94
1.635
—
Same, ~1.1 GB less RAM
MLX-6bit-vision
MLX
21.7 GB
110
1.668
✓
Fastest of the set
MLX-6bit
MLX
20.5 GB
108
1.668
—
Fastest text-only
GGUF Q8_0
GGUF
26.9 GB
75
—
✓*
Portable, effectively lossless
GGUF Q6_K
GGUF
22.6 GB
100
—
✓*
Norwegian imatrix
GGUF Q4_K_M
GGUF
16.8 GB
109
—
✓*
Smallest working build
(bf16 reference)
MLX
47 GB
—
1.630
—
Not published; used for validation
* GGUF vision needs the separate mmproj file from the GGUF repo.
On Apple Silicon MLX is meaningfully faster than GGUF at the same precision — 94 vs 75 tok/s at
8-bit, on identical hardware and prompts. MLX also won the Nynorsk OCR comparison: it transcribed
gjere correctly where GGUF Q8_0 produced bokmål gjøre.
GGUF is the right choice if you are not on Apple Silicon, or if you want the smallest build —
Q4_K_M at 16.8 GB has no MLX equivalent, because uniform 4-bit MLX quantization breaks this model
(NLL 2.47 → 9.50) while llama.cpp's K-quants protect the attention and dense-MLP tensors that are
the sensitive ones here.
Credit
Borealis 2 was built by the AI Lab at the National Library of Norway (Nasjonalbiblioteket),
starting from google/gemma-4-26B-A4B-it, continued pre-trained on the Aurora corpus and supervised
fine-tuned on NbAiLab/aurora-sft-2606.
Requires transformers >= 5.0 to run the converter; llama.cpp's pinned transformers==4.57.6
cannot read this repo's tokenizer (extra_special_tokens is a list in the transformers-5 format).
