To our knowledge, this is the first publicly released ternarized
Mixture-of-Experts model. All prior published ternary conversions
(PT-BitNet, PT2-LLM, Falcon3-1.58bit, Llama3-8B-1.58) target dense models.
Every one of the model's 24,576 expert weight matrices (512 experts x 48
layers, gate/up/down projections) was quantized to ternary values
{-1, 0, +1} with per-block scale and offset, using GPTQ-style error-
compensating reconstruction. The conversion ran for 34 hours on a single
NVIDIA Quadro RTX 4000 (8GB) in a 2016-era dual-Xeon workstation.
What was quantized
Component
Treatment
MoE experts (mlp.experts.*) — >90% of parameters
ternary (group-128, asymmetric)
Shared expert
ternary
Starved experts (<32 calibration tokens routed)
left bf16 (see below)
Router / gate
untouched bf16
DeltaNet linear attention, full attention
untouched bf16
Embeddings, norms, lm_head
untouched bf16
The trunk was deliberately left at full precision: it is a small fraction of
total parameters and by far the riskiest part to quantize aggressively.
Measured results
Reference machine: 2x Intel Xeon E5-2697 v4 (2016, no AVX-512, no AMX),
256GB DDR4-1866, Quadro RTX 4000 8GB. llama.cpp b10331,
flags -ngl 999 -ncmoe 999 -fa 1 -t 18 (attention on GPU, experts in system RAM).
Quality (wikitext2 perplexity, 2048 ctx)
Model
PPL
bf16 original
5.75
this ternary conversion
8.69 (+51%)
reference: naive rounding to ternary, no calibration
~8,960x worse
reference: PT2-LLM published dense LLaMA-2-7B ternary
11.56
Speed & size (vs the stock Q4_K_M release of the same model)
Q4_K_M
this model (Q2_K)
delta
generation
16.74 tok/s
23.74 tok/s
+42%
prompt processing
58.50 tok/s
107.08 tok/s
+83%
size
45.08 GB
27.25 GB
-40%
Coherence (greedy decode)
Q: What are the three primary colors?
A: The three primary colors are red, blue
"The capital of France is" -> "Paris. The capital of the United States"
"def reverse_string(s):" -> "\n # Write your code here.\n"
Honest limitations
This is meaningfully worse than the original. A +51% perplexity increase
is real and you will likely notice it on hard reasoning and long-form code.
If you need maximum quality, use the standard Q4_K_M release.
Calibration was wikitext2 only (128 x 2048 tokens) — generic English prose
for a model trained on code, math, multilingual and agentic data. Domain-
matched calibration would very likely improve results.
SSR (Structural Similarity-based Reordering) was not enabled, nor was a
distillation teacher used. Both are known to help.
The GGUF is Q2_K (2.94 bpw), not a true 1.58-bit format. llama.cpp's
native ternary formats (TQ1_0/TQ2_0) are symmetric with 256-element blocks;
the GPTQ reconstruction produces asymmetric group-128 ternary (measured:
exactly 3 distinct values per 128-block, offsets averaging |mu|/alpha =
0.10-0.19). Q2_K stores a scale and min per sub-block and represents this
far more faithfully. A native asymmetric group-128 ternary kernel would
recover the remaining ~0.9 bpw and push speed toward ~35 tok/s.
MTP (multi-token prediction) head is not included (--no-mtp), so
speculative decoding is unavailable.
Single eval set. No MMLU / HumanEval / GSM8K numbers yet.
Novel finding: expert utilization varies with depth
Experts routed fewer than 32 calibration tokens were left in bf16 rather than
quantized on unreliable statistics. The count of such "starved" experts is not
uniform across the network:
layers
starved experts (of 512)
0-31 (early/middle)
20-70
46-47 (late)
114-131
Later layers concentrate routing on fewer experts. We have not seen this
measured elsewhere and it has direct implications for MoE quantization: a
uniform calibration budget under-serves deep layers.
Bug found in PT2-LLM
solve_closed_form_alpha_mu() in the upstream PT2-LLM quantizer has its
epsilon commented out, so denom reaches exactly 0 for all-zero ternary rows,
producing NaN that silently propagates through the entire weight matrix.
This is harmless on large dense matrices but fatal on small expert matrices:
484 of 512 experts failed before the fix, 0 after. Fixed by clamping the
denominator and applying the AGA fallback per-row instead of per-matrix.
Tune -t to your core count; on dual-socket systems, single-socket thread
counts often beat using all cores (NUMA contention). On this machine 18 threads
was fastest for generation, 24 for prompt processing.
