GLM-4.7-Flash quantised to int4 with f32 scales per group of 64, in the
container format the TootegaEngine
reads. 19.3 GB, resident in RAM, decoding on CPU only.
This file does not load in transformers, vLLM, llama.cpp or anything else..tootega is not GGUF, not safetensors and not a variant of either — it is the
format of one engine, specified in
docs/14.
If you were looking for a quantisation that runs on your usual stack, this is
not it. What is here is the artefact that engine consumes, published so that
nobody has to download 62.4 GB and convert it again.
Files
file
size
what it is
model.tootega
19.3 GB
the container: weights, scales, index and the full config.json inside
tokenizer.json
20 MB
HuggingFace tokenizer, copied unchanged from the base model
chat_template.jinja
3 KB
reference only — the engine reproduces this template in C, without Jinja
config.json
1 KB
the architecture, also embedded in the container
The engine reads the architecture from the container, never from a file
alongside it. config.json is here so the metadata is visible on this page.
How to run
powershell
1git clone https://github.com/HermesSilva/TootegaInference
2cd TootegaInference
3.\build.ps1 # MSVC, found via vswhere; Linux uses the Makefile4.\tootega.exe serve model.tootega --tokenizer tokenizer.json
The web chat comes up at http://127.0.0.1:8080. Also available:
powershell
1.\tootega.exe chat model.tootega --tokenizer tokenizer.json
2.\tootega.exe generate model.tootega --tokenizer tokenizer.json --prompt "..."-n 64
3.\tootega.exe inspect model.tootega # header, index and budget4.\tootega.exe plan model.tootega --ram-gb 32
inspect opens and validates the file: magic, version, the 64-byte alignment of
every blob and the relationship between shape, group and each blob's size. If
it opens, the container is intact.
Measured, on the machine below
Reference machine: i9-14900K (8 P-cores), 64 GB DDR5-4800 dual channel,
RAID0 NVMe. No GPU is used — a CUDA backend was built and measured on a
separate branch, and rejected: the granularity was wrong, and no GPU code lands
on master until it shows a gain in the complete decode.
memory bandwidth 69.4 GB/s measured at runtime, not from the spec sheet
traffic per token 2.38 GB tensors a decode actually touches
roofline ceiling 29.1 tok/s bandwidth / traffic
warm decode ~50 ms/token = ~19.8 tok/s
mean decode 17.12 tok/s includes the cold first prompt (59% of ceiling)
cold decode 84 ms/token = 1.7x worse than warm
mean ttft 2.51 s
prefill 10.4 tok/s
RSS 16.69 GB peak
KV 108.3 KB per position; 221.8 MB for 2048
Matmul in isolation reaches 62 GB/s — 90% of bandwidth. The gap to the ceiling is
in attention and thread barriers, not in the matmuls.
The 2.38 GB of traffic per token is what makes the number: only 4 of 64 experts
run per layer, and the whole of attention is int4 here.
Quality
Generation against the original BF16 weights — twelve prompts, up to 96 tokens,
greedy, this container and the previous one measured in the same run against
a BF16 reference generated in that same run:
container
match@ref
rep4
tok/s
BF16 (reference)
96.0
7.3%
3.34
previous (plain scale, int8 attention)
32.4
5.3%
8.43
this one
42.7
6.8%
10.98
match@ref counts the initial tokens identical to the BF16 generation before
diverging; rep4 is the fraction of repeated 4-grams, where above ~15% means
degeneration. The scale search bought a third more matching tokens, and this
comparison is paired by construction — same prompts, same reference, one run.
property
result
measured on
teacher-forced oracle, independent implementation
32/32 identical tokens, 1e-4% relative error
this container
generation match@ref against BF16
42.7 against the previous container's 32.4
this container
perplexity against BF16, paired, 10 windows of 384 tokens
+2.86%, interval [-2.8%, +8.8%] — inconclusive
this container
determinism
byte-identical logits from 1 to 20 threads
this container
perplexity, 384 positions of prose, BF16 weights
engine 8.63 against transformers' 8.89
earlier container
tokenizer against HuggingFace tokenizers
36/36 cases, exact round-trip
independent of container
Determinism is not a nice property here, it is the one that makes every other
number reproducible: the reduction order is fixed and independent of how the work
is split, so --threads changes the time and nothing else. Verified by dumping
19.8 MB of f32 logits at 1 thread and at 20 and comparing the files.
The perplexity row says inconclusive and means it. The interval crosses zero
because one window of the corpus has a perplexity near 4,600 — a table-heavy
stretch where nothing is predictable — and it alone is four times any other
window's difference. Dropping it would give +5.4% with an interval clear of
zero, and that number is deliberately not the one published: choosing which
window to discard after seeing the result is how a measurement becomes an
opinion. The corpus is the thing to fix, not the arithmetic.
So: generation says this container is closer to the original weights than the
previous one, and perplexity on this corpus cannot tell them apart. Both are
stated because both were measured.
The paired comparison is the one that counts: comparing two configurations by the
global mean swallows the effect in the variance between windows, and that mistake
once produced a perplexity measurement with the sign inverted.
Precision policy, per tensor
Not everything is int4, and the split comes from a paired ablation, not from
taste:
group
precision
why
experts, attention
int4 g64
9,165 tensors, 16.5 GB — this is where the model's size is
lm_head, MTP layer
int8 g64
198 tensors, 1.3 GB; int4 on the MTP head drops draft acceptance to 0–4%
norms, router, e_score_correction_bias
f32
the router decides which experts run: an error swaps the expert, not the last bit
kv_a_proj_with_mqa
f32
feeds the compressed KV of every position
embed_tokens
bf16
read one row per token; never resident
The ablation found only one group with an effect above the noise: the experts
(−0.0225 ± 0.0076 nll when restored to BF16). Attention gives −0.0007 ± 0.0019 —
noise. An earlier version of this table said the opposite; that measurement had
the RoPE wrong and compared global means instead of paired windows.
Quantisation detail
Symmetric, per group of 64 along the input dimension, no zero-point, one f32
scale per group. Nibbles packed half/half inside the group: byte[j] carries
weight j in the low nibble and j + 32 in the high one, which unpacks with
AVX2 into two contiguous vectors with no activation shuffle.
The scale is notabsmax/qmax. That choice optimises the group's worst
weight instead of the group; shrinking it clips the extremes and tightens the
step for all the others. The converter searches 13 fractions of absmax between
0.70 and 1.00 and keeps the one with the lowest squared error.
Measured over every quantised tensor of this container, relative error of a
matmul's output — the metric that predicts quality; per-weight error in isolation
does not:
scale
int4 g64
worst int4
int8 g64
n (int4)
pure absmax/qmax
10.81%
14.13%
0.62%
9,161
13-point clip grid
9.52%
12.25%
0.62%
9,165
−11.9%, with no change to the format — still one f32 scale per group — and no
change to the kernel, which cannot tell the difference. int8 does not move: with
255 levels absmax is already optimal, and the search stays there only to avoid
opening a special case.
Side effect that pays for itself: with a scale below absmax/7 the code -8
becomes reachable. With the exact scale, round-to-nearest never emits it and int4
uses 15 of the 16 levels the dequantiser already knows how to read.
Sanity check on the numbers: 9.52/0.62 = 15.4, against a step ratio of
127/7 = 18.1. It sits below 18 precisely because only int4 gains from the
search. A ratio far outside 15–18 would be a bug, not quantisation.
Provenance and licence
Derived from zai-org/GLM-4.7-Flash,
MIT. The weights are theirs; the quantisation, the container format and the
engine are TootegaEngine, also MIT.
tokenizer.json and chat_template.jinja are copied unchanged from the base
model.
Reproducing this file from the original checkpoint:
The conversion takes 171 s on 8 threads and streams one tensor at a time —
the 62.4 GB source and the 19.3 GB output are never both in memory. --dry-run
prints the plan and the estimated size without writing anything.
What is not here
Batching and slots. One request at a time; a second client waits in the TCP
backlog, silently.
MTP speculation and persistent state: the tensors are in the container,
the paths are not implemented.
GPU. Master is pure CPU by decision, not by omission.
A guarantee that int4 is good enough for your case. +5.26% perplexity is a
measurement, not a verdict. Measure on your own task.
