The file needs the bananamend engine. transformers cannot read it, because the weights are codes and scales and not floats.
What is inside
Item
Value
Method
int8
Group size
64
Ternary matrices
0
8-bit matrices
99
Float size
100.72 MB
This file
26.79 MB
Smaller by
3.76 times
Made by
bananamendy 1.0.3
Measured quality
The numbers compare this checkpoint with the float checkpoint on a text that the quantizer never saw. The engine produced both sides.
Measure
Value
Same next token
96.8%
Next token inside the first five
100.0%
Divergence (KL)
0.0017
Perplexity
49.5 against 49.1
Identical greedy answers
7/8
Why the mixture
Ternary weights hold three values: minus one, zero and plus one. They are very small, and they lose much. Eight-bit weights are four times larger, and they lose almost nothing.
A model of this size cannot carry ternary weights everywhere. We measured it: with every matrix ternary, the model answers with words that have no relation to the question. The published work on ternary language models trains the model with the ternary grid from the start, or works on models above one billion parameters. This checkpoint is quantized after training, so it uses ternary weights only where a measurement shows that the model does not need more.
Run a calibration text through the model, and record what each matrix receives.
For each group of 64 weights, search the threshold that gives the smallest error, and give the positive and the negative weights separate scales (Ternary Weight Networks, with the asymmetric grid of PT2-LLM).
Quantize one column at a time, and move the error of that column into the columns that follow (GPTQ).
Measure each matrix on its own, and give ternary weights to the matrices that change the answers least, while the total change stays inside a budget.
Give every other matrix 8-bit weights.
quantization_report.json in this repository holds the result per tensor.
