GGUF quantizations of KSP-NMAI/Boris-125M-Instruct for llama.cpp and compatible runtimes (llama-server, LM Studio, koboldcpp, Jan).

The original safetensors weights live in the base repo — use those for finetuning or for any PyTorch-based runtime. GGUF is inference-only.

TQ1_0, TQ2_0 and Q1_0 were built and tested but produced unusable output at this scale, so they are deliberately not published here.

Boris-125M is a small model, and quantization behaves differently at this scale than it does for 7B+ models. The token embedding table is a large fraction of the parameters and is kept at high precision by llama.cpp, which sets a hard floor on file size. The practical result: every file here is between 49 MB and 240 MB. Dropping from Q8_0 to IQ1_S saves you a few tens of megabytes while degrading output substantially. The aggressive quants are provided for completeness, not because they are a good trade.

All quantizations below 8-bit were produced with an importance matrix calibrated on 100 chunks of held-out data drawn from the model's own training mixture (60% fineweb-edu / 40% dclm).

The Alpaca chat template is embedded in every file, so --jinja applies the correct prompt format automatically.

### Instruction:
{your instruction}

### Response:

This is a very small instruction-tuned model. It will produce text that is frequently inaccurate, inconsistent, or offensive, and has received no alignment or safety tuning beyond supervised fine-tuning on Alpaca. Do not rely on it for factual information or deploy it without supervision.

Apache 2.0. Copyright 2026 Joseph Jones. See the base repository for the full notice.