Iso-size Q4_K_M-class GGUF of meta-llama/Llama-3.1-8B-Instruct built with fraQtl calibration-aware per-tensor quantization: same file-size class as the leading community Q4_K_M (within 0.007%), measurably closer to the original bf16 model's output distribution on both locked evaluation slices.

This is the target half of the fraQtl Llama Draft Pair (its speculative-decoding draft is fraQtl/Llama-3.2-1B-Instruct-Hi-Fi-GGUF); the pair receipt ships in the collection when its benches complete.

Evaluation — fidelity to the original model

Metric: symmetric top-20 KLD on the support of a Q8_0 teacher quantized from the original pinned bf16 checkpoint (this measures fidelity-to-original — the axis a user migrating from the bf16 model experiences). 3 independent runs, run-to-run drift 0.0 (deterministic rig). Baseline credited: bartowski Q4_K_M — the canonical community quantization (pinned revision bf5b95e9…).

Arm	bytes	true bpw	KLD code/math ↓	KLD general ↓	top-1 agree cm	top-1 agree gen
bartowski Q4_K_M	4,920,739,232	4.9022	0.034463	0.022621	0.9532	0.9470
fraQtl Hi-Fi (this repo)	4,920,411,392 (−0.007%)	4.9019	0.028341 (−17.8%)	0.019196 (−15.1%)	0.9580	0.9510

Slices: code/math 400-record held-out (sha cce68602…b1665) and general chat/tool/wikitext 400-record (sha b10a79ca…276). Eval runtime: llama-cpp-python==0.3.33 (vendored llama.cpp 78d2f5246), A100-80GB, seed 0.

Long-context retrieval (needle-in-a-haystack)

Grid: contexts {8192, 32768} × depths {0.1, 0.5, 0.9} × 3 passkeys (18 cells), exact-match, greedy, 3 runs per arm. Both this artifact and the bartowski baseline score 18/18 on every run — the grid saturates at 8B scale; we state parity, not a win. Note: greedy decoding, but CUDA reduction order varies across instances, so retrieval scores are always reported as multi-run.

Provenance

Field	Value
Base model	`meta-llama/Llama-3.1-8B-Instruct` @ `0e9e39f249a16976918f6564b8830bc894c89659`
Conversion + quantization	llama.cpp @ `78d2f524682d9fee790a6460c93d018dafeb5229`; `convert_hf_to_gguf.py --outtype bf16`, then `llama-quantize` with per-tensor `--tensor-type` overrides
Calibration	fraQtl code+math packet (sha256 `9bec84a2…29e02`), llama-imatrix ctx 2048, BF16 substrate; `imatrix.dat` included in this repo
Teacher	Q8_0 quantized from the original bf16 at the same llama.cpp commit
Artifact sha256	`3c9656099140a679fe3a4061a61498dcb89fc324d978769312f4311a5de642da`
Receipts	`receipts/` in this repo: build + evaluation + retrieval JSONs, per-layer true-bpw table

Architecture / allocation

Standard dense Llama (32 layers, GQA). The type placement is chosen per tensor by calibration-aware allocation at the identical total byte budget as the baseline — the claim is smarter bits, not more bits. Attention V is the most protected family (Q8_0 on 29/32 layers); attention Q is never demoted below the baseline's native type. The lm_head stays at the baseline's Q6_K.

Usage

bash

# llama.cpp
llama-cli -m Llama-3.1-8B-Instruct-fraQtl-HiFi-Q4_K_M.gguf -ngl 99 -c 8192 \
  -p "your prompt" --temp 0.7

# speculative decoding with the fraQtl 1B draft (pair receipt to follow)
llama-speculative -m Llama-3.1-8B-Instruct-fraQtl-HiFi-Q4_K_M.gguf \
  -md Llama-3.2-1B-Instruct-fraQtl-HiFi-Q4_K_M.gguf \
  --draft-max 16 -ngl 99 -ngld 99 -p "your prompt"

The GGUF embeds the chat template; use llama_chat_apply_template / chat-completion APIs.

fraQtl ladder

Hi-Fi GGUFs (this repo + the 1B draft) — download-and-run quality tier
Qwen3-4B KV sidecars — the serving lane (long-context KV compression receipts)

License

Llama 3.1 Community License (full text in LICENSE; attribution in NOTICE). Derivative model name begins with "Llama" per the license. Use is subject to Meta's Acceptable Use Policy referenced therein.

Limitations

Fidelity numbers measure distributional closeness to the original model, not task accuracy; no task-benchmark claims are made on this card.
Q4_K_M-class quantization: outputs can differ from the bf16 model.
The retrieval grid saturates for both arms at 8B — it differentiates the 1B tier, not this one.

The fraQtl ladder

Tier	Job	Example
Hi-Fi	maximum fidelity at standard 4-bit size	this repo's Q4_K_M
Hi-Fi Phone	fits phone/laptop RAM, fidelity-first	E2B Phone, 2.86 GB
SmartEdge	smallest footprint that survives	E2B IQ3XXS, 2.45 GB

Same discipline at every tier: pinned provenance, measured numbers, losses disclosed.

More from fraQtl

The serving lane — KV-cache compression sidecars for vLLM — holds nine concurrent ≈128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14): fraQtl/qwen3-4b-instruct-2507-kv-sidecars. Org page: huggingface.co/fraQtl.

fraQtl/Llama-3.1-8B-Instruct-Hi-Fi-GGUF

Llama-3.1-8B-Instruct Q4_K_M GGUF — fraQtl Hi-Fi (calibration-aware quantization)