Neutrino‑1 8B — FV5 for BeeLlama 4.2 (Blackwell / sm_120)
Community port — this is not an original model.
It repackages Fermion Research — Neutrino‑1 8B
(a Qwen3‑8B model trained into Fermion's FV5 five‑value ternary format, Apache‑2.0) into GGUF files
that run on a BeeLlama 4.2 CUDA fork on NVIDIA Blackwell (sm_120),
plus a decode‑kernel optimization. All weights are Fermion's — we only changed the on‑disk byte layout and the
CUDA kernels. Numerics are bit‑identical to Fermion's reconstruction.
What FV5 is
FV5 = five‑value weights, not plain ternary. Each weight is one of {0, +s_lo, −s_lo, +s_hi, −s_hi} —
two magnitude levels, unlike ternary's single level. Per 256‑element block: two f32 scales + three 1‑bit planes
(bp = is +, bn = is −, br = hi‑vs‑lo magnitude) ⇒ 3.25 bpw. Reconstruction is exact:
w = (bp − bn) · (br ? s_hi : s_lo) # all f32
Fermion keep the activation path raw‑f32 (no int8 / dp4a / MMQ, TF32 disabled) — a deliberately quality‑first,
f32‑exact quant. We preserve that contract everywhere.
Files
file
plane layout
matmul bpw
size
runs on
neutrino-8b-fv5-v2.gguf
co‑located uchar4[bp,bn,br,pad]
4.25
4.96 GB
this branch (HEAD build)
neutrino-8b-fv5-t4950.gguf
standard SoA planes bp[]bn[]br[]
3.25
4.09 GB
base FV5 port / Fermion fork (see below)
Both decode to identical outputs. The v2 file trades +0.87 GB for +1.23× decode by packing the three
bit‑planes of each 8‑weight group into one 4‑byte word so a single coalesced load fetches all planes.
embed/lm_head stay int8 (8.125 bpw) in both, as Fermion ship them.
What we did & achieved
Hardware: RTX 5060 Ti 16 GB (Blackwell, sm_120), CUDA 12.8.
Ported FV5 into BeeLlama 4.2 — builds and runs natively on sm_120 (no dequant fallback for decode).
71 → 84 — co‑located "v2" layout: the 3 bit‑planes per 8‑weight group become one uchar4 so one
coalesced load replaces 2–3 separate/divergent loads → DRAM utilization 63 % → ~100 % on the heavy FFN
tensor (measured +1.23×, cold‑DRAM microbench on a real ffn_down tensor).
Batched verify kernel for speculative decoding (small ne11), bit‑identical to single‑token decode
(⇒ no rounding‑induced draft rejections).
Honest speculative‑decoding finding. With Fermion's official Neutrino‑0.6B drafter it works
(16.9 → 61 t/s after our batched kernel, 3.6×) but does not beat the 84 t/s single‑token decode: batched
verify is only ~1.09× faster per token than decode, because f32‑exact FV5 cannot use the cheap int8‑MMQ verify
path that makes drafting pay off for normal quants. Documented rather than hidden.
Benchmarks
workload
throughput
decode (single token), v2
84 t/s
decode, pre‑v2 (Attempt E)
71 t/s
decode, initial port
40 t/s
prefill (1659‑tok prompt)
841 t/s
speculative (0.6B draft, our batched kernel)
61 t/s (below decode — see finding above)
Size vs mainstream 4‑bit (Qwen3‑8B GGUF, unsloth)
v2 is 4.96 GB (4.84 bpw effective) — the matmul tensors are 4.25 bpw, but embed+lm_head are int8
(8.12 bpw, 1.26 GB), which is what keeps the whole file in Q4_K_M territory:
1# v2 (this build): ~84 t/s decode2llama-completion -m neutrino-8b-fv5-v2.gguf -ngl 99 --temp 0 -no-cnv \3 -p "The capital of France is"
The SoA file (t4950) is the standard plane layout; it runs on the base FV5 port (before the v2 layout commit)
or on Fermion's own fermion-fv5 fork after renumbering its type ids to this fork's enum — use gguf_renumber_fv5.py.
Recommended sampling (Fermion): --temp 0 … 0.01, --repeat-penalty 1.05.
