q4 that survive the squeeze.
normal q4 round the weight after training and hope. this one train WITH the
rounding: every linear weight fake-quantized to asymmetric int4 (group 32) on
each forward, straight-through gradient update the bf16 weight underneath. model
learn weight that still work after Q4_K_M round them. same recipe as grug-9b-qat
and grug-27b-qat.
trained on same data as
ProCreations/grug-3b,
so grug dialect and adaptive think length come through intact.
use the Q4_K_M one. plain (non-qat) quants live
here.
Nanbeige4.2 not in upstream llama.cpp yet (issue
#26086). Nanbeige team PR
#25994 add it - weight-shared
depth loop,
num_loops=2. until merge, build from that branch:
1git clone --depth 1 --branch nanbeige42 https://github.com/Nanbeige/llama.cpp
2cd llama.cpp && cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
3./build/bin/llama-cli -m grug-3b-Q4_K_M.gguf -p "What is 12 times 12?"
these gguf converted and load-probed with that branch.
