LFM2.5-8B-A1B-heretic 16e — the expert-cut, gate-schooled coffin brain
A leaner, fully-resident reincarnation of coder3101/LFM2.5-8B-A1B-heretic: the 32 routed experts per layer were cut to 16, and the router gates were retrained (teacher-prior school) so the surviving heads know where to route. Same soul, half the mass — built to live entirely inside a 4GB VRAM card with real context headroom.
Why this exists
The full 8B model is ~4.0G at IQ4_XS — it technically fits a 4GB card but leaves almost no room for context. Cutting experts cuts the file, not the brain per token: top-4 stays top-4, so the active parameters (~A1B) are identical to the original — same speed, half the weight.
Expert cut (32 → 16): per-layer expert tensors (w1/w2/w3) sliced to the first 16, gate rows and expert bias sliced to match, num_experts updated in config.
Gate school (teacher-prior proxy): the ORIGINAL 32-expert gates were kept as teacher. Per-token soft targets = softmax of the original routing preference over the surviving 16 experts, blended with a uniform prior. Only the 16-wide gate weights were trained (CE, AdamW) — no backprop through the body. The cut brain learns where to route using its own old wisdom.
The full bf16 safetensors (model.safetensors) ships in this repo — the gates are already patched in, so this is a complete, loadable model, not a patchset.
Files
File
Description
model.safetensors
Full bf16 weights, post-trained gates (8.6G)
lfm25-heretic-16e-iq4-xs.gguf
IQ4_XS quant, imatrix-calibrated (2.4G)
config.json / tokenizer*
Transformers-compatible config + tokenizer
Measured
Generation: ~29.7 tok/s on i5-12450H CPU (8 threads, -c 2048) — 4x faster than the 24-expert cut.
Prompt eval: ~58 tok/s.
Coherent chat output after school (no word salad; verified via llama-server and ollama).
Fits 4GB VRAM at IQ4_XS with ~1.5G left for context. Linear attention keeps KV tiny, so long context is cheap.
Usage
Ollama (import from this GGUF — a tuned Modelfile ships in this repo):
(Tuned sampling: temperature 0.8, top_p 0.95, top_k 60, repeat_penalty 1.15. Note: this is a thinking model — ollama run shows <think> blocks before the answer.)
llama.cpp (RECOMMENDED for harness/API use — clean reasoning split):
The GGUF carries the official LFM2.5 chat template (with the bos_token literal fix — llama.cpp doesn't pass bos_token as a template var, so the raw official template renders a 4-token prompt; <|startoftext|> is baked in). Responses arrive with the thinking in message.reasoning_content and the answer clean in message.content — verified: {"answer": "2+2 = 4", "reasoning_content": "..."}.
llama.cpp:
llama-server -m lfm25-heretic-16e-iq4-xs.gguf --jinja -c 8192
# or
llama-cli -m lfm25-heretic-16e-iq4-xs.gguf -p "Hi! Who are you?"
Small-active brain (~A1B per token): excellent for chat, explanation, code-adjacent work; arithmetic is not its superpower (a 1B active brain ceiling, not a defect of the cut).
The base model's "heretic" ablation carries through — this is an uncensored model. Use accordingly.
Built with the forge scripts: expert slice, teacher-prior gate school, imatrix IQ4_XS pipeline. The same pipeline makes any future cut (12e, 8e, ...) in about an hour.
