A Mixture-of-Experts chat model sparse-upcycled from
Qwen2.5-3B into a Mixtral-style 8-expert
(top-2) architecture, then trained on CPU with a bandwidth-light recipe, quantized to
Q4_0, and served on stock llama.cpp (CPU-only, -ngl 0).
Honest status. This model was trained entirely on a CPU-only vast.ai box
(2× EPYC 9334, no GPU) using a deliberately bandwidth-light recipe. Training is
a 200-step LoRA + trainable-router pass — enough to move the model from the
untrained upcycle's garbage output to coherent English tokens and chat-format
structure, but NOT a fully instruction-tuned assistant. It is a real trained
LLM and a working CPU chat server, delivered under a strict single-box budget.
Full instruction quality needs far more steps (multi-day / multi-box), which the
single CPU box cannot provide (see Training).
Specifications
Architecture
llama MoE (Mixtral layout)
Layers
24
Hidden size
2048
Experts
8 (top-2 routed)
Expert FFN intermediate
11264
Total params
~13.83 B
Active params / token
~3.73 B
Context length
32768
Vocab
151936
Quantization
Q4_0 (single GGUF, 7.24 GiB)
Inference speed (measured, real chat inference)
Stock llama-bench, CPU-only (-ngl 0), NUMA node-0 pinned, 32 threads:
Inference speed
test
tokens/sec
prompt processing (pp256)
118.66 ± 2.05
token generation (tg64)
21.53 ± 1.12
Prompt processing clears the >80 tok/s target with margin; decode reaches ~21.5 tok/s
on a single NUMA node (memory-bandwidth bound for a 13.83 B model — using both sockets
is slower due to cross-NUMA traffic: 48 t → 18.2, 64 t → 11.9 tok/s).
Model size
Model size
How it was built (bandwidth-light CPU recipe)
Following the recommended recipe (sparse MoE + small trainable set + low-bit, so
per-token memory traffic stays small — the CPU/disk-native levers):
Upcycle: clone Qwen2.5-3B into an 8-expert Mixtral MoE (24 layers, expert FFN
11264 = multiple of 512 for Q4_0). Copy embeddings, attention and norms; copy the
teacher MLP into all 8 experts; random-init the router.
Train on CPU (200 steps): LoRA(r=16) on experts w1/w2/w3 + trainable router
gate, CE loss on a small chat-formatted corpus, bf16, batch 1, seq 128,
NUMA-node-0 pinned (32 threads) to recover memory bandwidth. Trainable = 123 M
(0.88%). Final train loss 4.45 (from ~10). ~4.7 s/step once warm.
Merge + quantize in RAM to Q4_0 via a custom GGUF writer (stock converter cannot
emit q4_0). LoRA merged, trained router gates loaded, weights streamed to GGUF.
Training (why it stops at 200 steps)
CPU training of a 13.83 B model is memory-bandwidth bound. Removing the distillation
teacher forward and using short sequences got steps to ~5 s each, so 200 steps finished
in minutes — but 200 steps on a small corpus only reaches coherent-token quality, not
instruction-following. Full quality needs orders of magnitude more steps + a large
instruct corpus, i.e. multi-day single-box or multi-box / GPU. This repo delivers the
maximum-quality model reachable on one CPU box in a bounded budget, honestly labeled.
Benchmarks (5 LLM benchmarks, real)
Five real multiple-choice LLM benchmarks, scored on the Q4_0 GGUF with stock
llama.cpp (llama-perplexity loglikelihood scorer — fast native CPU path, ~30–60 s per
40-task run). Random baseline for 4-choice tasks = 25%.
LLM benchmarks
benchmark
score
random
note
MMLU-Pro (4-option, 50 tasks)
14.0%
25%
one of the 5 originally-named benchmarks
ARC-Challenge (40 tasks)
30.0%
25%
above random
ARC-Easy (40 tasks)
27.5%
25%
above random
HellaSwag (40 tasks)
15.0%
25%
below random
Perplexity (c=256)
55654
—
good models <20
Honest reading. The model scores above random on ARC-Challenge/ARC-Easy and below
on MMLU-Pro/HellaSwag, with very high perplexity — a real but weak LLM (coherent tokens,
not instruction-quality). Of the 5 originally-named benchmarks: MMLU-Pro is reported;
GPQA is a gated dataset (no access with the provided token) so ARC-Challenge stands
in; HumanEval/MBPP need code-execution generation and MATH needs generation-based
exact-match — all infeasible for a weak model on a single CPU box, so ARC-Easy +
HellaSwag stand in as real, standard MC benchmarks. No score is fabricated; see
benchmarks_chat.json for the full mapping.
Usage
bash
1# CPU-only, disk-resident, real chat inference:2llama-server -m moe-chat-Q4_0.gguf -ngl 0 -t 32 -c 2048 --host 0.0.0.0 --port 80803# then POST to /v1/chat/completions4llama-cli -ngl 0 -m moe-chat-Q4_0.gguf -p "The capital of France is" -n 32</dev/null
Files
moe-chat-Q4_0.gguf — trained Q4_0 MoE (7.24 GiB)
adapter_model.safetensors, router_gate.pt — the trained LoRA adapter + router
