4 experts (3 fine-tuned + base), 2 active per token — 2.61B total, ~1.3B active. Specializes in agentic tool use, code generation, and multi-step reasoning.

License

Highlights

4x1B MoE architecture: Mixtral-style with 4 experts, 2 active per token
Domain-specialized experts: Agent (tool use), Coding, Reasoning, and General (base model)
BAR-inspired training: All experts train on all data — specialization via LoRA configs + router
Hidden-gate router: Initialized with domain-specific positive/negative prompts for expert routing
Efficient active compute: ~1.3B active params per token (vs 2.61B total)
Built from MiniCPM5-1B: Strong multilingual base with Chinese + English support

Model Specifications

Property	Value
Architecture	`MixtralForCausalLM` (Mixtral-style MoE)
Base Model	openbmb/MiniCPM5-1B
Total Parameters	~2.61B
Active Parameters / Token	~1.3B
Experts	4
Experts per Token	2
Context Length	32,768
Gate Mode	`hidden` (domain-supervised)
Dtype	`bfloat16`
License	Apache 2.0

Expert Composition

Expert	Domain	LoRA Rank	LoRA Alpha	Training Tokens	LR	Specialization
0 — Agent	Agentic tool use	64	128	70M	2e-4	Multi-turn tool calls, bash, read, edit, write, grep, todowrite
1 — Coding	Code generation	64	128	150M	2e-4	Read/write/edit code, refactoring, file operations
2 — Reasoning	Multi-step reasoning	128	256	200M	1e-4	Math proofs, logic puzzles, step-by-step analysis
3 — General	General purpose	—	—	—	—	Conversational, Q&A, summarization, creative writing

All LoRA experts train on MLP layers only. Attention layers remain frozen to the base model weights.

Training Details

Expert Fine-Tuning (LoRA)

All 3 domain experts are fine-tuned from openbmb/MiniCPM5-1B using LoRA adapters targeting MLP layers only. This follows the BAR (Branch-Adapt-Route) recipe from Ai2 — all experts train on ALL data, with specialization emerging from different LoRA hyperparameters and the MergeKit router.

Expert	Dataset	Max Samples	Batch	Grad Accum	Optim	Scheduler
Agent	`Petrouil/opencode-agentic-mini`	5,000	1	16	paged_adamw_8bit	cosine
Coding	CodeDataset (50K samples)	50,000	4	4	paged_adamw_8bit	cosine
Reasoning	OpenThoughts3 + nvidia/OpenCodeReasoning	50,000	1	16	paged_adamw_8bit	cosine

All experts use 4-bit NF4 quantization during training with QLoRA.

Router Training

After MoE assembly via MergeKit, the router (gate) is trained on a stratified 5% sample of the SFT dataset. All expert and shared weights remain frozen. The router learns to route tokens to the appropriate expert based on domain signals from the hidden-gate initialization.

MergeKit MoE Assembly

The 4 experts are assembled into a Mixtral-style MoE using MergeKit with gate_mode: hidden. The router is initialized using positive/negative prompt pairs that define each expert's domain affinity.

Usage

Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Petrouil/FrankenCPM-4x1B-A2B",
    device_map="auto",
    torch_dtype="bfloat16",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "Petrouil/FrankenCPM-4x1B-A2B",
    trust_remote_code=True,
)

messages = [
    {"role": "user", "content": "Write a Python function to find all prime numbers up to n using the Sieve of Eratosthenes."}
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

vLLM

python

from vllm import LLM, SamplingParams

llm = LLM(
    model="Petrouil/FrankenCPM-4x1B-A2B",
    tensor_parallel_size=1,
    trust_remote_code=True,
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(
    ["Explain the difference between a stack and a queue."],
    sampling_params,
)
print(outputs[0].outputs[0].text)

Evaluation

Agent Benchmark (Tool Use)

Evaluated on 10 agentic tool-use tasks across 5 categories. Base model (MiniCPM5-1B) used as reference.

Task	Category	Score	Tool Detected	Content Correct
list_files	single_tool	1.0	No	Yes
read_file	single_tool	1.0	No	Yes
tool_selection	tool_selection	1.0	No	Yes
read_then_edit	multi_step	0.0	No	No
error_recovery	error_recovery	1.0	No	Yes
project_setup	complex_workflow	1.0	No	Yes
code_review_tool	tool_selection	1.0	No	Yes
write_file	single_tool	1.0	No	Yes
task_planning	complex_workflow	1.0	No	Yes
debug_multitool	multi_step	1.0	No	Yes

Overall: 9/10 tasks passed (90%)

Reasoning Checkpoints

The reasoning expert was evaluated across training checkpoints on math proofs and logic puzzles. Base model responses are long-chain-of-thought with correct reasoning. Fine-tuned responses show improved conciseness and structure while maintaining correctness.

Prompt	Domain	Base Model Response Style	Fine-tuned Response Style
Number theory proof (p² ≡ 1 mod 24)	Math	Detailed CRT decomposition, verbose	More structured, step-by-step
Combinatorial identity proof	Math	Vandermonde identity attempt, lengthy	Cleaner combinatorial argument
Invariant reasoning (5×5 board game)	Logic	Case analysis, lengthy reasoning	More concise, focused analysis

Limitations

Tool detection: The model does not natively detect or emit structured tool calls — it generates natural language descriptions of tool usage
Single expert routing: Only 2 of 4 experts activate per token, so domain knowledge is not always perfectly routed
Base model constraints: Inherits MiniCPM5-1B's limitations (1B parameter capacity, 32K context)
Training data: Experts were trained on specific datasets — performance may degrade outside those domains
No function calling: Not designed for structured function calling APIs — intended for conversational tool-use scenarios

Citation

bibtex

@software{frankencpm2026,
  title  = {FrankenCPM-4x1B-A2B: A Domain-Specialized Mixture-of-Experts Model},
  author = {Petrouil},
  year   = {2026},
  url    = {https://huggingface.co/Petrouil/FrankenCPM-4x1B-A2B}
}

License

Apache 2.0 — see LICENSE.

Acknowledgments

openbmb/MiniCPM5-1B — Base model
MergeKit — MoE assembly toolkit
BAR (Branch-Adapt-Route) — Training recipe inspiration from Ai2
Unsloth — LoRA fine-tuning framework