A Mixtral-style mixture-of-experts model built by fine-tuning 10 experts from Ministral-3B-Instruct (text-only, vision stripped) and merging with mergekit-moe. Intended as a coding assistant and agentic-tool-use backbone.

⚠️ EXPERIMENTAL. Model quality evaluation is pending. This is a hobby project by someone who barely knows what they're doing. It might be terrible. If you're brave enough to test it, please drop notes

Eval ran by user simonko912 (https://huggingface.co/simonko912)

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc		0.2521	_	0.0037
- humanities	2	none	0	acc	_	0.2618	_	0.0064
- formal_logic	1	none	0	acc	_	0.3492	_	0.0426
- high_school_european_history	1	none	0	acc	_	0.2242	_	0.0326
- high_school_us_history	1	none	0	acc	_	0.2794	_	0.0315
- high_school_world_history	1	none	0	acc	_	0.2785	_	0.0292
- international_law	1	none	0	acc	_	0.1818	_	0.0352
- jurisprudence	1	none	0	acc	_	0.3056	_	0.0445
- logical_fallacies	1	none	0	acc	_	0.2822	_	0.0354
- moral_disputes	1	none	0	acc	_	0.2399	_	0.0230
- moral_scenarios	1	none	0	acc	_	0.2737	_	0.0149
- philosophy	1	none	0	acc	_	0.2154	_	0.0234
- prehistory	1	none	0	acc	_	0.2716	_	0.0247
- professional_law	1	none	0	acc	_	0.2549	_	0.0111
- world_religions	1	none	0	acc	_	0.3099	_	0.0355
- other	2	none	0	acc	_	0.2533	_	0.0078
- business_ethics	1	none	0	acc	_	0.2800	_	0.0451
- clinical_knowledge	1	none	0	acc	_	0.2491	_	0.0266
- college_medicine	1	none	0	acc	_	0.2717	_	0.0339
- global_facts	1	none	0	acc	_	0.2400	_	0.0429
- human_aging	1	none	0	acc	_	0.2735	_	0.0299
- management	1	none	0	acc	_	0.2039	_	0.0399
- marketing	1	none	0	acc	_	0.2521	_	0.0284
- medical_genetics	1	none	0	acc	_	0.2600	_	0.0441
- miscellaneous	1	none	0	acc	_	0.2593	_	0.0157
- nutrition	1	none	0	acc	_	0.2190	_	0.0237
- professional_accounting	1	none	0	acc	_	0.2376	_	0.0254
- professional_medicine	1	none	0	acc	_	0.2279	_	0.0255
- virology	1	none	0	acc	_	0.3373	_	0.0368
- social sciences	2	none	0	acc	_	0.2340	_	0.0076
- econometrics	1	none	0	acc	_	0.2895	_	0.0427
- high_school_geography	1	none	0	acc	_	0.2273	_	0.0299
- high_school_government_and_politics	1	none	0	acc	_	0.2124	_	0.0295
- high_school_macroeconomics	1	none	0	acc	_	0.2359	_	0.0215
- high_school_microeconomics	1	none	0	acc	_	0.2521	_	0.0282
- high_school_psychology	1	none	0	acc	_	0.2330	_	0.0181
- human_sexuality	1	none	0	acc	_	0.2443	_	0.0377
- professional_psychology	1	none	0	acc	_	0.2255	_	0.0169
- public_relations	1	none	0	acc	_	0.2364	_	0.0407
- security_studies	1	none	0	acc	_	0.2327	_	0.0270
- sociology	1	none	0	acc	_	0.2040	_	0.0285
- us_foreign_policy	1	none	0	acc	_	0.2800	_	0.0451
- stem	2	none	0	acc	_	0.2540	_	0.0077
- abstract_algebra	1	none	0	acc	_	0.2000	_	0.0402
- anatomy	1	none	0	acc	_	0.2519	_	0.0375
- astronomy	1	none	0	acc	_	0.2500	_	0.0352
- college_biology	1	none	0	acc	_	0.3125	_	0.0388
- college_chemistry	1	none	0	acc	_	0.2900	_	0.0456
- college_computer_science	1	none	0	acc	_	0.3400	_	0.0476
- college_mathematics	1	none	0	acc	_	0.2900	_	0.0456
- college_physics	1	none	0	acc	_	0.2647	_	0.0439
- computer_security	1	none	0	acc	_	0.1800	_	0.0386
- conceptual_physics	1	none	0	acc	_	0.2596	_	0.0287
- electrical_engineering	1	none	0	acc	_	0.3034	_	0.0383
- elementary_mathematics	1	none	0	acc	_	0.2566	_	0.0225
- high_school_biology	1	none	0	acc	_	0.2774	_	0.0255
- high_school_chemistry	1	none	0	acc	_	0.2266	_	0.0295
- high_school_computer_science	1	none	0	acc	_	0.2300	_	0.0423
- high_school_mathematics	1	none	0	acc	_	0.2296	_	0.0256
- high_school_physics	1	none	0	acc	_	0.2252	_	0.0341
- high_school_statistics	1	none	0	acc	_	0.1991	_	0.0272
- machine_learning	1	none	0	acc	_	0.2768	_	0.0425
arc_challenge	1	none	0	acc	_	0.2099	_	0.0119
		none	0	acc_norm	_	0.2509	_	0.0127
boolq	2	none	0	acc	_	0.5859	_	0.0086
gsm8k	3	flexible-extract	5	exact_match	_	0.0000	_	0.0000
		strict-match	5	exact_match	_	0.0000	_	0.0000
hellaswag	1	none	0	acc	_	0.2542	_	0.0043
		none	0	acc_norm	_	0.2663	_	0.0044
winogrande	1	none	0	acc	_	0.4909	_	0.0141
Groups	Version	Filter	n-shot	Metric		Value		Stderr
------------------	------:	------	-----:	------	---	-----:	---	-----:
mmlu	2	none		acc		0.2521	_	0.0037
- humanities	2	none	0	acc	_	0.2618	_	0.0064
- other	2	none	0	acc	_	0.2533	_	0.0078
- social sciences	2	none	0	acc	_	0.2340	_	0.0076
- stem	2	none	0	acc	_	0.2540	_	0.0077

Quick facts


Architecture	Mixtral-style MoE
Parameters	23.3B (much smaller than planned — MoE layers replaced, vision encoder stripped)
Base model	`mistralai/Ministral-3-3B-Instruct-2512` (text-only)
Experts	10 routed, top-2 active per token
Training	Full fine-tune per expert, ~25k samples each (I'm broke)
Context length	128k (Ministral native)
Merge tool	mergekit-moe, hidden gate mode
Format	Ministral chat template (`[INST]...[/INST]`)

Why it's only 23.3B instead of ~40B

Honestly I aimed for 35-40B. The MoE merge strips the base model's dense MLP layers and replaces them with a router + shared expert MLPs. On top of that I ripped out the vision encoder since this model doesn't need images. Result: smaller. If I make a V2 I might bump the expert count and switch to DeepSeek MoE (shared + routed experts) to hit the size target.

Experts

Each expert was full fine-tuned on its own dataset mix so the router has something meaningful to route toward. Only 25k samples/epoch because GPU time costs money and I'm not made of it.

#	Expert	Specialty
1	Captain	General instruction following + safety boundaries
2	Python Coding	Core Python SFT (Magicoder OSS/Evol-Instruct)
3	Verified Python	Execution-filtered code + OPC educational instruct
4	Code Reasoning	Step-by-step algorithm design, CoT (OpenCodeReasoning + OpenThoughts)
5	Tool Calling	Function calling + structured JSON (Hermes-fc + ToolACE + xLAM)
6	Agentic Workflows	Multi-step agent planning, ReAct loops (Hermes-agent + Nemotron-agentic)
7	Code Editing	Diff generation, refactoring (commitpackft)
8	Code Review	Code critique + iterative improvement (CodeFeedback)
9	General Reasoning	Math, logic, non-code problem solving
10	Safety Net	Untuned base model — catch-all for chitchat, greetings, refusals

The captain (expert #1) donates self-attention and layer norms to the whole model. The safety net (expert #10) is the raw bf16 base with no fine-tuning at all — it catches prompts that don't match any specialty so you don't get the tool-calling expert trying to process "hello".

Limitations (there are many)

Not evaluated. I ran a local smoke test (3k samples/expert, q8_0 GGUF) and it spits out text, but I haven't done any real benchmarks. The production run trains at 25k or 75k samples per expert and I haven't tested the final merge yet.
Low sample counts. 25k samples per expert is tiny by modern standards. Quality is likely mediocre.
Training quality unknown. The local smoke run used 3k samples each (seq_len 1024) — results were... not great. May be the tiny dataset, may be the idea itself is bad. Production run on a RunPod A40 might fix it, might not. ¯\(ツ)/¯
No RLHF/DPO. This is SFT-only. No preference optimization, no reward modeling. It will not be aligned beyond what the datasets provide.
No vision. The vision encoder was completely stripped. This is text-only.

Planned V2 (if this isn't terrible)

DeepSeek MoE architecture: shared experts (always active) + routed experts, which should handle overlapping domains better.
~14 experts instead of 10, for more granular specialization.
Proper eval harness before shipping.
Maybe a separate "fanfiction" ( ͡° ͜ʖ ͡°) creative writing fine-tune if I feel like it. No promises.

How to use

This is a standard transformers model with a Mixtral architecture. The chat template is the Ministral format:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "HavocK1/Le-Chaton-Slim-MoE",
    torch_dtype="auto",
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained("HavocK1/Le-Chaton-Slim-MoE")

messages = [
    {"role": "user", "content": "Write a Python function to reverse a linked list."},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For GGUF (llama.cpp):

llama-server le_chaton_slim.gguf --n-gpu-layers 99

Datasets

The training datasets (deduped JSONL) are available at HavocK1/Le-Chaton-Slim-Experts.

Source datasets used:

HuggingFaceTB/smoltalk, HuggingFaceH4/no_robots, Magpie-Align/Magpie-Pro-300K-Filtered, teknium/OpenHermes-2.5, HuggingFaceH4/ultrachat_200k
ise-uiuc/Magicoder-OSS-Instruct-75K, ise-uiuc/Magicoder-Evol-Instruct-110K
bigcode/self-oss-instruct-sc2-exec-filter-50k, OpenCoder-LLM/opc-sft-stage2
nvidia/OpenCodeReasoning, open-thoughts/OpenThoughts-114k
NousResearch/hermes-function-calling-v1, Team-ACE/ToolACE, Salesforce/xlam-function-calling-60k
lambda/hermes-agent-reasoning-traces, nvidia/Nemotron-SFT-Agentic-v2
bigcode/commitpackft
m-a-p/CodeFeedback-Filtered-Instruction, m-a-p/Code-Feedback
QuixiAI/dolphin-r1, HuggingFaceTB/smoltalk (NuminaMath-CoT)

Credits

Built with:

Ministral-3B (Mistral AI)
mergekit (Arcee AI)
HuggingFace transformers + TRL + PEFT
All the amazing dataset authors above 🙏
An A40 GPU on RunPod that I paid for with actual money
Shout out to my homies Mistral 3.5 128B, GLM 5.2 and DS V4 pro. Couldn't have done it without them