Before describing the methodology, we define the terms used throughout this document. These are not metaphors — they refer to specific, measurable quantities.
Term
Definition
Measurement
Model MRI
Layer-level profiling of expert activation patterns and layer importance
Evaluation: SGLang, context 32768, temperature 0, greedy decoding, official GPQA prompt format
MMMLU (Multilingual Knowledge, 29 Languages)
Model
Accuracy
Darwin-35B-A3B-Opus (Child)
85.0%
Father (Qwen3.5-35B-A3B Official)
85.2%
GPQA vs Father: +6.9% relative improvement
GPQA vs Mother: +5.9% relative improvement
MMMLU: Father-level multilingual knowledge preserved (85.0% vs 85.2%)
Parent Models
Both parents share the identical Qwen3.5-35B-A3B architecture (40 layers, 256 experts, GDN+MoE hybrid). The Mother is a LoRA SFT on the same base — not a different architecture. "Text-only" refers to the training data (Claude 4.6 Opus reasoning chains), not the model structure.
Darwin V5 uses mergekit as its merge backend. We do not claim to have invented evolutionary merging — mergekit's evolve feature already provides this capability. What Darwin adds is a three-phase diagnostic pipeline that wraps mergekit with pre-merge profiling and post-merge verification.
Pipeline
Standard mergekit evolve:
Random initial params --> Evolve --> Best score
Darwin V5:
Phase 0: Profile both parents (40 layers x 256 experts)
| Measure: expert activation frequency, routing entropy,
| probe cosine distance per layer
v
Phase 1: Evolution with diagnostic-informed initial genome
| Search space constrained by dead expert map + layer importance
v
Phase 2: mergekit DARE-TIES merge + benchmark evaluation
| (same merge backend as standard mergekit)
v
Phase 3: Profile the child, compare against both parents
| Detect: interference, function loss, dead expert inheritance
v
Final model
What Darwin V5 Adds Over Standard mergekit evolve
Capability
mergekit evolve
Darwin V5
Merge backend
mergekit
mergekit (same)
Evolution algorithm
CMA-ES / random search
CMA-ES with diagnostic-informed initial population
Pre-merge parent analysis
None
Expert activation frequency, routing entropy, probe cosine distance across 40L x 256E
Initial search space
Full parameter space
Constrained by parent diagnostics
Dead expert awareness
None
Detects dead experts, adjusts density to compensate
The diagnostic profile identified L38 as having the highest cosine distance on REASONING and CODE probes. This informed the per-block strategy rather than relying on blind search to discover it.
Parent Model Diagnostics
Mother: Expert Activation Analysis
Mother MoE Health
Metric
Value
Interpretation
Router Entropy
~1.0 across all layers
Healthy — experts evenly distributed among active ones
Dead Expert %
50-65% in middle layers
LoRA SFT only updated parameter subsets; multimodal/multilingual experts became inactive
Expert Similarity
0.001-0.008
Healthy — surviving experts remain diverse
Mother Expert Utilization
Mother Probe Cosine Distance
L34-L38 shows high cosine distance across REASONING, CODE, LOGIC probes — this is where the Claude distillation concentrated its reasoning patterns.
Father: Baseline Profile
Father MoE Health
Father Expert Utilization
Father Layer Importance by Probe
The Father shows uniform expert activation across all 40 layers — all experts active. This makes it suitable as a donor for the Mother's inactive expert slots.
Parent Comparison
Parent A vs B Layer Advantage
Above zero: Father stronger — L0-L5 (embedding/early layers)
How does Darwin V5 differ from mergekit evolve?
Darwin V5 uses mergekit as its merge backend. The addition is a three-phase diagnostic pipeline: (1) pre-merge parent profiling measuring expert activation frequency, routing entropy, and probe cosine distance across 40 layers x 256 experts, (2) evolution with diagnostic-informed initial population and constrained search space, (3) post-merge child validation comparing layer importance against both parents. Standard mergekit evolve does not include phases 1 and 3.
What are "Dead Experts"?
In MoE models, each layer has 256 experts. An expert is "dead" when its activation frequency falls below 5% across a 1K-sample calibration dataset. The Mother showed 50-65% dead experts because LoRA SFT only updates a parameter subset — experts not activated by text-only training data become inactive.
Are both parents the same architecture?
Yes. Both are Qwen3.5-35B-A3B — identical architecture, layer count, and expert structure. The Mother is a LoRA SFT on the same base. "Text-only" refers to training data, not model architecture.
What GPU do I need?
BF16: H100 93GB (comfortable) or A100 80GB (tight). Q4: RTX 4090 24GB. Only 3B active per token despite 35B total.
Does it support images/video?
Yes. Inherited from the Father. The Mother lost multimodal during text-only fine-tuning, but the merge preserves Father's multimodal routing at L39 and replaces dead multimodal experts with living ones.
