The goal was a single E4B that keeps Musica's prose and stability while picking up range it doesn't have on its own — grounded slice-of-life dialogue, comic timing, and a persona voice that will answer a direct question directly instead of deflecting into vibes. In blind A/B judging across 24 roleplay scenarios it trades wins evenly with Musica overall, with its stable wins exactly where the recipe aimed: grounded, blunt, character-forward scenes.
Applied as weighted
task arithmetic over
google/gemma-4-E4B-it:
base + 0.8·Δmusica + 0.225·Δties3 + 0.3·Δbella. The multimodal (audio/vision) towers are inherited from base effectively unchanged — all three parents are language-stack tunes.
Nerd details — how the parents and weights were chosen
The premise: most merge quality is decided before anything is built. So instead of a merge→test→re-merge loop, the pool was measured first and exactly one candidate recipe was built and gated.
Scouting the pool
Every E4B derivative on the Hub (~1,100 repos at the time) was swept down to ten genuinely distinct creative-writing/RP fine-tunes — dropping quant/format re-uploads, code and domain tunes, and uncensored-only edits, then reading the cards of what remained.
Measure, then select
Per-tensor weight deltas (Δ = fine-tune − base) were extracted for all ten, over the language stack only (audio/vision towers excluded from all statistics): exact per-tensor delta norms plus a deterministic subsample of coordinates, sampled at identical positions in every model so the candidates can be compared directly.
Measurement did the pruning:
- One popular "family" of tunes was substantially the same vector uploaded twice (pairwise cosine 0.54–0.58; one member was recoverable as a weighted sum of two others) — redundant, dropped.
- Another candidate's headline delta norm was ~99% quantization noise in the per-layer embedding table, not trained signal — dropped.
- A pair of DPO variants moved the weights so little (‖Δ‖ ≈ 0.5, ~40× quieter than the pool median) that they couldn't survive at any reasonable merge weight — dropped.
What survived selection was a lead voice (musica — the strongest all-round RP tune in the pool) plus two complements chosen because each brings a direction the lead measurably lacks, and all three are near-orthogonal in delta space — they learned different things, so their contributions add instead of fighting:
| pair | cosine(Δ, Δ) |
|---|
| musica ↔ ties3 | 0.017 |
| musica ↔ bella | 0.004 |
| bella ↔ ties3 | 0.0008 |
Delta magnitudes (language stack ℓ₂): ties3 26.6, musica 7.5, bella 1.7 — i.e. bella is a light seasoning by construction, not just by weight.
Choosing the weights
Candidate weight vectors were swept on the artifacts alone — no builds — against the trained-envelope statistic (below). The recipe kept the lead near full strength (0.8, preserving musica's voice and stability), scaled ties3 down to 0.225 (its raw delta is 3.5× the lead's; at higher weight it dominates the mix), and added bella at 0.3 — deliberately below her full strength, because a single-human-voice tune is a garnish that becomes an identity at high weight. Each parent's documented quirks cap its weight: dilution does not reliably remove a known failure mode, as the four-parent variant below demonstrated.
Constrain the build
Two guardrails picked the weights:
- Trained envelope: at the chosen weights, almost no coordinate of the merged delta exceeds the largest movement any single parent trained at that coordinate (violation rate ≈ 0.7% after flooring out bf16 rounding noise, which is within the "clean" band calibrated on known-good and known-broken merges).
- Weight-sum budget: Σw = 1.325. Empirically (on this family and a 12B sibling project), merges hold together up to roughly Σw ≈ 1.3 and degrade beyond it regardless of per-coordinate statistics — a heavier four-parent variant of this same recipe (Σw = 1.87) passed the envelope check and still lost a blind A/B against both Musica and this model, 3–9 and 2–10, reproducing its parents' documented failure modes at merge scale. This recipe is the one that survived.
As a free integrity check, regressing the built model's delta against the parent subsamples recovers the recipe weights (0.80 / 0.21 / 0.30) — confirming nothing unexpected leaked into the build.
Blind gate
Two independent rounds of 12 roleplay/creative scenarios (fresh seeds per round, identical samplers, per-prompt shuffled A/B, judged blind at Q8_0): 11–13 vs Musica — statistical parity, with stable wins split by territory: this merge on grounded/comedic/blunt scenes, Musica on atmosphere and literary continuation. Per the one-build philosophy, parity against the strongest parent plus added range = ship it, don't start a weight-tuning loop.
