GGUF iMat

This is a roleplay and creative-writing merge of google/gemma-4-31B-it.

I built it because the usual merge recipe was failing in a familiar way: it got louder before it got better. The model would pick up flavor from the sources, but long generations started to ramble, repeat, or lose the thread. That looked less like a bad source model and more like a bad operation.

The bet here is that the useful part of a finetune is not just its delta magnitude. It is the direction it turns the base.

I have argued elsewhere that by the time a transformer reaches its readout, the useful linearity has migrated to the residual-to-lm_head interface, and that RMSNorm makes that interface primarily directional: magnitude is suppressed as an independent degree of freedom in the last step before unembedding. If that is true, it has a consequence for merging that I had not seen drawn out. Combining two fine-tunes by adding their weight deltas and renormalizing the result works against that geometry. The operation it rewards is a rotation of the base toward them. This is the first model I built on that principle.

The body is composed with an operator I am calling an isometric merge. Each source delta is reduced to the rotation it induces on the base weight inside the subspace it actually moved; those rotations are averaged on the orthogonal group and applied as W → W + Z(R − I)Zᵀ W. Because R is orthogonal, the merged weight carries the same norm it started with, to machine precision. There is nothing to restore afterward.

The conventional path interpolates the deltas and rescales the result back to a target norm. That is not an isometry, and the leftover magnitude has to be put back by hand. In my runs that final rescale is exactly what produces repetition loops — the same failure that appears when you edit late-layer behavior with a crude magnitude change instead of a norm-preserving one. The architecture is fairly clear about which edit it prefers; the isometric merge takes it at its word, and stays coherent under load where additive and slerp merges of the same ingredients did not.

Two further choices follow from the same view. The subspace each rotation acts in is chosen in the activation metric Δ · H^½, with H = E[xxᵀ] harvested from the base — the directions the activations actually use, not the raw Frobenius directions — and each source is weighted by how much it moves the states the model visits rather than by the size of its weight change, so that a large but mostly-noisy dense tune does not drown the low-norm adapters that carry the actual flavor. Pure rotation by itself is nearly inert, so a bounded radial term reintroduces finetune strength, placed in the low-rank signal head rather than the quantization-noise tail, with an on-policy generation check as the ceiling on how far it is allowed to go.

Rotated into the body:

Routed in directly, not merged: input embeddings from google/gemma-4-31B-it-qat-q4_0-unquantized, and lm_head from Gryphe/Gemma-4-31B-StyleTune (the model is untied).

This model should compress well and run well at low precision, and the reasons are structural: The input embeddings come from Google's quantization-aware-trained Gemma-4 checkpoint; they were trained to survive q4_0 rounding, so taking them back down to four bits costs far less than quantizing ordinary weights. And because the merge is norm-preserving, it introduces no inflated weight magnitudes for a low-bit quantizer to clip or to spend its dynamic range on. This is the same property that makes structured hidden state survive rotation and aggressive quantization in the first place: what matters here is direction, and direction is what these formats keep.

The merged model's first-token distribution moves toward the sources it was built from, and does so selectively — it leans into the directions where a source has signal and stays near the base where it does not. Over long generations it holds: on four-thousand-token rollouts it kept its footing on dark fantasy, on literary prose, and on a multi-step reasoning problem that it solved correctly, in a setting where the louder variant of the same merge lost coherence partway through.

It is a roleplay and creative-writing model first. It will reason and it will code, but that is not what it was tuned for.

This is an experimental merge built around the isometric merge algorithm described above. The radial energy knob still matters, and too much of it can bring back the same instability the rotation was meant to avoid.

The model is also untied: the input embeddings and lm_head are routed in from different sources. Preserve that layout when converting or quantizing.

Standard Gemma instruct chat template, included. It prefers a little room: temperature around 0.8–1.0, top-p around 0.95. Untied head, bf16, 17 shards.

If you have repetition problems, dialing up min-p helps. Gryphe's StyleTune head causes quite a bit of instability for the sampler. It also helps to use instruction + thinking.