Diagonal Fisher Information masks for Gemma 4 family models. Used to scale per-layer LoRA updates by parameter importance, and to derive instruct-subspace SVD masks for training-set isolation.
For each model parameter θ_i we estimate the diagonal of the Fisher Information matrix:
In plain terms: for each weight, how much does the model's output distribution depend on it? Weights with high Fisher are "load-bearing" for the model's current capabilities; weights with low Fisher are nearly free to move without disturbing existing behavior.
-
Per-layer scaling factor. Sum Fisher over each layer to get a layer-importance vector. During LoRA SFT, scale each layer's gradient (or the merge-time blend factor) by this importance — touch the load-bearing layers more gently, write more freely in the lightweight layers. Cheap defense against catastrophic forgetting.
-
Instruct-subspace SVD mask. Treat the per-tensor Fisher as a saliency map, take its top-k SVD modes, and project gradient updates away from that subspace. Effectively says: "do whatever you want, but don't touch the directions the instruct-tuned base model cares about." This is what powers our "release-merge" pipeline — style/RP CPT or SFT can happen on top of an it-base without losing instruction following.
Apache 2.0 — these are derived statistics, not model weights.
