🧠 Research & Optimization: Custom APEX Quants
This repository hosts a curated suite of APEX (Adaptive Precision for EXpert Models) configurations. This collection bridges the gap between the standard APEX layouts and experimental mixed-precision configurations, all specifically adjusted to adhere with the structural nuances and naming conventions of the Gemma 4 MoE architecture.
⚠️ Disclaimer: Some of these layouts are custom-crafted and experimental. All credit for the underlying architecture, style fine-tune, original model weights and so on, belongs entirely to the upstream authors and developers. The foundational imatrix data utilized across this suite was generated by mradermacher.
🔬 The Mixed-Precision Philosophy
- The APEX Edge: Uniform quantization compresses all layers equally, which degrades MoE routing logic. The APEX approach locks core routing blocks at high precision, protects vital shared experts, and offsets file size by compressing less sensitive mid-layer mixed expert blocks.
- The Custom Sweet Spot: Standard automated APEX scripts lack a balanced midrange option, forcing users to choose between a big
Q6 or a small Q3/Q4 config. This project largely aims for Q5-equivalent precision at a rough Q4+ memory footprint.
⚡ Standard APEX Suite
The default APEX configurations.
- i-Mini:
12,052.38 MiB MiB (3.89 BPW) — Extremely nimble, stable low-VRAM baseline.
- i-Compact:
13,341.05 MiB (4.31 BPW) — The all-rounder baseline sweet spot.
- i-Balanced:
19,436.36 MiB (6.28 BPW) — High-precision layer tracking with mid-expert compression.
- i-Quality:
18,226.36 MiB (5.89 BPW) — Advanced Importance Quantization (iq4_xs/nl) core valley logic.
🚀 Custom Quants
Continuing to build on the APEX logic, while introducing additional tweaks.
💎 The Upscale Engine
Legacy attempt at an intermediary tier, using an upscaled Compact as base layout, while testing different types of mid-layer gradient setups. Versions Econ/Balanced no longer being considered as standard, leaving only one remaining variant.
- i-Upscale.Premium:
14,996.94 MiB (4.84 BPW) — Stable, tested, high-performance configuration. Employs a [Q5_K -> Q4_K -> Q3_K] stepped gradient across the mid-layer experts while shielding context gates.
🎵 The Harmonic Series (Pulsed Attention)
The Harmonic layouts optimize layer contiguity, by following strict precision tiers across all 30 layers, and explicitly shield Global Attention (GA) layers from aggressive compression to anchor deep-context tracking.
- i-Harmonic.Minus:
14,853.74 MiB (4.80 BPW) — Experimental high-performance configuration. Condenses the architecture into 3 strict tiers (5/15/10). Shifts high-precision blocks to coincide with global attention pulses. In testing: Faster than Upscale.Premium, with better long-context detail attention.
- i-Harmonic.Plus:
16,288.70 MiB (5.26 BPW) — Experimental high-precision configuration. Eliminates the Q3_K compression valley entirely to enforce a flat, 2-tier matrix (10/20). In testing: Replaces and outperforms Upscale.Balanced at slightly larger overhead, while shielding long-context detail attention.
🧠 The Mindful Dynamics Series (Asymmetric Shock-Absorption)
The Mindful layouts build on the Harmonic framework, balancing precision-tier distributions while implementing additional buffers around the GA-layers.
- i-Mindful.Minus:
16,079.18 MiB (5.19 BPW) — Experimental Semantic Anchor. Follows 15/9/6 distribution. Trades uniform end-precision for comprehensive buffering of all Global Attention layers. Expectations: Slight edge in context stability and logical coherence under heavy prompt load.
- i-Mindful.Plus:
16,064.38 MiB (5.19 BPW) — Experimental Fluidity Engine. Built on a 16/7/7 topology. Trades early intermediate buffers for an uncompromised, clean high-precision runway across the final formatting layers. Expectations: Slight edge in inference speed and final execution.
💬 Join the Discussion & Co-Design!
- All configurations get uploaded to the repo. Everyone welcome to create or suggest their own designs, whether driven by specific hardware needs or fresh ideas.
- Observations are not gospel. Your hardware environment and prompt styles can be expected to create unique conditions for these layouts.
- Knowing when a custom quant is in demand saves it from being pruned. This is a gauntlet, where only a handful of layouts are expected to survive.
- Did you notice a specific variant handling a character's tone perfectly?
- Did one layout throw an odd loop that another avoided?
- Are you seeing different token-per-second (T/s) behavior on your rig?
Please open a thread, share your experience, and post your impressions.
Gemma-4-26B-A4B-StyleTune-V2
As promised, a slightly less wonky 26B-A4B Style Tune! Turns out you really shouldn't use this technique with multiple epochs. I delved deep into the data and found that a second epoch does all sorts of nasty stuff to MoE models, so V2 is a single epoch of an otherwise unchanged technique. The metrics have barely changed, but stability should be far, far better this time around!
Where V1 had a 54% reduction in cliches V2 offers 52%, but otherwise still an entirely new writing style, and the same Gemma 4 26B-A4B you already know underneath. One tensor changed out of 659. For realsies this time, I promise.
Also available in
31B version and a
12B version!
What is a style tune?
Normally when I finetune a model I train as much of it as possible, loading every tensor and transforming it to better approximate whatever's in my data. Not this time. This time I trained precisely one tensor: the lm_head output projection - the layer that decides which token to emit. Literally the last stop before text appears on your screen.
This specific tensor has a massive influence on a model's writing style, something I first discovered building MythoMax years ago. Gemma 31B (the first style tune) is a VRAM-hungry monster, so the question became: how do I have the maximum impact with the minimum hardware requirements?
The answer: freeze everything else. All 30 transformer layers, all the attention heads, all the MLPs — completely untouched. Only lm_head trains, which means VRAM requirements drop dramatically, training completes in a single overnight run on consumer hardware, and every single one of Gemma's capabilities remains fully intact. The model hasn't changed. Only the voice has, and it's done so in the best way possible. (Obligatory disclaimer: I might be biased towards my own data.)
I used the same data I had on me for my last Pantheon Reasoning release, with one notable exception - No instruct 24k set. 100% narrative data, certified cliché free.
What changed?
Benchmarked against 200 diverse roleplay prompts versus the base instruct model:
- 52% fewer clichés per 100 words (1.141 → 0.551)
- Only 19.9% shared trigram vocabulary - the model reaches for an almost entirely different set of phrases, with responses feeling much less sloppy as a result.
Considering we're talking about narrative data it's hard to provide you with many other meaningful statistics - It's one of those "try it to understand it" kinda situations.
What didn't change?
Everything else. All the reasoning capability, world knowledge, instruction following, and language understanding are completely intact - none of those live in lm_head. This isn't a full finetune. It's a targeted style replacement on a single tensor.
Inference
Whatever you prefer, Gemma seems remarkably flexible in that regard. I run with temp 1.0, 0.10 MinP and the DRY sampler.
Prompt Format
Gemma 4's native chat template applies automatically.
Credits
- Everyone from Anthracite! Hi, guys!
- Latitude, for which I am still producing finetunes on a regular basis, helping me keep my skills sharp and up-to-date!
- All the folks I chat with on a daily basis on Discord! You know who you are.
- Anyone I forgot to mention, just in case!
