Tested hardware: one Intel Arc Pro B70, 32 GB VRAM, on Linux. No B50, Windows, CPU, or offload performance claim.
Latest uncached 24K prefill: 4,841 → 5,653 tokens/s (+16.8%) in a matched Heretic-reference comparison; the clean deployment build confirmed 5,667 tokens/s. Prefix reuse was OFF, kernels were warmed, and PP means input tokens divided by time to first token. This is not a cache-hit benchmark or a 6K+ result at 24K.
Shorter prompts / generation: the matched test retained about 7.4K PP at 6,622 input tokens and 112 output tokens/s on short prompts. These are reference-model figures, not guaranteed speeds for this exact fine-tune or every RP request.
Memory: about 26.8 GiB sampled peak card usage at 24K with an 8 GiB KV pool. Download size is not total VRAM. A 12 GB minimum has not been established.
Custom fork required for this gain: Wondernuttz/OpenVINO, wide-query commit, plus matching OpenVINO GenAI/tokenizers. This combines custom tuning with credited Intel/OpenVINO kernels and fixes; it is not an unmodified stock wheel.
Inference: use the complete OpenVINO export with OpenVINO GenAI, not direct Transformers loading. The tested 26B/B70 profile uses PA, DQ128, U4 KV, cache8 GiB, batch16384 and one sequence. Set GEMMA_MIXED_512_TILE=wideq before Python starts, alongside MOE_USE_GROUPED_GEMM_PREFILL=1 and MOE_GROUPED_BINARY_LOOKUP=1. RP prefix caching may stay enabled. See full instructions and rollback.
Quality / scope: all 17 matched Heretic output strings were identical; Chimera passed separate 24K recall and prefix-update checks. The clean kernel passed 24 repeated U4 reference tests with bit-identical repeats. The old baseline showed U4 variability, and an extra OFF/ON comparison gate did not pass; that caveat remains documented. This is bounded validation, not a universal intelligence guarantee.
Context and installation: the live Heretic bots received the new kernel while keeping their existing 16K context and settings. Valid exports need no recompression. Rebuild custom runtime wheels/container to get the new kernel—git pull alone does not replace OpenVINO inside an image. No new prebuilt image is supplied. 131K RoPE LUT coverage is not validated 131K usable context; reserve output space.

StyleTune V2 remains HELD on its prior 16K profile. This update does not approve the wide-query/24K profile for StyleTune or resolve its earlier RP discrepancy. Do not enable the new profile based on the reference benchmarks alone.

Earlier binary-lookup results (including 7,435 PP at 6,622 and 6,866 PP at 15,872) remain historical measurements with their original profiles. The wider-query result above is an additional cached/chunked-prefill improvement, not a replacement claim for every model or context size. Original model/merge credits, license and detailed historical notes below remain unchanged.

Detailed September runtime notes, configuration, and validation history

September 6 runtime update: faster long-prompt processing

Clean Release follow-up and rollout status

StyleTune runtime rollout is HELD. Automated 24K factual/cache checks passed, but a later RP sample had broken phrasing. The 6.6K lookup-off/on outputs matched; the separate 24K comparison remains inconsistent and needs repeated controls. The existing local StyleTune runtime/settings are unchanged. Do not treat this as a clean all-quality validation.

The clean Release report records a matched Heretic 24K result of 2,711 → 4,771 PP tok/sec (+76%), the per-checkpoint decode/cache measurements, exact runtime hashes, and limitations. The kernel journal also recorded engine resets when the temporary canary server was terminated after its successful replies; shutdown/model-switch stability is not certified. The old runtime was retained for rollback. These updates change runtime documentation, not model weights or repository visibility.

The numbers below are reference-stack measurements on Gemma 4 26B A4B Heretic, not a benchmark of this exact fine-tune. This checkpoint belongs to the related Gemma 26B A4B MoE family; per-checkpoint speed, memory, and quality need their own validation.

The Wondernuttz OpenVINO fork update and full report adds an opt-in binary search for grouped-MoE token-row lookup and incorporates eight credited Intel upstream backports. No model reconversion or weight download is required to get this runtime optimization. It requires the patched, ABI-matched OpenVINO runtime; an environment variable alone cannot add it to a stock wheel.

Heretic reference input	Previous lookup	New lookup	Matched PP gain
6,622 tokens	6,028 PP tok/sec	7,435 PP tok/sec	23%
15,872 tokens	2,921 PP tok/sec	6,866 PP tok/sec	135% / 2.35x
24,576 tokens	2,735 PP tok/sec	4,767 PP tok/sec	74%

One Intel Arc Pro B70 (32 GB), DQ128, U4 KV, 8 GiB cache allocation, batch16384, one sequence, prefix reuse off. PP is input tokens divided by TTFT. At 24K, TTFT was ~5.15 seconds, long-context decode ~93 tok/sec, sampled VRAM ~26.5 GiB. Decode was essentially unchanged. The 24K matched pair produced identical output strings and passed 11 automated checks; this is bounded coherence evidence, not a universal intelligence or production-stability guarantee.

With prefix caching on, an exact 24K repeat reached first token in 173 ms. That is cache-hit latency, not raw prefill throughput. An updated-fact probe retrieved the corrected password but still answered earlier embedded questions, so strict output-only formatting was not perfect. Separate patched-only 32K probes also passed; do not automatically raise this fine-tune's serving cap to match a reference experiment.

To opt in, set these before pipeline creation, using the matching fork runtime:

bash

export MOE_USE_GROUPED_GEMM_PREFILL=1
export MOE_GROUPED_BINARY_LOOKUP=1

The report includes the PA scheduler configuration, exact revisions, binary hash, matched methodology, limitations, and rollback switch. The published measurements use a Release/O3 binary with profiling capability compiled in but measurement counters disabled; clean-release deployment validation is separate. Native vision/audio are not validated by these text tests. Preserve your model's chat template, reasoning framing, and sampler.

Correction to older tuning attribution: MOE_MICRO_GEMM_N_HINT is bypassed by the active grouped path and must not be credited as a grouped tile optimization. Older benchmark/setup sections below document their historical builds, not updated limits or identical per-tune guarantees. The new lookup does not apply to dense Gemma 31B.

Original base-model, fine-tune, merge credits and licenses below are unchanged. Wondernuttz's contribution is the OpenVINO conversion/runtime work and testing.

OpenVINO INT4 AWQ - asymmetric 4-bit weight compression, verified group size 64.

Gemma-4 26B-A4B StyleTune V2 — OpenVINO INT4

OpenVINO INT4 conversion of Gryphe/Gemma-4-26B-A4B-StyleTune-V2 for Intel CPU/GPU inference.

Gryphe created the StyleTune and deserves the model credit. Wondernutts performed the OpenVINO conversion and graph preparation. Read the original card for Gryphe's full explanation, methodology, metrics, and credits.

About the original tune

This is Gryphe's single-epoch V2 style tune of Gemma 4 26B-A4B. It surgically trains the lm_head output projection while leaving the transformer body untouched. Gryphe reports a 52% reduction in clichés and substantially different phrase selection across a 200-prompt roleplay benchmark, while retaining the underlying model's other capabilities. Those figures belong to the source model; this conversion has not rerun that benchmark.

The intended strengths are roleplay, character voice, narrative prose, and creative writing. Gemma 4's native bundled chat template is the correct prompt format.

Conversion details

Item	Value
Source	`Gryphe/Gemma-4-26B-A4B-StyleTune-V2`
Architecture	Gemma 4 26B-A4B mixture of experts; conditional-generation/VLM export
Language-model compression	INT4 asymmetric AWQ, group size 64, ratio 1.0
Router	Excluded from INT4 compression
Group fallback	Adjust incompatible groups
Dynamic quantization group	64 in export metadata
OpenVINO binary payload	approximately 15.02 GiB
Arc graph preparation	precomputed RoPE lookup-table path included by the conversion recipe

The model contains language, text-embedding, per-layer embedding, vision, tokenizer, and detokenizer IRs. The graph advertises the source architecture's long context, but this specific StyleTune conversion has not been independently benchmarked across that full range.

Download

This repository is private, so authenticate first.

bash

python -m pip install "openvino-genai==2026.2.1.0" "huggingface_hub>=1.0"
hf auth login
hf download Wondernutts/Gemma-4-26B-A4B-StyleTune-V2-int4-ov --local-dir ./gemma4-26b-styletune-v2-int4-ov

Run on Intel GPU

python

import openvino_genai as ov_genai

model_dir = "./gemma4-26b-styletune-v2-int4-ov"
pipe = ov_genai.VLMPipeline(model_dir, "GPU", CACHE_DIR="./ov_cache")

config = ov_genai.GenerationConfig()
config.max_new_tokens = 768
config.do_sample = True
config.temperature = 1.0
config.top_p = 0.95
config.repetition_penalty = 1.15

pipe.start_chat("You are a sharp-witted fantasy companion with a distinct voice.")
print(pipe.generate("The tavern has gone silent. Continue the scene.", generation_config=config))
pipe.finish_chat()

start_chat() uses the bundled Gemma chat template and maintains KV cache across turns. Gryphe's own sampling preference is temperature 1.0, Min-P 0.10, and DRY; OpenVINO GenAI exposes temperature and nucleus sampling directly, while Min-P/DRY availability depends on the serving layer.

Use VLMPipeline, even for text-only chat, because this is the conditional-generation export. For continuous-batching servers on Intel GPU, use DYNAMIC_QUANTIZATION_GROUP_SIZE=0 if the default dynamic quantization path produces garbled output. No throughput claims are made until this exact conversion is benchmarked.

License and attribution

Apache-2.0, following Gryphe's source repository and the upstream Gemma release. StyleTune authorship and training credit: Gryphe. OpenVINO conversion: Wondernutts.