You can deploy this model and start earning money today!

Wondernutts/
gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic-int4-ov

Name: gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic-int4-ov
Availability: InStock

llmfan46/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic

finetune

apache-2.0

Views

No views yet

Gemma 4 26B-A4B — custom OpenVINO runtime update

Tested hardware: one Intel Arc Pro B70, 32 GB VRAM, on Linux. No B50, Windows, CPU, or offload performance claim.
Latest uncached 24K prefill: 4,841 → 5,653 tokens/s (+16.8%) in a matched Heretic-reference comparison; the clean deployment build confirmed 5,667 tokens/s. Prefix reuse was OFF, kernels were warmed, and PP means input tokens divided by time to first token. This is not a cache-hit benchmark or a 6K+ result at 24K.
Shorter prompts / generation: the matched test retained about 7.4K PP at 6,622 input tokens and 112 output tokens/s on short prompts. These are reference-model figures, not guaranteed speeds for this exact fine-tune or every RP request.
Memory: about 26.8 GiB sampled peak card usage at 24K with an 8 GiB KV pool. Download size is not total VRAM. A 12 GB minimum has not been established.
Custom fork required for this gain: Wondernuttz/OpenVINO, wide-query commit, plus matching OpenVINO GenAI/tokenizers. This combines custom tuning with credited Intel/OpenVINO kernels and fixes; it is not an unmodified stock wheel.
Inference: use the complete OpenVINO export with OpenVINO GenAI, not direct Transformers loading. The tested 26B/B70 profile uses PA, DQ128, U4 KV, cache8 GiB, batch16384 and one sequence. Set GEMMA_MIXED_512_TILE=wideq before Python starts, alongside MOE_USE_GROUPED_GEMM_PREFILL=1 and MOE_GROUPED_BINARY_LOOKUP=1. RP prefix caching may stay enabled. See full instructions and rollback.
Quality / scope: all 17 matched Heretic output strings were identical; Chimera passed separate 24K recall and prefix-update checks. The clean kernel passed 24 repeated U4 reference tests with bit-identical repeats. The old baseline showed U4 variability, and an extra OFF/ON comparison gate did not pass; that caveat remains documented. This is bounded validation, not a universal intelligence guarantee.
Context and installation: the live Heretic bots received the new kernel while keeping their existing 16K context and settings. Valid exports need no recompression. Rebuild custom runtime wheels/container to get the new kernel—git pull alone does not replace OpenVINO inside an image. No new prebuilt image is supplied. 131K RoPE LUT coverage is not validated 131K usable context; reserve output space.

This checkpoint: the headline benchmark used the local LUT131K deployment of this Heretic model. Its bot runtime has been updated; current bot context remains 16K, with separate bounded 24K tests.

Earlier binary-lookup results (including 7,435 PP at 6,622 and 6,866 PP at 15,872) remain historical measurements with their original profiles. The wider-query result above is an additional cached/chunked-prefill improvement, not a replacement claim for every model or context size. Original model/merge credits, license and detailed historical notes below remain unchanged.

Detailed September runtime notes, configuration, and validation history

September 6 runtime update: faster long-prompt processing

Clean Release follow-up and rollout status

This checkpoint completed the clean Release 24K factual/boundary/prefix-cache gates and is included in the local runtime rollout. Manual RP review found broadly coherent prose, not perfect instruction following; see the per-model caveats.

The clean Release report records a matched Heretic 24K result of 2,711 → 4,771 PP tok/sec (+76%), the per-checkpoint decode/cache measurements, exact runtime hashes, and limitations. The kernel journal also recorded engine resets when the temporary canary server was terminated after its successful replies; shutdown/model-switch stability is not certified. The old runtime was retained for rollback. These updates change runtime documentation, not model weights or repository visibility.

The matched benchmark used the local LUT131K Heretic deployment of this model. The runtime update does not change the weights in this repository.

The Wondernuttz OpenVINO fork update and full report adds an opt-in binary search for grouped-MoE token-row lookup and incorporates eight credited Intel upstream backports. No model reconversion or weight download is required to get this runtime optimization. It requires the patched, ABI-matched OpenVINO runtime; an environment variable alone cannot add it to a stock wheel.

Heretic reference input	Previous lookup	New lookup	Matched PP gain
6,622 tokens	6,028 PP tok/sec	7,435 PP tok/sec	23%
15,872 tokens	2,921 PP tok/sec	6,866 PP tok/sec	135% / 2.35x
24,576 tokens	2,735 PP tok/sec	4,767 PP tok/sec	74%

One Intel Arc Pro B70 (32 GB), DQ128, U4 KV, 8 GiB cache allocation, batch16384, one sequence, prefix reuse off. PP is input tokens divided by TTFT. At 24K, TTFT was ~5.15 seconds, long-context decode ~93 tok/sec, sampled VRAM ~26.5 GiB. Decode was essentially unchanged. The 24K matched pair produced identical output strings and passed 11 automated checks; this is bounded coherence evidence, not a universal intelligence or production-stability guarantee.

With prefix caching on, an exact 24K repeat reached first token in 173 ms. That is cache-hit latency, not raw prefill throughput. An updated-fact probe retrieved the corrected password but still answered earlier embedded questions, so strict output-only formatting was not perfect. Separate patched-only 32K probes also passed; do not automatically raise this fine-tune's serving cap to match a reference experiment.

To opt in, set these before pipeline creation, using the matching fork runtime:

bash

export MOE_USE_GROUPED_GEMM_PREFILL=1
export MOE_GROUPED_BINARY_LOOKUP=1

The report includes the PA scheduler configuration, exact revisions, binary hash, matched methodology, limitations, and rollback switch. The published measurements use a Release/O3 binary with profiling capability compiled in but measurement counters disabled; clean-release deployment validation is separate. Native vision/audio are not validated by these text tests. Preserve your model's chat template, reasoning framing, and sampler.

Correction to older tuning attribution: MOE_MICRO_GEMM_N_HINT is bypassed by the active grouped path and must not be credited as a grouped tile optimization. Older benchmark/setup sections below document their historical builds, not updated limits or identical per-tune guarantees. The new lookup does not apply to dense Gemma 31B.

Original base-model, fine-tune, merge credits and licenses below are unchanged. Wondernuttz's contribution is the OpenVINO conversion/runtime work and testing.

OpenVINO INT4 AWQ - asymmetric 4-bit weight compression, verified group size 64.

Gemma-4 26B-A4B Heretic, OpenVINO INT4. Coherent to 32K on ONE Intel Arc B70 (32 GB).

This is the abliterated ("heretic") Gemma-4 26B-A4B mixture-of-experts, exported to OpenVINO INT4 and graph-patched so it stays coherent at long context on Intel GPUs. Stock exports of this model break down into word salad past about 16K tokens on Arc. This build does not.

It is a general instruction model with the refusal behavior removed, so you can use it for anything you would use the base model for. I personally run it for uncensored roleplay (Skyrim companion AI and Discord character bots), and the defaults below are tuned from hundreds of hours of that use.

Everything below was measured on one Intel Arc Pro B70 with 32 GB VRAM. The current fast path uses the Wondernuttz OpenVINO 2026.4 fork on Linux. The older stock OpenVINO 2026.2 VLMPipeline path remains documented because it is the path separately verified at 32K. No llama.cpp and no CUDA were used for either set of results.

Toolkit: patches, OpenAI-compatible server, quickstart, and the full bug catalog live at github.com/Wondernuttz/OpenVino-For-Gemma-4.

Measured performance (single Arc Pro B70)

Metric	Current OpenVINO 2026.4 fork result
Decode	112.2 tok/s short context; 94.9 tok/s after 6,622 tokens
Prefill	5,827.2 tok/s sustained mean at 6,622 tokens; peak measured point 6,500.2 tok/s at 4K
Coherence	passed 4/4 long-prompt retrieval and style checks
Hardware	one Arc Pro B70, 32 GB

Fork branch: arc-xe2-gemma4-pa-2026.4, commit 2c82358676.

Benchmark history

Stage	6,622-token PP	Decode	Coherence
July 6 nightly continuous batching	971.6 tok/s	93.2 tok/s	failed
July 23 stock continuous batching	1,048.4 tok/s	95.6 tok/s	passed
Fork, one scheduler pass	4,143.5 tok/s	94.6 tok/s	passed
Fork, grouped MoE and N128	4,448.5 tok/s	95.1 tok/s	passed
Fork, 512-head Xe2 micro-SDPA	5,827.2 tok/s	112.2 tok/s short	passed

The last patch is a full-model 31.0% PP gain over the previous accepted fork result and a 5.56x gain over the coherent July 23 stock continuous-batching result. Prompt contents differed between runs, so these are not prefix-cache hits. The accepted coherence output remained bit-identical with SHA-256 b123146233af2aac9e725826d9011513ee4c3dc9ec4634fd892e6937f06afb58.

The full history, exact settings, profiles, build commands and rejected runs are in BENCHMARK_HISTORY_GEMMA4_26B.md.

Current context curve

Release plugin, DQ128, second unique request at each shape:

Prompt tokens	Throughput
966	5,158.9 tok/s
2,048	6,359.5 tok/s
4,096	6,500.2 tok/s
6,622	5,869.5 tok/s

At 6,622 tokens, two sustained DQ64 runs measured 5,821.9 and 5,832.5 tok/s. DQ128 tied within measurement noise and stays in the deployment configuration.

Older stock single-stream context curve

These OpenVINO 2026.2 measurements use the compatibility path. They are kept because this is the path that was actually exercised at 32K.

Prompt tokens	Throughput	Time to ingest
512	~2,900 tok/s	0.2 s
2K	~4,700 tok/s	0.5 s
6K	~3,200 tok/s	2.2 s
16K	~1,300 tok/s	14 s
32K	~530 tok/s	61 s

Fastest short-prompt prefill in my fleet (2.5x SYCL at matched pp512). Past roughly 5K my Qwen3.6-35B MoE overtakes it, and at 32K the Qwen ingests nearly 3x faster; full attention pays a quadratic tax at depth that the Qwen's hybrid linear attention mostly avoids.

Long-context capability, verified by needle retrieval

A password fact was planted early in the prompt and the model was asked to retrieve it at the end. Pass means it answered with the exact password.

Context	Thinking OFF	Thinking ON
8K	PASS (3.4 s)	PASS (6.9 s)
16K	PASS (10.3 s)	PASS (16.6 s)
32K	PASS (54 s)	PASS (63 s, clean structured reasoning)

Thinking mode reasoning correctly over 32,000 tokens of context is the standout result. Without the rope patch in this build, thinking degrades within a few thousand tokens on Intel GPUs. Past 32K the limit on my box is host RAM during prefill, not the model or the card; the architecture is rated to 262K positions and this build ships position tables to 131K.

Those 32K results belong to the older stock single-stream path. The current 2026.4 fork path is validated through 16K. Do not split a 16K continuous-batching prefill with an 8,192-token scheduler limit; that test triggered an Xe GPU fault. A 30K continuous-batching test also faulted after the larger scheduler shape exceeded the GPU maximum allocation size. The fast path is not advertised beyond 16K yet.

How this compares to published numbers

I found no other public single-B70 result matching the current OpenVINO speed as of 2026-07-26. The public PMZFX repository reports the same Gemma-4 26B-A4B class at 1,129 pp512 and 52.6 tg128 with llama.cpp SYCL.

Same card (Arc Pro B70), decode	tok/s
llama.cpp SYCL, Q4_K_M (PMZFX benchmarks)	52.6
This build, OpenVINO INT4, no speculative decoding	112.2

That is 2.13x the public B70 decode result. The current nearest short-prefill point is 5,158.9 tok/s at 966 tokens, so it must not be called a matched pp512 comparison. The older stock path did measure about 2,900 tok/s at pp512.

For an NVIDIA reference, a public single-RTX-3090 replication of this model reports 129 to 131 tok/s with llama.cpp and n-gram speculative decoding. The current B70 result is 112.2 tok/s without speculative decoding, so RTX 3090-class decode is a fair description. A public RTX 5090 vLLM result reports about 228 tok/s without DFlash and 578 tok/s with DFlash. This B70 is not at RTX 5090 decode speed.

NVIDIA sources: single RTX 3090 Gemma-4 replication and the single RTX 5090 DFlash run.

What was fixed (why this is not just another export)

The MoE router is kept unquantized (AWQ INT4, group size 64, ignored_scope on the router, matching Intel's own recipe). Quantizing the router breaks GPU MoE fusion and the model will not load at all.
RoPE lookup-table patch. Intel GPUs execute the rope angle math in fp16, which cannot represent the rotation angles past roughly 16K positions, so long context collapses into gibberish. This build replaces the runtime sin/cos computation with precomputed tables and a Gather, which fp16 execution cannot corrupt. Note for anyone patching other Gemma-4 exports: the global attention uses proportional (partial) RoPE, so three quarters of the frequency values are zero BY DESIGN. Do not "repair" them; preserve the zeros.
The matching July 23 OpenVINO and GenAI commits fix Gemma-4 PagedAttention conversion, token_type_ids layout and the sliding-window graph path. Earlier continuous-batching builds either aborted or returned garbage at long prompt lengths.
max_num_batched_tokens=8192 keeps the 6,622-token benchmark in one scheduler pass. This was the jump from roughly 1,014 to 4,143 PP tok/s.
Grouped MoE prefill, DQ128 and the N128 tile selection raised the accepted result to 4,448 PP tok/s.
Commit 2c82358676 raises Xe2 micro-SDPA eligibility from 256 to 512 head size. Gemma's five global-attention layers then use the existing XMX route, raising the sustained result to 5,827 PP tok/s.

How to run

The stock single-stream path is the easiest compatibility route:

bash

pip install openvino-genai==2026.2.0 huggingface_hub
hf download Wondernutts/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic-int4-ov --local-dir ./gemma4-26b-heretic-ov

python

import openvino_genai as g

# "GPU" is your Arc card. On multi-GPU systems check
# openvino.Core().get_property("GPU.N", "DEVICE_PCI_INFO") first.
pipe = g.VLMPipeline("./gemma4-26b-heretic-ov", "GPU")

def chat(system, user, think=False):
    p = "<bos><|turn>system\n" + system + ("\n<|think|>" if think else "") + "<turn|>\n"
    p += "<|turn>user\n" + user + "<turn|>\n<|turn>model\n"
    if not think:
        p += "<|channel>thought\n<channel|>"   # pre-closed thought channel = fast direct replies
    c = g.GenerationConfig()
    c.max_new_tokens = 512 if not think else 1536
    c.do_sample = True; c.temperature = 0.9; c.top_p = 0.95
    try: c.repetition_penalty = 1.2            # recommended, see notes
    except Exception: pass
    try: c.apply_chat_template = False
    except Exception: pass
    return str(pipe.generate(p, generation_config=c))

print(chat("You are a witty tavern keeper in Whiterun.", "Rough night?"))

Headline-performance path

The 5,827 PP and 112 decode results require the Linux fork branch above, the matching OpenVINO 2026.4 ABI and the matching GenAI build. A 2026.4 GPU plugin cannot be dropped into a 2026.2 runtime. Build instructions and pinned commits are in the benchmark history.

The runtime configuration is:

python

import os
import openvino_genai as g

os.environ["MOE_USE_GROUPED_GEMM_PREFILL"] = "1"
os.environ["MOE_MICRO_GEMM_N_HINT"] = "128"

scheduler = g.SchedulerConfig()
scheduler.cache_size = 8
scheduler.enable_prefix_caching = True
scheduler.max_num_batched_tokens = 8192

pipe = g.VLMPipeline(
    "./gemma4-26b-heretic-ov",
    "GPU",
    DYNAMIC_QUANTIZATION_GROUP_SIZE=128,
    scheduler_config=scheduler,
)

Use max_num_batched_tokens=16384 for the separately validated 16K shape. Do not use 8,192 to split that prompt. The optimized branch is Linux-tested. Windows is untested for this fork path.

Notes that will save you pain:

The prompt format is <|turn>role ... <turn|> (this model's native template, check chat_template.jinja), not classic Gemma <start_of_turn>. The classic format "works" but leaks a stray thought prefix and runs about 20% slower.
Thinking is binary. Pre-close the thought channel as above for fast direct replies. Put <|think|> in the system turn and do not pre-close for reasoning-first answers. When thinking, give it at least 1024 max_new_tokens or the answer gets truncated.
Do not use grammar-constrained or JSON mode (response_format). Gemma-4 has a documented repetition-collapse bug (google-deepmind/gemma#622) that fires most reliably under constrained JSON decoding. Asking for JSON in the prompt is fine.
Repetition penalty around 1.2 holds INT4 coherence at temperature 1.0 in long roleplay sessions. Values near 1.05 were observed to degenerate.
On the stock 2026.2 continuous-batching path, use DYNAMIC_QUANTIZATION_GROUP_SIZE=0. The corrected 2026.4 fork path is coherent at DQ128 and uses it for deployment.

Provenance

google/gemma-4-26B-A4B-it (QAT q4_0 unquantized), heretic abliteration by llmfan46 (directional ablation, norm-preserving), then OpenVINO INT4 AWQ export with the router excluded, then the RoPE lookup-table graph patch (this repo).

The 31B dense sibling is also published: richer prose, slower replies, honest single-card limits on its own card.

Intended use and content notice

This is an uncensored general model. Built and tested for roleplay and creative writing on local Intel hardware; The abliteration removes refusal behavior and outputs are unfiltered; you are responsible for lawful and appropriate use. Licensed under Apache 2.0, same as the upstream Gemma 4 release.