NVFP4 quantized version of huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated — an abliterated (uncensored) variant of Google's Gemma 4 26B-A4B Mixture-of-Experts model.

49 GB → 16.5 GB — runs on a single NVIDIA Blackwell GPU at ~160 tok/s.

Known Issue: Japanese / Non-English Long-Form Generation

NVFP4 quantization of the 26B (128-expert) model causes intermittent repetition collapse on long Japanese text generation. The model may produce degenerate output like get(get) get(get)... when generating 500+ tokens in Japanese. English tasks (code, math, reasoning) are unaffected.

This appears to be inherent to the 128-expert MoE architecture under FP4 quantization — with fewer experts, quantization noise can corrupt the specific expert combinations needed for non-English generation.

For multilingual / Japanese workloads, we recommend the 48B (256-expert) variant instead: sakamakismile/Huihui4-48B-A4B-abliterated-NVFP4 — same inference speed, stable across all languages.

Key Specs


Base model	huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated
Architecture	Gemma 4 MoE — 25.2B total, 3.8B active per token
Quantization	NVFP4 (W4A4 — weights FP4, activations FP4, scales FP8)
Format	`compressed-tensors` (native vLLM support)
Tool	vllm-project/llm-compressor (main)
Size	16.5 GB (single safetensors shard)
Requires	NVIDIA Blackwell GPU (SM 120), vLLM nightly (cu130)

Quickstart

vLLM (recommended)

bash

vllm serve Lna-Lab/Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4 \
    --max-model-len 8192

No --quantization flag needed — vLLM auto-detects compressed-tensors format.

Docker

bash

docker run --gpus '"device=0"' -p 8016:8016 \
    -v /path/to/model:/models/current:ro \
    --shm-size 16gb \
    vllm/vllm-openai:cu130-nightly \
    vllm serve /models/current --port 8016 --max-model-len 8192

Python (vLLM)

python

from vllm import LLM, SamplingParams

llm = LLM(
    model="Lna-Lab/Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4",
    max_model_len=8192,
    gpu_memory_utilization=0.85,
)

output = llm.generate(
    ["Explain quantum entanglement in simple terms."],
    SamplingParams(max_tokens=256, temperature=0.7),
)
print(output[0].outputs[0].text)

Benchmark

Tested on a single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM), vLLM 0.19.1+, max_model_len=8192, temperature=0.0.

Task	Tokens	Time (s)	Speed (tok/s)	Quality
Japanese essay (方丈記)	2,048	14.05	145.8	FAIL (repetition collapse)
Python code generation	1,904	12.04	158.1	PASS
Math reasoning (EN)	1,090	6.86	158.8	PASS
VLM image description	271	1.80	150.2	PASS

Sustained throughput: ~150–160 tok/s (post-warmup, single GPU).

VRAM Usage

State	GPU Memory
After model load	89,678 MiB
Peak (during inference)	90,074 MiB

Comparison with 48B (256-expert) variant

	26B (this model)	48B-NVFP4
Experts	128	256
Active params	3.8B	3.8B (same)
Speed	~158 tok/s	~153 tok/s
Disk size	16.5 GB	27.3 GB
English quality	Good	Good
Japanese quality	Unstable	Stable
VLM	Good	Good

Quantization Details

Recipe

yaml

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head, 're:.*embed.*', 're:.*router', 're:.*vision_tower.*']
      scheme: NVFP4

What's quantized, what's not

Quantized (NVFP4): All Linear layers in the language model, including MoE expert layers
Kept in BF16: lm_head, all embedding layers, MoE routers, entire vision tower

Calibration

Dataset: neuralmagic/calibration (LLM split)
Samples: 20
Max sequence length: 8192
MoE expert calibration handled automatically by llm-compressor's SequentialGemma4TextExperts

Reproduction

python

from datasets import load_dataset
from transformers import AutoProcessor, Gemma4ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = Gemma4ForConditionalGeneration.from_pretrained(
    "huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated", dtype="auto"
)
processor = AutoProcessor.from_pretrained(
    "huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated"
)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["lm_head", "re:.*embed.*", "re:.*router", "re:.*vision_tower.*"],
)

ds = load_dataset("neuralmagic/calibration", name="LLM", split="train[:20]")

def preprocess_function(example):
    messages = [
        {"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
        for m in example["messages"]
    ]
    return processor.apply_chat_template(
        messages, return_tensors="pt", padding=False, truncation=True,
        max_length=8192, tokenize=True, add_special_tokens=False,
        return_dict=True, add_generation_prompt=False,
    )

ds = ds.map(preprocess_function, batched=False, remove_columns=ds.column_names)

import torch
def data_collator(batch):
    assert len(batch) == 1
    return {
        key: (torch.tensor(value) if key != "pixel_values"
              else torch.tensor(value, dtype=torch.bfloat16).squeeze(0))
        for key, value in batch[0].items()
    }

oneshot(
    model=model, recipe=recipe, dataset=ds,
    max_seq_length=8192, num_calibration_samples=20,
    data_collator=data_collator,
)

model.save_pretrained("output-NVFP4", save_compressed=True)
processor.save_pretrained("output-NVFP4")

Environment

Package	Version
torch	2.11.0+cu130
transformers	5.5.4
llmcompressor	0.1.dev (main @ 3084520)
compressed-tensors	0.15.1a20260414
safetensors	0.7.0
CUDA	13.0

Requirements

GPU: NVIDIA Blackwell (RTX 5090, RTX PRO 6000, B200, etc.) — NVFP4 requires SM 120
VRAM: ~16 GB minimum
Software: vLLM nightly (cu130 build), or any framework supporting compressed-tensors NVFP4

Notes

This is an abliterated (uncensored) model. The base model has had safety training removed. Use responsibly.
Vision tower is kept in BF16 — multimodal capabilities are preserved at full precision.
NVFP4 is a Blackwell-specific format. This checkpoint will not work on Ampere/Hopper GPUs.

Credits

Base model: huihui-ai (abliteration)
Original model: Google DeepMind (Gemma 4)
NVFP4 quantization & benchmarking: Lna-Lab
Blackwell NVFP4 GEMM kernels: lna-lab/blackwell-geforce-nvfp4-gemm
Quantization framework: vllm-project/llm-compressor
Quantization method follows RedHatAI/gemma-4-26B-A4B-it-NVFP4 (proven path)

Support the Base Model Author

If you find this model useful, please consider supporting huihui-ai — the creator of the abliterated base model:

Ko-fi: https://ko-fi.com/huihuiai
Bitcoin: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge

sakamakismile/Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4

Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4

Known Issue: Japanese / Non-English Long-Form Generation