Benchmark	This Model	RedHatAI FP8	Delta
GSM8k-Platinum (5-shot)	97.68%	97.82%	-0.14
MMLU-CoT (5-shot)	89.97%	90.70%	-0.73
MMLU-Pro (5-shot)	84.52%	84.92%	-0.40
IFEval inst_strict	93.41%	93.84%	-0.43
IFEval prompt_strict	90.76%	91.31%	-0.55

Generation parameters: temperature=1.0, top_p=0.95, top_k=64, max_gen_toks=8192

Usage

Method: FP8_BLOCK (weight_block_size=[128,128], activation_scheme=dynamic)
Tool: <a href="https://github.com/vllm-project/llmcompressor" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">llmcompressor model_free_ptq
Format: compressed-tensors (vLLM native)
Ignored layers: vision tower, lm_head, embedding layers
Model size: ~31.5 GB (vs ~62 GB BF16)

python

from vllm import LLM
model = LLM("kakrotto/gemma-4-31B-it-uncensored-heretic-FP8")

Disclaimer

This is an uncensored model. The quantizer (kakrotto) is not responsible for the model's outputs or any misuse. This FP8 quantization preserves the original model's behavior. Please use responsibly.

Attribution

Source model: llmfan46/gemma-4-31B-it-uncensored-heretic
Quantization recipe reference: RedHatAI/gemma-4-31B-it-FP8-block