Property	Value
Base Model	google/gemma-4-12b-it
Quantization Method	RTN (round-to-nearest)
Format	float-quantized (FP8)
Weight Strategy	channel (per-channel)
Activation Strategy	token (per-token, dynamic)
Weight Bits	8 (FP8 E4M3)
Model Size	~15 GB
Calibration Data	None (data-free)

Benchmark Results

Evaluated with lm-evaluation-harness v0.4.12 using vLLM v0.25.1 on 2x NVIDIA H100 80GB.

Variant	GSM8K (flexible)	GSM8K (strict)
Baseline (BF16)	87.57%	86.58%
FP8 Dynamic	87.95%	86.96%
Delta	+0.38%	+0.38%

FP8 Dynamic quantization shows no accuracy loss on GSM8K — the delta is within noise (stderr ~0.9%).

Quantization Recipe

Quantized with llm-compressor v0.12.0 on 2x NVIDIA H100 80GB GPUs.

python

from transformers import AutoModelForImageTextToText, AutoProcessor
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "google/gemma-4-12b-it"
model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, dtype="auto")

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=[
        "lm_head",
        "re:.*embed_vision.*",
        "re:.*embed_audio.*",
        "re:.*vision_embedder.*",
    ],
)

oneshot(model=model, recipe=recipe)
model.save_pretrained("gemma-4-12b-it-FP8-Dynamic", save_compressed=True)

Serving with vLLM

bash

pip install vllm
vllm serve soyrsoyr/gemma-4-12b-it-FP8-Dynamic --max-model-len 4096 --tensor-parallel-size 2

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="soyrsoyr/gemma-4-12b-it-FP8-Dynamic",
    messages=[{"role": "user", "content": "Explain quantization in one sentence."}],
    max_tokens=128,
)
print(response.choices[0].message.content)

Running on Kubernetes / OpenShift

yaml

containers:
  - name: vllm
    image: vllm/vllm-openai:latest
    args:
      - --model
      - soyrsoyr/gemma-4-12b-it-FP8-Dynamic
      - --max-model-len
      - "4096"
      - --tensor-parallel-size
      - "2"
    resources:
      limits:
        nvidia.com/gpu: "2"