Model Architecture: Gemma3ForConditionalGeneration
- Input: Text, Image
- Output: Text
Model Optimizations:
- Weight quantization: NVFP4
- Activation quantization: NVFP4
Intended Use Cases: Intended for commercial and research use.
Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws).
Release Date: 2026-08-02
Version: 1.0
Model Developers: Syafiq Kamarul Azman, Claude

Quantized version of google/gemma-3-4b-it, targeting NVIDIA Blackwell GPUs.

Model Optimizations

This model was obtained by quantizing the weights and activations of the linear layers in the language model of google/gemma-3-4b-it to NVFP4 data type, using llm-compressor. This optimization significantly reduces GPU memory requirements (by approximately 56%) and increases inference throughput.

The vision tower (SigLIP), multi-modal projector, lm_head, and token embeddings are kept in bf16. See Development Notes below for why.

Deployment

Use with vLLM

This model is tested against vllm/vllm-openai:v0.26.0 on an NVIDIA RTX 5050 (8GB VRAM). Tested launch flags:

bash

docker run \
  --gpus=all \
  --rm -d \
  -p 8080:8080 \
  -v /path/to/this/model:/model \
  --name gemma3-4b-nvfp4 \
  vllm/vllm-openai:v0.26.0-ubuntu2404 \
  /model \
  --served-model-name syaffers/gemma-3-4b-it-NVFP4 \
  --max-model-len 4096 \
  --max-num-seqs 8 \
  --kv-cache-dtype fp8_e4m3 \
  --host 0.0.0.0 \
  --port 8080

--max-model-len 4096 — the full 131k context in the base model's config is not realistic on 8GB; 4096 leaves enough headroom for KV cache after the ~5.6GB of weights are loaded.
--max-num-seqs — this is the biggest lever for VRAM. Start at 8; drop to 1 or 2 if you see KV-cache-related OOMs at startup (vLLM's own startup log prints how much KV cache memory it actually has available — check that line first).
--kv-cache-dtype fp8_e4m3 — halves KV cache memory at negligible quality cost, buying back some of the concurrency --max-num-seqs costs you. This is a serving-time flag, not something baked into the checkpoint (kv_cache_scheme in config.json is unset).

Send a request:

python

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8080/v1")

response = client.chat.completions.create(
    model="syaffers/gemma-3-4b-it-NVFP4",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image in one sentence."},
            {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
        ],
    }],
    max_tokens=60,
)
print(response.choices[0].message.content)

Use with transformers

Also loadable directly via transformers + compressed-tensors (no vLLM required), though without NVFP4 GEMM kernels it dequantizes to bf16 on first forward pass, so peak memory briefly needs close to the full bf16 footprint (~8GB) — tight but workable on an 8GB card, more comfortable on CPU or a larger GPU.

python

import torch
from transformers import Gemma3Processor, Gemma3ForConditionalGeneration

model_id = "syaffers/gemma-3-4b-it-NVFP4"
processor = Gemma3Processor.from_pretrained(model_id)
model = Gemma3ForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda:0")

Creation

Creation details

This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor), calibrating on 256 samples from [derek-thomas/ScienceQA](https://huggingface.co/datasets/derek-thomas/ScienceQA) (a VQA dataset, chosen so calibration exercises both the vision and text paths).

Run the following with: uv run quantize.py

python

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "torch>=2.11.0",
#     "transformers",
#     "datasets",
#     "llmcompressor==0.12.0",
# ]
#
# [tool.uv.sources]
# torch = { index = "pytorch-cu128" }
#
# [[tool.uv.index]]
# name = "pytorch-cu128"
# url = "https://download.pytorch.org/whl/cu128"
# explicit = true
# ///

import json

import torch
from datasets import Dataset, load_dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from transformers import AutoProcessor, Gemma3ForConditionalGeneration

MODEL_ID = "google/gemma-3-4b-it"
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-NVFP4"
NUM_CALIBRATION_SAMPLES = 256
MAX_SEQUENCE_LENGTH = 2048
BATCH_SIZE = 1


def build_calib_dataset(processor, num_samples: int):
    """Pre-tokenized calibration set."""
    ds = load_dataset("derek-thomas/ScienceQA", split="train").filter(
        lambda ex: ex["image"] is not None
    )
    ds = ds.select(range(min(num_samples, len(ds))))

    rows = []
    for ex in ds:
        question = f"{ex['question']}\nChoices: {', '.join(ex['choices'])}"
        msgs = [
            {
                "role": "user",
                "content": [{"type": "text", "text": question}, {"type": "image"}],
            }
        ]
        prompt = processor.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
        enc = processor(text=prompt, images=[ex["image"].convert("RGB")], return_tensors="pt")
        rows.append({k: v[0].tolist() for k, v in enc.items()})

    return Dataset.from_list(rows)


def patch_vision_tower_ignore_list(output_dir: str):
    """Fix the vision-tower module paths in config.json's `ignore` list."""
    config_path = f"{output_dir}/config.json"
    with open(config_path) as f:
        config = json.load(f)

    ignore = config["quantization_config"]["ignore"]
    fixed = [
        e.replace("model.vision_tower.", "model.vision_tower.vision_model.", 1)
        if e.startswith("model.vision_tower.")
        and not e.startswith("model.vision_tower.vision_model.")
        else e
        for e in ignore
    ]
    config["quantization_config"]["ignore"] = fixed
    with open(config_path, "w") as f:
        json.dump(config, f, indent=2)

    print(
        f"Patched {sum(a != b for a, b in zip(ignore, fixed))} ignore-list entries "
        f"to match the real vision_tower.vision_model.* module path (in config.json)"
    )


def main():
    print(f"[1/5] ## Loading model from {MODEL_ID}")
    model = Gemma3ForConditionalGeneration.from_pretrained(MODEL_ID, dtype=torch.bfloat16)
    processor = AutoProcessor.from_pretrained(MODEL_ID)

    recipe = [
        QuantizationModifier(
            targets="Linear",
            scheme="NVFP4",
            ignore=[
                "lm_head",
                r"re:model\.vision_tower.*",
                r"re:model\.multi_modal_projector.*",
            ],
        )
    ]

    print(f"[2/5] ## Building calibration dataset (scienceqa, n={NUM_CALIBRATION_SAMPLES})")
    calib_dataset = build_calib_dataset(processor, NUM_CALIBRATION_SAMPLES)

    print(f"[3/5] ## Oneshot NVFP4 quantization (n={NUM_CALIBRATION_SAMPLES})")
    oneshot(
        model=model,
        processor=processor,
        dataset=calib_dataset,
        recipe=recipe,
        batch_size=BATCH_SIZE,
        shuffle_calibration_samples=False,
        max_seq_length=MAX_SEQUENCE_LENGTH,
        num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    )

    print("[4/5] ## Sanity generation check")
    msgs = [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "In one sentence, what is the capital of France?",
                }
            ],
        }
    ]
    prompt = processor.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    inputs = processor(text=prompt, return_tensors="pt").to(model.device)
    out = model.generate(**inputs, max_new_tokens=30, disable_compile=True)
    print("===\n", processor.decode(out[0], skip_special_tokens=True), "\n===")

    print(f"[5/5] ## Saving to {SAVE_DIR}")
    model.save_pretrained(SAVE_DIR, save_compressed=True)
    processor.save_pretrained(SAVE_DIR)
    patch_vision_tower_ignore_list(SAVE_DIR)

    print("Done.")


if __name__ == "__main__":
    main()

Post-processing fix

After export, config.json's quantization_config.ignore list needs one correction: llmcompressor==0.12.0 records vision-tower entries as model.vision_tower.encoder...., but the real module path (and the safetensors tensor names) is vision_tower.vision_model.encoder.... — missing the vision_model segment. This doesn't affect the weights themselves (the vision tower is genuinely plain bf16 either way), but vLLM builds each layer's quant method from this list at model-construction time, before reading any tensor, so a name that doesn't match makes it treat an unquantized bf16 layer as if it were quantized — and it crashes on load with KeyError/AttributeError deep in the SigLIP weight loader. transformers doesn't hit this because it decides per-tensor from what's actually in the checkpoint. Fixed with:

python

import json

path = "gemma-3-4b-it-NVFP4/config.json"
config = json.load(open(path))
ignore = config["quantization_config"]["ignore"]
config["quantization_config"]["ignore"] = [
    e.replace("model.vision_tower.", "model.vision_tower.vision_model.", 1)
    if e.startswith("model.vision_tower.") and not e.startswith("model.vision_tower.vision_model.")
    else e
    for e in ignore
]
json.dump(config, open(path, "w"), indent=2)

Evaluation

Benchmarked with lm-evaluation-harness (lm-eval[api]==0.4.12) against this checkpoint and the unquantized google/gemma-3-4b-it bf16 baseline, both served via vLLM (local-completions backend, --apply_chat_template). Shot counts replicate google/gemma-3-4b-it's own benchmark table for consistency.

Sorted by bf16 score:

Benchmark	Shots	Metric	bf16	NVFP4	Recovery
BoolQ	0	acc	83.73	82.45	98.47%
GSM8K	8	exact_match (flexible)	77.26	69.67	90.18%
PIQA	0	acc_norm	68.66	69.10	100.64%
WinoGrande	5	acc	64.72	63.22	97.68%
ARC-easy	0	acc_norm	63.34	61.20	96.62%
ARC-challenge	25	acc_norm	60.75	57.42	94.52%
MMLU	5	acc	59.58	56.13	94.21%
HellaSwag	10	acc_norm	58.81	61.12	103.93%
TriviaQA	5	exact_match	44.26	39.29	88.77%
DROP	1	f1	13.70	14.28	104.23%
Natural Questions	5	exact_match	10.86	9.78	90.06%

Reading it: simple/short-context tasks (BoolQ, PIQA, ARC-easy) show 96-101% recovery — negligible loss. The real cost shows up on tasks requiring longer reasoning chains — GSM8K (90.2%), TriviaQA (88.8%), MMLU (94.2%), ARC-challenge (94.5%) — where NVFP4 quantization of the language model measurably hurts accuracy. HellaSwag's >100% recovery is plausibly noise (single run, no repeated seeds).

Also verified directly (not part of the harness run):

Text generation: coherent, on-topic completions via both transformers (bf16 dequant) and vLLM (native NVFP4 GEMM via FlashInfer).
Image understanding: correct image descriptions via both serving paths, using COCO sample images.

Evaluation details

The following script was used to evaluate the NVFP4/BF16 model served via vLLM's OpenAI-compatible API.

python

"""Runs the Gemma 3 PT benchmark scheme against a served checkpoint via lm-eval.

Replicates the shot counts from google/gemma-3-4b-it's own README (HellaSwag
10-shot, ARC-e 0-shot, ARC-c 25-shot, MMLU 5-shot, GSM8K 8-shot, etc.) so the
result is comparable across checkpoints. Targets any OpenAI-completions-
compatible endpoint (vLLM, in our case), not a specific checkpoint, so the
same script produces both the NVFP4 and bf16 numbers in this README.

Calls lm-eval's own Python entry point (`lm_eval.simple_evaluate`, the same
function the `lm_eval` CLI calls internally) directly, rather than shelling
out to the CLI, so results come back as a plain dict instead of scraped
terminal output.

Requires a running server, e.g.:
    docker run --gpus=all --rm -d -p 8080:8080 \\
      -v /path/to/this/model:/model \\
      vllm/vllm-openai:v0.26.0-ubuntu2404 /model \\
      --served-model-name syaffers/gemma-3-4b-it-NVFP4 \\
      --max-model-len 8192 --max-num-seqs 2 --gpu-memory-utilization 0.85 \\
      --kv-cache-dtype fp8_e4m3 --host 0.0.0.0 --port 8080

Then run with:
    uv run evaluate.py \\
      --base-url http://localhost:8080/v1/completions \\
      --model syaffers/gemma-3-4b-it-NVFP4 \\
      --tokenizer . \\
      --output ./eval_results

See README.md ("Evaluation") for the results this produced.
"""

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "lm-eval[api]",
#     "transformers",
# ]
# ///

import argparse
import json

from lm_eval import simple_evaluate
from lm_eval.loggers import EvaluationTracker
from lm_eval.utils import handle_non_serializable, make_table

# (tasks, num_fewshot) groups, batched by shot count to minimize server
# restarts.
FEWSHOT_GROUPS = [
    (["boolq", "piqa"], 0),
    (["drop"], 1),
    (["triviaqa", "nq_open", "winogrande", "mmlu"], 5),
    (["gsm8k"], 8),
    (["hellaswag"], 10),
    (["arc_challenge"], 25),
]


def run(tasks: list[str], num_fewshot: int, model_args: str, output: str, limit: int | None) -> dict:
    print(f"=== {','.join(tasks)} (n={num_fewshot}) ===")
    tracker = EvaluationTracker(output_path=output)
    results = simple_evaluate(
        model="local-completions",
        model_args=model_args,
        tasks=tasks,
        num_fewshot=num_fewshot,
        apply_chat_template=True,
        log_samples=True,
        evaluation_tracker=tracker,
        limit=limit,
    )
    if results is None:
        return {}

    # simple_evaluate() only uses evaluation_tracker for config metadata --
    # actually persisting results/samples to --output needs these explicit
    # calls, same as the lm_eval CLI does after its own simple_evaluate() call.
    samples = results.pop("samples")
    tracker.save_results_aggregated(results=results, samples=samples)
    for task_name in results["configs"]:
        tracker.save_results_samples(task_name=task_name, samples=samples[task_name])

    print(make_table(results))
    if "groups" in results:
        print(make_table(results, "groups"))
    return results


def main() -> None:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--base-url", required=True, help="e.g. http://localhost:8080/v1/completions")
    ap.add_argument("--model", required=True, help="served model name")
    ap.add_argument("--tokenizer", required=True, help="local checkpoint dir or HF repo id")
    ap.add_argument("--output", default="./eval_results")
    ap.add_argument(
        "--num-concurrent", type=int, default=1,
        help="Keep this low (1-2) on consumer GPUs -- lm-eval's multiple-choice "
            "tasks request logprobs over the *entire* prompt, which some vLLM "
            "versions don't account for in startup memory profiling. This can "
            "CUDA OOM mid-run even with modest --max-num-seqs. If that happens, "
            "lower this further and/or relaunch the server with a lower "
            "--gpu-memory-utilization (0.85 worked for an 8GB/12GB card here).",
    )
    ap.add_argument("--max-retries", type=int, default=3)
    ap.add_argument(
        "--limit", type=int, default=None,
        help="Cap examples per task (for smoke-testing the script itself, not for "
            "real numbers -- results with --limit set are not comparable to the "
            "README's).",
    )
    args = ap.parse_args()

    model_args = (
        f"model={args.model},base_url={args.base_url},tokenizer={args.tokenizer},"
        f"num_concurrent={args.num_concurrent},max_retries={args.max_retries}"
    )

    all_results = {}
    for tasks, num_fewshot in FEWSHOT_GROUPS:
        all_results.update(run(tasks, num_fewshot, model_args, args.output, args.limit).get("results", {}))

    summary_path = f"{args.output}/summary.json"
    with open(summary_path, "w") as f:
        json.dump(all_results, f, indent=2, default=handle_non_serializable, ensure_ascii=False)
    print(f"Wrote consolidated summary to {summary_path}")
    print("ALL_TASKS_COMPLETE")


if __name__ == "__main__":
    main()

Development Notes

Development notes

A few things worth recording here for anyone extending this work:

NVIDIA TensorRT Model Optimizer (nvidia-modelopt) was tried as an alternative to LLM Compressor. It produced a same-size, same-scope checkpoint, but its export_hf_checkpoint() writes quant_method: "modelopt" into config.json, which plain transformers and vLLM don't recognize — loading it silently skips quantization handling entirely (and then OOMs trying to allocate full-size tensors). That export path is meant for TensorRT-LLM or a modelopt-aware runtime, not the transformers/vLLM stack this checkpoint targets, so we standardized on LLM Compressor's native compressed-tensors output instead.
Quantizing the vision tower too was tried, to shrink further (landed around 3.2GB). The resulting checkpoint then failed to load in plain transformers (AttributeError: 'Linear' object has no attribute 'weight' in SigLIP's _init_weights) and in vLLM (KeyError/AttributeError in the SigLIP weight loader) — a current gap in Gemma3-VLM + compressed-tensors support, not something specific to our recipe or calibration. The vision tower is kept unquantized for this reason.
8GB VRAM (RTX 5050) is tight for calibration, not just serving.
- NVFP4 fake-quantize calibration requires CUDA (asserts amax.is_cuda), but the 8.1GB bf16 model doesn't fit on an 8GB GPU with any headroom for activations. LLM Compressor's SequentialPipeline calibrates one decoder layer at a time, which sidesteps this cleanly — this is the main reason LLM Compressor was easier to work with here than modelopt, which required CPU-offload workarounds via accelerate on this hardware.
- NVFP4 was served on an RTX 5050 (8GB), bf16 on an RTX 3060 (12GB); both needed --max-num-seqs 2 --gpu-memory-utilization 0.85 — lower than normal serving — because lm-eval's multiple-choice tasks request logprobs over the entire prompt, which vLLM's startup memory profiling doesn't account for and which OOM'd at the normal-serving settings. See Evaluation details for the exact script.