INT4 weight-only GPTQ quantization of google/medgemma-27b-text-it, produced with llm-compressor using 512 medical-domain calibration samples (PubMedQA abstracts + Q&A, 2048-token sequences) — not generic web text — so weight rounding is optimized for clinical language.

Quantized and benchmarked on an NVIDIA DGX Spark (GB10, 128 GB unified memory).

Before / after

Metric	BF16 baseline	INT4 GPTQ	Change
Disk size	54.1 GB	16.1 GB	-70.3%
Wikitext word PPL	11.416	11.696	+2.5%
Throughput, batch 32 (tok/s)	111	296	+165.8%
Single stream (tok/s)	4.3	12.9	+200.0%

Medical benchmark accuracy (lm-evaluation-harness)

Task	BF16	INT4 GPTQ	Change
PubMedQA	0.7460	0.7420	-0.5%
MMLU Clinical Knowledge	0.8377	0.8189	-2.3%
MMLU Professional Medicine	0.8493	0.8566	+0.9%
MMLU College Medicine	0.7803	0.7688	-1.5%
MMLU Medical Genetics	0.8800	0.8800	+0.0%
MMLU Anatomy	0.7556	0.7111	-5.9%
Average (medical)	0.8082	0.7962	-1.5%

Throughput measured with vLLM on the GB10 (memory-bandwidth-bound hardware — INT4 speedup is larger on bandwidth-rich datacenter GPUs).

Where this is useful

Single-GPU serving: ~16 GB of weights fits a 24 GB GPU (L4, A10G, RTX 4090) with room for KV cache — the BF16 original (54 GB) needs 80 GB-class hardware.
Snowpark Container Services / vLLM: loads directly in vLLM (compressed-tensors format), suitable as the hard-text-reasoning tier behind clinical document pipelines.
Higher throughput per dollar for medical summarization, QA, and structured extraction where the accuracy table above shows near-baseline quality.

Usage (vLLM)

python

from vllm import LLM, SamplingParams

llm = LLM(model="sarathi-balakrishnan/medgemma-27b-text-it-GPTQ-INT4", max_model_len=8192)
out = llm.generate(["Summarize the key contraindications of metformin."],
                   SamplingParams(temperature=0.2, max_tokens=512))
print(out[0].outputs[0].text)

License and intended use

This is a quantized derivative of google/medgemma-27b-text-it and is governed by the Health AI Developer Foundations Terms of Use. MedGemma is intended for developer research and development of healthcare applications. It is not a clinical-grade product; outputs require independent clinical validation before any use in patient care.