File	Format	Size	Bits	Quality	Best For
`medgemma-health-chat-Q4_K_M.gguf`	Q4_K_M	~2.5 GB	4.5	High	Recommended — best quality/size balance
`medgemma-health-chat-Q8_0.gguf`	Q8_0	~4.3 GB	8.0	Excellent	Near-lossless, when memory allows
`medgemma-health-chat-F16.gguf`	F16	~7.8 GB	16.0	Lossless	Full precision, conversion baseline

Quantization Method

These GGUF files were produced using llama.cpp's official conversion and quantization pipeline — the same method used by bartowski and the llama.cpp community.

Process

Base model: The merged 16-bit model (bisonnetworking/medgemma-health-chat-merged) was downloaded from HF Hub
Conversion: python convert_hf_to_gguf.py converted the HuggingFace safetensors to GGUF F16 format (444 tensors, 7.24 GB)
Quantization: llama-quantize applied K-quant methods to produce Q4_K_M and Q8_0 from the F16 baseline

Why K-quants?

K-quant methods (Q4_K_M, Q8_0) use mixed-precision tensor quantization — different tensor types get different bit depths based on their sensitivity:

Attention weights (q_proj, k_proj, o_proj): quantized to Q4_K
Value and FFN down weights (v_proj, ffn_down): quantized to Q6_K (higher precision)
FFN gate/up weights: quantized to Q4_K
Norm weights: kept at F32 (full precision)

This mixed approach preserves more quality than uniform quantization at the same average bit rate. Q4_K_M specifically balances size and quality — it's the community standard for 4-bit GGUF.

No imatrix

These quantizations were produced without an importance matrix (imatrix). For a 4B model at Q4_K_M, the quality difference is minimal. If you need maximum quality at low bitrates (Q2_K/Q3_K), consider generating an imatrix from your calibration data and re-quantizing.

How to Use

With llama.cpp

bash

# Download the Q4_K_M (recommended)
huggingface-cli download bisonnetworking/medgemma-health-chat-gguf \
  medgemma-health-chat-Q4_K_M.gguf \
  --local-dir ./models

# Interactive chat
./llama-cli -m ./models/medgemma-health-chat-Q4_K_M.gguf \
  --interactive \
  -ngl 99 \
  -c 2048

# Single prompt with system message
./llama-cli -m ./models/medgemma-health-chat-Q4_K_M.gguf \
  -sys "You are a board-certified Primary Care Physician. Provide direct, clinical guidance. No tables. No AI disclaimers." \
  -p "I've had a sore throat and low-grade fever for 3 days. What should I do?" \
  -ngl 99 \
  -c 2048

With Ollama

bash

# Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./medgemma-health-chat-Q4_K_M.gguf
TEMPLATE """{{ if .System }}<start_of_turn>user
{{ .System }}<end_of_turn>
{{ end }}<start_of_turn>user
{{ .Prompt }}<end_of_turn>
<start_of_turn>model
"""
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 2048
PARAMETER stop "<end_of_turn>"
EOF

# Create and run
ollama create medgemma-health-chat -f Modelfile
ollama run medgemma-health-chat "I've had a sore throat for 3 days. What should I do?"

With LM Studio

Download the GGUF file to ~/.cache/lm-studio/models/bisonnetworking/medgemma-health-chat-gguf/
Open LM Studio and select the model
Set GPU offload layers to maximum (all layers fit on most modern GPUs)
Set context length to 2048
Start chatting

With llama-cpp-python

python

from llama_cpp import Llama

llm = Llama(
    model_path="./medgemma-health-chat-Q4_K_M.gguf",
    n_ctx=2048,
    n_gpu_layers=99,  # Apple Metal or CUDA offload
    chat_format="gemma3",
)

messages = [
    {
        "role": "system",
        "content": "You are a board-certified Primary Care Physician. Provide direct, clinical guidance. No tables. No AI disclaimers.",
    },
    {
        "role": "user",
        "content": "I've had a sore throat and low-grade fever (100.4) for 3 days. No cough, no swollen lymph nodes. What should I do?",
    },
]

response = llm.create_chat_completion(
    messages=messages,
    max_tokens=256,
    temperature=0.7,
    top_p=0.9,
)
print(response["choices"][0]["message"]["content"])

As an OpenAI-Compatible Server

bash

# Start server
./llama-server -m ./medgemma-health-chat-Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 99 \
  -c 2048

# Use with any OpenAI client
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "medgemma-health-chat",
    "messages": [
      {"role": "system", "content": "You are a Primary Care Physician..."},
      {"role": "user", "content": "I have a sore throat. What should I do?"}
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'

Model Details

Base model: google/medgemma-1.5-4b-it (Gemma 3 4B architecture)
Architecture: Gemma3ForConditionalGeneration
Parameters: 4,365,656,432 (4.4B)
Layers: 34
Embedding dimension: 2560
Attention heads: 8 (4 KV heads, GQA)
Context length: 131,072 (use 2048 for health chat)
Chat template: Gemma3 (start_of_turn/end_of_turn format)

Training

See the merged model card for full training details. Summary:

LoRA fine-tune (r=32, alpha=64) on 49,500 health chat conversations
2 epochs, 12,376 steps, H100 80GB via Modal
Final training loss: 0.7981, eval loss: 0.8779
6 medical personas: primary care, internal medicine, clinical nutritionist, exercise specialist, best doctor, chronic health

Limitations

GGUF quantization introduces minor precision loss vs the F16 original
Q4_K_M may occasionally produce slightly different phrasing than the full-precision model
The Gemma3 chat template must be used correctly — mismatched templates will produce poor output
Not a substitute for professional medical advice
MedGemma base model is gated — users need approved access to the original model

Related Models

LoRA adapter — 250MB, for use with PEFT
Merged 16-bit — 8.6GB, full precision
Training dataset — 49,500 conversations

Model Card Contact

bisonnetworking