FP8 format
- 8‑bit floating‑point (1 sign bit + 5 exponent bits + 2 mantissa bits).
- Drastically shrinks weight/activation size while keeping floating‑point behavior.
Dynamic scheme (FP8_DYNAMIC)
- Weights: static, per‑channel quantization (each out‑feature channel has its own scale).
- Activations: dynamic, per‑token quantization (scales are recomputed on‑the‑fly for every input token).
RTN (Round‑To‑Nearest) PTQ
- Post‑training; no back‑prop required.
- No calibration dataset needed because:
  - Weights use symmetric RTN.
  - Activations are quantized dynamically at inference time.

2. Serving the FP8 Model with vLLM

vllm serve BCCard/gemma-3-27b-it-FP8-Dynamic \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9  \
  --max-model-len 8192 \  
  --enforce-eager \  
  --api-key bccard \
  --served-model-name gemma-3-27b-it

3. Quantization Code Walk‑Through (Shared Knowledges)

LLM Compressor is an easy-to-use library for optimizing models for deployment with vllm, including:

Comprehensive set of quantization algorithms for weight-only and activation quantization Seamless integration with Hugging Face models and repositories safetensors-based file format compatible with vllm Large model support via accelerate

from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.transformers import oneshot

model_name = "google/gemma-3-27b-it"

processor = AutoProcessor.from_pretrained(model_name)
model = Gemma3ForConditionalGeneration.from_pretrained(model_name, device_map="auto", torch_dtype="auto", trust_remote_code=True)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=['re:.*lm_head', 're:vision_tower.*', 're:multi_modal_projector.*'],
)

SAVE_DIR = "gemma-3-27b-it-FP8-Dynamic"
oneshot(model=model, recipe=recipe, output_dir=SAVE_DIR)
processor.save_pretrained(SAVE_DIR)

4. Gemma 3 model card

Model Page: Gemma

Terms of Use: [Terms][terms]

Authors: Google DeepMind, BC Card (Quatization)

Description

Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. Gemma 3 models are multimodal, handling text and image input and generating text output, with open weights for both pre-trained variants and instruction-tuned variants. Gemma 3 has a large, 128K context window, multilingual support in over 140 languages, and is available in more sizes than previous versions. Gemma 3 models are well-suited for a variety of text generation and image understanding tasks, including question answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as laptops, desktops or your own cloud infrastructure, democratizing access to state of the art AI models and helping foster innovation for everyone.

Inputs and outputs

Input:
- Text string, such as a question, a prompt, or a document to be summarized
- Images, normalized to 896 x 896 resolution and encoded to 256 tokens each
- Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and 32K tokens for the 1B size
Output:
- Generated text in response to the input, such as an answer to a question, analysis of image content, or a summary of a document
- Total output context of 8192 tokens

Citation

none

@article{gemma_2025,
    title={Gemma 3 FP8 Dynamic},
    url={https://bccard.ai},    
    author={BC Card},
    year={2025}
}