Quality: Q8 retains 92.9% token agreement with FP16 — nearly identical output
VRAM: Saves 2.1 GB vs FP16 (1.3x compression)
Speed: 14.5 tok/s — slower than Q4 due to bfloat16→float16 cast overhead in bitsandbytes on Blackwell GPUs
Best for: Cases where quality is priority over speed

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

model = AutoModelForCausalLM.from_pretrained(
    "MichaelLowrance/gemma-4-e2b-q8",
    quantization_config=BitsAndBytesConfig(load_in_8bit=True),
    device_map="cuda",
)
tokenizer = AutoTokenizer.from_pretrained("MichaelLowrance/gemma-4-e2b-q8")

messages = [{"role": "user", "content": "Hello!"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda")

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=200, do_sample=False)

print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Hardware

Tested on: NVIDIA RTX 5090 (Blackwell, sm_120, 32GB GDDR7)
CUDA: 12.8 | Python: 3.12 | transformers: 5.6.0.dev

dahus/gemma-4-e2b-it-q8

Gemma 4 E2B it — INT8 Quantization (bitsandbytes)

Benchmark Results

Results by Category

Key Findings

Usage

Hardware

dahus/
gemma-4-e2b-it-q8