First Quantization-Aware Training (QAT) for Gemma 4 31B — achieving quantization quality on par with 16-bit (BF16) inference.
🎯 Project Goal
Produce a 4-bit quantized Gemma 4 31B that is indistinguishable from the original BF16 model in output quality.
🏆 Hackathon Impact
Novelty: No QAT models exist for Gemma 4 (Google only released QAT for Gemma 3)
Impact: Enables running Gemma 4 31B on consumer hardware (~20GB VRAM vs 62GB)
Technical depth: Combines cutting-edge QAT research with practical deployment
🔬 Technical Approach
Method: QAT + LoRA via Unsloth + TorchAO
Load Gemma-4-31B-it in 4-bit (base model)
Apply QAT with LoRA adapters (qat_scheme="int4")
Fine-tune on high-quality instruction data to adapt weights to quantization noise
Export to TorchAO Int4WeightOnlyConfig for inference
Convert to GGUF for broad llama.cpp deployment
Why QAT Beats PTQ
Post-training quantization (PTQ) simply rounds weights to lower precision, causing accuracy loss. QAT simulates quantization during training, allowing the model to learn how to compensate for quantization noise. Results from Unsloth on Gemma 3:
Gemma 3 4B: Recovered 66.9% of lost accuracy, +1.0% raw improvement
Gemma 3 12B: Recovered 45.5% of lost accuracy, +2.1% raw improvement
For Gemma 4 31B, we expect even better recovery due to larger model capacity (scaling law for QAT shows larger models → better quantization recovery).
📁 Repository Structure
File
Purpose
train_gemma4_qat_32gb.py
Main training script (optimized for RTX 5090 / 32GB VRAM)
gemma4_qat_train.py
Full-featured training with Trackio monitoring
evaluate_gemma4_qat.py
KL divergence evaluation vs BF16 baseline
benchmark_qat.py
Comprehensive benchmark suite (KL, PPL, speed, generation)
convert_to_gguf.py
Convert trained model to GGUF format
gemma4_advanced_ptq.py
Advanced PTQ with imatrix calibration (no training required)
