This study investigates whether extreme quantization down to 1-bit precision is viable for the Gemma-4 architecture. Using google/gemma-4-E2B-it (5.12B parameters), we ran a full quantization sweep from FP16 baseline down to 1-bit BitNet-style ternary weights, combined with a layer-by-layer cosine similarity sensitivity analysis across all 566 linear layers.
Bottom line: 1-bit and INT4 quantization are not feasible without dedicated BitNet-native training. However, INT8 quantization outperforms FP16 by 31.7% on perplexity, and — unlike prior Qwen results — 26.1% of Gemma-4 layers tolerate 1-bit quantization, opening a potential hybrid quantization path.
Note: Perplexity values are higher than typical base-model results because gemma-4-E2B-it is instruction-tuned. Instruction-tuned models are optimized for conversation, not raw next-token prediction on Wikipedia text. The relative comparisons between precision levels are the meaningful metric.
What We Tested
Precision
Bits/Weight
Method
FP16
16
Standard half-precision (baseline)
INT8
8
Symmetric per-tensor linear quantization
INT4
4
Symmetric per-tensor 4-bit quantization
1-bit (W1.58A8)
~1.58
BitNet ternary {−1, 0, +1} scaled by mean absolute value
Results
Benchmark Summary
Quantization
Perplexity ↓
vs FP16
Inference (ms)
Status
FP16 (baseline)
127,575
—
67.6ms
✓
INT8
87,205
+31.7% better
67.2ms
✓ recommended
INT4
3.59 × 10¹⁵
28 billion× worse
68.9ms
✗ catastrophic
1-bit W1.58A8
6.53 × 10¹⁰
512,000× worse
70.7ms
✗ catastrophic
Perplexity Comparison (Log Scale)
Perplexity Comparison
Memory & Inference Speed
Inference Speed and Memory
Layer Sensitivity Analysis
Every linear layer was analyzed by computing cosine similarity between original FP16 weights and 1-bit quantized weights. Layers with cosine similarity ≥ 0.90 are classified as "tolerant" (safe for 1-bit); below 0.90 as "sensitive."
Sensitivity Heatmap
Layer Sensitivity Heatmap
Results
Metric
Value
Total layers analyzed
566
Sensitive (cosine sim < 0.90)
418 (73.9%)
Tolerant (cosine sim ≥ 0.90)
148 (26.1%) — hybrid path viable
Cosine similarity range
0.661 – 0.967
Mean cosine similarity
~0.848
Threshold
0.90
Key Contrast vs Prior Studies
Unlike the Qwen3.5-2B study (0/187 tolerant layers), Gemma-4 has 148 tolerant layers (26.1%). This is a significant architectural difference — Gemma-4's weight distributions in certain layers are compact enough to survive ternary projection. A hybrid quantization strategy (tolerant → 1-bit, sensitive → INT8) is architecturally feasible for Gemma-4, though it requires dedicated hardware kernels (BitNet) to realize actual memory savings.
Key Findings
1. INT8 Beats FP16 by 31.7%
INT8 perplexity of 87,205 vs 127,575 for FP16 — a 31.7% improvement. This is consistent with uniform quantization noise acting as mild L2 regularization. INT8 is the recommended deployment precision for Gemma-4.
2. INT4 and 1-bit Both Fail Catastrophically
INT4: 3.59 × 10¹⁵ perplexity — 28 billion times worse than FP16
1-bit: 6.53 × 10¹⁰ perplexity — 512,000 times worse than FP16
Both produce effectively random output. Simulated quantization applied post-training cannot preserve the weight distributions. BitNet-native training from scratch is required.
3. Gemma-4 Has a Viable Hybrid Path (26.1% Tolerant Layers)
This is the first documented evidence that 26.1% of Gemma-4 linear layers survive 1-bit quantization with cosine similarity ≥ 0.90. The tolerant layers are distributed across both attention and MLP projections, suggesting Gemma-4's architecture may be inherently more quantization-friendly than comparable models.
4. Inference Speed Unaffected in Simulation
All four configurations ran at ~67–71ms per sample. Real-world deployment with hardware-native INT8 kernels (e.g., bitsandbytes, GPTQ) would show 1.5–2× speedup and true 2× memory reduction.
