Hardware/runtime: NVIDIA L4, CUDA, dense BF16 evaluation after applying the quantized weights.
This supports a narrow claim: BF16-baseline-equivalent perplexity on this exact Gemma4/WikiText/Colab runner at about 4.00 BPW. It is not an FP16 result, not an FP8 comparison, and not a throughput result. The current evaluator reconstructs/applies weights into a normal dense model for correctness.
1pip install "transformers>=5.5.0" torch accelerate safetensors huggingface_hub
2
3python -c "from huggingface_hub import snapshot_download; snapshot_download('google/gemma-4-E2B', local_dir='./models/gemma-4-E2B')"
4
5python scripts/limited_ppl_bench.py \
6 --label mixed_budget_full_g128_target4p0 \
7 --model-dir models/gemma-4-E2B \
8 --wikitext data/wiki.test.txt \
9 --quantized-pt quantized/gemma_mixed_budget_full_g128_target4p0.pt \
10 --tokens 1000000000 \
11 --max-length 512 \
12 --stride 512 \
13 --device cuda \
14 --output eval_results/mixed_budget_full_g128_target4p0_ppl_live.json
The quantization code and metadata in this repository are Apache-2.0. The base model remains governed by Google's Gemma license.
