Gemma-4-26B-A4B FP8 — Quality Benchmark vs BF16

TL;DR

• Overall accuracy: BF16 81.5% vs FP8 81.5% (Δ = 0.00 pp) — parity at n=270
• Win/Tie/Loss (across 270 task pairs): BF16-only wins = 7, FP8-only wins = 7, both pass = 213, both fail = 34, evaluator skipped = 9 (IFEval rules not implemented) — the win/loss ratio is perfectly symmetric
• Largest degradation: MATH -6.7 pp and GSM8K -6.7 pp — both driven by reasoning-loop timeouts, not arithmetic errors
• Wins for FP8: MMLU-thinking +6.7 pp and MMLU no-think +3.3 pp — FP8 actually scores higher on knowledge MCQ (within sampling noise at n=30, but consistently across both modes)
• Worst case: on the trivial MATH problem "sum 3/10 + 3/1000 to the nearest thousandth" FP8 enters an infinite repetition loop and never produces \boxed{0.303} within 8 192 tokens; BF16 solves it in 1 416 tokens.
• Reasoning efficiency: FP8 spends on average +8% more tokens than BF16 (1 195 vs 1 105). On logical-reasoning curated prompts FP8 is actually -46% shorter than BF16.
• Max-token cap reached: FP8 hit finish_reason=length on 20 / 270 requests (vs 19 for BF16 — essentially identical).

Methodology

Generation parameters

• temperature = 0 (greedy / deterministic mode)
• seed = 42 (fixed for reproducibility)
• top_p = 1.0
• enable_thinking is toggled via chat_template_kwargs (supported by sglang for Gemma-3/4 chat templates).

Categories and thinking mode

For MCQ benchmarks (MMLU, ARC) both variants were run: *_thinking (thinking=on) and the base (thinking=off), to cover both the "fast" and "reasoning" modes.

Datasets

Evaluators

Results by category

Token usage (reasoning + completion)

Token takeaway: FP8 is roughly as token-efficient as BF16 overall (+8%). The only categories with a noticeable overhead are GSM8K (+27%, mostly from the same loop cases that hit max_tokens) and curated translation (+40%, on 4 samples — too small to be meaningful).

Worst-case reasoning loops

• BF16 solves it in 1 416 tokens, finishes with \boxed{0.303}.
• FP8 runs for 8 192 tokens (hits finish_reason=length), repeating the same four-line block indefinitely:

  *  The sum is 0.303.
  *  The decimal form is 0.303.
  *  The thousandths place is the third decimal place.
  *  0.303 is already at the thousandths place.

  ... (repeats until the token budget is exhausted; never emits \boxed{...})

• BF16 finishes in 854 tokens with the correct answer.
• FP8 runs for 8 192 tokens and never produces Answer: N.

Conclusions

1. Overall accuracy: zero measurable gap between FP8 and BF16 at n=270 (both 81.5%). The win/loss distribution is perfectly symmetric (7 BF16-only wins, 7 FP8-only wins).
2. Standard benchmarks with deep CoT (MMLU-thinking, GSM8K, MATH): the only measurable gaps are -6.7 pp on GSM8K and -6.7 pp on MATH, both dominated by reasoning-loop timeouts, not arithmetic errors. On MMLU FP8 slightly outperforms BF16 (+3.3 pp no-think, +6.7 pp thinking) — within sampling noise at n=30 but consistent across both modes.
3. Standard benchmarks without deep CoT (ARC, HumanEval, IFEval): identical quality (0.00 pp on every category).
4. Reasoning stability: FP8 hits the max-token cap on 20/270 requests, vs 19/270 for BF16 — essentially identical. The remaining FP8 timeouts are concentrated on the same hard MATH problems where BF16 also often runs out of budget.
5. Simple tasks (knowledge QA, translation, multilingual reasoning, code-explain, safety, creative): identical quality. FP8 quantization does not damage the model's "fast" modes.
6. Instruction following (IFEval): no measurable difference at n=30, but the implemented rule subset is limited and does not include the hardest categories (combination:*, detectable_format:* beyond bullets). A larger sample would be needed to draw a confident conclusion here.
7. Token efficiency: FP8 produces +8% more tokens on average (1 195 vs 1 105) — close to BF16. On curated logical-reasoning prompts FP8 is actually 46% more efficient than BF16.