We compared BF16 and FP8 weights of Mellum2-12B-A2.5B-Thinking across four evaluation suites covering 134 prompts and 80 functional tests. Across logit-level, distribution-level, perplexity, and generative metrics, the FP8 variant matches the BF16 baseline within statistical noise:
| Metric | Value | Interpretation |
|---|
| Top-1 token agreement | 95.5% | BF16 and FP8 pick the same next token in 64/67 prompts |
| Avg KL(BF16 ‖ FP8) | 0.037 | Very small divergence |
| Avg Jensen-Shannon divergence | 0.0058 | Effectively identical distributions |
| Avg relative perplexity change | +0.14% | Inside sampling noise |
| Greedy generation byte-identical | 81.8% | Of 33 prompts |
| Token-level overlap | 98.7% | Of the generated continuations |
| Pass@1 (15 code tasks, 80 tests) | 78/78 | Both models, no disagreements |
Conclusion. FP8 weight quantization of this checkpoint is a near-lossless compression. The model can be served in FP8 without measurable quality regression for typical code-completion workloads.
1. Background
Modern LLM serving increasingly relies on lower-precision inference (FP8, INT8, INT4) to reduce memory footprint and improve throughput on Hopper / Blackwell GPUs. The risk is that aggressive quantization can shift the output distribution, biasing generation or degrading accuracy on structured tasks.
This report quantifies that risk for Mellum2-12B-A2.5B-Thinking, a 12B code-completion model with an extended thinking mode. We measured three orthogonal classes of effect:
- Probability-distribution shift — do the next-token logits change?
- Sequence-likelihood shift — does perplexity on natural code change?
- End-to-end behavior — do greedy completions diverge, and do functional tests still pass?
Two identical model copies are exposed through an OpenAI-compatible inference server: one in BF16 and one in FP8 weights. All requests below use temperature=0 so that any divergence is attributable to quantization, not sampling.
2. Methodology
2.1 Endpoint contract
Both checkpoints are served via the standard v1/completions interface. All requests used:
temperature=0logprobs=20 (top-20 token log-probabilities)max_tokens adapted per test (see below)- For greedy generation tests: explicit
stop sequences to keep continuations on a single function body
2.2 Test suites
| # | Suite | Prompts / tasks | What it measures |
|---|
| 1 | Next-token distribution | 67 Python code-completion prompts | KL / JSD divergence on the top-20 token distribution |
| 2 | Long-document perplexity | 4 long Python modules (≈500–700 tokens each) | Mean negative log-likelihood under each model |
| 3 | Pass@1 on code tasks | 15 short coding problems (80 unit tests) | Functional correctness via AST extraction + exec |
| 4 | Greedy generation comparison | 33 Python continuation prompts | Byte- and token-level similarity of greedy outputs |
2.3 Metrics
Suite 1 — Distribution shift. For each prompt we extract the top-20 next-token distribution from each model:
top1_agreement — whether both models place the same token in rank 1KL(P_BF16 ‖ P_FP8) — forward KL divergence on the union of top-20 tokens, padded with eps = 1e-10 for missing entriesJSD(P_BF16, P_FP8) — symmetric Jensen-Shannon divergencemean |Δlogprob| — average absolute log-probability difference across the union of tokens
Suite 2 — Perplexity. Long Python snippets are sent with echo=true, logprobs=1. We average the non-null token log-probabilities and compute PPL = exp(-mean_logp).
Suite 3 — Pass@1. Each task provides a minimal prompt (signature + 1-line docstring) and a list of (call, expected) test cases. The model is asked to complete the function. We extract the first def block via Python's ast module (with textwrap.dedent fallback for malformed indentation) and execute the resulting code in a fresh namespace, running every test.
Suite 4 — Generation equivalence. Identical prompts are sent to both endpoints with temperature=0. Outputs are compared byte-for-byte and token-for-token (str.split() whitespace tokenizer).
2.4 Reproducibility
- Randomness: none — all decodes are greedy.
- Stopping criteria: explicit stop strings in Suite 3 and 4 (no top-level
def, no class, no import, no print(, no assert ).
- Padding: when KL is computed, tokens absent from one model's top-20 are treated as having probability
1e-10.
- All artifacts (raw responses, extracted function sources, summary files) are persisted alongside the report.
3. Results
3.1 Distribution-level divergence
| Value |
|---|
| Prompts tested | 67 |
| Top-1 agreement | 95.5% |
| Avg KL(P_BF16 ‖ P_FP8) | 0.037 |
| Avg Jensen-Shannon divergence | 0.0058 |
| Avg |Δlogprob| on top-1 | 0.064 nats |
| Avg P(top1_BF16) under BF16 | 0.816 |
| Avg P(top1_BF16) under FP8 | 0.809 |
| Prob-mass shift on top-1 | 0.0064 |
For reference, a JSD of 0.0058 corresponds to an effective per-token information loss of ~6 millinats — orders of magnitude below the entropy of typical next-token distributions (~1 nat). The 3 prompts where the top-1 token disagrees (4.5% of cases) involve tokens that were already very close in probability under the BF16 model.
3.2 Perplexity on long Python code
| Document | Tokens | BF16 PPL | FP8 PPL | Δ |
|---|
| Doc 1 (DataProcessor module) | 419 | 1.5029 | 1.5153 | +0.83% |
| Doc 2 (async scraper) | 317 | 1.3920 | 1.3919 | −0.00% |
| Doc 3 (sorting algorithms) | 682 | 1.0677 | 1.0634 | −0.40% |
| Doc 4 (hashmap + linked list) | 618 | 1.1299 | 1.1313 | +0.13% |
| Mean | — | 1.2731 | 1.2755 | +0.137% |
One document shows +0.83% degradation, two show essentially zero change, and one actually improves marginally. Averaged across 2,036 tokens of real code, FP8 is +0.14% in perplexity — well within typical run-to-run noise.
3.3 Pass@1 on coding tasks
15 short, self-contained coding problems with 80 unit tests total (greedy decoding, minimal prompt = signature + 1-line docstring):
| BF16 | FP8 |
|---|
| Tests passed | 78 / 80 | 78 / 80 |
| Pass rate | 97.5% | 97.5% |
| Tasks fully solved | 14 / 15 | 14 / 15 |
| Tasks with disagreement | — | 0 / 15 |
Both models fail the same 2 unit tests on the same task (is_palindrome), and solve the remaining 14 tasks identically.
3.4 Greedy-generation comparison
| Metric | Value |
|---|
| Prompts | 33 |
| Byte-identical generations | 27 / 33 (81.8%) |
| Avg character-level similarity (SequenceMatcher) | 0.979 |
| Avg token-level overlap (whitespace tokenizer) | 0.987 |
| Avg length difference | 2.33% |
Of the 6 non-identical generations:
- 4 differ only in whitespace (e.g. one extra space or newline). These are tokens that are indistinguishable in their effect on downstream parsing.
- 1 (
gen_09) is a 44-char vs 29-char continuation — the FP8 model stopped one statement earlier, after the same token sequence on the prefix.
- 1 (
gen_16) is a 103-char vs 159-char continuation with 57% token overlap — the only prompt where FP8 and BF16 take genuinely different code paths.
Manual inspection of gen_16 (an async context-manager completion) shows both continuations are syntactically valid Python; the BF16 path explores a more verbose error-handling branch, the FP8 path a leaner one. Functionally both are reasonable, and neither is clearly "wrong" without the surrounding program.
4. Discussion
What does FP8 preserve?
Across all four suites, FP8 quantization preserves:
- The argmax of the next-token distribution in 95.5% of cases — sufficient to keep greedy decoding byte-identical 81.8% of the time and token-identical 98.7% of the time.
- Mean negative log-likelihood within +0.14% on multi-hundred-token code documents.
- Functional correctness on standard code-completion tasks (97.5% pass@1, no FP8-only failures).
Where does divergence show up?
The only measurable signals are at the tail of the distribution:
- Tiny log-probability shifts (≈0.06 nats on the top-1 token).
- ≈4.5% of prompts where the top-1 token disagrees; in all observed cases these are tokens that BF16 itself assigned close probabilities to (within ~1 nat of the runner-up).
- One out of 33 prompts where the generation path visibly diverges after a few dozen tokens.
Operational implications
For serving the model in FP8:
- Quality — negligible degradation on standard code-completion workloads.
- Memory — ~2× weight-memory reduction vs BF16 (FP8 weights ≈ half the bytes of BF16 weights; KV cache still BF16 in this setup).
- Throughput — depends on GPU architecture and runtime, but Hopper / Blackwell gain meaningful speedups from native FP8 matmul.
- Risk surface — tasks with very low-entropy decoding paths (highly constrained completions, single-token outputs after a long deterministic prefix) are the most likely to surface sub-token-level differences; in our suite these did not affect functional outcomes.
5. Limitations
- Single language. All prompts are Python; the conclusions may differ for other languages, especially low-resource ones where token distributions are sharper.
- Single decoding mode. We tested greedy only. Sampling-based evaluation (e.g., temperature ≥ 0.3) would amplify any tail differences.
- Functional tests are small. 80 unit tests across 15 tasks is enough to detect large regressions but cannot rule out subtle bugs.
- No human evaluation. All "correctness" judgments are mechanical (string / list equality on deterministic inputs).
- One model, one quant recipe. Generalization to other checkpoints (different size, different quant scheme) is not addressed.
