A language model does not give you an answer. It gives you a probability
distribution over answers, and generating text draws one sample from it. Ask
the same question twice and you can get two different results. Most people
treat that as a defect.
We treat it as an untapped resource. Sample the same frozen model N times and
select well, and accuracy climbs sharply — no training, no fine-tuning, no
new weights. The knowledge was always in there. It just needed asking more
than once.
single sample
Best-of-128
GSM8K, Qwen2.5-0.5B frozen
45.3%
66.5%
+21.2 points. Nothing was trained.
Why voting works so well
Here is the asymmetry that makes the whole thing run, and it is more elegant
than it first looks:
Correct answers agree with each other. Wrong answers scatter.
A wrong trajectory has a thousand different ways to be wrong and picks a
different one each time, so errors split into singletons. The correct answer is
the only thing several attempts can converge on together.
That is why counting votes recovers answers only a small minority reached. The
model does not need to be right most of the time — it needs to be right more
consistently than it is wrong in any one particular way.
Quickstart
python
1from bestofn import BestOfN
23engine = BestOfN("Qwen/Qwen2.5-0.5B-Instruct", n=16)4r = engine.solve("A train travels at 60 km/h for 3 hours. How far does it go?")56r.answer # '180'7r.agreement # 0.81 how strongly the trajectories agreed8r.effective_n # 15 how many produced a usable answer
Both backends are tested on real hardware: transformers runs anywhere torch
runs, and vllm is dramatically faster at large N — it is the one the
measurements below were generated with.
Measured results
Qwen2.5-0.5B-Instruct on GSM8K, 200 problems, 128 trajectories each,
weights frozen. Every figure is recomputed from the published trajectories by
scripts/analyse.py, which re-runs extraction over the raw reasoning text.
GSM8K accuracy against N
N
random
majority
95% CI
coverage
1
45.3%
45.3%
[44.7, 45.9]
45.3%
2
46.4%
46.5%
[37.0, 50.5]
56.9%
4
46.1%
53.0%
[45.0, 58.5]
66.4%
8
46.2%
58.3%
[52.0, 65.5]
75.0%
16
46.1%
61.8%
[55.0, 68.0]
81.7%
32
46.5%
64.0%
[56.0, 69.0]
86.8%
64
46.1%
65.5%
[58.5, 72.0]
90.6%
128
46.3%
66.5%
[60.5, 73.0]
93.5%
45.3% to 66.5% on a half-billion-parameter model, with the weights
frozen throughout. Against random selection at the same N, exact McNemar gives
p ≤ 6.6 × 10⁻⁵ at every one of 200 random seeds, with a median of 8.2 × 10⁻¹⁰.
We quote the worst seed rather than the best, and we say what it is: the worst
of the 200 we enumerated, not a bound. random draws differently on every run,
so its p-value has a distribution and a wider sweep will find a worse seed. The
conclusion is what does not move — significant at all 200.
And coverage reaches 93.5%: on more than nine problems out of ten, this
small model does find the right answer somewhere in its 128 attempts. That is
the number that says how much is still on the table.
How much of the gain is really selection
Most reports skip this, and it is the part that decides whether a headline
means anything:
N=1, a single sample
45.3%
N=128, random among the trajectories that answered
46.3%
+1.0
N=128, majority vote
66.5%
+20.2
+21.2 total
Random selection improves slightly with N without selecting anything, because
with more trajectories one of them usually did not abstain. Separating the two
shows that 20.2 of the 21.2 points — 95% of the gain — is genuine
selection, not an artefact of comparing a one-sample baseline against an
N-sample system.
We report it this way because that comparison quietly folds the first number
into the second, and the size of the fold is not knowable in advance. Here it
is small. It is small because the token budget lets trajectories finish; at a
tighter budget the same experiment would have credited five times as much of
the gain to the method.
The selectors we can measure here, against the baseline
selector at N=128
accuracy
95% CI
random (exact expectation)
46.3%
[38.0, 52.0]
majority
66.5%
[60.5, 73.0]
self_certainty
66.5%
[60.5, 73.0]
oracle (diagnostic ceiling)
93.5%
[90.0, 96.5]
verifier and verifier_argmax are absent because the published trajectories
carry no reward-model scores — this release ships no reward model, so there was
nothing to score them with. Plug one in and the same table prints them.
The accounting
Trajectories generated
25,600
Cast a vote
24,797 — 96.9%
Abstained
803 — 3.1%
Truncated at the token limit
216 — 0.8% (215 of them abstained)
Tokens generated
8,434,157
Re-extraction drift on replay
0
All 25,600 trajectories are published in full.
It tells you what to do next
Beyond raising accuracy, Best-of-N measures the two halves of the problem
separately and tells you which one you are actually facing. The framing is
not ours — pass@k beside maj@n appears in the evaluation harnesses and in
Snell et al. (2024). What is ours is that it takes one call, and that the
answer arrives as a decision rather than as two numbers to interpret:
python
1r = engine.solve(problem)23r.is_correct(gold)# did the system return the right answer?4r.covered(gold)# did any trajectory find it at all?
returned
reachable
What it means
What to do
✓
✓
Working
You are done. Consider whether you need this much N
✗
✓
The answer is in there
A selection problem — a verifier recovers it
✗
✗
Not yet reachable
Raise N, improve the prompt, or use a stronger model
Two numbers, one decision. Without them you are guessing whether to spend your
next hour on sampling or on selection.
The library also reports the accounting most tooling hides:
python
1r.effective_n # trajectories that actually voted2r.n_abstained # produced no usable answer3r.n_truncated # ran out of tokens4r.total_tokens # what it cost
The same pool always gives the same answer
Shuffle the trajectory pool and every selector returns the same answer.
random is the exception, and only because picking at random is what it is
for.
That sounds like it should be free. It is not, and it took three separate
things to make true:
The equivalence classes cannot be built greedily. Symbolic equivalence is
not transitive — a parser accepts 0.3333333333 against 1/3, and 1/3
against 0.33333333333333, while rejecting the two decimals against each
other. Compare each new answer against one representative per class and the
partition you get depends on which answer the model emitted first. We take the
transitive closure over all pairs instead.
Weights cannot be accumulated in arrival order. Float addition is not
associative, so two permutations of the same weights can differ in the last
bit — enough to flip a near-tie in a verifier-weighted vote. Totals are summed
with math.fsum over a sorted list, which is exactly rounded.
Ties cannot break on whichever came first. Every tie, in every selector,
resolves on the canonical key.
We check it in the test suite, not by hand. tests/test_selectors.py fuzzes
all five deterministic selectors over pools built to hit the cases that break
invariance — exact ties, duplicate scores, weights spread across many orders of
magnitude, and pools past the merge cap — and separately asserts that the
weight tally itself is unchanged by reordering. Each of the four historical
bugs has been confirmed to turn that suite red when reintroduced.
It is visible in the published output too: at N=128 the resampled curve draws
128 trajectories from a pool of 128, so every draw is a permutation of one pool
and the reported spread is 0.00. That is a necessary condition rather than
a proof, which is why the tests carry the weight.
The habit worth building: always print the baseline
python
1r = engine.solve(problem, n=16)23r.select_with("random", seed=0)# the honest baseline4r.select_with("majority")# how much better is it, really?
Re-running a selector over an existing result is free — generation is the
expensive part — so there is no reason not to. Our own published tables carry
the random row, because a gain is only a gain relative to something.
Method
Needs
What it is for
random
nothing
The baseline. Print it every time
majority
nothing
The default, and it is strong
self_certainty
logprobs=True
Weighs votes by the model's own confidence
verifier
a verifier callable
The one that can promote a minority answer
verifier_argmax
a verifier callable
Single best trajectory, no vote
oracle
the gold answer
Measures your headroom during development
Bring your own reward model
Best-of-N works with any published reward model, and it makes them work
correctly:
Reward models emit unbounded logits, not probabilities, and a naive
implementation silently degrades to a plain majority vote when you hand it one
— leaving you convinced your verifier is running when it is not. This one
catches it, tells you exactly what to do, and the adapters apply the sigmoid
for you.
Licences differ and they govern how you may use the scores. from_hub flags
anything non-permissive, and verifiers.license_of(model_id) checks the Hub
live. The full table is in USAGE.md.
Everything here is checkable
bash
1pip install"bestofn[math]"2python scripts/analyse.py # no GPU needed
The [math] extra pulls in math-verify, which is what makes 1/2, 0.5 and
\frac{1}{2} count as one answer. Without it the script still runs and still
reports, but equivalent answers written differently vote separately and the
numbers come out slightly lower. Regenerating the figures additionally needs
matplotlib; the analysis itself does not.
The published dataset contains the complete reasoning text of every
trajectory, with finish_reason, log-probabilities and token counts — not
answers extracted earlier by someone else. The analysis re-runs extraction over
that raw text, computes exact McNemar against the random baseline, and reports
bootstrap confidence intervals.
That means the numbers above are not asserted, they are reproduced — by
you, in about two minutes on a laptop CPU, with no hardware. Very little
published work in this area can say that.
Tasks with one comparable final answer — mathematics, multiple choice,
short factual questions, unit-testable code.
Models that are sometimes right. The gain is largest when per-sample
accuracy sits around 20–60%, which is exactly where small models live on
hard problems.
Hardware you already own. On your own GPU the extra samples cost you
nothing but time you were not using. This is the one setting where the
economics are simply free.
Practical guidance on choosing N, plugging in a verifier and measuring your own
task is in USAGE.md.
Built on solid ground
Inference-time compute is one of the most active areas in the field, and this
sits squarely inside it:
Cobbe et al., Training Verifiers to Solve Math Word Problems, 2021
Wang et al., Self-Consistency Improves Chain of Thought Reasoning, 2022
Lightman et al., Let's Verify Step by Step, 2023
Snell et al., Scaling LLM Test-Time Compute Optimally, 2024
Brown et al., Large Language Monkeys, 2024
What this adds: the selection layer the serving stacks deliberately leave
out — vLLM removed best_of in 2025, SGLang discourages n>1, LMDeploy
supports only 1 — a single small API over six interchangeable selectors, and
the raw trajectories behind every number we publish.
Citation
bibtex
1@software{arecesrivera2026bestofn,
2 title = {Best-of-N: inference-time compute for language models},
3 author = {Areces Rivera, Alejandro},
4 year = {2026},
5 doi = {10.5281/zenodo.21936832},
6 url = {https://github.com/voidlinestudios12-jpg/Interlace-AI}
7}
License
Apache License 2.0 — free to use, modify and redistribute, including
commercially.
Copyright 2026 Alejandro Areces Rivera — Interlace AI
Questions and collaboration: interlaceIA@gmail.com
Release notes: CHANGELOG.md
