The base model completes text. This one answers questions, and - more reliably than
it answers them - declines when the context it is given does not contain the answer.
Chat format
Match this byte-for-byte. Deviating from it measurably degrades output.
cross-entropy + 0.05 x KL against the frozen DPO model
RAFT uses a KL penalty rather than plain cross-entropy so that a narrow,
retrieval-shaped dataset cannot drag the policy away from everything learned
earlier. Validation CE 1.214 to 1.146.
Results
stage
qa
instruction
grounded
refusal
mean
95% CI
base-v1 (ppl 11.06)
0.33
0.23
0.66
0.37
0.38
[0.27, 0.52]
base (ppl 9.71)
0.40
0.40
0.88
0.26
0.47
[0.35, 0.61]
+ instruction tuning
1.85
1.20
1.81
1.11
1.49
[1.31, 1.69]
+ SFT v2 (refusals)
1.65
1.32
3.50
5.89
2.97
[2.56, 3.40]
+ SFT v3
1.90
1.40
3.84
5.29
2.99
[2.61, 3.39]
+ DPO
2.00
1.35
3.38
6.23
3.13
[2.71, 3.56]
+ RAFT
1.75
1.30
2.97
6.89
3.12
[2.68, 3.56]
147 held-out questions, decontaminated against the SFT training passages, scored
0-10 by an LLM judge. Every model was scored on the same item by the same judge
with model identities hidden and re-shuffled per item, so a gap between two rows
is a gap between two models rather than between two judging runs. Intervals are
10,000-sample bootstraps.
Paired step-over-step change on the same items:
step
delta
95% CI
verdict
base-v1 to base
+0.09
[-0.08, +0.25]
not distinguishable from noise
base to instruct
+1.02
[+0.80, +1.24]
real
instruct to SFT v2
+1.48
[+1.06, +1.92]
real
SFT v2 to DPO
+0.16
[+0.02, +0.31]
real
DPO to RAFT
-0.01
[-0.35, +0.32]
not distinguishable from noise
Read the refusal column. It is the only metric where a model this size reaches
useful numbers, and it is the one that makes retrieval usable: a model that
declines when the passage does not support an answer can be trusted with a
retrieved passage.
DPO and RAFT are statistically tied on the mean. RAFT is published here because it
is the pipeline endpoint, it has the best refusal calibration (6.89), and it
degenerates least - it scores 0 on 0.7% of items, against 68.7% for the base model.
Honest limitations
This is not a reliable question-answering system. Closed-book factual accuracy
is 1.75/10. At 125M parameters, post-training teaches response format, register
and calibration - it cannot install knowledge the base model does not hold.
Sampling the same question six times at temperature 0.7 gave three correct answers,
two that inverted the legal standard, and one that answered a different question.
A single good screenshot of this model proves nothing.
It will state incorrect things fluently and invent plausible citations. Not legal
or financial advice.
