SchLAIMS Claim Selector
SchLAIMS Claim Selector is a Longformer-based paragraph selector. Given an
ordered scientific paragraph, it returns the one sentence that states the
paragraph's central claim, or null when no sentence does.
The model was trained on AI-consensus reference labels, not fully
human-adjudicated labels.
We used agents to create the training labels only
after a hidden test showed that they could follow the human-written guidelines
and usually select the same claims as human annotators. The agents never saw
the human answers. Across 10 seeded low-reasoning GPT-5.4 mini runs, the
retained human-consensus test had about 0.84 mean claim F1 and 0.948 mean
sentence accuracy. The complete hand-labeled set, including seven paragraphs
marked for adjudication, had about 0.69 mean claim F1 and 0.908 mean sentence
accuracy. This validates the labeling procedure; it does not make every
training or test label human gold. See the
SchLAIMS validation pipeline.
Fixed Test Results
Both results below come from one inference pass and must be reported together.
| Gold view | Scorable paragraphs | Precision | Recall | Claim F1 |
|---|
| Full random-resolved | 4,603 | 0.7032 | 0.7039 | 0.7035238841 |
| Majority-only | 4,538 | 0.7117 | 0.7106 | 0.7111252586 |
The full view includes deterministic selections from observed votes for
three-way label disagreements. The majority view excludes those paragraphs,
so it measures a different population. It is not simply a corrected score.
Use
Install the project environment and run its supported loader:
1git clone https://github.com/danielscottsmith/SchLAIMS.git
2cd SchLAIMS
3conda env create -f 03_ml_training/environment.yml
4conda run -n schlaims_ml python 03_ml_training/scripts/canonical_run.py predict \
5 --input paragraphs.jsonl \
6 --output predictions.jsonl \
7 --model-repo danielscottsmith/schlaims-claim-selector \
8 --revision v1
Each input row needs a unique paragraph_id and an ordered sentences list;
each sentence needs consecutive sentence_n values starting at 1 and text.
The output records the selected sentence or null plus every choice score.
The custom sentence-selector head requires the SchLAIMS loader and is not a
generic Transformers text-classification pipeline.
On the dataset card's example paragraph (triad_4498__original__p0037), this
checkpoint selects sentence 1, matching the unanimous reference label.
Training And Limits
The checkpoint is run_20260711_1135: Longformer base 4096, seed 2027,
learning rate 6e-6, weight decay 0.01, clip norm 1.0, 300 warmup updates,
and a three-epoch horizon. Checkpoints were selected only on the full
random-resolved development view, with 1,000-update checks, minimum improvement
0.002, patience 5, and a 21-check one-epoch floor.
The release has mixed but recorded preparation lineage: the first 150 triads
were prepared and carried forward before the remaining 350 were added. One
test paragraph was excluded because it exceeded model context. The benchmark
is a retrospective baseline refined through iterative diagnostics, not a new
prospective sealed test. The labels lack final human adjudication, and the
model does not check truth, evidence quality, or scientific importance.
The checkpoint and selector head are released under Apache 2.0. Exact
reproduction files for Version 1 are also available in the hashed canonical
model archive.