Semantic Highlighting Classifier (ModernBERT + CRF)
Tags rhetorical-role spans in technical/academic abstracts -- Contribution,
Method, Result, Evidence, Limitation, FutureWork, Safety
-- for a semantic highlighting accessibility tool aimed at readers with
dyslexia/ADHD. Instead of highlighting keywords, it labels what role each
span of text plays, so a reader can skim a paper by color rather than reading
every word.
Architecture: a
ModernBERT-base
encoder with a linear classification head, decoded with a CRF (Conditional
Random Field) layer instead of independent per-token argmax. The CRF was
added specifically after an earlier version of this model was found to
change its predicted label on nearly every token (even splitting labels
mid-word) -- independent per-token classification has no mechanism to
penalize that, while a CRF explicitly learns which label transitions are
plausible and decodes the whole sequence jointly. Full diagnosis and
before/after examples in the
project's findings log.
This is the model used by the interactive Gradio demo in the project repo
(
app.py, run locally with
uv run python app.py or
make demo) and is
the fast, real-time-capable option in this project (see the
paired
Qwen2.5-1.5B LoRA adapter
for a generative alternative that scores higher but is ~20x+ slower and only
reliable on a minority of inputs -- both models and the full comparison are
documented in the findings log linked above).
Usage
This is a custom architecture (not a standard transformers
AutoModelForTokenClassification, since it wraps a CRF decoding step), so
it needs the model class from the project repo rather than transformers
alone:
1import torch
2from transformers import AutoTokenizer
3# pip install the project, or copy classifier/crf_model.py + classifier/labels.py
4from semantic_highlighting.classifier.crf_model import ModernBertCrfForTokenClassification
5from semantic_highlighting.classifier.predict import predict_spans_crf
6
7tokenizer = AutoTokenizer.from_pretrained("Rychanfox/semantic-highlighting-modernbert-crf")
8model = ModernBertCrfForTokenClassification.from_pretrained("Rychanfox/semantic-highlighting-modernbert-crf")
9model.eval()
10
11text = "We propose a new method for X. It achieves strong results using only 12 minutes of data."
12spans = predict_spans_crf(text, model, tokenizer)
13for span in spans:
14 print(span.label, repr(span.text))
For real documents (not just single short abstracts), see deploy/chunking.py
in the repo for sentence-aware chunking plus a postprocessing pipeline
(merging fragments, fixing boundaries, capping total highlighted coverage to
~20% of the document) that the raw model output on its own does not do.
Training data
Trained on Rychanfox/semantic-highlighting-abstracts -- arXiv abstracts
weak-labeled by a local LLM and partially hand-corrected. See that dataset's
card for the full collection/cleaning methodology.
Evaluation
Exact-match span-level F1 against a 21-abstract hand-corrected gold set
(small -- treat these numbers as directional, not precise):
| Metric | Score |
|---|
| Micro F1 (exact match) | 0.16 |
| Macro F1 (exact match) | 0.09 |
| Char-overlap F1 | 0.86 |
For context: a zero-training-cost baseline (frozen sentence embeddings +
logistic regression) scores 0.14/0.10 on the same metrics -- the fine-tuning,
CRF architecture, and data-cleaning work in this project bought a real but
modest improvement over a baseline that trains in about a second. That's an
intentionally reported honest comparison, not a caveat to bury -- full
context in the findings log.
Limitations
- Small gold evaluation set (21 abstracts) -- reported numbers carry real
variance.
- Exact-match performance is modest in absolute terms; character-overlap
(does it highlight roughly the right region) is much stronger (0.86) than
exact-match (does it get the precise boundary right).
- Trained predominantly on ML/NLP arXiv abstracts; domain transfer to other
fields or full paper bodies (vs. abstracts) is untested.
Safety and FutureWork labels are rare in the training data and the
model underperforms on them specifically.
