Base Model: AllenAI's SciBERT pre-trained on scientific text with a domain-specific vocabulary.
Task: Citation Intent Classification — predicting why an author is citing another work (background, method, or result).
Labels:
background (0): Citations providing prior work context
method (1): Citations of techniques or methodologies
result (2): Citations comparing or contrasting experimental results
Results
Achieved on the SciCite test set:
Metric
Score
Accuracy
85.60%
Macro F1
0.8431
Weighted F1
0.8566
Per-class performance:
Class
Precision
Recall
F1-Score
Support
background
0.88
0.87
0.88
997
method
0.88
0.81
0.84
605
result
0.74
0.89
0.81
259
Intended Uses & Limitations
Intended Use: Automatically classify citation intents in academic papers to improve literature mining, knowledge graph construction, and semantic search applications.
Limitations: Model trained on arXiv scientific abstracts; may not generalize to other domains (biomedical, legal, etc.). Best performance on background/method classes; result class has lower precision due to class imbalance.
Training and Evaluation Data
Dataset:SciCite — 8,243 training examples, 916 validation, 1,861 test (citation contexts from arXiv papers).
Format: Citation sentence + class label. Max length: 256 tokens. Split: 80% train, 10% val, 10% test.
How to Use
Installation
pip install transformers torch
Inference
python
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
34tokenizer = AutoTokenizer.from_pretrained("lostelf/scibert_scicite_finetuned")5model = AutoModelForSequenceClassification.from_pretrained("lostelf/scibert_scicite_finetuned")67text ="We use the BERT architecture as in Devlin et al."8inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)910with torch.no_grad():11 outputs = model(**inputs)12 logits = outputs.logits
13 predicted_class = logits.argmax(dim=-1).item()1415labels ={0:"background",1:"method",2:"result"}16print(f"Predicted: {labels[predicted_class]}")
Batch Prediction
python
1texts =[2"We build on the transformer framework introduced by Vaswani et al.",3"Our implementation follows the optimization procedure in Kingma & Ba.",4"These results exceed prior work by Devlin et al. (BERT)."5]67inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True, max_length=256)8outputs = model(**inputs)9predictions = outputs.logits.argmax(dim=-1)10print([labels[p]for p in predictions])
Training Hyperparameters
Parameter
Value
Model
allenai/scibert_scivocab_uncased
Epochs
8
Batch Size
32
Learning Rate
1e-05
Warmup Steps
25 (~10% of training)
Weight Decay
0.01
Optimizer
AdamW
LR Scheduler
linear
Gradient Accumulation
2 steps
FP16
Enabled
Training Results
Best checkpoint: Epoch 8 (macro F1 = 0.8431). Early stopping patience = 4 epochs.
