RWGBench Citation Frame Classifier
This model is the citation-frame classifier used by RWGBench to compute
Citation Frame Alignment (CFA). CFA evaluates whether a generated related work
section uses citations with a rhetorical frame distribution similar to the
author-written reference section.
The classifier is a DeBERTa-v3-large sequence-classification model fine-tuned
for multi-label citation-function prediction. It predicts seven
MultiCite-derived labels:
| Label | Intended citation function |
|---|
motivation | motivates the target problem or research direction |
background | provides general background or prior findings |
uses | describes methods, data, tools, or results used by the target work |
extends | indicates that the target work builds on or extends prior work |
similarities | marks similarity or close relationship to prior work |
differences | marks contrast, limitation, gap, or distinction |
future_work | discusses future directions |
Intended Use
The model is intended for automatic evaluation and analysis of related work
generation systems. In RWGBench, citation contexts are extracted from generated
and reference related work sections, classified with this model, aggregated
into frame-count profiles, and compared with macro count F1.
It is not intended as a general-purpose judge of paper quality or citation
correctness. It should be used together with citation-selection and
citation-support metrics.
Files
| File | Description |
|---|
config.json, model.safetensors | Transformers sequence-classification checkpoint |
tokenizer.json, tokenizer_config.json, special_tokens_map.json, spm.model, added_tokens.json | Tokenizer files |
thresholds.json | Per-label development thresholds used for multi-label decoding |
metrics.json | Development and held-out MultiCite evaluation metrics |
label_map.json | Label mapping used during training |
rwgb_classifier_config.json | RWGBench-specific usage metadata |
Evaluation
Held-out MultiCite test performance:
| Metric | Score |
|---|
| Macro F1 | 0.5585 |
| Micro F1 | 0.6751 |
| Samples F1 | 0.6933 |
| Weighted F1 | 0.6750 |
| Subset accuracy | 0.5119 |
Per-label held-out F1:
| Label | F1 |
|---|
motivation | 0.3523 |
background | 0.7756 |
uses | 0.7372 |
extends | 0.4450 |
similarities | 0.5508 |
differences | 0.6274 |
future_work | 0.4211 |
The thresholds in thresholds.json are applied after sigmoid probabilities.
If no label passes its threshold, RWGBench assigns the highest-probability label
so that every citation context contributes at least one frame.
Usage With RWGBench
Download the model into the RWGBench code repository:
1models/citation_frame_multicite/
2 config.json
3 model.safetensors
4 tokenizer.json
5 thresholds.json
6 ...
run:
1python src/evaluation/evaluate_generated_results.py \
2 --input results/your_results.json \
3 --output results/your_eval.json \
4 --gold data/gold100_papers.json \
5 --corpus data/corpus.json \
6 --citation-frame-model-path models/citation_frame_multicite
RWGBench automatically marks numbered citations with <cite>...</cite> before
classification, matching the input format used during training.
