You can deploy this model and start earning money today!

mgroenendyk/
bert-gov-canada-data-citation-classifier

Name: bert-gov-canada-data-citation-classifier
Availability: InStock

pytorch

safetensors

bert

google-bert/bert-base-uncased

finetune

mit

Views

No views yet

BERT Canadian Government Data Citation Classifier

Model Description

This model is a fine-tuned BERT classifier designed to identify citations of Government of Canada open datasets in academic and news literature. It was trained to recognize various citation patterns specific to Canadian government data sources, particularly datasets published through open.canada.ca.

Model Details

Model Type: BERT-base-uncased fine-tuned for binary classification
Language: English (with some capability for French citations)
Domain: Citation detection, bibliometrics, government data
Base Model: bert-base-uncased
Training Framework: Hugging Face Transformers 4.26, PyTorch 1.13
Fine-tuning Task: Binary classification (citation vs. non-citation)

Performance

The model achieves strong performance on citation detection:

Metric	Score
Accuracy	0.91
Precision	0.90
Recall	0.93
F1-Score	0.91
ROC-AUC	0.92

Baseline Comparisons

Significantly outperforms alternative approaches:

Model	Accuracy	Precision	Recall	F1-Score	ROC-AUC
Fine-tuned BERT (This Model)	0.910	0.900	0.930	0.910	0.920
BERT Zero-shot	0.657	0.599	1.000	0.749	0.914
Enhanced Keyword Baseline	0.857	0.894	0.818	0.854	0.864
Scientific Embedding Model	0.697	0.909	0.455	0.606	0.843
Keyword Matching Baseline	0.730	0.670	0.620	0.640	0.710

Training Data

Training Examples: 6,514 sentences (3,257 citations + 3,257 non-citations)
Sources: Academic articles and Canadian news media
Data Collection: Manual verification with Cohen's κ = 0.85 inter-annotator agreement
Split: 70% training, 15% validation, 15% testing
Balance: 50/50 positive/negative examples

Citation Examples

The model was trained to recognize patterns like:

"Data for this study was taken from Fisheries and Oceans Canada."
"According to Health Canada data, the findings show..."
"Information obtained from the Government of Canada portal."
"Statistics Canada reported that..."

Usage

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("mgroenendyk/bert-gov-canada-data-citation-classifier")
model = AutoModelForSequenceClassification.from_pretrained("mgroenendyk/bert-gov-canada-data-citation-classifier")

# Example text
text = "Data for this study was taken from Statistics Canada."

# Tokenize and predict
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=128)
with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
    
# Get prediction
citation_probability = predictions[0][1].item()
is_citation = citation_probability > 0.5

print(f"Citation probability: {citation_probability:.3f}")
print(f"Is citation: {is_citation}")

Pipeline Usage

python

from transformers import pipeline

classifier = pipeline("text-classification", 
                     model="mgroenendyk/bert-gov-canada-data-citation-classifier",
                     tokenizer="mgroenendyk/bert-gov-canada-data-citation-classifier")

result = classifier("Data for this study was taken from Statistics Canada.")
print(result)
# [{'label': 'CITATION', 'score': 0.89}]

Intended Use

Primary Applications

Bibliometric Analysis: Track impact of Canadian government open data
Research Evaluation: Measure data reuse in academic literature
Policy Analysis: Understand how government data is utilized
Citation Mining: Automated extraction of data citations from literature

Suitable Text Types

Academic papers and articles
News articles and reports
Policy documents
Research reports

Limitations

Known Issues

Multilingual Performance: Occasionally struggles with French-language citations
Ambiguous References: May misclassify vague references like "federal statistics"
Novel Patterns: Performance may degrade on citation formats not in training data
Domain Specificity: Optimized for Canadian government data; may not generalize to other government data sources

Edge Cases

Footnote-style references
Embedded citations in complex sentences
Mixed language (English/French) citations
Internal government reports vs. open data

Training Details

Hyperparameters

Learning Rate: 2e-5
Batch Size: 16
Epochs: 3
Max Sequence Length: 128 tokens
Optimizer: AdamW with linear learning rate scheduler
Warmup: 10% of training steps

Infrastructure

Platform: AWS SageMaker
Instance Type: ml.p3.2xlarge (training), ml.m5.large (inference)
Framework: Transformers 4.26, PyTorch 1.13, Python 3.9

Validation

5-fold stratified cross-validation
Standard deviation < 0.02 across all metrics
Early stopping based on validation F1-score

Ethical Considerations

Bias and Fairness

Training data focused on English-language sources with some French content
May have bias toward formal citation styles found in academic literature
Performance on informal or non-standard citation formats may vary

Privacy

Training data consists of publicly available academic and news content
No personal information or proprietary data used
Model outputs are classifications, not content generation

Citation

If you use this model in your research, please cite:

bibtex

@article{groenendyk2025bert,
  title={The Effectiveness of Fine-Tuning BERT to Identify Citations of Government of Canada Open Data},
  author={Groenendyk, Michael},
  journal={[Pre-Publication]},
  year={2025},
  note={Model available at: https://huggingface.co/mgroenendyk/bert-gov-canada-data-citation-classifier}
}

Additional Resources

Training Code: https://github.com/mikeglibrary/bert-data-citation
Dataset: https://github.com/mikeglibrary/bert-data-citation

Model Card Authors

Michael Groenendyk, Concordia University

Model Card Contact

For questions about this model, please contact michael.groenendyk@concordia.ca or open an issue in the GitHub repository.

Disclaimer: This model is provided for research purposes. Users should validate performance on their specific use cases and data before deployment in production systems.