This is a bert-base-cased model fine-tuned for multi-label country classification. It identifies which countries are mentioned in a paragraph of text from a news article.

This model was developed as part of the research project "Distilling Large Language Models for Efficient Tagging in Open-Source Intelligence Data" by Rajan Paneru.

How to Use

This is a multi-label classification model, so it returns a score for every possible country. To get the final predicted labels, you should apply a confidence threshold. The recommended thresholds for each country are provided in the class_thresholds.json file in this repository.

Here is how to use the model with the transformers pipeline:

Install python package: uv add transformers[torch] requests

python

from transformers import pipeline
import json
import requests

# Load Model and Thresholds
model_id = "paneru-rajan/bert-news-paragraph-country-classify"

# Load the classification pipeline from the Hub
pipe = pipeline("text-classification", model=model_id, truncation=True, padding=True, max_length=512)

# Download the recommended thresholds file from the Hub
try:
    thresholds_url = f"https://huggingface.co/{model_id}/raw/main/class_thresholds.json"
    thresholds = requests.get(thresholds_url).json()
except Exception as e:
    print(f"Could not download thresholds, using 0.5 for all classes. Error: {e}")
    thresholds = {}

# Make a Prediction
text = "Donald Trump has strongly endorsed the Aukus pact and praised prime minister Anthony Albanese as a “great” leader, but the president’s navy secretary says the US may seek to “clarify some ambiguities” in the nuclear submarine deal.\n\nTrump and Albanese also signed a multibillion-dollar agreement for Australia to supply the United States with critical minerals, amid a deepening trade war as China threatens to cut its supply of rare earth elements. But the president also downplayed any prospect of cutting tariffs on Australian goods."

# Get scores for all labels
predictions = pipe(text, top_k=None)

# Apply Thresholds to Get Final Labels
final_labels = []
for pred in predictions:
    # The model returns country codes (e.g., 'au' for Australia)
    country_code = pred['label']
    # Use the specific threshold for that country, or a default of 0.5
    if pred['score'] > thresholds.get(country_code, 0.5):
        final_labels.append(country_code)

print(f"Text: '{text}'")
print(f"Predicted Countries: {final_labels}")
# Expected output for this example: ['au', 'us', 'cn']

Model Details

Base Model: google-bert/bert-base-cased
Training Data: This model was trained using a black-box distillation approach. The training data consists of 209,052 paragraphs from a large corpus of OSINT news articles. These paragraphs were labeled using a large language model (LLM) as the "teacher" model. The dataset was curated to include approximately 1,000 paragraphs for each of the 200 target countries, plus 10,000 negative samples.
Training Procedure: Fine-tuned for 15 epochs with a learning rate of 2e-5 and a max sequence length of 512.

Evaluation & Performance

The model achieves the following performance on the held-out test set:

Macro F1-Score: 0.929
Macro ROC AUC: 0.993
Subset Accuracy (Exact Match): 0.748

The training and validation loss, F1 scores, and learning rate schedule can be seen below:

Model Training and Validation Performance

Model Performance on Validation Set

Model Performance on Test Set

Contact

For any questions or collaboration inquiries, please feel free to reach out:

GitHub: paneru-rajan
Email: paneru.rajan@gmail.com