This model is a fine-tuned mDeBERTa-v3-base that performs stance classification using Natural Language Inference (NLI) to determine whether political text expresses positive, negative, or neutral stance towards specific target groups.

Developed by: Will Horne, Alona O. Dolinsky and Lena Maria Huber
Model type: Sequence Classification (NLI-based stance detection)
Language(s) (NLP): English, German (multilingual)
Finetuned from model: MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7

Model Sources

Repository: rwillh11/mdeberta_NLI_stance_NoContext
Base Model: MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7

Uses

Direct Use

The model is designed for researchers analyzing political discourse and stance towards specific groups in political text, trained and validated using party manifestos. It takes a natural sentence and a target group as input and classifies the stance as positive, negative, or neutral.

Downstream Use

This model can be integrated into larger political text analysis pipelines for:

Political manifestos analysis
Group appeals detection in political communication
Comparative political research across countries and languages

Out-of-Scope Use

This model should not be used for:

General sentiment analysis (not group-specific)
Real-time social media monitoring without human oversight
Making decisions about individuals or groups
Content moderation without additional validation

Bias, Risks, and Limitations

Technical Limitations

Trained specifically on political manifesto text; performance may vary on other text types
Limited to three stance categories (positive, negative, neutral)

Bias Considerations

Training data consists of political manifestos from specific countries and time periods
May reflect biases present in political discourse of training data
Stance classification may vary across different political contexts

Recommendations

Users should be aware that this model:

Is designed for research purposes in political science
Should be validated on specific domains before deployment
May require human oversight for sensitive applications
Performance may vary across different types of groups and political contexts

How to Get Started with the Model

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_name = "rwillh11/mdeberta_NLI_stance_NoContext"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Example usage
text = "We will increase funding for schools to better support students."
target_group = "students"

# Create hypotheses for each stance
hypotheses = {
    "positive": f"The text is positive towards {target_group}.",
    "negative": f"The text is negative towards {target_group}.",
    "neutral": f"The text is neutral, or contains no stance, towards {target_group}."
}

# Get predictions for each hypothesis
results = {}
for stance, hypothesis in hypotheses.items():
    inputs = tokenizer(text, hypothesis, return_tensors="pt", truncation=True)
    with torch.no_grad():
        outputs = model(**inputs)
        probs = torch.softmax(outputs.logits, dim=-1)
        entailment_prob = probs[0][0].item()  # Probability of entailment
        results[stance] = entailment_prob

# Select stance with highest entailment probability
predicted_stance = max(results, key=results.get)
print(f"Predicted stance towards '{target_group}': {predicted_stance}")

Training Details

Training Data

The model was trained on political manifesto data containing:

Languages: English and German
Text Type: Political manifesto sentences at natural sentence level
Labels: Three-class stance classification (positive, negative, neutral)
Groups: Various political target groups (citizens, specific demographics, etc.)
Original dataset: 7,567 text-group pairs
- English: 4,110 examples (54.3%)
- German: 3,457 examples (45.7%)
Training Size: 12,104 expanded training examples (~6,054 original texts × 2 hypotheses each)
Test Size: 4,539 expanded test examples (~1,513 original texts × 3 hypotheses each)

Training Procedure

Preprocessing

Texts tokenized using mDeBERTa tokenizer with max length 512
NLI format: premise (political text) + hypothesis (stance towards group)
Each text paired with both true and false hypotheses for binary classification

Training Hyperparameters

Training regime: Mixed precision training
Optimizer: AdamW with weight decay
Learning rate: Optimized via Optuna (range: 1e-5 to 4e-5)
Weight decay: Optimized via Optuna (range: 0.01 to 0.3)
Warmup ratio: Optimized via Optuna (range: 0.0 to 0.1)
Epochs: 10 per trial
Batch size: 16 (train and eval)
Trials: 20 total (10 + 10 batches)
Metric for selection: F1 Macro
Seed: 42 (deterministic training)

Training Infrastructure

Hardware: CUDA-enabled GPU
Framework: Transformers, PyTorch
Hyperparameter optimization: Optuna
Deterministic training: All random seeds fixed

Evaluation

Testing Data, Factors & Metrics

Testing Data

20% holdout from original dataset
Multilingual political manifesto sentences

Factors

The model was evaluated across:

Languages: English and German text
Additional validation on held out sets: English, German, Dutch, Danish, Spanish, French, Italian, Swedish
Stance classes: Positive, negative, neutral
Group types: Various socia-demographic groups

Metrics

Primary metrics used for evaluation:

F1 Macro: Primary optimization metric (treats all classes equally)
Balanced Accuracy: Accounts for class imbalance
Precision/Recall (Macro): Detailed performance measures

Results

Best Model Performance (Trial 19, Epoch 9):

Accuracy: 0.86
Balanced Accuracy: 0.81
Precision: 0.83
Recall: 0.81
F1 Macro: 0.81

Additional validation on held-out sets return the following metrics:

English

Accuracy: 0.86
Precision: 0.84
Recall: 0.81
F1 Macro: 0.82

German (using texts translated from English)

Accuracy: 0.86
Precision: 0.85
Recall: 0.79
F1 Macro: 0.82

Dutch (using texts translated from English)

Accuracy: 0.82
Precision: 0.80
Recall: 0.74
F1 Macro: 0.76

Danish (using texts translated from English)

Accuracy: 0.83
Precision: 0.80
Recall: 0.75
F1 Macro: 0.77

Spanish (using texts translated from English)

Accuracy: 0.85
Precision: 0.84
Recall: 0.77
F1 Macro: 0.80

French (using texts translated from English)

Accuracy: 0.84
Precision: 0.84
Recall: 0.76
F1 Macro: 0.79

Italian (using texts translated from English)

Accuracy: 0.86
Precision: 0.86
Recall: 0.77
F1 Macro: 0.81

Swedish (using texts translated from English)

Accuracy: 0.82
Precision: 0.79
Recall: 0.77
F1 Macro: 0.78

The model demonstrates strong performance across stance categories with deterministic results confirmed through multiple prediction runs.

Model Examination

The model uses Natural Language Inference to transform stance detection into a binary entailment task:

For each text-group pair, generates three hypotheses (positive/negative/neutral stance)
Selects the hypothesis with highest entailment probability
This approach leverages pre-trained NLI capabilities for stance classification

Environmental Impact

Training involved hyperparameter optimization with 20 trials, each training for 10 epochs.

Hardware Type: CUDA-enabled GPU
Hours used: Estimated 10-15 hours (including hyperparameter search)
Cloud Provider: Google Colab
Compute Region: Variable
Carbon Emitted: Not precisely measured

Technical Specifications

Model Architecture and Objective

Base Architecture: mDeBERTa-v3-base (278M parameters)
Task: Natural Language Inference for stance detection
Input: Text pair (political sentence + stance hypothesis)
Output: Binary classification (entailment/non-entailment)
Objective: Cross-entropy loss with F1 Macro optimization

Compute Infrastructure

Hardware

GPU-accelerated training (CUDA)
Mixed precision training support

Software

Transformers library
PyTorch framework
Optuna for hyperparameter optimization
scikit-learn for metrics

Citation

If you use this model in your research, please cite:

BibTeX:

bibtex

@misc{mdeberta_stance_nocontext,
  title={mDeBERTa Stance Detection Model for Political Group Appeals},
  author={Research Team},
  year={2024},
  url={https://huggingface.co/rwillh11/mdeberta_NLI_stance_NoContext}
}

Model Card Authors

Research team studying group appeals in political discourse.

Model Card Contact

For questions about this model, please open an issue in the repository or contact the research team through appropriate academic channels.