Model Card for Model ID
Model Details
This model is a modified version of
ruRoberta-large for Target sentiment Analysis (TSA) using training data from the
RuSentNE-2023 collection .
This model is designed to analyze news texts in Russian.
Given an input sentence and a specified entity (target object) within it, this model determines the sentiment directed toward that entity and classifies it into one of the following categories:
['positive', 'negative', 'neutral'].
Model Description
Model type: ruRoberta-large Language(s) (NLP): RussianLicense: Apache License 2.0
Uses
Direct Use
This is a ruRoberta-large model with the addition of a linear layer for classification.
Loading model and tokenizer
1 from model import TargetSentimentClassifier
2
3 model = TargetSentimentClassifier (
4 model_name = "sberbank-ai/ruRoberta-large" ,
5 use_multi_sample_dropout = True ,
6 device = "cuda"
7 )
8 model . load_state_dict ( torch . load ( "pytorch_model.bin" , map_location = "cuda" ) )
Predict sentiment for a named entity in a sentence
1 text = "Джеймс «Бадди» Макгирт ... спортсмен остановить бой..."
2 target = "спортсмен"
3 entity_type = "PROFESSION"
4
5 prediction = model . predict ( text , target , entity_type )
6 print ( prediction ) # Output: 0, 1 or 2
Input Format
The input sentence must include a marked entity using the following format:
<en> ENTITY_TEXT <|ENTITY_TAG|> </en>Example:
Джеймс «Бадди» Макгирт ... <en> спортсмен <|PROFESSION|> </en> остановить бой...Labels
The model predicts one of the following labels:
Label Description 0 Neutral 1 Positive 2 Negative
Training Details
Training Data
For training, the data published for the RuSentNE-2023 competition, available via repository, was used:
https://github.com/dialogue-evaluation/RuSentNE-evaluation To increase the training sample, data from the Sentiment Analysis in Russian dataset was used:
https://www.kaggle.com/c/sentiment-analysis-in-russian/overview Additionally, to expand the dataset with entity-level sentiment annotations, an automatic annotation algorithm was used:
https://github.com/RyuminV01/markup-tsa-news/tree/main
This algorithm enabled the generation of additional labeled data based on named entity recognition and sentiment alignment in Russian news texts.
Evaluation
Testing Data, Factors & Metrics
Testing Data
Metrics
For the model evaluation, two metrics were used:
F1_posneg_macro -- F1-measure over positive and negative classes;
F1_macro -- F1-measure over positive, negative, and neutral classes;
Results
The test evaluation for this model, as shown on the
showcases , demonstrates the following performance:
Metric Score F1_posneg_macro 61.84 F1_macro 70.38
