Model Card for orYx-models/finetuned-roberta-leadership-sentiment-analysis

• Model Description: This model is a finetuned version of the RoBERTa text classifier(cardiffnlp/twitter-roberta-base-sentiment-latest). It has been trained on a dataset comprising communications from corporate executives to their therapists. Its primary function is to determine whether statements from corporate executives convey a "Positive," "Negative," or "Neutral" sentiment, accompanied by a confidence level indicating the percentage of sentiment expressed in a statement. Being a prototype tool by orYx Models, all feedbacks and insights will be used to further refine the model.

Model Details

Model Information

• Model Type: Text Classifier
• Language(s): English
• License: Creative Commons license family
• Finetuned from Model: cardiffnlp/twitter-roberta-base-sentiment-latest

Model Sources

• HuggingFace Model ID: cardiffnlp/twitter-roberta-base-2021-124m
• Paper: TimeLMs - Link

Uses

• Use case: This sentiment analysis tool can analyze text from any user within an organization, such as executives, employees, or clients, and assign a sentiment to it.
• Outcomes: The tool generates a "Scored sentiment" which can be used to assess the likelihood of events occurring or vice versa. It can also facilitate the creation of a rating system based on the sentiments expressed in texts.

Direct Use

• Based on the text the outcomes can be "Positive, Negative, Neutral" along with their confidence score.

Recommendations

• Continuous Monitoring: Regularly monitor the model's performance on new data to ensure its effectiveness and reliability over time.
• Error Analysis: Conduct thorough error analysis to identify common patterns of misclassifications and areas for improvement.
• Fine-Tuning: Consider fine-tuning the model further based on feedback and insights from users, to enhance its domain-specific performance.
• Model Interpretability: Explore techniques for explaining the model's predictions, such as attention mechanisms or feature importance analysis, to increase trust and understanding of its decisions.

Training Details

X_train, X_val, y_train, y_val = train_test_split(X,y, test_size = 0.2, stratify = y)

• Train data: 80% of 4396 records = 3516
• Test data: 20% of 4396 records = 879

Training Procedure

• Dataset Split: Data divided into 80% training and 20% validation sets.
• Preprocessing: Input data tokenized into 'input_ids' and 'attention_mask' tensors.
• Training Hyperparameters: Set for training, evaluation, and optimization, including batch size, epochs, and logging strategies.
• Training Execution: Model trained with specified hyperparameters, monitored with metrics, and logged for evaluation.
• Evaluation Metrics: Model evaluated on loss, accuracy, F1 score, precision, and recall for both training and validation sets.

Preprocessing [optional]

'input_ids': tensor
'attention_mask': tensor
'label': tensor(2)

Training Hyperparameters

args = TrainingArguments(
    output_dir="output",
    do_train = True,
    do_eval = True,
    num_train_epochs = 1,
    per_device_train_batch_size = 4,
    per_device_eval_batch_size = 8,
    warmup_steps = 50,
    weight_decay = 0.01,
    logging_strategy= "steps",
    logging_dir= "logging",
    logging_steps = 50,
    eval_steps = 50,
    save_strategy = "steps",
    fp16 = True,
    #load_best_model_at_end = True
)

Speeds, Sizes, Times [optional]

• TrainOutput

global_step=879,
training_loss=0.1825900522650848,

• Metrics

'train_runtime': 101.6309,
'train_samples_per_second': 34.596,
'train_steps_per_second': 8.649,
'total_flos': 346915041274368.0,
'train_loss': 0.1825900522650848,
'epoch': 1.0

Evaluation Metrics Results

# Assuming you have a list of evaluation results q and want to create a DataFrame with it
q = [Trainer.evaluate(eval_dataset=df) for df in [train_dataset, val_dataset]]

# Create DataFrame with index and select only the first 5 columns
result_df = pd.DataFrame(q, index=["train", "val"]).iloc[:,:5]

# Display the resulting DataFrame
print(result_df)

______________________________________________________________________
eval_loss  eval_Accuracy   eval_F1  eval_Precision  eval_Recall
train   0.049349       0.988908  0.987063        0.982160     0.992357
val     0.108378       0.976136  0.972464        0.965982     0.979861
______________________________________________________________________

• train 0.049349
• val 0.108378

• train 0.988908 - 98.8%
• val 0.976136 - 97.6%

• train 0.987063 - 98.7%
• val 0.972464 - 97.2%

• train 0.982160 - 98.2%
• val 0.965982 - 96.5%

• train 0.992357 - 99.2%
• val 0.979861 - 97.9%

Environmental Impact

• Hardware Type: T4 GPU
• Hours used: 2
• Cloud Provider: Google
• Compute Region: India
• Carbon Emitted: No Information Available

Compute Infrastructure

References

@inproceedings{camacho-collados-etal-2022-tweetnlp,
    title = "{T}weet{NLP}: Cutting-Edge Natural Language Processing for Social Media",
    author = "Camacho-collados, Jose  and
      Rezaee, Kiamehr  and
      Riahi, Talayeh  and
      Ushio, Asahi  and
      Loureiro, Daniel  and
      Antypas, Dimosthenis  and
      Boisson, Joanne  and
      Espinosa Anke, Luis  and
      Liu, Fangyu  and
      Mart{\'\i}nez C{\'a}mara, Eugenio" and others,
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.emnlp-demos.5",
    pages = "38--49"
}

Model Card Authors [optional]

Model Card Contact