You can deploy this model and start earning money today!

StanfordSCALE/
tutor_talkmoves_roberta_large_classifier

Name: tutor_talkmoves_roberta_large_classifier
Availability: InStock

text-embeddings-inference

endpoints_compatible

Views

No views yet

Tutor Talk Moves Classifier (RoBERTa Large)

Model Description

This model is a fine-tuned version of roberta-large that classifies tutor utterances based on teacher talk moves from talkmoves.com. The model identifies whether a tutor's message represents one of the following three categories, or none of them:

Classroom management
Pressing for accuracy or reasoning
Restating or revoicing

Training Data

This model is trained on text of tutoring sessions from three different tutoring providers, annotated by two raters.

Inter-Rater Reliability with Krippendorff's Alpha: 0.83
Total Training Examples: 1,849
Class Distribution: | Class | Examples | Percentage | |-------|----------|------------| | None | 987 | 53.4% | | Classroom management | 284 | 15.4% | | Pressing for accuracy or reasoning | 474 | 25.6% | | Restating or revoicing | 104 | 5.6% |

Data Format

The is trained on utterances with the following format: [PRETEXT] {3 previous messages} [TEXT] {target message}, where [PRETEXT] and [TEXT] are special tokens. Names are anonymized, message text is lowercased, and leading and trailing whitespace is removed.

Example:

[PRETEXT] tutor: hello there [student]
tutor: what is the answer to this problem?
student: the answer is 6 [TEXT] tutor: why do you say the answer is 6?

Performance

Test set results (264 examples):

Class	Precision	Recall	F1-Score	Support
None	0.9368	0.9271	0.9319	192
Classroom Management	0.6800	0.7727	0.7234	22
Pressing for Accuracy or Reasoning	0.8605	0.8810	0.8706	42
Restating or Revoicing	0.3333	0.2500	0.2857	8
Macro Average	0.7027	0.7077	0.7029	264
Weighted Average	0.8850	0.8864	0.8852	264

StanfordSCALE/tutor_talkmoves_roberta_large_classifier