metadata
library_name: transformers
tags: []
Tutor Talk Moves Classifier (RoBERTa Large)
Model Description
This model is a fine-tuned version of roberta-large that classifies tutor utterances based on teacher talk moves from talkmoves.com. The model identifies whether a tutor's message represents one of the following three categories, or none of them:
- Classroom management
- Pressing for accuracy or reasoning
- Restating or revoicing
Training Data
This model is trained on text of tutoring sessions from three different tutoring providers, annotated by two raters.
- Inter-Rater Reliability with Krippendorff's Alpha: 0.83
- Total Training Examples: 1,849
- Class Distribution:
Class Examples Percentage None 987 53.4% Classroom management 284 15.4% Pressing for accuracy or reasoning 474 25.6% Restating or revoicing 104 5.6%
Data Format
The is trained on utterances with the following format: [PRETEXT] {3 previous messages} [TEXT] {target message},
where [PRETEXT] and [TEXT] are special tokens.
Names are anonymized, message text is lowercased, and leading and trailing whitespace is removed.
Example:
[PRETEXT] tutor: hello there [student]
tutor: what is the answer to this problem?
student: the answer is 6 [TEXT] tutor: why do you say the answer is 6?
Performance
Test set results (264 examples):
| Class | Precision | Recall | F1-Score | Support |
|---|---|---|---|---|
| None | 0.9368 | 0.9271 | 0.9319 | 192 |
| Classroom Management | 0.6800 | 0.7727 | 0.7234 | 22 |
| Pressing for Accuracy or Reasoning | 0.8605 | 0.8810 | 0.8706 | 42 |
| Restating or Revoicing | 0.3333 | 0.2500 | 0.2857 | 8 |
| Macro Average | 0.7027 | 0.7077 | 0.7029 | 264 |
| Weighted Average | 0.8850 | 0.8864 | 0.8852 | 264 |