StanfordSCALE/tutor_talkmoves_modernbert_base_classifier

Tutor Talk Moves Classifier (ModernBERT)

Model Description

This model is a fine-tuned version of answerdotai/ModernBERT-base that classifies tutor utterances based on teacher talk moves from talkmoves.com. The model identifies whether a tutor's message represents one of the following three categories, or none of them:

• Classroom management
• Pressing for accuracy or reasoning
• Restating or revoicing

Training Data

This model is trained on text of tutoring sessions from three different tutoring providers, annotated by two raters.

• Inter-Rater Reliability with Krippendorff's Alpha: 0.83
• Total Training Examples: 1,849
• Class Distribution:

  Class	Examples	Percentage
  None	987	53.4%
  Classroom management	284	15.4%
  Pressing for accuracy or reasoning	474	25.6%
  Restating or revoicing	104	5.6%

Data Format

The is trained on utterances with the following format: [PRETEXT] {3 previous messages} [TEXT] {target message}, where [PRETEXT] and [TEXT] are special tokens. Names are anonymized, message text is lowercased, and leading and trailing whitespace is removed.

Example:

[PRETEXT] tutor: hello there [student]
tutor: what is the answer to this problem?
student: the answer is 6 [TEXT] tutor: why do you say the answer is 6?

Performance

Test set results (264 examples):

Class	Precision	Recall	F1-Score	Support
0 - None	0.9158	0.9062	0.9110	192
1 - Classroom Management	0.6818	0.6818	0.6818	22
2 - Pressing for Accuracy or Reasoning	0.7826	0.8571	0.8182	42
3 - Restating or Revoicing	0.5000	0.3750	0.4286	8
Macro Average	0.7201	0.7051	0.7099	264
Weighted Average	0.8625	0.8636	0.8625	264