|
|
--- |
|
|
library_name: transformers |
|
|
tags: [] |
|
|
--- |
|
|
|
|
|
# Tutor Talk Moves Classifier (ModernBERT) |
|
|
|
|
|
## Model Description |
|
|
|
|
|
This model is a fine-tuned version of [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) that classifies tutor utterances based on teacher talk moves from [talkmoves.com](https://www.talkmoves.com/). |
|
|
The model identifies whether a tutor's message represents one of the following three categories, or none of them: |
|
|
|
|
|
- Classroom management |
|
|
- Pressing for accuracy or reasoning |
|
|
- Restating or revoicing |
|
|
|
|
|
## Training Data |
|
|
|
|
|
This model is trained on text of tutoring sessions from three different tutoring providers, annotated by two raters. |
|
|
|
|
|
- **Inter-Rater Reliability with Krippendorff's Alpha**: 0.83 |
|
|
- **Total Training Examples**: 1,849 |
|
|
- **Class Distribution**: |
|
|
| Class | Examples | Percentage | |
|
|
|-------|----------|------------| |
|
|
| None | 987 | 53.4% | |
|
|
| Classroom management | 284 | 15.4% | |
|
|
| Pressing for accuracy or reasoning | 474 | 25.6% | |
|
|
| Restating or revoicing | 104 | 5.6% | |
|
|
|
|
|
## Data Format |
|
|
|
|
|
The is trained on utterances with the following format: `[PRETEXT] {3 previous messages} [TEXT] {target message}`, |
|
|
where `[PRETEXT]` and `[TEXT]` are special tokens. |
|
|
Names are anonymized, message text is lowercased, and leading and trailing whitespace is removed. |
|
|
|
|
|
**Example**: |
|
|
``` |
|
|
[PRETEXT] tutor: hello there [student] |
|
|
tutor: what is the answer to this problem? |
|
|
student: the answer is 6 [TEXT] tutor: why do you say the answer is 6? |
|
|
``` |
|
|
|
|
|
## Performance |
|
|
|
|
|
Test set results (264 examples): |
|
|
|
|
|
| Class | Precision | Recall | F1-Score | Support | |
|
|
|-------|-----------|--------|----------|---------| |
|
|
| 0 - None | 0.9158 | 0.9062 | 0.9110 | 192 | |
|
|
| 1 - Classroom Management | 0.6818 | 0.6818 | 0.6818 | 22 | |
|
|
| 2 - Pressing for Accuracy or Reasoning | 0.7826 | 0.8571 | 0.8182 | 42 | |
|
|
| 3 - Restating or Revoicing | 0.5000 | 0.3750 | 0.4286 | 8 | |
|
|
| **Macro Average** | **0.7201** | **0.7051** | **0.7099** | **264** | |
|
|
| **Weighted Average** | **0.8625** | **0.8636** | **0.8625** | **264** | |
