Text Classification
setfit
Safetensors
English
mpnet
education
classroom-discourse
edubehaviors
assertion
Eval Results (legacy)
Instructions to use StanfordSCALE/assertion_contains_self_correction with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- setfit
How to use StanfordSCALE/assertion_contains_self_correction with setfit:
from setfit import SetFitModel model = SetFitModel.from_pretrained("StanfordSCALE/assertion_contains_self_correction") preds = model.predict(["i loved the spiderman movie!", "pineapple on pizza is the worst"]) print(preds) - Notebooks
- Google Colab
- Kaggle
|
Download README.md from StanfordSCALE/assertion_contains_self_correction: direct link, hf CLI and curl.
- Browser
- Download file 4.26 kB
-
https://huggingface.co/StanfordSCALE/assertion_contains_self_correction/resolve/main/README.md
- Command line
-
hf download hf://StanfordSCALE/assertion_contains_self_correction/README.md
-
curl -L -o README.md https://huggingface.co/StanfordSCALE/assertion_contains_self_correction/resolve/main/README.md
4.26 kB
metadata
language: en
library_name: setfit
pipeline_tag: text-classification
base_model: sentence-transformers/paraphrase-mpnet-base-v2
datasets:
- StanfordSCALE/assertions_llm_annotated_talkmoves
metrics:
- f1
- precision
- recall
- roc_auc
tags:
- setfit
- text-classification
- education
- classroom-discourse
- edubehaviors
- assertion
model-index:
- name: assertion_contains_self_correction
results:
- task:
type: text-classification
dataset:
name: assertions_llm_annotated_talkmoves
type: StanfordSCALE/assertions_llm_annotated_talkmoves
split: test
metrics:
- type: f1
value: 0.2491
name: F1 (positive class, test)
- type: precision
value: 0.3913
name: Precision (positive class, test)
- type: recall
value: 0.1827
name: Recall (positive class, test)
- type: roc_auc
value: 0.6867
name: ROC-AUC (test)
Assertion: contains self correction
This classifier was trained for EduBehaviors: Assertion-based schemas for auditable dialogue coding and is usable through the Python package EduBehaviors-kit. This classifier was trained on an LLM-annotated subset of teacher utterances from the TalkMoves Dataset. See the Datasets section below for more information.
Training Details
Datasets
| Dataset | Split | Size |
|---|---|---|
| StanfordSCALE/assertions_llm_annotated_talkmoves | train | 13,179 (53.3%) |
| StanfordSCALE/assertions_llm_annotated_talkmoves | dev | 3,296 (13.3%) |
| StanfordSCALE/assertions_llm_annotated_talkmoves | test | 8,240 (33.3%) |
This model's columns are assertion_contains_self_correction and split_contains_self_correction.
Base rate (share of rows labeled as True): 2.3% overall — 2.2% train, 2.2% dev, 2.4% test.
Labels and annotation
Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.570.
Hyperparameters
| Parameter | Value |
|---|---|
| Base model (body) | sentence-transformers/paraphrase-mpnet-base-v2 |
| Head | LogisticRegression |
| Body learning rate | 2e-05 |
| Head learning rate | 0.01 |
| Batch size | 16 (contrastive phase) / 32 (head) |
| Epochs | 10 |
| Max steps | 5000 (contrastive phase) |
| Eval max steps | 100 |
| Seed | 20260904 |
| Mixed precision | enabled on GPU |
Evaluation
Results
| Split | n | Base rate | Precision | Recall | F1 (positive class) | ROC-AUC | Average precision |
|---|---|---|---|---|---|---|---|
| dev | 3,296 | 2.2% | 0.438 | 0.194 | 0.269 | 0.727 | 0.196 |
| test | 8,240 | 2.4% | 0.391 | 0.183 | 0.249 | 0.687 | 0.189 |
Limitations
- Labels come from LLM annotators, not human coders. Agreement between annotators with Krippendorff's Alpha is 0.570.
- Trained on teacher utterances only. Behaviour on student speech is untested.
- Test F1 is 0.249. This model does not work well enough to be used on its own.
How to Use
Message Structure
The model was trained on text built as:
{utterance}
The utterance is passed through as-is.
Running instructions
pip install setfit
from setfit import SetFitModel
model = SetFitModel.from_pretrained("StanfordSCALE/assertion_contains_self_correction")
text = 'That wasn t what he was focusing on but I think suppose changed this suppose we took this story and made it a math problem'
model.predict([text]) # -> array([1]) when the assertion holds
model.predict_proba([text]) # -> [[P(no), P(yes)]]
Citation
@misc{assertion_contains_self_correction,
author = {Stanford SCALE Initiative},
title = {Assertion classifier: contains self correction},
year = {2026},
url = {https://huggingface.co/StanfordSCALE/assertion_contains_self_correction}
}