On-Task Participation Detector
Detects whether a student is on-task based on their daily check-ins.
Model Details
- Embedder:
sentence-transformers/all-MiniLM-L6-v2 - Classifier: XGBoost
- Threshold: 0.40
- AUC-ROC: 0.8483
- CV Mean AUC: 0.8030 ± 0.0903
- Training data: 2,296 labeled check-in pairs
What it detects
Given a student's yesterday and today check-in, predicts whether the student is actively participating and on-task.
ON-TASK means:
- Work is connected to or progressing from yesterday
- Check-in is specific enough to show real work happened
- Non-technical work (meetings, community work) counts if participation is clear
NOT ON-TASK means:
- Copy-paste of yesterday's check-in
- Unexplained project jumping
- Purely future tense with no evidence of work done
- Too vague to verify participation
Usage
from sentence_transformers import SentenceTransformer
import joblib, json
embedder = joblib.load("embedder.pkl")
clf = joblib.load("classifier.pkl")
metadata = json.load(open("metadata.json"))
THRESHOLD = metadata["threshold"]
def predict_on_task(yesterday: str, today: str) -> dict:
text = f"Yesterday: {yesterday} Today: {today}"
emb = embedder.encode([text])
proba = clf.predict_proba(emb)[0][1]
return {
"on_task": bool(proba >= THRESHOLD),
"confidence": round(float(proba), 3)
}