text/summary.md

Threshold-conversion reproduction: raw summary

• generated: 2026-10-20T21:43:52.319972+00:00
• upstream repo: https://github.com/isllabres/python-multiagent-system
• execution fork: https://github.com/jomasego/python-multiagent-system
• branch: orx/real-text-classifiers-2-3-and-6-classes
• commit: 48269177ba20f5b97e496ab5bcd71c5513461f6b
• device: auto (text=cpu, vision=mps if available, digits=cpu)

Theory

Models

model	C	n	argmax acc	delta_0.5	coverage	selective acc	ECE	exact adaptive
binary control: DistilBERT SST-2	2	872	91.06%	0.00%	100.00%	91.06%	0.0800	True
3-class sentiment: Twitter-RoBERTa	3	2000	71.50%	4.65%	95.35%	72.36%	0.0573	True
6-class emotion: DistilBERT emotion	6	2000	92.70%	0.20%	99.80%	92.69%	0.0385	True

Temperature sweep

model	T	delta_0.5	argmax acc	ECE
binary control: DistilBERT SST-2	0.1	0.00%	91.06%	0.0906
binary control: DistilBERT SST-2	0.25	0.00%	91.06%	0.0905
binary control: DistilBERT SST-2	0.5	0.00%	91.06%	0.0876
binary control: DistilBERT SST-2	1.0	0.00%	91.06%	0.0800
binary control: DistilBERT SST-2	2.0	0.00%	91.06%	0.0412
binary control: DistilBERT SST-2	4.0	0.00%	91.06%	0.0931
binary control: DistilBERT SST-2	8.0	0.00%	91.06%	0.2117
3-class sentiment: Twitter-RoBERTa	0.1	0.00%	71.50%	0.2625
3-class sentiment: Twitter-RoBERTa	0.25	0.10%	71.50%	0.2248
3-class sentiment: Twitter-RoBERTa	0.5	0.55%	71.50%	0.1587
3-class sentiment: Twitter-RoBERTa	1.0	4.65%	71.50%	0.0573
3-class sentiment: Twitter-RoBERTa	2.0	19.10%	71.50%	0.1086
3-class sentiment: Twitter-RoBERTa	4.0	62.20%	71.50%	0.2329
3-class sentiment: Twitter-RoBERTa	8.0	99.70%	71.50%	0.3062
6-class emotion: DistilBERT emotion	0.1	0.00%	92.70%	0.0700
6-class emotion: DistilBERT emotion	0.25	0.00%	92.70%	0.0627
6-class emotion: DistilBERT emotion	0.5	0.00%	92.70%	0.0533
6-class emotion: DistilBERT emotion	1.0	0.20%	92.70%	0.0385
6-class emotion: DistilBERT emotion	2.0	2.90%	92.70%	0.0830
6-class emotion: DistilBERT emotion	4.0	14.15%	92.70%	0.3522
6-class emotion: DistilBERT emotion	8.0	100.00%	92.70%	0.5820

Per-class ROC operating points (0.5 vs best F1)

model	class	AUC	TPR@0.5	FPR@0.5	best F1 thr	best F1
binary control: DistilBERT SST-2	NEGATIVE	0.972	0.890	0.070	0.020	0.908
binary control: DistilBERT SST-2	POSITIVE	0.972	0.930	0.112	0.525	0.914
3-class sentiment: Twitter-RoBERTa	negative	0.891	0.773	0.173	0.448	0.719
3-class sentiment: Twitter-RoBERTa	neutral	0.815	0.629	0.184	0.231	0.758
3-class sentiment: Twitter-RoBERTa	positive	0.936	0.717	0.063	0.479	0.726
6-class emotion: DistilBERT emotion	sadness	0.999	0.966	0.013	0.103	0.973
6-class emotion: DistilBERT emotion	joy	0.997	0.942	0.027	0.961	0.951
6-class emotion: DistilBERT emotion	love	0.993	0.855	0.018	0.192	0.860
6-class emotion: DistilBERT emotion	anger	0.996	0.920	0.012	0.041	0.931
6-class emotion: DistilBERT emotion	fear	0.996	0.906	0.016	0.098	0.902
6-class emotion: DistilBERT emotion	surprise	0.994	0.682	0.006	0.344	0.803