| # Threshold-conversion reproduction: raw summary | |
| - generated: 2026-10-20T21:43:52.319972+00:00 | |
| - upstream repo: https://github.com/isllabres/python-multiagent-system | |
| - execution fork: https://github.com/jomasego/python-multiagent-system | |
| - branch: `orx/real-text-classifiers-2-3-and-6-classes` | |
| - commit: `48269177ba20f5b97e496ab5bcd71c5513461f6b` | |
| - device: auto (text=cpu, vision=mps if available, digits=cpu) | |
| ## Theory | |
| ## Models | |
| | model | C | n | argmax acc | delta_0.5 | coverage | selective acc | ECE | exact adaptive | | |
| |---|---:|---:|---:|---:|---:|---:|---:|---:| | |
| | binary control: DistilBERT SST-2 | 2 | 872 | 91.06% | 0.00% | 100.00% | 91.06% | 0.0800 | True | | |
| | 3-class sentiment: Twitter-RoBERTa | 3 | 2000 | 71.50% | 4.65% | 95.35% | 72.36% | 0.0573 | True | | |
| | 6-class emotion: DistilBERT emotion | 6 | 2000 | 92.70% | 0.20% | 99.80% | 92.69% | 0.0385 | True | | |
| ### Temperature sweep | |
| | model | T | delta_0.5 | argmax acc | ECE | | |
| |---|---:|---:|---:|---:| | |
| | binary control: DistilBERT SST-2 | 0.1 | 0.00% | 91.06% | 0.0906 | | |
| | binary control: DistilBERT SST-2 | 0.25 | 0.00% | 91.06% | 0.0905 | | |
| | binary control: DistilBERT SST-2 | 0.5 | 0.00% | 91.06% | 0.0876 | | |
| | binary control: DistilBERT SST-2 | 1.0 | 0.00% | 91.06% | 0.0800 | | |
| | binary control: DistilBERT SST-2 | 2.0 | 0.00% | 91.06% | 0.0412 | | |
| | binary control: DistilBERT SST-2 | 4.0 | 0.00% | 91.06% | 0.0931 | | |
| | binary control: DistilBERT SST-2 | 8.0 | 0.00% | 91.06% | 0.2117 | | |
| | 3-class sentiment: Twitter-RoBERTa | 0.1 | 0.00% | 71.50% | 0.2625 | | |
| | 3-class sentiment: Twitter-RoBERTa | 0.25 | 0.10% | 71.50% | 0.2248 | | |
| | 3-class sentiment: Twitter-RoBERTa | 0.5 | 0.55% | 71.50% | 0.1587 | | |
| | 3-class sentiment: Twitter-RoBERTa | 1.0 | 4.65% | 71.50% | 0.0573 | | |
| | 3-class sentiment: Twitter-RoBERTa | 2.0 | 19.10% | 71.50% | 0.1086 | | |
| | 3-class sentiment: Twitter-RoBERTa | 4.0 | 62.20% | 71.50% | 0.2329 | | |
| | 3-class sentiment: Twitter-RoBERTa | 8.0 | 99.70% | 71.50% | 0.3062 | | |
| | 6-class emotion: DistilBERT emotion | 0.1 | 0.00% | 92.70% | 0.0700 | | |
| | 6-class emotion: DistilBERT emotion | 0.25 | 0.00% | 92.70% | 0.0627 | | |
| | 6-class emotion: DistilBERT emotion | 0.5 | 0.00% | 92.70% | 0.0533 | | |
| | 6-class emotion: DistilBERT emotion | 1.0 | 0.20% | 92.70% | 0.0385 | | |
| | 6-class emotion: DistilBERT emotion | 2.0 | 2.90% | 92.70% | 0.0830 | | |
| | 6-class emotion: DistilBERT emotion | 4.0 | 14.15% | 92.70% | 0.3522 | | |
| | 6-class emotion: DistilBERT emotion | 8.0 | 100.00% | 92.70% | 0.5820 | | |
| ### Per-class ROC operating points (0.5 vs best F1) | |
| | model | class | AUC | TPR@0.5 | FPR@0.5 | best F1 thr | best F1 | | |
| |---|---|---:|---:|---:|---:|---:| | |
| | binary control: DistilBERT SST-2 | NEGATIVE | 0.972 | 0.890 | 0.070 | 0.020 | 0.908 | | |
| | binary control: DistilBERT SST-2 | POSITIVE | 0.972 | 0.930 | 0.112 | 0.525 | 0.914 | | |
| | 3-class sentiment: Twitter-RoBERTa | negative | 0.891 | 0.773 | 0.173 | 0.448 | 0.719 | | |
| | 3-class sentiment: Twitter-RoBERTa | neutral | 0.815 | 0.629 | 0.184 | 0.231 | 0.758 | | |
| | 3-class sentiment: Twitter-RoBERTa | positive | 0.936 | 0.717 | 0.063 | 0.479 | 0.726 | | |
| | 6-class emotion: DistilBERT emotion | sadness | 0.999 | 0.966 | 0.013 | 0.103 | 0.973 | | |
| | 6-class emotion: DistilBERT emotion | joy | 0.997 | 0.942 | 0.027 | 0.961 | 0.951 | | |
| | 6-class emotion: DistilBERT emotion | love | 0.993 | 0.855 | 0.018 | 0.192 | 0.860 | | |
| | 6-class emotion: DistilBERT emotion | anger | 0.996 | 0.920 | 0.012 | 0.041 | 0.931 | | |
| | 6-class emotion: DistilBERT emotion | fear | 0.996 | 0.906 | 0.016 | 0.098 | 0.902 | | |
| | 6-class emotion: DistilBERT emotion | surprise | 0.994 | 0.682 | 0.006 | 0.344 | 0.803 | | |
Xet Storage Details
- Size:
- 3.59 kB
- Xet hash:
- aa67ed759bd779546d43c75646b5c10f0ebe3e5032e54516e507b0be6b2e4e92
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.