Barist.1 / reports /comparison.md
KucLab's picture
Add report comparison.md
b6d3ae7 verified
|
Raw History Blame Contribute Delete
1.88 kB

Comparison with other public safety classifiers

Same held-out test split for every model: data/processed/test.jsonl (n=6496), threshold 0.5. Hardware: AMD Radeon Graphics.

Model Params Checkpoint Accuracy Precision Recall F1 Latency Throughput (b=32) Peak VRAM
Censer.5 (prev) 0.9M 3.6 MB 0.876 0.864 0.834 0.849 1.35 ms 1828 texts/s 97 MB
Censer.4 (old) 0.9M 3.6 MB 0.874 0.868 0.824 0.845 1.39 ms 1708 texts/s 95 MB
Censer.1 (old) 1.1M 4.5 MB 0.865 0.848 0.823 0.835 1.68 ms 1697 texts/s 97 MB
Censer.2 (prev) 7.3M 29.4 MB 0.863 0.861 0.799 0.829 2.77 ms 551 texts/s 171 MB
Censer.9 (this work) 0.9M 3.6 MB 0.858 0.851 0.799 0.824 1.39 ms 1860 texts/s 105 MB
Censer.3 (old) 14.4M 57.7 MB 0.857 0.855 0.792 0.822 4.36 ms 270 texts/s 240 MB
unbiased-toxic-roberta 124.7M 0.0 MB 0.822 0.940 0.612 0.741 14.85 ms 65 texts/s 771 MB
toxic-bert 109.5M 0.0 MB 0.794 0.965 0.526 0.681 14.08 ms 68 texts/s 708 MB
toxic-comment-model 67.0M 0.0 MB 0.744 0.961 0.402 0.566 7.81 ms 129 texts/s 540 MB
bert-base-uncased-hatexplain 109.5M 0.0 MB 0.704 0.761 0.423 0.543 14.15 ms 67 texts/s 692 MB
dehatebert-mono-english 167.4M 0.0 MB 0.636 0.875 0.147 0.252 14.50 ms 66 texts/s 923 MB
keyword baseline 0.0M 0.0 MB 0.568 0.374 0.054 0.094 0.03 ms 29801 texts/s 0 MB

overview f1 vs size metrics

Reproduce: .venv/bin/python scripts/compare_models.py && .venv/bin/python scripts/make_comparison_figs.py