| # Threshold conversion, max-F1, and the multiclass 0.5 rule | |
| **Verdict.** The whiteboard claim is true only as a *reparameterization*: for any chosen threshold `tau`, a strictly increasing affine map sends `tau` to `0.5` and leaves the ROC/F1 curve unchanged. That is the max-F1 conversion. It is false as a claim that one fixed `0.5` rule reproduces multiclass argmax: argmax is not a threshold rule, and for `C >= 3` a single conversion vector cannot put all class thresholds at `0.5`. The max-F1 version is exact per class, but it does not improve F1 and does not replace argmax. | |
| The experiments below separate those two readings. Binary conversions are exact and trivial. Multiclass `0.5` rules are abstention rules, not argmax: on a 20-class random forest, 998 of 1000 test points fall below `0.5`, so the rule answers almost nothing while argmax still has macro-F1 0.5495. | |
| ## 1. The claim | |
| For a multiclass classifier with probabilities `p in Delta^(C-1)`, the prediction is `argmax_c p_c`. The whiteboard proposes a conversion vector `w` such that a `0.5` threshold on the converted score reproduces the classifier's decision, and states that the conversion is chosen to keep the maximum F1. | |
| There are three distinct mathematical objects behind that sentence: | |
| 1. A **per-threshold affine reparameterization**: choose `tau`, map it to `0.5`. | |
| 2. A **per-class** conversion: one map per one-vs-rest problem, so each class's F1-optimal threshold sits at `0.5`. | |
| 3. A **single fixed vector** `w` (dot product, elementwise, or normalized) meant to reproduce argmax for every input. | |
| (1) and (2) are true. (3) is false for `C >= 3`. The whiteboard's own binary example already fails under the literal dot-product reading. | |
| ## 2. Theory | |
| ### 2.1 Binary: every threshold is a 0.5 level set | |
| For `p1 in (0,1)` and `tau in (0,1)`, define | |
| ``` | |
| T_tau(p1) = 1/2 + (p1 - tau) / (2 (1 - tau)). | |
| ``` | |
| `T_tau` is strictly increasing and `T_tau(tau) = 1/2`, hence | |
| ``` | |
| T_tau(p1) >= 1/2 <=> p1 >= tau. | |
| ``` | |
| The ROC curve, AUC, TPR, FPR and F1 at `tau` are unchanged; only the coordinate is rescaled. This is the exact sense in which "the conversion puts the operating point at 0.5". If `tau` maximizes F1 on a validation set, the converted rule at `0.5` has exactly the same validation F1, because it makes exactly the same decisions. It cannot have a higher F1: the decision set is identical. Figure `images/binary_conversion.png` shows the construction. | |
|  | |
| ### 2.2 Max-F1 conversion | |
| For a chosen threshold `tau*` (for example the validation F1 maximizer), the conversion above is the max-F1 conversion. The word "keep" is exact: on the same data, `F1(converted 0.5) = F1(tau*)`. The conversion does not create a better threshold, and it does not make `0.5` special; it moves `tau*` to `0.5`. | |
| For multiclass one-vs-rest, apply `T_{tau_c}` to class `c`, where `tau_c` is that class's F1 maximizer. Each class's `0.5` indicator is exactly its `tau_c` indicator. This requires **C** conversions (a diagonal matrix of scales and biases), not one vector. If one then takes `argmax_c T_{tau_c}(p_c)`, the slopes differ and the argmax can change; this is measured in Section 3.5. | |
| ### 2.3 No fixed 0.5 rule for multiclass argmax | |
| A single affine score `s = w · p + b` has a `0.5` level set that is one hyperplane. Reproducing the argmax regions for `C >= 3` would require the `C` pairwise decision boundaries to coincide with one hyperplane, which they do not. Equivalently, a fixed elementwise conversion `a_c p_c` with a shared threshold has to satisfy `a_c tau_c = 0.5` for all classes; unless all `tau_c` agree, no single `a` does it. | |
| The `0.5` rule on `max_c p_c` is not argmax. It selects the argmax class only when the maximum exceeds `0.5`; otherwise it abstains. It never selects a wrong class, but it loses coverage. Argmax is the total completion of that partial rule. | |
| The exact instance-adaptive fix is | |
| ``` | |
| m = max_c p_c, q_c = p_c / (2 m). | |
| ``` | |
| The winner has `q = 0.5` and every loser has `q < 0.5`, for every `C`. This is exact, but it uses `m`, i.e. the argmax, to define the conversion; it is a representation of argmax, not a fixed conversion vector learned from the classifier. | |
| ### 2.4 How often does the 0.5 rule abstain? | |
| For `p` uniform on the simplex, | |
| ``` | |
| P(max_c p_c < 0.5) = 1 - C / 2^(C-1). | |
| ``` | |
| Values: `C=2: 0`, `C=3: 0.25`, `C=4: 0.5`, `C=5: 0.6875`, `C=6: 0.8125`, `C=10: 0.98046875`, `C=20: 0.9999619`. The Monte Carlo check (40,000 draws per `C`) is in `images/delta_vs_c.png`; the observed values are 0, 24.86%, 49.46%, 68.62%, 80.90%, 98.06%, 99.995%. | |
|  | |
|  | |
| For a symmetric Dirichlet `Dir(alpha)`, with `I_x` the regularized incomplete beta function, | |
| ``` | |
| delta(alpha, C) = 1 - C + C * I_{1/2}(alpha, (C-1) alpha). | |
| ``` | |
| Concentration matters: at `C=3`, `alpha=0.3` gives 6.36%, `alpha=1` gives 25%, `alpha=3` gives 56.64%; at `C=10`, the same values are 65.61%, 98.05%, 99.999%. Uniform probabilities are not the worst case. | |
| ### 2.5 Temperature | |
| Softmax temperature `T > 0` changes `p(T)` but not `argmax`. The `0.5` rule is not invariant: `delta(T)` and every F1 threshold move with `T`. Figure `images/temperature_text.png` and `images/temperature_vision.png` show this. A rule that depends on `0.5` is therefore a statement about calibration, not about the classifier's ranking. | |
| ### 2.6 The whiteboard's numbers | |
| With `p = [0.07, 0.93]` and `w = [0.6, 0.4]`: | |
| - dot product: `0.6*0.07 + 0.4*0.93 = 0.414`, below `0.5`, so a literal dot-product rule predicts class 0; argmax is class 1. | |
| - Hadamard product `[0.042, 0.372]`: argmax is class 1. | |
| - normalized weighted score `[0.1014, 0.8986]`: argmax is class 1. | |
| So the diagram cannot mean the literal dot product `w · p`; it means an elementwise or normalized weighting. Even then, a positive weight vector can change argmax in general: ranking by `w_c p_c` is not ranking by `p_c` unless `w` is constant. The whiteboard's own binary example is internally inconsistent under the dot-product reading. | |
| ## 3. Experiments | |
| All runs use `bash repro/run.sh`, seed 0, and the fixed run command. The core table: | |
| | model | C | n | argmax accuracy | delta at 0.5 | coverage | selective accuracy | ECE (15 bins) | adaptive exact | | |
| |---|---:|---:|---:|---:|---:|---:|---:|---| | |
| | sklearn digits, logistic regression | 10 | 540 | 97.22% | 0.74% | 99.26% | 97.76% | 0.0222 | yes | | |
| | DistilBERT SST-2 | 2 | 872 | 91.06% | 0.00% | 100.00% | 91.06% | 0.0800 | yes | | |
| | Twitter-RoBERTa sentiment | 3 | 2000 | 71.50% | 4.65% | 95.35% | 72.36% | 0.0573 | yes | | |
| | DistilBERT emotion | 6 | 2000 | 92.70% | 0.20% | 99.80% | 92.69% | 0.0385 | yes | | |
| | ViT-B/16 ImageNet head on CIFAR-10 | 1000 | 1000 | n/a | 51.00% | 49.00% | n/a | n/a | yes | | |
| `delta` is `P(max_c p_c < 0.5)` on the test set. Coverage is `1 - delta`. Selective accuracy is accuracy on the covered subset. The uniform-simplex formula is a reference, not a law for trained models: real `delta` depends on confidence, calibration and domain shift. Emotion (`C=6`) has a *smaller* delta than sentiment (`C=3`) because it is much more confident. | |
| ### 3.1 Binary control | |
| SST-2 is exactly the theorem: `delta = 0`, every example has `max p >= 0.5`, and the adaptive conversion matches argmax. The per-class F1-optimal thresholds are 0.0204 (negative) and 0.5249 (positive); at `0.5` the F1s are 0.9071 and 0.9127, at the optima 0.9081 and 0.9137. The conversion moves those thresholds to `0.5`; the decisions and F1 are unchanged. Figure `images/binary_conversion.png`. | |
| ### 3.2 Text: class-specific max-F1 thresholds | |
| The text models are not calibrated to a common `0.5`. Per-class one-vs-rest F1: | |
| | model | class | F1 @0.5 | F1 @tau* | tau* | | |
| |---|---|---:|---:|---:| | |
| | Twitter-RoBERTa | negative | 0.715 | 0.719 | 0.448 | | |
| | Twitter-RoBERTa | neutral | 0.693 | 0.758 | 0.231 | | |
| | Twitter-RoBERTa | positive | 0.725 | 0.726 | 0.479 | | |
| | DistilBERT emotion | sadness | 0.967 | 0.973 | 0.103 | | |
| | DistilBERT emotion | joy | 0.946 | 0.951 | 0.961 | | |
| | DistilBERT emotion | love | 0.829 | 0.860 | 0.192 | | |
| | DistilBERT emotion | anger | 0.923 | 0.931 | 0.041 | | |
| | DistilBERT emotion | fear | 0.890 | 0.902 | 0.098 | | |
| | DistilBERT emotion | surprise | 0.738 | 0.803 | 0.344 | | |
| The thresholds are not close to `0.5`, and they are not equal to each other (`0.041` to `0.961` for emotion). A single `0.5` rule cannot be simultaneously F1-optimal for all six classes. The per-class conversion can put each `tau*` at `0.5`, but that is six binary rules. The ROC curves are in `images/roc_text.png`; the model-level deltas are in `images/model_delta_text.png`. | |
|  | |
|  | |
| Temperature sweep (same argmax accuracy at every `T`, because argmax is invariant): | |
| | model | T=0.5 delta | T=1 delta | T=2 delta | T=4 delta | | |
| |---|---:|---:|---:|---:| | |
| | SST-2 | 0.00% | 0.00% | 0.00% | 0.00% | | |
| | sentiment | 0.55% | 4.65% | 19.10% | 62.20% | | |
| | emotion | 0.00% | 0.20% | 2.90% | 14.15% | | |
| This is the cleanest falsification of "0.5 is intrinsic": the classifier's ranking is unchanged, but the `0.5` rule's behavior changes from 0.2% abstention to 14% (emotion) or from 4.65% to 62% (sentiment). Figure `images/temperature_text.png`. | |
|  | |
| ### 3.3 Vision: high C is not the whole story | |
| ViT-B/16 has a 1000-class ImageNet head; the images are CIFAR-10, so this is out-of-domain. At `T=1`, `delta = 51.00%`, coverage 49.00%, mean max probability 0.523, median 0.489. The uniform 1000-simplex reference would predict 100% abstention, but the model is not uniform: half the images have a class above `0.5`. Temperature again moves the rule: `T=0.5` delta 12.7%, `T=2` 99.5%. Figure `images/temperature_vision.png`. | |
|  | |
| This is still a strong practical result: a 1000-way classifier on shifted images loses half its coverage to a `0.5` rule, while argmax always answers. | |
| ### 3.4 A fixed map can mimic argmax, but that is not the conversion | |
| A supervised `C x C` linear map fitted to argmax labels gets: | |
| | data | train | test | chance | | |
| |---|---:|---:|---:| | |
| | SST-2 | 100.00% | 100.00% | 50.00% | | |
| | sentiment | 99.14% | 99.33% | 33.33% | | |
| | emotion | 99.50% | 99.83% | 16.67% | | |
| This is a classifier trained to reproduce argmax, not a `0.5` threshold rule. It reintroduces argmax as the target. The exact fixed elementwise conversion for `C=3` is much weaker: best `a = [1.4, 1.4, 1.0]`, agreement 89.5%, 4.0% wrong selected, 4.3% abstain, versus identity delta 24.66%. A fixed conversion can approximate the decision regions but cannot be exact for all `p`. | |
| ### 3.5 Tabular: 30 dataset/classifier pairs, max-F1 conversion | |
| We ran logistic regression, random forest, histogram gradient boosting, Gaussian NB and kNN on breast cancer, wine, iris, digits, synthetic 5-class and synthetic 20-class data (30 pairs; 5 binary, 25 multiclass). The threshold `tau*` was selected on a validation split and evaluated on a held-out test split. For each pair we computed `T_{tau*}` and checked the converted `0.5` decisions. | |
| **Exactness.** All 30 conversions are exact: `conversion_exact = True`, and `F1(test, tau*) = F1(test, converted 0.5)` to floating precision. This is the max-F1 conversion doing exactly what it claims: it preserves the selected threshold, not the test-optimal threshold. | |
| **The selected threshold is not 0.5.** Binary `tau*` ranges from 0.4819 (breast cancer/logreg) to 0.9984 (breast cancer/Gaussian NB). Multiclass per-class `tau*` spans up to 1.0, with mean spread 0.479. Figure `images/tabular_tau.png` shows every class threshold; the red line at `0.5` is rarely where the classes sit. | |
|  | |
| **Binary pairs.** | |
| | dataset / classifier | tau* | F1val @0.5 | F1val @tau* | F1test @0.5 | F1test @tau* | | |
| |---|---:|---:|---:|---:|---:| | |
| | breast cancer / logreg | 0.4819 | 0.9888 | 0.9944 | 0.9775 | 0.9775 | | |
| | breast cancer / random forest | 0.5400 | 0.9551 | 0.9714 | 0.9551 | 0.9492 | | |
| | breast cancer / hist-GB | 0.5280 | 0.9832 | 0.9832 | 0.9609 | 0.9609 | | |
| | breast cancer / Gaussian NB | 0.9984 | 0.9392 | 0.9655 | 0.9497 | 0.9425 | | |
| | breast cancer / kNN | 0.6655 | 0.9674 | 0.9724 | 0.9727 | 0.9600 | | |
| **Selected multiclass pairs.** | |
| | dataset / classifier | tau* range | macro-F1val @0.5 | macro-F1val @tau* | macro-F1test @0.5 | macro-F1test @tau* | argmax F1 | converted-argmax F1 | agreement | | |
| |---|---:|---:|---:|---:|---:|---:|---:|---:| | |
| | digits / logreg | 0.0748-0.8163 | 0.9605 | 0.9769 | 0.9606 | 0.9571 | 0.9647 | 0.9561 | 97.8% | | |
| | digits / random forest | 0.2467-0.5300 | 0.9111 | 0.9787 | 0.9229 | 0.9698 | 0.9778 | 0.9626 | 97.3% | | |
| | synthetic_5 / random forest | 0.3033-0.4267 | 0.7990 | 0.8749 | 0.7829 | 0.8454 | 0.8600 | 0.8524 | 96.7% | | |
| | synthetic_20 / random forest | 0.1167-0.1867 | 0.0133 | 0.6046 | 0.0040 | 0.5415 | 0.5495 | 0.5585 | 87.2% | | |
| | synthetic_20 / kNN | 0.1365-0.2667 | 0.0490 | 0.4946 | 0.0797 | 0.4332 | 0.4512 | 0.4184 | 76.9% | | |
| **F1 on validation is preserved; F1 on test can move either way.** On validation the conversion moves `0.5` to the F1 maximum by construction. On test, comparing the converted rule to the raw `0.5` rule, 13 pairs improve and 15 worsen. For the five binary pairs, three are lower on test, two equal, none higher. The changes are threshold-selection/generalization effects, not a benefit of the conversion. For example, iris/random forest has validation F1 1.000 at `tau*` but test F1 0.8006 after conversion, versus 0.8932 at raw `0.5`; the validation split is tiny. Conversely, digits/random forest improves from 0.9229 to 0.9698 on test. Figure `images/tabular_f1.png`. | |
|  | |
| **The 20-class cases make the distinction sharp.** In the next table, `macro-F1 at 0.5` is the mean of the C one-vs-rest F1 scores obtained by thresholding each class at `0.5` independently. It is not the macro-F1 of argmax; the argmax F1 is its own column. The independent-threshold rule can select zero or several classes, which is exactly why it is not argmax. | |
| | model | delta at 0.5 | per-class macro-F1 at 0.5 | argmax F1 | converted-argmax F1 | argmax agreement | | |
| |---|---:|---:|---:|---:|---:| | |
| | synthetic_20 / random forest | 99.80% | 0.0040 | 0.5495 | 0.5585 | 87.2% | | |
| | synthetic_20 / kNN | 95.30% | 0.0797 | 0.4512 | 0.4184 | 76.9% | | |
| | synthetic_20 / logreg | 65.20% | 0.2670 | 0.3850 | 0.3850 | 85.2% | | |
| For the 20-class random forest, the `0.5` rule abstains on 998 of 1000 test points and has macro-F1 0.004. Argmax has F1 0.5495 without abstaining. Per-class max-F1 conversions give each class a `0.5` threshold, but `argmax` over the converted scores agrees with the original argmax only 87.2% of the time, and its F1 is 0.5585, not a recovery of the original rule. The conversion is a different decision rule, not an equivalent one. | |
| ## 4. What is true, and what is false | |
| **True.** | |
| - For any binary threshold `tau`, `T_tau` is a strictly increasing affine conversion with `T_tau(tau)=0.5`; the ROC/AUC and F1 at `tau` are unchanged. | |
| - For any multiclass one-vs-rest problem, the same is true per class; this is the max-F1 conversion, and it is exact. | |
| - The instance-adaptive `q = p/(2 max p)` makes the winner exactly `0.5` and every loser below `0.5`, for all `C`. | |
| - The `0.5` rule on `max p` never picks a wrong class; it only abstains. Argmax is its total completion. | |
| **False.** | |
| - That a single fixed conversion vector can reproduce multiclass argmax for `C >= 3`. | |
| - That a `0.5` threshold on a single score is equivalent to argmax in general. | |
| - That the max-F1 conversion improves F1 or makes `0.5` the "correct" threshold. It is a reparameterization; the selected threshold is model- and data-dependent. | |
| - That high `C` alone implies `delta ~ 1`. Under uniform probabilities it does (98% at `C=10`), but trained models can be confident (emotion 0.20%) or unconfident (ViT-1000 51%). | |
| **The practical rule.** If you want max F1 for a binary classifier, select `tau*` and either use `tau*` directly or convert it to `0.5`; the two are identical. If you want a single label for multiclass, use argmax. If you want per-class operating points, use C one-vs-rest thresholds (and accept that the result is a set of binary decisions, not one label). A confidence threshold at `0.5` is an abstention rule, and its coverage depends on calibration and domain. | |
| ## 5. Reproduction | |
| All code is in `repro/` on each branch. The fixed command is `bash repro/run.sh`; every run starts from the branch tip on GitHub. | |
| | experiment | id | branch | run | | |
| |---|---|---|---| | |
| | baseline theory + digits | `8a044950-bedc-4084-8507-75e4555d9650` | `orx/threshold-conversion-baseline-theory-and-simplex` | `cff2a16d-f190-42ed-8510-38d21ec359ad` | | |
| | text | `9512af43-d119-4b35-b9d0-b11b727b28e3` | `orx/real-text-classifiers-2-3-and-6-classes` | `18fb96ec-2a40-43dc-ac78-6343bdc067aa` | | |
| | vision | `493a2ee5-43dd-4e01-b541-3aedf0bbab4f` | `orx/1000-class-vision-vit-b-16-on-cifar-10` | `a5f6fc9f-88fb-415e-92b1-fc064fa95d96` | | |
| | fixed fit | `3ab3fa4c-eab3-4cc1-96f9-126f68d0a24a` | `orx/can-a-fitted-fixed-conversion-replace-the-adapti` | `6af95ffd-982c-45ca-b87c-65f817fb1aef` | | |
| | tabular max-F1 | `b117d2f9-70a2-4eb7-82cd-98cd1f99e632` | `orx/tabular-classifiers-and-the-max-f1-0-5-conversio` | `576b9f04-e295-4300-b42b-de57f96a54bb` | | |
| Launch with `orx exp run <id> --backend local`. Logs are the evidence channel: `orx logs <runId>`. Artifacts: `hf://buckets/jomasego/repro-threshold-conversion-artifacts` (public). | |
| Environment: macOS 26.7.1 arm64, Apple M4 Max, Python 3.9.6, torch 2.8.0 (MPS), transformers 4.57.6, datasets 3.6.0, scikit-learn 1.5.2, numpy 2.0.2, scipy 1.13.1. | |
| ### Links | |
| - Upstream: https://github.com/isllabres/python-multiagent-system | |
| - Fork: https://github.com/jomasego/python-multiagent-system | |
| - Branch (baseline): https://github.com/jomasego/python-multiagent-system/tree/orx/threshold-conversion-baseline-theory-and-simplex | |
| - Branch (text): https://github.com/jomasego/python-multiagent-system/tree/orx/real-text-classifiers-2-3-and-6-classes | |
| - Branch (vision): https://github.com/jomasego/python-multiagent-system/tree/orx/1000-class-vision-vit-b-16-on-cifar-10 | |
| - Branch (fixed fit): https://github.com/jomasego/python-multiagent-system/tree/orx/can-a-fitted-fixed-conversion-replace-the-adapti | |
| - Branch (tabular): https://github.com/jomasego/python-multiagent-system/tree/orx/tabular-classifiers-and-the-max-f1-0-5-conversio | |
| - Bucket: https://huggingface.co/buckets/jomasego/repro-threshold-conversion-artifacts | |
| - Models: https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment-latest , https://huggingface.co/bhadresh-savani/distilbert-base-uncased-emotion , https://huggingface.co/google/vit-base-patch16-224 , https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english | |
| - Datasets: https://huggingface.co/datasets/cardiffnlp/tweet_eval , https://huggingface.co/datasets/dair-ai/emotion , https://huggingface.co/datasets/uoft-cs/cifar10 , https://huggingface.co/datasets/stanfordnlp/sst2 | |
| - sklearn digits: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html | |
| **HF Job.** A text run on `--backend hf --flavor cpu-upgrade --image python:3.12` was attempted from branch `orx/real-text-classifiers-2-3-and-6-classes`. The submit was rejected: | |
| ``` | |
| Hugging Face job submit failed (402): {"error":"Pre-paid credit balance is insufficient - add more credits to your account to use Jobs."} | |
| ``` | |
| No Job URL exists. The local CPU/MPS runs are the evidence. No ZeroGPU Space was used; no Modal token was available. | |
| ## 6. Limitations | |
| - The vision test is out-of-domain (CIFAR-10 images through an ImageNet head), so its 51% delta is not a clean `C`-scaling measurement. It is a calibration/domain-shift measurement. | |
| - Text evaluation uses the SST-2 validation split (872 examples) and the first 2000 test examples for sentiment/emotion; it is not a full benchmark. | |
| - Tabular iris/wine test splits are 38-45 points, and threshold selection on a 37-75 point validation split overfits. The test F1 changes should be read as noise, not as reliable ranking. | |
| - One seed; no confidence intervals. The exactness statements are mathematical and deterministic; the F1 comparisons are not. | |
| - The fixed `C x C` map is supervised on argmax labels, so its near-perfect accuracy is expected and is not evidence for the conversion claim. | |
| - HF Jobs were blocked by the account's pre-paid balance; no Job link can be provided. | |
| ## 7. Files | |
| - `results.json` / `tabular_results.json` in the bucket: all metrics, per-class thresholds, temperature sweeps. | |
| - `summary.md` and `tabular_summary.md`: per-run tables. | |
| - `images/`: all figures referenced here. | |
| - `logbook.ipynb`: the same results as an executable notebook, with cells for the theory, the measured tables and every link. | |
Xet Storage Details
- Size:
- 20.8 kB
- Xet hash:
- eb23fc043050a4e30973d40f92638aa348d7af7fee43773ed4e87830a9782128
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.