report.md

Threshold conversion, max-F1, and the multiclass 0.5 rule

Verdict. The whiteboard claim is true only as a reparameterization: for any chosen threshold tau, a strictly increasing affine map sends tau to 0.5 and leaves the ROC/F1 curve unchanged. That is the max-F1 conversion. It is false as a claim that one fixed 0.5 rule reproduces multiclass argmax: argmax is not a threshold rule, and for C >= 3 a single conversion vector cannot put all class thresholds at 0.5. The max-F1 version is exact per class, but it does not improve F1 and does not replace argmax.

The experiments below separate those two readings. Binary conversions are exact and trivial. Multiclass 0.5 rules are abstention rules, not argmax: on a 20-class random forest, 998 of 1000 test points fall below 0.5, so the rule answers almost nothing while argmax still has macro-F1 0.5495.

1. The claim

For a multiclass classifier with probabilities p in Delta^(C-1), the prediction is argmax_c p_c. The whiteboard proposes a conversion vector w such that a 0.5 threshold on the converted score reproduces the classifier's decision, and states that the conversion is chosen to keep the maximum F1.

There are three distinct mathematical objects behind that sentence:

1. A per-threshold affine reparameterization: choose tau, map it to 0.5.
2. A per-class conversion: one map per one-vs-rest problem, so each class's F1-optimal threshold sits at 0.5.
3. A single fixed vector w (dot product, elementwise, or normalized) meant to reproduce argmax for every input.

(1) and (2) are true. (3) is false for C >= 3. The whiteboard's own binary example already fails under the literal dot-product reading.

2. Theory

2.1 Binary: every threshold is a 0.5 level set

For p1 in (0,1) and tau in (0,1), define

T_tau(p1) = 1/2 + (p1 - tau) / (2 (1 - tau)).

T_tau is strictly increasing and T_tau(tau) = 1/2, hence

T_tau(p1) >= 1/2  <=>  p1 >= tau.

The ROC curve, AUC, TPR, FPR and F1 at tau are unchanged; only the coordinate is rescaled. This is the exact sense in which "the conversion puts the operating point at 0.5". If tau maximizes F1 on a validation set, the converted rule at 0.5 has exactly the same validation F1, because it makes exactly the same decisions. It cannot have a higher F1: the decision set is identical. Figure images/binary_conversion.png shows the construction.

2.2 Max-F1 conversion

For a chosen threshold tau* (for example the validation F1 maximizer), the conversion above is the max-F1 conversion. The word "keep" is exact: on the same data, F1(converted 0.5) = F1(tau*). The conversion does not create a better threshold, and it does not make 0.5 special; it moves tau* to 0.5.

For multiclass one-vs-rest, apply T_{tau_c} to class c, where tau_c is that class's F1 maximizer. Each class's 0.5 indicator is exactly its tau_c indicator. This requires C conversions (a diagonal matrix of scales and biases), not one vector. If one then takes argmax_c T_{tau_c}(p_c), the slopes differ and the argmax can change; this is measured in Section 3.5.

2.3 No fixed 0.5 rule for multiclass argmax

A single affine score s = w · p + b has a 0.5 level set that is one hyperplane. Reproducing the argmax regions for C >= 3 would require the C pairwise decision boundaries to coincide with one hyperplane, which they do not. Equivalently, a fixed elementwise conversion a_c p_c with a shared threshold has to satisfy a_c tau_c = 0.5 for all classes; unless all tau_c agree, no single a does it.

The 0.5 rule on max_c p_c is not argmax. It selects the argmax class only when the maximum exceeds 0.5; otherwise it abstains. It never selects a wrong class, but it loses coverage. Argmax is the total completion of that partial rule.

The exact instance-adaptive fix is

m = max_c p_c,    q_c = p_c / (2 m).

The winner has q = 0.5 and every loser has q < 0.5, for every C. This is exact, but it uses m, i.e. the argmax, to define the conversion; it is a representation of argmax, not a fixed conversion vector learned from the classifier.

2.4 How often does the 0.5 rule abstain?

For p uniform on the simplex,

P(max_c p_c < 0.5) = 1 - C / 2^(C-1).

Values: C=2: 0, C=3: 0.25, C=4: 0.5, C=5: 0.6875, C=6: 0.8125, C=10: 0.98046875, C=20: 0.9999619. The Monte Carlo check (40,000 draws per C) is in images/delta_vs_c.png; the observed values are 0, 24.86%, 49.46%, 68.62%, 80.90%, 98.06%, 99.995%.

For a symmetric Dirichlet Dir(alpha), with I_x the regularized incomplete beta function,

delta(alpha, C) = 1 - C + C * I_{1/2}(alpha, (C-1) alpha).

Concentration matters: at C=3, alpha=0.3 gives 6.36%, alpha=1 gives 25%, alpha=3 gives 56.64%; at C=10, the same values are 65.61%, 98.05%, 99.999%. Uniform probabilities are not the worst case.

2.5 Temperature

Softmax temperature T > 0 changes p(T) but not argmax. The 0.5 rule is not invariant: delta(T) and every F1 threshold move with T. Figure images/temperature_text.png and images/temperature_vision.png show this. A rule that depends on 0.5 is therefore a statement about calibration, not about the classifier's ranking.

2.6 The whiteboard's numbers

With p = [0.07, 0.93] and w = [0.6, 0.4]:

• dot product: 0.6*0.07 + 0.4*0.93 = 0.414, below 0.5, so a literal dot-product rule predicts class 0; argmax is class 1.
• Hadamard product [0.042, 0.372]: argmax is class 1.
• normalized weighted score [0.1014, 0.8986]: argmax is class 1.

So the diagram cannot mean the literal dot product w · p; it means an elementwise or normalized weighting. Even then, a positive weight vector can change argmax in general: ranking by w_c p_c is not ranking by p_c unless w is constant. The whiteboard's own binary example is internally inconsistent under the dot-product reading.

3. Experiments

All runs use bash repro/run.sh, seed 0, and the fixed run command. The core table:

model	C	n	argmax accuracy	delta at 0.5	coverage	selective accuracy	ECE (15 bins)	adaptive exact
sklearn digits, logistic regression	10	540	97.22%	0.74%	99.26%	97.76%	0.0222	yes
DistilBERT SST-2	2	872	91.06%	0.00%	100.00%	91.06%	0.0800	yes
Twitter-RoBERTa sentiment	3	2000	71.50%	4.65%	95.35%	72.36%	0.0573	yes
DistilBERT emotion	6	2000	92.70%	0.20%	99.80%	92.69%	0.0385	yes
ViT-B/16 ImageNet head on CIFAR-10	1000	1000	n/a	51.00%	49.00%	n/a	n/a	yes

delta is P(max_c p_c < 0.5) on the test set. Coverage is 1 - delta. Selective accuracy is accuracy on the covered subset. The uniform-simplex formula is a reference, not a law for trained models: real delta depends on confidence, calibration and domain shift. Emotion (C=6) has a smaller delta than sentiment (C=3) because it is much more confident.

3.1 Binary control

SST-2 is exactly the theorem: delta = 0, every example has max p >= 0.5, and the adaptive conversion matches argmax. The per-class F1-optimal thresholds are 0.0204 (negative) and 0.5249 (positive); at 0.5 the F1s are 0.9071 and 0.9127, at the optima 0.9081 and 0.9137. The conversion moves those thresholds to 0.5; the decisions and F1 are unchanged. Figure images/binary_conversion.png.

3.2 Text: class-specific max-F1 thresholds

The text models are not calibrated to a common 0.5. Per-class one-vs-rest F1:

model	class	F1 @0.5	F1 @tau*	tau*
Twitter-RoBERTa	negative	0.715	0.719	0.448
Twitter-RoBERTa	neutral	0.693	0.758	0.231
Twitter-RoBERTa	positive	0.725	0.726	0.479
DistilBERT emotion	sadness	0.967	0.973	0.103
DistilBERT emotion	joy	0.946	0.951	0.961
DistilBERT emotion	love	0.829	0.860	0.192
DistilBERT emotion	anger	0.923	0.931	0.041
DistilBERT emotion	fear	0.890	0.902	0.098
DistilBERT emotion	surprise	0.738	0.803	0.344

The thresholds are not close to 0.5, and they are not equal to each other (0.041 to 0.961 for emotion). A single 0.5 rule cannot be simultaneously F1-optimal for all six classes. The per-class conversion can put each tau* at 0.5, but that is six binary rules. The ROC curves are in images/roc_text.png; the model-level deltas are in images/model_delta_text.png.

Temperature sweep (same argmax accuracy at every T, because argmax is invariant):

model	T=0.5 delta	T=1 delta	T=2 delta	T=4 delta
SST-2	0.00%	0.00%	0.00%	0.00%
sentiment	0.55%	4.65%	19.10%	62.20%
emotion	0.00%	0.20%	2.90%	14.15%

This is the cleanest falsification of "0.5 is intrinsic": the classifier's ranking is unchanged, but the 0.5 rule's behavior changes from 0.2% abstention to 14% (emotion) or from 4.65% to 62% (sentiment). Figure images/temperature_text.png.

3.3 Vision: high C is not the whole story

ViT-B/16 has a 1000-class ImageNet head; the images are CIFAR-10, so this is out-of-domain. At T=1, delta = 51.00%, coverage 49.00%, mean max probability 0.523, median 0.489. The uniform 1000-simplex reference would predict 100% abstention, but the model is not uniform: half the images have a class above 0.5. Temperature again moves the rule: T=0.5 delta 12.7%, T=2 99.5%. Figure images/temperature_vision.png.

This is still a strong practical result: a 1000-way classifier on shifted images loses half its coverage to a 0.5 rule, while argmax always answers.

3.4 A fixed map can mimic argmax, but that is not the conversion

A supervised C x C linear map fitted to argmax labels gets:

data	train	test	chance
SST-2	100.00%	100.00%	50.00%
sentiment	99.14%	99.33%	33.33%
emotion	99.50%	99.83%	16.67%

This is a classifier trained to reproduce argmax, not a 0.5 threshold rule. It reintroduces argmax as the target. The exact fixed elementwise conversion for C=3 is much weaker: best a = [1.4, 1.4, 1.0], agreement 89.5%, 4.0% wrong selected, 4.3% abstain, versus identity delta 24.66%. A fixed conversion can approximate the decision regions but cannot be exact for all p.

3.5 Tabular: 30 dataset/classifier pairs, max-F1 conversion

We ran logistic regression, random forest, histogram gradient boosting, Gaussian NB and kNN on breast cancer, wine, iris, digits, synthetic 5-class and synthetic 20-class data (30 pairs; 5 binary, 25 multiclass). The threshold tau* was selected on a validation split and evaluated on a held-out test split. For each pair we computed T_{tau*} and checked the converted 0.5 decisions.

Exactness. All 30 conversions are exact: conversion_exact = True, and F1(test, tau*) = F1(test, converted 0.5) to floating precision. This is the max-F1 conversion doing exactly what it claims: it preserves the selected threshold, not the test-optimal threshold.

The selected threshold is not 0.5. Binary tau* ranges from 0.4819 (breast cancer/logreg) to 0.9984 (breast cancer/Gaussian NB). Multiclass per-class tau* spans up to 1.0, with mean spread 0.479. Figure images/tabular_tau.png shows every class threshold; the red line at 0.5 is rarely where the classes sit.

Binary pairs.

dataset / classifier	tau*	F1val @0.5	F1val @tau*	F1test @0.5	F1test @tau*
breast cancer / logreg	0.4819	0.9888	0.9944	0.9775	0.9775
breast cancer / random forest	0.5400	0.9551	0.9714	0.9551	0.9492
breast cancer / hist-GB	0.5280	0.9832	0.9832	0.9609	0.9609
breast cancer / Gaussian NB	0.9984	0.9392	0.9655	0.9497	0.9425
breast cancer / kNN	0.6655	0.9674	0.9724	0.9727	0.9600

Selected multiclass pairs.

dataset / classifier	tau* range	macro-F1val @0.5	macro-F1val @tau*	macro-F1test @0.5	macro-F1test @tau*	argmax F1	converted-argmax F1	agreement
digits / logreg	0.0748-0.8163	0.9605	0.9769	0.9606	0.9571	0.9647	0.9561	97.8%
digits / random forest	0.2467-0.5300	0.9111	0.9787	0.9229	0.9698	0.9778	0.9626	97.3%
synthetic_5 / random forest	0.3033-0.4267	0.7990	0.8749	0.7829	0.8454	0.8600	0.8524	96.7%
synthetic_20 / random forest	0.1167-0.1867	0.0133	0.6046	0.0040	0.5415	0.5495	0.5585	87.2%
synthetic_20 / kNN	0.1365-0.2667	0.0490	0.4946	0.0797	0.4332	0.4512	0.4184	76.9%

F1 on validation is preserved; F1 on test can move either way. On validation the conversion moves 0.5 to the F1 maximum by construction. On test, comparing the converted rule to the raw 0.5 rule, 13 pairs improve and 15 worsen. For the five binary pairs, three are lower on test, two equal, none higher. The changes are threshold-selection/generalization effects, not a benefit of the conversion. For example, iris/random forest has validation F1 1.000 at tau* but test F1 0.8006 after conversion, versus 0.8932 at raw 0.5; the validation split is tiny. Conversely, digits/random forest improves from 0.9229 to 0.9698 on test. Figure images/tabular_f1.png.

The 20-class cases make the distinction sharp. In the next table, macro-F1 at 0.5 is the mean of the C one-vs-rest F1 scores obtained by thresholding each class at 0.5 independently. It is not the macro-F1 of argmax; the argmax F1 is its own column. The independent-threshold rule can select zero or several classes, which is exactly why it is not argmax.

model	delta at 0.5	per-class macro-F1 at 0.5	argmax F1	converted-argmax F1	argmax agreement
synthetic_20 / random forest	99.80%	0.0040	0.5495	0.5585	87.2%
synthetic_20 / kNN	95.30%	0.0797	0.4512	0.4184	76.9%
synthetic_20 / logreg	65.20%	0.2670	0.3850	0.3850	85.2%

For the 20-class random forest, the 0.5 rule abstains on 998 of 1000 test points and has macro-F1 0.004. Argmax has F1 0.5495 without abstaining. Per-class max-F1 conversions give each class a 0.5 threshold, but argmax over the converted scores agrees with the original argmax only 87.2% of the time, and its F1 is 0.5585, not a recovery of the original rule. The conversion is a different decision rule, not an equivalent one.

4. What is true, and what is false

True.

• For any binary threshold tau, T_tau is a strictly increasing affine conversion with T_tau(tau)=0.5; the ROC/AUC and F1 at tau are unchanged.
• For any multiclass one-vs-rest problem, the same is true per class; this is the max-F1 conversion, and it is exact.
• The instance-adaptive q = p/(2 max p) makes the winner exactly 0.5 and every loser below 0.5, for all C.
• The 0.5 rule on max p never picks a wrong class; it only abstains. Argmax is its total completion.

False.

• That a single fixed conversion vector can reproduce multiclass argmax for C >= 3.
• That a 0.5 threshold on a single score is equivalent to argmax in general.
• That the max-F1 conversion improves F1 or makes 0.5 the "correct" threshold. It is a reparameterization; the selected threshold is model- and data-dependent.
• That high C alone implies delta ~ 1. Under uniform probabilities it does (98% at C=10), but trained models can be confident (emotion 0.20%) or unconfident (ViT-1000 51%).

The practical rule. If you want max F1 for a binary classifier, select tau* and either use tau* directly or convert it to 0.5; the two are identical. If you want a single label for multiclass, use argmax. If you want per-class operating points, use C one-vs-rest thresholds (and accept that the result is a set of binary decisions, not one label). A confidence threshold at 0.5 is an abstention rule, and its coverage depends on calibration and domain.

5. Reproduction

All code is in repro/ on each branch. The fixed command is bash repro/run.sh; every run starts from the branch tip on GitHub.

experiment	id	branch	run
baseline theory + digits	8a044950-bedc-4084-8507-75e4555d9650	orx/threshold-conversion-baseline-theory-and-simplex	cff2a16d-f190-42ed-8510-38d21ec359ad
text	9512af43-d119-4b35-b9d0-b11b727b28e3	orx/real-text-classifiers-2-3-and-6-classes	18fb96ec-2a40-43dc-ac78-6343bdc067aa
vision	493a2ee5-43dd-4e01-b541-3aedf0bbab4f	orx/1000-class-vision-vit-b-16-on-cifar-10	a5f6fc9f-88fb-415e-92b1-fc064fa95d96
fixed fit	3ab3fa4c-eab3-4cc1-96f9-126f68d0a24a	orx/can-a-fitted-fixed-conversion-replace-the-adapti	6af95ffd-982c-45ca-b87c-65f817fb1aef
tabular max-F1	b117d2f9-70a2-4eb7-82cd-98cd1f99e632	orx/tabular-classifiers-and-the-max-f1-0-5-conversio	576b9f04-e295-4300-b42b-de57f96a54bb

Launch with orx exp run <id> --backend local. Logs are the evidence channel: orx logs <runId>. Artifacts: hf://buckets/jomasego/repro-threshold-conversion-artifacts (public).

Environment: macOS 26.7.1 arm64, Apple M4 Max, Python 3.9.6, torch 2.8.0 (MPS), transformers 4.57.6, datasets 3.6.0, scikit-learn 1.5.2, numpy 2.0.2, scipy 1.13.1.

Links

• Upstream: https://github.com/isllabres/python-multiagent-system
• Fork: https://github.com/jomasego/python-multiagent-system
• Branch (baseline): https://github.com/jomasego/python-multiagent-system/tree/orx/threshold-conversion-baseline-theory-and-simplex
• Branch (text): https://github.com/jomasego/python-multiagent-system/tree/orx/real-text-classifiers-2-3-and-6-classes
• Branch (vision): https://github.com/jomasego/python-multiagent-system/tree/orx/1000-class-vision-vit-b-16-on-cifar-10
• Branch (fixed fit): https://github.com/jomasego/python-multiagent-system/tree/orx/can-a-fitted-fixed-conversion-replace-the-adapti
• Branch (tabular): https://github.com/jomasego/python-multiagent-system/tree/orx/tabular-classifiers-and-the-max-f1-0-5-conversio
• Bucket: https://huggingface.co/buckets/jomasego/repro-threshold-conversion-artifacts
• Models: https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment-latest , https://huggingface.co/bhadresh-savani/distilbert-base-uncased-emotion , https://huggingface.co/google/vit-base-patch16-224 , https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english
• Datasets: https://huggingface.co/datasets/cardiffnlp/tweet_eval , https://huggingface.co/datasets/dair-ai/emotion , https://huggingface.co/datasets/uoft-cs/cifar10 , https://huggingface.co/datasets/stanfordnlp/sst2
• sklearn digits: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html

HF Job. A text run on --backend hf --flavor cpu-upgrade --image python:3.12 was attempted from branch orx/real-text-classifiers-2-3-and-6-classes. The submit was rejected:

Hugging Face job submit failed (402): {"error":"Pre-paid credit balance is insufficient - add more credits to your account to use Jobs."}

No Job URL exists. The local CPU/MPS runs are the evidence. No ZeroGPU Space was used; no Modal token was available.

6. Limitations

• The vision test is out-of-domain (CIFAR-10 images through an ImageNet head), so its 51% delta is not a clean C-scaling measurement. It is a calibration/domain-shift measurement.
• Text evaluation uses the SST-2 validation split (872 examples) and the first 2000 test examples for sentiment/emotion; it is not a full benchmark.
• Tabular iris/wine test splits are 38-45 points, and threshold selection on a 37-75 point validation split overfits. The test F1 changes should be read as noise, not as reliable ranking.
• One seed; no confidence intervals. The exactness statements are mathematical and deterministic; the F1 comparisons are not.
• The fixed C x C map is supervised on argmax labels, so its near-perfect accuracy is expected and is not evidence for the conversion claim.
• HF Jobs were blocked by the account's pre-paid balance; no Job link can be provided.

7. Files

• results.json / tabular_results.json in the bucket: all metrics, per-class thresholds, temperature sweeps.
• summary.md and tabular_summary.md: per-run tables.
• images/: all figures referenced here.
• logbook.ipynb: the same results as an executable notebook, with cells for the theory, the measured tables and every link.