| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "# Threshold conversion logbook\n", | |
| "\n", | |
| "**Question.** Can any multiclass classifier's threshold curve be converted to a\n", | |
| "0.5-threshold rule equivalent to argmax, and can the conversion keep maximum F1?\n", | |
| "\n", | |
| "**Verdict.** The conversion is an exact reparameterization of a chosen threshold\n", | |
| "(binary), or of each class's chosen threshold (multiclass). It cannot turn one\n", | |
| "fixed 0.5 rule into argmax for C >= 3, and it cannot improve F1; it only moves the\n", | |
| "selected operating point to 0.5. The max-F1 version is exact per class, but it is\n", | |
| "a different decision rule, not argmax.\n", | |
| "\n", | |
| "**Where to look.**\n", | |
| "- Report: `report.md` in this folder.\n", | |
| "- Bucket: https://huggingface.co/buckets/jomasego/repro-threshold-conversion-artifacts\n", | |
| "- Fork: https://github.com/jomasego/python-multiagent-system\n", | |
| "- Upstream: https://github.com/isllabres/python-multiagent-system\n", | |
| "\n", | |
| "All measurements are CPU/MPS local runs launched through `orx exp run --backend local`." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "import json\n", | |
| "import math\n", | |
| "import numpy as np\n", | |
| "\n", | |
| "RUNS = {\n", | |
| " \"baseline\": {\"id\": \"8a044950-bedc-4084-8507-75e4555d9650\", \"run\": \"cff2a16d-f190-42ed-8510-38d21ec359ad\", \"branch\": \"orx/threshold-conversion-baseline-theory-and-simplex\"},\n", | |
| " \"text\": {\"id\": \"9512af43-d119-4b35-b9d0-b11b727b28e3\", \"run\": \"18fb96ec-2a40-43dc-ac78-6343bdc067aa\", \"branch\": \"orx/real-text-classifiers-2-3-and-6-classes\"},\n", | |
| " \"vision\": {\"id\": \"493a2ee5-43dd-4e01-b541-3aedf0bbab4f\", \"run\": \"a5f6fc9f-88fb-415e-92b1-fc064fa95d96\", \"branch\": \"orx/1000-class-vision-vit-b-16-on-cifar-10\"},\n", | |
| " \"fixed_fit\": {\"id\": \"3ab3fa4c-eab3-4cc1-96f9-126f68d0a24a\", \"run\": \"6af95ffd-982c-45ca-b87c-65f817fb1aef\", \"branch\": \"orx/can-a-fitted-fixed-conversion-replace-the-adapti\"},\n", | |
| " \"tabular\": {\"id\": \"b117d2f9-70a2-4eb7-82cd-98cd1f99e632\", \"run\": \"576b9f04-e295-4300-b42b-de57f96a54bb\", \"branch\": \"orx/tabular-classifiers-and-the-max-f1-0-5-conversio\"},\n", | |
| "}\n", | |
| "BUCKET = \"https://huggingface.co/buckets/jomasego/repro-threshold-conversion-artifacts\"\n", | |
| "print(json.dumps(RUNS, indent=2))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 1. Binary conversion: every threshold is a 0.5 level set\n", | |
| "\n", | |
| "For `p1` and `tau` in `(0,1)`:\n", | |
| "\n", | |
| "```\n", | |
| "T_tau(p1) = 1/2 + (p1 - tau) / (2 (1 - tau))\n", | |
| "```\n", | |
| "\n", | |
| "`T_tau(tau) = 1/2` and `T_tau` is strictly increasing, so\n", | |
| "`T_tau(p1) >= 1/2 <=> p1 >= tau`. ROC, AUC and F1 at `tau` are unchanged." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "def T(p, tau):\n", | |
| " return 0.5 + (np.asarray(p) - tau) / (2.0 * (1.0 - tau))\n", | |
| "\n", | |
| "for tau in (0.1, 0.25, 0.5, 0.75, 0.9):\n", | |
| " p = np.linspace(0, 1, 20001)\n", | |
| " same = np.all((T(p, tau) >= 0.5) == (p >= tau))\n", | |
| " print(f\"tau={tau:.2f} T(tau)={T(tau, tau):.1f} decisions_same={same}\")\n", | |
| "\n", | |
| "p = np.array([0.07, 0.93])\n", | |
| "w = np.array([0.6, 0.4])\n", | |
| "print(\"argmax\", int(np.argmax(p)))\n", | |
| "print(\"dot\", float(w @ p), \"dot_pred\", int(w @ p >= 0.5))\n", | |
| "print(\"hadamard_argmax\", int(np.argmax(w * p)))\n", | |
| "print(\"renormalized\", (w * p) / (w * p).sum(), \"argmax\", int(np.argmax((w * p) / (w * p).sum())))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 2. Uniform simplex: the 0.5 rule's abstention rate\n", | |
| "\n", | |
| "`P(max p < 0.5) = 1 - C / 2^(C-1)`. The table contrasts the closed form with a\n", | |
| "40,000-draw Monte Carlo." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "rng = np.random.default_rng(0)\n", | |
| "print(f\"{'C':>3} {'closed':>10} {'MC':>10}\")\n", | |
| "for C in (2, 3, 4, 5, 6, 10, 20):\n", | |
| " closed = 1.0 - C / 2.0 ** (C - 1)\n", | |
| " p = rng.dirichlet(np.ones(C), size=40000)\n", | |
| " mc = float(np.mean(p.max(axis=1) < 0.5))\n", | |
| " print(f\"{C:>3} {100*closed:9.2f}% {100*mc:9.2f}%\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "### Dirichlet concentration\n", | |
| "\n", | |
| "`delta(alpha, C) = 1 - C + C * I_{1/2}(alpha, (C-1) alpha)` where `I` is the\n", | |
| "regularized incomplete beta. Concentration raises delta." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "from scipy.special import betainc\n", | |
| "print(f\"{'C':>3} {'alpha=0.3':>10} {'alpha=1':>10} {'alpha=3':>10}\")\n", | |
| "for C in (3, 6, 10):\n", | |
| " vals = [1 - C + C * betainc(a, (C - 1) * a, 0.5) for a in (0.3, 1.0, 3.0)]\n", | |
| " print(f\"{C:>3} \" + \" \".join(f\"{100*v:9.2f}%\" for v in vals))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 3. Measured models\n", | |
| "\n", | |
| "| model | C | n | argmax | delta at 0.5 | coverage | selective | ECE |\n", | |
| "|---|---:|---:|---:|---:|---:|---:|---:|\n", | |
| "| sklearn digits, logreg | 10 | 540 | 97.22% | 0.74% | 99.26% | 97.76% | 0.0222 |\n", | |
| "| DistilBERT SST-2 | 2 | 872 | 91.06% | 0.00% | 100.00% | 91.06% | 0.0800 |\n", | |
| "| Twitter-RoBERTa sentiment | 3 | 2000 | 71.50% | 4.65% | 95.35% | 72.36% | 0.0573 |\n", | |
| "| DistilBERT emotion | 6 | 2000 | 92.70% | 0.20% | 99.80% | 92.69% | 0.0385 |\n", | |
| "| ViT-B/16 on CIFAR-10 (OOD) | 1000 | 1000 | n/a | 51.00% | 49.00% | n/a | n/a |\n", | |
| "\n", | |
| "`delta = P(max p < 0.5)`. The uniform formula is a reference, not a law: emotion\n", | |
| "(C=6) has a smaller delta than sentiment (C=3) because it is much more confident." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "models = [\n", | |
| " (\"sklearn digits, logreg\", 10, 540, 0.9722, 0.0074, 0.9926, 0.9776, 0.0222),\n", | |
| " (\"DistilBERT SST-2\", 2, 872, 0.9106, 0.0000, 1.0000, 0.9106, 0.0800),\n", | |
| " (\"Twitter-RoBERTa sentiment\", 3, 2000, 0.7150, 0.0465, 0.9535, 0.7236, 0.0573),\n", | |
| " (\"DistilBERT emotion\", 6, 2000, 0.9270, 0.0020, 0.9980, 0.9269, 0.0385),\n", | |
| " (\"ViT-B/16 on CIFAR-10 (OOD)\", 1000, 1000, None, 0.5100, 0.4900, None, None),\n", | |
| "]\n", | |
| "for name, C, n, acc, delta, cov, sel, ece in models:\n", | |
| " acc_s = \"n/a\" if acc is None else f\"{100*acc:.2f}%\"\n", | |
| " sel_s = \"n/a\" if sel is None else f\"{100*sel:.2f}%\"\n", | |
| " ece_s = \"n/a\" if ece is None else f\"{ece:.4f}\"\n", | |
| " print(f\"{name:34s} C={C:<4d} n={n:<5d} argmax={acc_s:>7s} delta={100*delta:5.2f}% cov={100*cov:5.2f}% sel={sel_s:>7s} ECE={ece_s}\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "### Text: max-F1 thresholds are class-specific\n", | |
| "\n", | |
| "| model | class | F1 @0.5 | F1 @tau* | tau* |\n", | |
| "|---|---|---:|---:|---:|\n", | |
| "| sentiment | negative | 0.715 | 0.719 | 0.448 |\n", | |
| "| sentiment | neutral | 0.693 | 0.758 | 0.231 |\n", | |
| "| sentiment | positive | 0.725 | 0.726 | 0.479 |\n", | |
| "| emotion | sadness | 0.967 | 0.973 | 0.103 |\n", | |
| "| emotion | joy | 0.946 | 0.951 | 0.961 |\n", | |
| "| emotion | love | 0.829 | 0.860 | 0.192 |\n", | |
| "| emotion | anger | 0.923 | 0.931 | 0.041 |\n", | |
| "| emotion | fear | 0.890 | 0.902 | 0.098 |\n", | |
| "| emotion | surprise | 0.738 | 0.803 | 0.344 |\n", | |
| "\n", | |
| "A single 0.5 threshold cannot be F1-optimal for all classes; the per-class\n", | |
| "conversion puts each tau* at 0.5 but creates C binary rules, not argmax." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 4. Tabular max-F1 test\n", | |
| "\n", | |
| "30 dataset/classifier pairs: logreg, random forest, hist-GB, Gaussian NB and kNN\n", | |
| "on breast cancer, wine, iris, digits, synthetic 5-class and synthetic 20-class.\n", | |
| "`tau*` was selected on validation and evaluated on a held-out test split.\n", | |
| "\n", | |
| "All 30 conversions are exact: `F1(test, tau*) = F1(test, converted 0.5)`.\n", | |
| "The selected threshold is not 0.5: binary `tau*` ranges 0.4819-0.9984, and\n", | |
| "multiclass per-class thresholds span up to 1.0. On test, converted vs raw 0.5\n", | |
| "F1 improves in 13 pairs and worsens in 15; the differences are threshold\n", | |
| "selection/generalization, not a benefit of conversion." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "For multiclass, `F1@0.5` and `F1@tau` below are the mean of the C one-vs-rest F1 scores (independent per-class thresholds), not the macro-F1 of argmax. The argmax F1 is in its own column.\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "tab = [\n", | |
| " (\"digits/logreg\", \"0.0748-0.8163\", 0.9606, 0.9571, 0.9647, 0.9561, 0.978),\n", | |
| " (\"digits/random_forest\", \"0.2467-0.5300\", 0.9229, 0.9698, 0.9778, 0.9626, 0.973),\n", | |
| " (\"synthetic_5/random_forest\", \"0.3033-0.4267\", 0.7829, 0.8454, 0.8600, 0.8524, 0.967),\n", | |
| " (\"synthetic_20/random_forest\", \"0.1167-0.1867\", 0.0040, 0.5415, 0.5495, 0.5585, 0.872),\n", | |
| " (\"synthetic_20/kNN\", \"0.1365-0.2667\", 0.0797, 0.4332, 0.4512, 0.4184, 0.769),\n", | |
| "]\n", | |
| "print(f\"{'pair':28s} {'tau range':>17s} {'F1@0.5':>8s} {'F1@tau':>8s} {'argmax':>8s} {'convArgmax':>11s} {'agree':>7s}\")\n", | |
| "for name, rng, f05, ftau, farg, fconv, agree in tab:\n", | |
| " print(f\"{name:28s} {rng:>17s} {f05:8.4f} {ftau:8.4f} {farg:8.4f} {fconv:11.4f} {100*agree:6.1f}%\")\n", | |
| "\n", | |
| "print()\n", | |
| "print(\"20-class random forest: 998/1000 test points below 0.5; per-class macro-F1 at 0.5 = 0.004; argmax F1 = 0.5495.\")\n", | |
| "print(\"The 0.5 rule abstains; argmax always answers. Per-class conversions are not argmax (agreement 87.2%).\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 5. Temperature: argmax invariant, 0.5 not\n", | |
| "\n", | |
| "| model | T=0.5 delta | T=1 delta | T=2 delta | T=4 delta |\n", | |
| "|---|---:|---:|---:|---:|\n", | |
| "| SST-2 | 0.00% | 0.00% | 0.00% | 0.00% |\n", | |
| "| sentiment | 0.55% | 4.65% | 19.10% | 62.20% |\n", | |
| "| emotion | 0.00% | 0.20% | 2.90% | 14.15% |\n", | |
| "\n", | |
| "Argmax accuracy is unchanged at every T; the 0.5 rule's coverage moves from\n", | |
| "0.2% to 14% (emotion) or 4.65% to 62% (sentiment). The 0.5 threshold is a\n", | |
| "calibration statement, not an intrinsic property of the classifier." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 6. Links\n", | |
| "\n", | |
| "Every model, dataset, repository, run and bucket used. The HF Job was attempted\n", | |
| "and rejected for insufficient pre-paid Jobs credit; there is no Job URL." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "links = {\n", | |
| " \"upstream_repo\": \"https://github.com/isllabres/python-multiagent-system\",\n", | |
| " \"fork_repo\": \"https://github.com/jomasego/python-multiagent-system\",\n", | |
| " \"branches\": {\n", | |
| " \"baseline\": \"https://github.com/jomasego/python-multiagent-system/tree/orx/threshold-conversion-baseline-theory-and-simplex\",\n", | |
| " \"text\": \"https://github.com/jomasego/python-multiagent-system/tree/orx/real-text-classifiers-2-3-and-6-classes\",\n", | |
| " \"vision\": \"https://github.com/jomasego/python-multiagent-system/tree/orx/1000-class-vision-vit-b-16-on-cifar-10\",\n", | |
| " \"fixed_fit\": \"https://github.com/jomasego/python-multiagent-system/tree/orx/can-a-fitted-fixed-conversion-replace-the-adapti\",\n", | |
| " \"tabular\": \"https://github.com/jomasego/python-multiagent-system/tree/orx/tabular-classifiers-and-the-max-f1-0-5-conversio\",\n", | |
| " },\n", | |
| " \"bucket\": BUCKET,\n", | |
| " \"models\": [\n", | |
| " \"https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment-latest\",\n", | |
| " \"https://huggingface.co/bhadresh-savani/distilbert-base-uncased-emotion\",\n", | |
| " \"https://huggingface.co/google/vit-base-patch16-224\",\n", | |
| " \"https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english\",\n", | |
| " ],\n", | |
| " \"datasets\": [\n", | |
| " \"https://huggingface.co/datasets/cardiffnlp/tweet_eval\",\n", | |
| " \"https://huggingface.co/datasets/dair-ai/emotion\",\n", | |
| " \"https://huggingface.co/datasets/uoft-cs/cifar10\",\n", | |
| " \"https://huggingface.co/datasets/stanfordnlp/sst2\",\n", | |
| " \"https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html\",\n", | |
| " ],\n", | |
| " \"hf_job\": {\n", | |
| " \"status\": \"blocked\",\n", | |
| " \"attempted\": \"orx exp run 9512af43-d119-4b35-b9d0-b11b727b28e3 --backend hf --flavor cpu-upgrade --image python:3.12 --timeout 1h\",\n", | |
| " \"error\": \"Hugging Face job submit failed (402): Pre-paid credit balance is insufficient\",\n", | |
| " \"url\": None,\n", | |
| " },\n", | |
| "}\n", | |
| "print(json.dumps(links, indent=2))" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 7. Reproduction\n", | |
| "\n", | |
| "```bash\n", | |
| "# from the fork's worktree\n", | |
| "orx exp run 8a044950-bedc-4084-8507-75e4555d9650 --backend local # baseline\n", | |
| "orx exp run 9512af43-d119-4b35-b9d0-b11b727b28e3 --backend local # text\n", | |
| "orx exp run 493a2ee5-43dd-4e01-b541-3aedf0bbab4f --backend local # vision\n", | |
| "orx exp run 3ab3fa4c-eab3-4cc1-96f9-126f68d0a24a --backend local # fixed fit\n", | |
| "orx exp run b117d2f9-70a2-4eb7-82cd-98cd1f99e632 --backend local # tabular\n", | |
| "orx logs <runId>\n", | |
| "```\n", | |
| "\n", | |
| "Run IDs: baseline `cff2a16d-f190-42ed-8510-38d21ec359ad`, text\n", | |
| "`18fb96ec-2a40-43dc-ac78-6343bdc067aa`, vision\n", | |
| "`a5f6fc9f-88fb-415e-92b1-fc064fa95d96`, fixed fit\n", | |
| "`6af95ffd-982c-45ca-b87c-65f817fb1aef`, tabular\n", | |
| "`576b9f04-e295-4300-b42b-de57f96a54bb`.\n", | |
| "\n", | |
| "Environment: macOS 26.7.1 arm64, M4 Max, Python 3.9.6, torch 2.8.0 (MPS),\n", | |
| "transformers 4.57.6, datasets 3.6.0, scikit-learn 1.5.2." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## 8. Limitations\n", | |
| "\n", | |
| "- Vision is out-of-domain (CIFAR-10 through an ImageNet head), so 51% is a\n", | |
| " calibration/domain-shift result, not a clean C-scaling measurement.\n", | |
| "- Text uses 872 SST-2 validation examples and the first 2000 sentiment/emotion\n", | |
| " test examples; it is not a full benchmark.\n", | |
| "- Tabular iris/wine test splits are 38-45 points and threshold selection\n", | |
| " overfits the small validation split; test F1 changes are noisy.\n", | |
| "- One seed; no confidence intervals. Exactness claims are deterministic.\n", | |
| "- The fixed C x C map is supervised on argmax labels, so its high accuracy is\n", | |
| " expected and is not evidence for the whiteboard conversion.\n", | |
| "- HF Jobs were blocked by the account balance; no Job URL exists." | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "kernelspec": { | |
| "display_name": "Python 3", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "name": "python", | |
| "version": "3.9.6" | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 5 | |
| } |
Xet Storage Details
- Size:
- 15.8 kB
- Xet hash:
- 488f54c98b98cca7f56f1a53d9dee0b5c0f4514021867f4ea2a4b0f46c3adc1c
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.