areai51/propensity-one

Propensity One

Research preview. Propensity One ranks headline variants for one story by how likely each one is to get the higher click-through rate in a randomised A/B test. It is answerdotai/ModernBERT-large fine-tuned on the randomised headline tests of the Upworthy Research Archive (CC BY 4.0), and only on those.

On 1,945 held-out A/B tests it picks the better headline of a pair 0.636 of the time. Zero-shot GPT-6.1 Sol gets 0.596 on the same tests.

What it is, and what it isn't

• What it is: a ModernBERT-large encoder with a linear head. It is trained on two objectives together: the chance each variant beats the others in its test, and a pairwise ranking loss within each test. So it learns what makes one headline beat another for the same story, not which stories are popular.
• Trained on news and social-issue headlines from upworthy.com, 2013-2015. It hasn't been validated on product ads, other sites, other audiences or other languages.
• Scores only rank variants against each other. One score on its own means nothing.
• A lighter version: Propensity One lite is a TF-IDF ridge that runs on CPU and in the browser. It is less accurate (0.589 on the same tests).

Use

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("areai51/propensity-one")
sys.path.insert(0, path)
from propensity_one import PropensityOne

model = PropensityOne(path)            # picks cuda, then mps, then cpu
for r in model.rank(["Headline A", "Headline B", "Headline C"],
                    excerpts="Optional teaser text shown under each headline",
                    story="Optional: the story's first paragraph, the same for every variant"):
    print(r["rank"], round(r["score"], 3), r["p_top_beats"], r["headline"])

It needs torch 2.14.1, transformers 5.20.2 and safetensors 0.8.0 (other versions may work). The weights are safetensors, and encoder/ holds the base model's config and tokenizer. The loader builds the same tagged input text the model was trained on. If the text is longer than 512 tokens, it shortens the story first, then the excerpt; the headline is never cut. Before release, the loader's scores were checked against the training run's on every validation and test headline.

Evaluation

These numbers are for the test split of upworthy_brx-pipe-0.2.0: 1,945 randomised tests with 8,654 headline variants. Splits are by family, which groups variants of one story with exact and near-duplicate headlines. So no test story or test headline appears in training. Nothing was tuned on test, and a sealed holdout is kept for the final release check.

metric (test)	Propensity One	GPT-6.1 Sol, zero-shot	Propensity One lite	random
pairwise accuracy	0.636 [0.625, 0.647]	0.596	0.589	0.5
top-1 on the 323 tests with a clear winner (P(best beats runner-up) >= 0.95)	0.588	0.570		0.266

• Pairwise accuracy is the share of within-test pairs in which the higher-scored headline had the higher observed click-through rate. Validation gives 0.636.

• Other metrics: weighted by how certain each observed winner is, pairwise accuracy is 0.684. Top-1 (the top pick is the observed best) is 0.405, against 0.242 for a random pick.

• Against the other models, on identical tests: Propensity One minus GPT-6.1 Sol is +0.040 [0.025, 0.054]. Minus the lite model it is +0.048 [0.035, 0.060] (95% bootstrap intervals). A plain TF-IDF ridge scores 0.572 and a random score 0.503.

• GPT-6.1 Sol may have seen the answers. The archive has been public since 2021, so its winners may be in GPT's training data. The comparison flatters GPT, if anything.

• The archive's 2024 correction: its Author Correction (Sci. Data 11:765) says tests created between 2013-06-25 and 2014-01-10 may have been badly randomised. Without the tests created in that window (1,503 tests left), Propensity One scores 0.634 and GPT-6.1 Sol 0.598 (+0.036).

• Retraining gives somewhat different results. We trained this recipe three times. On the same test:

  • this checkpoint: 0.636;
  • a run on the same Upworthy data plus research-only TikTok ad data: 0.635;
  • an earlier run of that mixed recipe, which hit a spike in gradient norm during training: 0.596.

  If you retrain, expect differences of up to about 0.04.

Chance the top pick wins. p_top_beats is sigmoid(k x score difference), with k fitted on validation pairs. On test pairs, the predicted chance that the model's preferred headline wins averages 0.640, and it won 0.640 of the time. Log loss is 0.6286, against 0.6931 for always saying 50%. Most headline differences are small, so the probabilities stay close to 50%. That reflects real uncertainty; it isn't a bug.

When it should abstain. It does better when it says nothing. Requiring a score margin between the top pick and the runner-up — threshold chosen on validation, applied unchanged to test (2026-11-14, eval_harness/abstain.py):

calling	test coverage	tests called	pairwise accuracy on called tests	top-1 on called tests
always (default)	100%	1,945	0.636 [0.625, 0.647]	0.405
clearest half	50%	973	0.668 [0.655, 0.683]	0.475
clearest third	33%	643	0.679 [0.662, 0.697]	0.513
clearest quarter	25%	487	0.692 [0.673, 0.711]	0.548

Validation tracks test closely (0.667 / 0.543 / 0.686 at 100/50/33%). A random-score model held to the same abstention rule stays at chance at every coverage, so these curves are signal, not thresholding artefacts. If your use of this model has a cost of being wrong, prefer "only when the margin is large" over "always" — and treat every call short of that as a tie-breaker to shortlist both versions for a live test.

Training

• Data: the Upworthy Research Archive, randomised headline tests run on upworthy.com in 2013-2015, with impressions and clicks for each variant.
  • We keep tests with at least two headline variants.
  • We drop 2,007 tests where the teaser image varied but wasn't crossed with the headline, so the image could have caused the difference.
  • That leaves 19,268 tests; 61,925 variants are in the training split.
• Target: each variant's posterior probability of beating the others in its test.
• Run: 2 epochs on one NVIDIA T4, 4.2 h. Base model answerdotai/ModernBERT-large at revision 45bb465. Input text up to 512 tokens.

Licence and attribution

• This model: CC BY 4.0, like its training data.
• The base model: ModernBERT-large, by Answer.AI and LightOn, Apache-2.0.
• The data: Matias, J. N., Munger, K., Aubin Le Quere, M., & Ebersole, C. (2021), The Upworthy Research Archive, a time series of 32,487 experiments in U.S. media, Scientific Data 8, 195.

Files

• model.safetensors: encoder and head weights (fp32).
• encoder/: the base model's config and tokenizer.
• config.json: the input length limit, pairwise calibration, training snapshot hashes and library versions.
• eval.json: the full validation and test report, the sensitivity checks and the export parity check.
• propensity_one.py: the loader.