Model card: ballot-E-v0 (v1 release)

In short

• What it does: you give it a document and a multiple-choice/yes-no/rating question, it gives you back a probability for each answer — instantly, no text generation. This is a "System One" model in TypeSafe's sense: fast, narrow, one decision per call, not a chatbot.
• What's different about it: every other model like this (kev, laya, and TypeSafe's own Jev) reads all the answer options together in one pass, which means the order you list the options in can quietly change the answer — we measured this at 16.7% of the time for kev and 43.3% for laya (a number nobody had published before). Ballot reads each option on its own, so reordering the options cannot change its answer — measured at 0.0%.
• What it's not, yet: this is an early v1 — one short training pass, not the full recipe. Its accuracy (68%) is behind kev's (79%). The order-robustness result is real and already proven; catching up on accuracy is the next step, not something this release claims.
• Try it: jump to Usage for a code snippet, or 10 real examples below to see actual inputs and outputs.

What it is, more precisely

ballot is a decision model: it takes one document (the state) and a set of typed questions, and returns a probability distribution for each question in one forward pass. It does not generate text. It follows the same input/output contract as TypeSafe's /v1/systemone API — the same one kev, laya, and cua-s1-forms all target.

This is v1: a small, honest, single-epoch checkpoint, not a finished production model. It exists to establish one specific, measured claim — see Results — before scaling up.

• Code: the four files under ballot/ in this repo (hf_backbone.py, hf_model.py, checkpoint.py, model.py + encode.py for a shared constant) are the complete, minimal loading path. There is no separate package yet; this repo is the source right now.
• Weights: ballot.safetensors + ballot.json (config, training metadata, and a SHA-256 signature over the weights file) — the same safetensors-plus-signed-JSON-sidecar shape cua-s1-forms uses, adopted here rather than inventing a new format. cua_s1's own loader rejects pickled checkpoints by design (arbitrary pickle is a code-execution risk for a public checkpoint); so does this one.

What this model actually is

• Backbone: jhu-clsp/mmBERT-base (ModernBERT architecture, 22 layers, hidden 768, ~149M parameters), bidirectional, fully fine-tuned.
• Head: "Arm E" — a listwise set-attention readout. Each option in a question is encoded independently (its own forward pass through the backbone, batched with the other options in one call for speed, but never sharing a sequence or attention computation with them), then a single unmasked self-attention layer over [decide, opt_1, ..., opt_K] lets options interact in the readout only, followed by a pointer dot-product against the decide vector.
• Why this shape, specifically: every option's tokens get their own fixed position-id base, and no option's encoding ever depends on which other options are present or what order they arrived in. This is not a training-time mitigation (order shuffling during training, which kev and Nimble both also use) — it is a structural guarantee, true for any input, checkable on any single forward pass. See Results.

Results

Measured here (bench/quick_compare.py), on 100 held-out records from ballot's own 9-source test split (30 usable for the flip-rate check: choice questions with >=3 options), using each model's real weights and real inference code — kev's own kev/model.py, laya's own rl_agent_api.py, not reimplementations:

model	accuracy	permutation flip rate (option reversal)
kev-0.5b	0.790	0.167
laya	0.510	0.433
ballot-E-v0 (this model)	0.680	0.000

Flip rate is the fraction of choice questions (K>=3 options) where the model's picked option (by content, not slot index) changes between the original order and the fully-reversed order. It is 0.0 for ballot by construction — verified on these actual trained weights, not just an untrained baseline (max_delta under reversal is 1.19e-7, float-precision noise, identical to the pre-training number).

What this table does and doesn't show:

• kev's causal block-mask lets a later-packed option attend to an earlier one in the same question branch — kev's own README reports a 7.4% flip rate on its own eval; 16.7% here is the same defect measured on a different (wider, 9-source) test mix, not a new problem.
• Laya's 43.3% flip rate has not been published anywhere before this. Laya packs every option into one bidirectional sequence — "bidirectional" is not the same claim as "order-invariant," and this is the first direct measurement showing the gap between those two properties for that architecture.
• Laya's low accuracy here (0.510) is expected, not a foul result: laya's own README documents near-chance zero-shot accuracy outside its RLCD fine-tuning distribution (0.362 on its own typed-decisions zero-shot split). We ran its base English checkpoint on ballot's own task mix, which it was never tuned on — exactly the regime laya's own docs warn about.
• ballot's accuracy (0.680) honestly trails kev's (0.790). This checkpoint is cross-entropy only, one epoch, 300 records per source — about 1/5 of kev's per-source training depth, and with no calibration/RLCD stage yet. The permutation-robustness result is real and already measured; closing the accuracy gap is the next round of training, not a claim this release makes.
• Accuracy here is a single simplified metric (plain argmax-matches-label over all question types), not kev's or laya's fuller ECE/NLL/per-source-family suite — a fast sanity comparison, not a replacement for a full benchmark pass.

What tasks ballot can do — 10 real examples

Every example below is a real call against this exact checkpoint (ballot-E-v0, bench/ scripts, run just before publishing this card) — genuine output probabilities, not illustrative numbers. Ballot answers three question shapes (choice, score, noul) over whatever state + options you give it; these ten cover the range of domains it was trained on.

Being upfront: this is a shallow v1 checkpoint (68% accuracy on held-out data), so a few of these are visibly wrong — marked below rather than edited out. That's the honest picture of what one epoch on 300 records/source currently buys.

#	task	state (truncated)	question	options	prediction	correct?
1	Support ticket routing	"Order #4471: wireless mouse, delivered 2 days late... wants a partial refund"	Which team should this ticket be routed to?	billing / logistics / returns / technical_support	technical_support (42.1%)	✗ questionable — a refund request on a working item reads more like returns or billing
2	Banking intent detection	"ATM never dispensed the money, and my balance was still deducted"	Which intent does this message express?	cash_withdrawal_not_recognised / card_swallowed / balance_not_updated_after_bank_transfer / declined_cash_withdrawal	declined_cash_withdrawal (55.2%)	✗ debatable — "never dispensed but still charged" is the textbook cash_withdrawal_not_recognised (41.4%) case
3	Yes/no reading comprehension	"...it is not visible from space with the naked eye."	Is the Great Wall of China visible from space with the naked eye?	false / true	false (63.1%)	✓
4	News topic classification	"The central bank raised interest rates by half a percentage point..."	What topic is this article about?	World / Sports / Business / Sci/Tech	Business (52.0%)	✓
5	Natural language inference	"A man is playing a guitar on a busy street corner while a small crowd gathers to watch."	Does this hold: "The man is performing music in public."	entailment / neutral / contradiction	neutral (44.1%)	✗ near-miss — entailment (40.3%) is a hair behind; a genuinely close call
6	Sentiment scoring	"The food arrived cold... but the dessert was surprisingly good."	How positive is the sentiment?	very negative / negative / neutral / positive / very positive	negative (34.3%)	✓ reasonable for a mostly-negative mixed review
7	Review rating	"Absolutely fantastic stay... would come back in a heartbeat."	How many stars, 1 to 5?	1-5 stars	5 stars (31.1%)	✓ (correct top pick; note the whole distribution is fairly flat — a calibration gap, expected pre-RLCD)
8	Emotion detection	"I cannot believe they cancelled the trip... I am so frustrated right now."	Which emotion does this text primarily express?	sadness / joy / love / anger / fear / surprise	fear (36.8%)	✗ wrong — this is anger (29.1%), not fear
9	Dynamic-option routing	"What is my current checking account balance and when is my next payment due?"	Which of these destinations should handle this?	balance / bill_due / pto_balance / credit_score / routing	pto_balance (47.2%)	✗ wrong — a checking-account question landing on vacation-time balance is a real miss
10	Content moderation	"I disagree with your policy but I think it is worth a calm discussion..."	Is this comment toxic or inappropriate?	false / true	false (95.7%)	✓ high-confidence, correct

Reproduce any of these yourself with the snippet in Usage — swap in the state/instruction/ options from the table.

Training data

Nine public sources, six of them kev's own training mix plus three added for broader coverage: Banking77 (choice, K=77), BoolQ (noul), AG News (choice K=4 + 2 derived noul), MultiNLI (choice K=3), SST-5 (score, 5 levels), Yelp Review Full (score + noul), plus dair-ai/emotion (choice K=6, held out entirely as a zero-shot family, matching how laya reports a held-out "emotion and tone" split), clinc_oos (choice, a random 4-12-option subset per record including the gold intent — a realistic routing call, not the full 151-way space), and google/civil_comments (noul, toxicity >= 0.5).

Augmentation (train split only): option order shuffled, 10% the true option's text replaced with a generic "other: None of the above", 15% an irrelevant distractor option added — all scoped to choice questions only (score options are ordered levels; noul is a fixed 2-way pair). Rendering variation: ~32% of states wrapped as structured JSON objects/arrays, ~15% of instructions wrapped as structured {"question", "focus"} objects, re-rolled fresh on every use rather than fixed at conversion time.

This release: 300 records per source (2,400 total), 1 epoch, cross-entropy loss (+ an ordinal regularizer for score questions), AdamW lr 2e-5, gradient accumulation 8, linear warmup/decay. Trained on a Mac (CPU; MPS hit a memory ceiling from another process on that machine and was not used for this run), ~42 minutes wall clock.

Intended use

Intended: research on decision models, specifically permutation robustness / order sensitivity in typed multiple-choice and routing tasks. Local demos, teaching, a base to fine-tune further.

Not intended: any production decision affecting people (moderation, fraud, credit, hiring, medical/legal routing). This is a v1 research checkpoint with a known accuracy gap versus kev and no calibration stage yet — its raw probabilities have not been checked for calibration at all (no temperature scaling has been applied, unlike kev's post-hoc scaling or laya's RLCD training).

Usage

import sys
sys.path.insert(0, ".")  # or wherever you've placed this repo's `ballot/` folder
from ballot.checkpoint import load_checkpoint
from ballot.hf_backbone import ModernBertBackbone
from ballot.hf_model import HFDecisionModelE
import torch

def factory(config):
    backbone = ModernBertBackbone(config["backbone_dir"], device="cpu")
    return HFDecisionModelE(backbone)

model, sidecar = load_checkpoint(".", factory)  # dir containing ballot.safetensors + ballot.json
model.eval()

with torch.no_grad():
    logits = model.logits(
        state="Order #4471: wireless mouse, delivered 2 days late.",
        instruction="Which team should this ticket be routed to?",
        options=["billing", "logistics", "returns", "technical_support"],
    )
    print(torch.softmax(logits, -1))

Limitations

• Shallow by design, for now. 1 epoch, 300 records/source vs. kev's 2 epochs / 1500 per source. Accuracy trails kev by 11 points on this quick eval.
• No calibration stage. Raw softmax probabilities are not temperature-scaled or otherwise calibrated. Do not read confidence values from this checkpoint as verified probabilities.
• No RLCD / proper-scoring-rule training yet — the training recipe this checkpoint uses is cross-entropy only, the part of the plan closest to kev's approach, not laya's.
• Single arm. Arm S (independent pointer readout, no listwise mixing) was designed as an A/B alternative and has not been trained or compared yet.
• In-distribution only, same caveat every model in this family carries: all numbers above are on held-out splits of the training sources (except the emotion family, held out entirely). Out-of-source generalization is unmeasured.
• No code package yet. The loading code lives directly in this repo's ballot/ folder, not a pip-installable library.

License

Apache-2.0. Base model jhu-clsp/mmBERT-base carries its own license (MIT); training datasets carry their own licenses.