ModernBERT-JEV (tasksource/modernbert-tasksource-jev)

ModernBERT-JEV is an open-source, high-throughput foundation model for typed decisions built on answerdotai/ModernBERT-base (149M parameters, 22 layers, bidirectional encoder, native context up to 8,192 tokens) equipped with an option-query cross-attention decision head.

Unlike traditional sequence classification (which concatenates each candidate option with the context and re-runs the entire encoder $K$ times, incurring $\mathcal{O}(K \times L)$ cost) or shared option-token budget architectures (such as Laya, which suffers severe token starvation on high-cardinality tasks like Banking77), ModernBERT-JEV:

1. Encodes context once ($H_{\text{context}} \in \mathbb{R}^{L \times d}$).
2. Encodes each candidate option independently with the shared ModernBERT encoder ($q_k \in \mathbb{R}^d$).
3. Queries context via cross-attention ($Q \to H_{\text{context}}$).
4. Is structurally permutation-equivariant: $\text{permute}(\text{options}) \equiv \text{permute}(\text{logits})$, guaranteeing zero position bias.

---

Architecture Specification

                         ┌────────────────────┐
question + state ───────►│                    │──── Hcontext [L, d]
                         │   ModernBERT-base  │
option 1 ───────────────►│   shared weights   │──── q1 [d]
option 2 ───────────────►│                    │──── q2 [d]
...                      └────────────────────┘
Q = [q1 ... qK]
         │
         ▼
cross-attention (8 heads)
Q queries Hcontext
         │
         ▼
Residual LayerNorm + MLP → K logits
         │
         ▼
Softmax over K options

Computational Complexity

Architecture	Context Encoding Cost	Option Encoding Cost	Cross-Attention / Interaction	Total Complexity for $K$ Options
Cross-Encoder per option (Standard CE)	$K \times \text{Cost}(L + M)$	Included in encoder	Self-attention over $(L+M)$	$\mathcal{O}(K \times (L + M)^2)$
Shared Option Budget (e.g. Laya)	$\text{Cost}(L + K \times M_{\text{fixed}})$	Shared token pool	Self-attention over sequence	$\mathcal{O}((L + K \cdot M_{\text{fixed}})^2)$ (cliff at $K > 30$)
ModernBERT-JEV (Ours)	$1 \times \text{Cost}(L)$	$\sum_{k=1}^K \text{Cost}(M_k)$	Cross-attention: $K \times L$	$\mathcal{O}(L^2) + \mathcal{O}(\sum M_k^2) + \mathcal{O}(K \times L)$

---

Canonical Decision Primitives

ModernBERT-JEV natively handles three canonical decision types conditioning on primitive embeddings:

1. choice: Categorical multiple choice (intent routing, topic classification, multi-class triage).
2. noul: Nuanced binary & policy compliance judgments (["no", "yes"]).
3. score: Bounded ordinal rating scales (e.g. 0 to 5) trained with universal Cross-Entropy combined with Ranked Probability Score (RPS) loss to preserve numerical distance geometry.

---

Empirical Benchmark Results

1. Zero-Shot NLP Transfer

The model was evaluated zero-shot across public benchmarks strictly firewalled from training data:

Benchmark	Classes ($K$)	ModernBERT-JEV Latency	ModernBERT-JEV Accuracy	ModernBERT-JEV NLL	ModernBERT-JEV ECE
AG News	4	5.18 ms/decision	24.07%	1.4056	0.0475
DAIR Emotion	6	4.74 ms/decision	29.55%	1.7587	0.0160
Banking77	77	11.93 ms/decision	2.05%	4.3652	0.0019

Throughput note: ModernBERT-JEV processes a 77-class decision on Banking77 in only 11.93 ms on an NVIDIA A30, demonstrating smooth scalability without out-of-memory errors or token truncation.

2. Typed Decisions Benchmark (LocalLLaMA/typed-decisions)

Evaluated across all 2,000 canonical decisions:

Primitive Kind	Accuracy	NLL	Brier Score	ECE / Metric
noul (Policy/Binary)	62.67%	0.6589	0.1671	—
score (Ordinal Rating)	20.88%	1.4371	0.2414	MAE: 0.7051
choice (Categorical)	16.83%	1.4747	0.2936	—
Overall Macro	32.20%	1.2149	0.2348	ECE: 0.0790

3. Internal Tasksource & Sarge Heldouts

Evaluation Split	Metric	Result
Sarge Test Set (30k)	Accuracy / NLL	68.20% (NLL: 0.7045)
Sarge Out-of-Distribution (OOD)	Accuracy / NLL	57.00% (NLL: 0.8802)
Tasksource Unseen Tasks (Test)	Accuracy / NLL	54.80% (NLL: 0.6908)
Tasksource Unseen Tasks (Dev)	Accuracy / NLL	49.20% (NLL: 0.6933)
Tasksource Seen Tasks (Dev)	Accuracy / NLL	44.60% (NLL: 0.9123)

---

Calibration & Behavioral Stress Tests

1. Probability Quality

ModernBERT-JEV achieves exceptional calibration natively from soft-target distillation:

Metric	Raw ($T=1.0$)	Status
Expected Calibration Error (ECE)	0.0291 (< 3.0%)	Optimal calibration directly out of the box
Negative Log-Likelihood (NLL)	0.9598	Cross-entropy across diverse validation mixture
Brier Score	0.4795	Multi-class proper scoring rule

2. Permutation Equivariance Verification

Tested across 150 validation decisions with 5 random option order permutations per item:

• Argmax Flip Rate: 0.0000% (100% stable predictions)
• Mean Jensen-Shannon Divergence: 0.000000e+00
• Max Jensen-Shannon Divergence: 0.000000e+00

3. Cardinality Scaling ($K = 2 \dots 255$)

Measured inference latency and GPU memory across variable option counts:

Candidate Options ($K$)	Latency (ms)	Peak VRAM (MB)	Scaling Behavior
$K = 2$	64.1 ms	311.3 MB	Baseline
$K = 4$	64.9 ms	311.3 MB	+0.8 ms
$K = 8$	71.2 ms	311.4 MB	+7.1 ms
$K = 16$	72.5 ms	311.9 MB	+8.4 ms
$K = 32$	74.1 ms	313.9 MB	+10.0 ms
$K = 64$	66.8 ms	318.8 ms	Sub-linear
$K = 128$	68.6 ms	326.8 MB	Constant overhead
$K = 255$	75.7 ms	342.5 MB	Only +31.2 MB VRAM for 255 options

---

Quickstart & Inference

import torch
from transformers import AutoTokenizer
from modernjev.model import ModernJevModel
from modernjev.inference import predict

device = "cuda" if torch.cuda.is_available() else "cpu"
repo_id = "tasksource/modernbert-tasksource-jev"

# 1. Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = ModernJevModel.from_pretrained(repo_id).to(device)

# 2. Categorical Choice (e.g. customer request intent)
choice_probs = predict(
    state="The customer contacted support saying: 'I lost my card while traveling abroad and need a replacement urgently.'",
    question="Identify the primary intent:",
    options=[
        "card_replacement",
        "pin_reset",
        "international_transaction_fees",
        "account_statement",
    ],
    kind="choice",
    model=model,
    tokenizer=tokenizer,
    device=device,
)
print("Choice probabilities:", choice_probs)

# 3. Policy Compliance / Noul Judgment
noul_probs = predict(
    state="Company Policy: Travel expenses above $100 require pre-approval from a director. An employee submitted $140 for meals without pre-approval.",
    question="Is this expense claim approved?",
    options=["no", "yes"],
    kind="noul",
    model=model,
    tokenizer=tokenizer,
    device=device,
)
print("Policy decision:", noul_probs)

# 4. Ordinal Score (0 to 5)
score_probs = predict(
    state="User Feedback: 'The product arrived on time, was packaged well, and works as described.'",
    question="Rate satisfaction from 0 to 5:",
    options=["0", "1", "2", "3", "4", "5"],
    kind="score",
    model=model,
    tokenizer=tokenizer,
    device=device,
)
print("Score distribution:", score_probs)

---

Training Reproducibility

The training procedure is 100% deterministic and reproducible.

1. One-Line Reproduction

git clone https://github.com/tasksource/train_jev.git
cd train_jev
bash scripts/reproduce_training.sh

2. Hyperparameter Specifications

Parameter	Value	Rationale
backbone	answerdotai/ModernBERT-base	149M parameters, 22 layers, 768 hidden dimension
data_mixture	70% Tasksource / 30% Jev-native	Hierarchical sampling ($P(t) \propto n_t^{0.30}$)
effective_batch_size	128	Micro-batch 32 $\times$ 4 gradient accumulation steps
optimizer	AdamW	Backbone LR: $2.0 \times 10^{-5}$, Head LR: $2.0 \times 10^{-4}$
scheduler	Linear with warmup	5% warmup ratio, linear decay to 0
weight_decay	0.01	Applied to non-bias weights
precision	bfloat16	Native bf16 mixed precision
context_max_len	512	Question first, truncates state before question
option_max_len	32	Flattened options tokenized independently
score_rps_weight	0.20	Ranked Probability Score weight for ordinal distance
seed	42	Fixed RNG seed across PyTorch, NumPy, and Python

---

License & Citation

• License: Apache 2.0
• Model Organization: tasksource