KYS-Modernbert-Quality-Scorer

The distilled document-quality scorer from Know Your Sources: Data Selection Matters when Rewriting for Data-Constrained Pretraining — one of the three scorers whose disagreement drives the DISAGREEMENT-AWARE selection strategy.

It distils Claude Haiku 4.5's judgements into something cheap enough to run over 100M documents: a closed-form ridge head on frozen ModernBERT embeddings.

Architecture

text -> nomic-ai/modernbert-embed-base (FROZEN, fp16, max_len 2048)
     -> mean-pool over non-padding tokens -> 768-d
     -> L2-normalise
     -> Ridge(alpha=1.0) -> continuous score in [0, 5]

The backbone is frozen and unmodified, so it is not redistributed here. Pull it from nomic-ai/modernbert-embed-base at the pinned revision d556a88e332558790b210f7bdbe87da2fa94a8d8. The only trained artifact in this repo is the 3.5 KB ridge head.

⚠️ L2 normalisation is applied at inference time, not baked into a pipeline. Skipping it changes the scores. There is no scaler.

Files

File	Size	Contents
model.pkl	3,512 B	pickled sklearn.linear_model.Ridge, alpha=1.0, n_features_in_=768, float32 coef_
model_meta.json	323 B	backbone id, embed dim, n_train=50427, in-sample Spearman 0.7406
metrics_summary.json	488 B	the headline CV numbers
results_per_seed.json, results_table.csv		per-seed / per-fold detail
args.json	808 B	full training hyperparameters
summary.md		the backbone × normalisation ablation table
embeddings.npy	155 MB	float32 (50427, 768) — the frozen embeddings the head was fitted on
scores.npy		the matching Claude Haiku targets
embeddings_metadata.json		pooling / max_length / fp16 / prefix settings
train_and_eval.py		fits the head and runs the CV protocol
score_modernbert.py		applies the scorer at corpus scale

model.pkl was written by scikit-learn 1.8.0; load it with a compatible version.

Performance

5-fold stratified CV over ten score bins, repeated across seeds {42, 123, 777, 2024, 2025} — 25 runs.

Metric	Value
Spearman ρ	0.7314 ± 0.0037
MSE	0.8532 ± 0.0086
Top-10% precision	0.3288 ± 0.0122

Backbone and normalisation ablation (all ridge, α = 1.0):

Backbone	L2 norm	Spearman ρ	MSE	Top-10%
nomic-embed-text-v1.5	no	0.6717 ± 0.0055	1.0342 ± 0.0128	0.2874 ± 0.0095
nomic-embed-text-v1.5	yes	0.6723 ± 0.0060	1.0353 ± 0.0132	0.2950 ± 0.0099
modernbert-embed-base	no	0.7300 ± 0.0038	0.8548 ± 0.0090	0.3253 ± 0.0101
modernbert-embed-base	yes	0.7314 ± 0.0037	0.8532 ± 0.0086	0.3288 ± 0.0122

Usage

import pickle, torch, numpy as np
from transformers import AutoTokenizer, AutoModel
from huggingface_hub import hf_hub_download

REV = "d556a88e332558790b210f7bdbe87da2fa94a8d8"
tok = AutoTokenizer.from_pretrained("nomic-ai/modernbert-embed-base", revision=REV)
enc = AutoModel.from_pretrained("nomic-ai/modernbert-embed-base", revision=REV,
                                torch_dtype=torch.float16).eval().cuda()
ridge = pickle.load(open(hf_hub_download("blab-jhu/KYS-Modernbert-Quality-Scorer", "model.pkl"), "rb"))

def score(texts):
    b = tok(texts, padding=True, truncation=True, max_length=2048, return_tensors="pt").to("cuda")
    with torch.no_grad():
        h = enc(**b).last_hidden_state
    m = b["attention_mask"].unsqueeze(-1).to(h.dtype)          # mean-pool over non-padding
    e = (h * m).sum(1) / m.sum(1)
    e = torch.nn.functional.normalize(e.float(), p=2, dim=1)   # L2 -- required
    return ridge.predict(e.cpu().numpy())

print(score(["The mitochondrion is the powerhouse of the cell.", "CLICK HERE!!! buy now cheap"]))

No prefix is prepended to the text (prefix: ""), unlike the nomic-embed convention.

How it was built

1. 50,427 documents annotated by Claude Haiku 4.5 on a five-criterion binary rubric summing to 0–5 → KYS-Claude-Haiku-50K-Labeled.
2. Each encoded with frozen ModernBERT, mean-pooled, L2-normalised → embeddings.npy.
3. A closed-form ridge head fitted on those fixed embeddings.

The other two scorers

This is the only scorer trained for the paper. The other two are used as published:

Scorer	Repo	Revision
DCLM fastText	mlfoundations/fasttext-oh-eli5	cd8b714a90f2dbcd3b02cf5fc972e5d7c7f4f107
FineWeb-Edu classifier	HuggingFaceFW/fineweb-edu-classifier	284663cbb2dabf9bda30d8f8cc49601251ee1631

The three reward different notions of quality — similarity to curated reference data, educational value, and general pretraining utility — and agree only moderately (pairwise Spearman 0.414–0.451). That disagreement is the signal the DISAGREEMENT-AWARE strategy exploits.

The rest of the release

Repo	What it holds
KYS-1.5B-Quality-Base	QUALITY-BASE — non-rewritten baseline
KYS-1.5B-Quality-First	QUALITY-FIRST
KYS-1.5B-Diversity-Oriented	DIVERSITY-ORIENTED
KYS-1.5B-Disagreement-Aware	DISAGREEMENT-AWARE (λ = 0.5)
KYS-1.5B-Wrap-Inspired	WRAP-INSPIRED
KYS-1.5B-Rewire-Inspired	REWIRE-INSPIRED
KYS-Modernbert-Quality-Scorer	the distilled ModernBERT ridge quality scorer
KYS-DCLM-Refinedweb-100M-Scored	the candidate pool with all scores
KYS-Claude-Haiku-50K-Labeled	the Claude Haiku annotations behind the scorer
KYS-1.5B-Pretraining-Corpora	the shared anchor + six strategy remainders
KYS-Configs	prompts, vLLM, Nanotron and eval configs + shared init weights

Citation

@misc{kys2026,
  title  = {Know Your Sources: Data Selection Matters when Rewriting for Data-Constrained Pretraining},
  author = {TODO},
  year   = {2026},
  note   = {TODO: fill in venue / arXiv id / URL}
}