influence-taste-56m: a 17-taste data-influence predictor

A 56M-parameter predictor that scores web documents by their predicted training influence on 17 held-out text categories — not "quality" as a scalar, but a 17-dimensional taste vector. Trained to predict gradient-alignment influence labels: for each document, the cosine between its training gradient and the mean held-out gradient of each category, measured at a mid-training reference checkpoint.

Built and validated end-to-end: selecting pretraining data with these scores produces the predicted effects in actual continued-pretraining runs (details below).

Categories (17 heads, val Spearman vs true gradient labels)

head	ρ		head	ρ
ao3_english	0.605		github_cpp	0.480
ao3_nonenglish	0.332		github_javascript	0.503
arxiv_cs	0.602		github_markdown	0.573
arxiv_math	0.453		github_other	0.574
arxiv_other	0.629		github_python	0.585
arxiv_physics	0.533		wikipedia_english	0.489
bbc_news	0.635		wikipedia_nonenglish	0.330
biorxiv_all	0.618		hn_threads	0.528
			reddit_threads	0.578

Mean 0.532 (tf-idf ridge baseline: 0.457). The 15 base categories are the UncheatableEval 2026-01 test sets; hn_threads and reddit_threads are our additions — linearized full-thread documents from the Hacker News and Reddit January 2026 firehoses, covering the natural-dialogue register UncheatableEval lacks.

How the labels were made

1. A 56M LLaMA-style reference model (PleIAs/Monad 8k tokenizer, 512×12) was pretrained on ~0.9B tokens of dclm-baseline to its end-of-stable-LR checkpoint.
2. Per category: mean gradient of per-doc mean-token CE over the held-out month, unit-normalized.
3. Per candidate doc (100k fresh dclm docs): cosine of the doc's gradient against each category gradient → 17 labels.
4. This model = the same 56M backbone + mean-pool + 17-way linear head, regression on z-scored labels.

Validated selection results (56M student, 90M-token arms)

• top-by-mean-score < random < bottom on macro held-out loss (2.74 / 2.82 / 2.97); ordering holds on 14/15 categories.
• Scalar selection starves minority registers: top-selection makes fiction and forum worse than random — their supply lives in the middle and bottom of any scalar quality ranking (49% of the best forum docs are in the bottom half).
• Best use found: train one cheap specialist per taste-cluster (fiction / sci / code / news-wiki / forum) on its own head's ranking, then equal linear-merge the specialists. The 5-specialist panel merge beat every data-mixing strategy tried, on every category.

Scope and limitations — read before using

• This is a mid-training curriculum signal, not a polish for mature models. Transferring the same selections to a fully-pretrained Qwen3.5-0.8B reproduced the directions (bottom-data harm, fiction/forum starvation) but ~20× weaker, with no macro gain from top-selection: a converged model has already extracted what ranked common-crawl teaches. Use it to order/filter data for models still in their steep regime.
• English-centric: Monad's 8k vocab byte-falls-back on non-English; the two nonenglish heads are the weakest (ρ ≈ 0.33).
• The taste space is UncheatableEval's plus our forum additions. No math- problem, textbook, or chat-assistant category; "good for X" here means good for predicting X-register text.
• Scores are z-scored per head (mu/sd in the checkpoint); compare docs within a head, don't compare raw magnitudes across heads.
• Intended for docs of roughly 200+ chars (trained on 64–2048-token docs); scores on short fragments are unreliable. Longer docs score somewhat higher on average (length is partially bundled into influence).

Usage

from score_documents import load_predictor, score
model, tok, cats, mu, sd = load_predictor(".", device="cuda")
preds = score(model, tok, ["your document text ..."])   # (N, 17)

Or python score_documents.py docs.txt (one document per line, JSONL out). Throughput ≈ 200 docs/s on one RTX 3090.

Files

• predictor.pt — full state dict (backbone + head), z-score stats, head names
• backbone_config.json — 56M LLaMA config for AutoModel.from_config
• score_documents.py — standalone loader + scorer (tokenizer pulled from PleIAs/Monad)