scienceverse/bibr-paper-classifier

paper-classifier-papertype-selfdistill

paper-classifier-papertype-selfdistill is a model for the bibr stage extract/core metadata. Input: title + abstract, max 256 tokens. bibr loads it through the setting ML_PAPER_CLASSIFIER_MODEL_ID on the default, fast path. It replaces paper-classifier (scienceverse/bibr-paper-classifier at 6046171).

Files

File	Bytes	sha256 (first 16)
bibr_model.json	6,710	d78a035c09795389
onnx/bibr_onnx.json	2,198	3a99003f337514ff
onnx/model.onnx	90,470,082	63fa8d0f45c83605
onnx/tokenizer.json	711,649	851ca67100d372ca

The full checksums are in checksums.json.

Architecture

• encoder: sentence-transformers/all-MiniLM-L6-v2, initialised from the shipped checkpoint (6046171)
• head: mean pool + three linear heads (OECD L1 6, L2 32, paper type 7), class order unchanged
• input: title + abstract, max 256 tokens
• format: ONNX fp32 (90 MB), opset 17, parity with torch 1.5e-6 (max abs logit difference)

Training data

• rows: 24796 train (16796 PMC with a paper type, 8000 arXiv distillation-only); dev 3646, test 8304 (each with 1000 arXiv rows)
• papers: 47201 labelled PMC papers after the registry and the split (journal-disjoint train / dev 2646 / test 7304; 2209 dev and 2255 test papers share journals with train and are scored separately)
• labels: paper type from JATS article-type plus MEDLINE publication types plus title rules (src/bibr_training/paper_classifier/paper_type_labels.py); OECD heads get no hard labels
• languages: train rows English 24590, de 43, es 17, fr 60, pt 38, ru 5, zh 43 (non-English train papers kept)
• licences: CC BY, CC BY-SA, CC0 only (train rows 23714 / 528 / 554)
• notes: empirical capped at 9000 of the train papers; retraction has 19 train rows. The journal-disjoint step is a fixed-salt hash (195) on top of corpus-split-v1, kept so that the journal assignment is the first candidate's; with it the PMC test holds no retraction paper (dev has 4), so retraction is judged on the gold only

Every evaluation set in the exclusion registry was kept out of training. Registry: exclusions-b8, build manifest sha256 b7ec1d94d0915498. Split: corpus-split-v1, sha256 c6fc246571f435d7.

Evaluation

Set	Metric	Value	Shipped	Difference	Interval	Notes
PMC journal-disjoint test (7304)	paper_type_macro_f1	0.864	0.495	+0.369	difference 95% [0.342, 0.394]	vs_first_candidate: 0.86; diff_vs_first_ci95: -0.003, 0.009
PMC journal-disjoint test (7304)	paper_type_accuracy	0.952	0.919	+0.033		vs_first_candidate: 0.95
PMC test, class-capped at 300	paper_type_macro_f1	0.919	0.5	+0.419		vs_first_candidate: 0.917
PMC test, 5 classes the shipped head can emit	paper_type_macro_f1	0.858	0.599	+0.259		vs_first_candidate: 0.853
PMC test, meta-analysis merged into review	paper_type_macro_f1	0.859	0.596	+0.263		vs_first_candidate: 0.853
PMC in-journal test (2255)	paper_type_macro_f1	0.862	0.435	+0.427		vs_first_candidate: 0.831
non-English rows of the non-train PMC sets	paper_type_macro_f1	0.66	0.299	+0.361		tiny slice, interval wide; vs_first_candidate: 0.425
10000 held-out rows (OpenAlex dump 6000, arXiv test 2000, PMC test 2000)	l1_top1_agreement with shipped	0.973
10000 held-out rows	l2_top1_agreement with shipped	0.94
OpenAlex-topic sample (6000)	l1_accuracy	0.71	0.713	-0.003		vs_first_candidate: 0.711
OpenAlex-topic sample (6000)	l2_accuracy	0.547	0.553	-0.006		vs_first_candidate: 0.547
arXiv test proxy labels (2000)	l1_accuracy	0.864	0.863	+0.001		vs_first_candidate: 0.861
arXiv test proxy labels (2000)	l2_accuracy	0.768	0.773	-0.005		vs_first_candidate: 0.769
paper-type gold v1, all (362)	accuracy	0.768	0.442	+0.326	difference 95% [0.268, 0.387]	vs_first_candidate: 0.768; diff_vs_first_ci95: -0.019, 0.022
paper-type gold v1, all (362)	macro_f1	0.822	0.29	+0.532		vs_first_candidate: 0.809; diff_vs_first_ci95: -0.014, 0.054
paper-type gold v1, PMC (270)	accuracy	0.837	0.459	+0.378	difference 95% [0.304, 0.448]	vs_first_candidate: 0.819; diff_vs_first_ci95: -0.004, 0.041
paper-type gold v1, OpenAlex non-PMC (92)	accuracy	0.565	0.391	+0.174	difference 95% [0.076, 0.283]	vs_first_candidate: 0.62; diff_vs_first_ci95: -0.109, 0.0; macro_f1_diff_vs_first_ci95: -0.111, -0.01
paper-type gold v1, PMC where the two models disagree (60)	accuracy	0.683	0.3	+0.383		vs_first_candidate: 0.617
paper-type gold v1, OpenAlex where the two models disagree (54)	accuracy	0.407	0.111	+0.296		vs_first_candidate: 0.5
paper-type gold v1, recall per class	recall	case-study 0.732, commentary 0.698, empirical 0.755, erratum 0.972, meta-analysis 0.973, retraction 0.9, review 0.649	case-study 0.683, commentary 0.175, empirical 0.776, erratum 0.083, meta-analysis 0.0, retraction 0.0, review 0.545

Known weaknesses

• second look at the gold with a candidate. The gold papers were not a training or selection input, but a third look would no longer be a clean test; new data should go to a new gold
• on the gold the model ties the first candidate overall (0.768 vs 0.768, difference -0.019 to +0.022). Against it the model is better on PMC (+0.019) and worse on OpenAlex non-PMC papers (-0.054, interval -0.109 to 0.000; macro-F1 -0.050, interval -0.111 to -0.010), so outside PMC it is not an improvement on the first candidate
• derived labels (JATS + MEDLINE) agree with the gold on 86 percent of the sampled PMC papers; by derived class 83 percent (commentary) to 100 (erratum, retraction); where MEDLINE is silent and the label comes from JATS alone, 75 percent
• outside PMC the candidate scores 0.565 on 92 gold papers, below the first candidate's 0.620; training rows are PMC and arXiv only, and non-PMC rows with distant labels are the next step (not done). It is weak on case studies and commentary there; the training data has no non-biomedical paper-type labels
• on PMC it reads empirical papers as reviews, most of them in the disagreement sample
• the PMC test holds no retraction paper (the pinned journal salt left none there); retraction is judged on the 10 gold retraction papers (0.9 recall) and the 4 dev papers, which supports no per-class claim
• the PMC test macro-F1 of the shipped model reads 0.495 here against 0.416 in the first candidate's entry because the retraction class (F1 0) is no longer in the test; compare systems on the same rows only
• shipped accuracy on the gold reads 0.442 here and 0.439 in the first candidate's entry (same 362 papers; the cause of the 0.003 was not traced); the differences to the shipped model agree to about 0.005
• the corpus is biomedical; the OECD heads were checked on OpenAlex-topic and arXiv-category proxy labels, not on gold
• the non-English slice is small (tens of papers per language); no per-language claim
• the shipped head has 7 paper-type classes (case-study, commentary, empirical, erratum, meta-analysis, retraction, review; bibr_onnx.json paper_type_classes). The "8" counted before is bibr's output vocabulary (PAPER_TYPE_LABELS in bibr/structure/paper_classifier.py), which adds corrigendum. Neither the shipped head nor this one emits corrigendum; the training table has no such label
• the bundle loads through bibr's own OnnxPaperClassifierModel. In the whole pipeline with only this model changed (60-paper screening set) every front-matter score equals the shipped model's; the paper type itself has no pipeline gold there. The run's torch weights give the ONNX file's logits (largest difference 2.9e-6 on 12 inputs)

How it was made

• trainer script: scripts/train_paper_classifier_oecd_transformer.py
• config: configs/paper_classifier_paper_type_distill.yaml
• recipe: lr 2e-5, batch 32, bf16, 4 epochs (best = epoch 4 by dev paper-type macro-F1, 0.763; still rising), inverse-sqrt class weights on the paper-type head, KL self-distillation of L1 and L2 logits to the shipped model (T=2, weight 1 each) on every row including 8,000 unlabelled arXiv abstracts; 3 minutes on one GPU slot
• commit: a4cca55c2b0b42f7b767c94b4500eb4c56ae2f21
• seed: 42
• run record sha256: 52d8d8e3477b16bfe6d3045ebdd44f365aa01e4bcfeca0239977a63485e7e2e8

Licence and attribution

• model licence: apache-2.0
• training data licences: CC BY, CC BY-SA, CC0 only (train rows 23714 / 528 / 554)
• starts from: sentence-transformers/all-MiniLM-L6-v2, initialised from the shipped checkpoint (6046171); that model's own licence and card apply