piedomains-text

Classifies a website into one of 47 content categories from its page text. The default model behind piedomains.

Numbers, and which one to believe

measured on	what it is	accuracy	macro-F1
Shallalist held-out	the corpus it trained on	0.797	0.727
Curlie x Tranco	independent human labels, popular domains	—	0.543 agreement
hand-labelled eval	49 domains, single gold label	0.714	—

Trust the Curlie figure. Those labels come from Curlie's human editors and share neither the taxonomy nor the selection bias of the training corpus; the held-out number is measured on the same blocklist-derived corpus the model learned, so it flatters.

Temperature-scaled: T = 1.7757, ECE 0.0997 to 0.0095.

parked is the best class in the model

F1 0.992 on 378 held-out documents. Parking placeholders were 7.9% of the training corpus and concentrated hard -- 42% of drugs, 23% of webmail -- because expired domains in those niches get parked. The previous model had learned that a "this domain is for sale" template means drugs and returned it for zappos.com and suicidepreventionlifeline.org. It is now its own answer.

Punctuation is kept, and that was measured

An earlier candidate stripped standalone punctuation -- table pipes and layout dashes are 4.8% of tokens and 40% of the p99 page, so removing them looked obviously right. Trained both ways on otherwise identical corpora it was worse: Curlie 0.543 to 0.523, held-out macro-F1 0.7267 to 0.7134, and deadspin.com went back to gamble. Structural punctuation says something about what kind of page it is.

What the corpus cannot tell you

Trained on Shallalist, a blocklist: 59% of it is blockable categories, porn alone was 53%, and 39% of its domains no longer resolve. It over-specifies what a content filter cares about and collapses the ordinary web. aggressive is F1 0.154 on 9 examples -- treat the long tail accordingly.

Reproducing this

pip install piedomains
cd "$(classify_domains --training-scripts)"
cat kaggle/train_text_kaggle.py