PakLID v1

Character 1-5-gram TF-IDF + calibrated linear SVM that identifies the Perso-Arabic-script languages of Pakistan (plus Roman Urdu, Persian, Arabic and English as confusers). Built by Balagh (Ownova) to re-sort web-corpus buckets whose upstream language labels are unreliable.

Held-out (document-grouped) macro F1: 0.974, accuracy 0.981 on 7,209 chunks.

label	precision	recall	f1	support
arb_Arab	0.965	0.995	0.980	442
bal_Arab	0.963	0.892	0.926	204
brh_Arab	0.976	0.929	0.952	309
eng_Latn	0.987	1.000	0.993	444
fas_Arab	0.985	0.996	0.990	453
hno_Arab	0.933	0.970	0.951	101
kas_Arab	1.000	0.986	0.993	422
pbt_Arab	0.998	0.995	0.997	613
phr_Arab	1.000	0.899	0.947	109
pnb_Arab	0.975	0.991	0.983	799
skr_Arab	0.984	0.954	0.968	693
snd_Arab	1.000	0.993	0.997	764
urd_Arab	0.930	0.979	0.954	729
urd_Latn	1.000	0.995	0.997	1127

Usage: normalise exactly as in training (digits to ASCII, tatweel/zero-width removed, Arabic yeh/kaf folded to Urdu forms, lower-cased), then joblib.load(hf_hub_download('balagh-lm/paklid', 'paklid-v1.joblib')).predict_proba([text[:500]]). Code: lid_norm and base_norm in phase0/colab/01_build_corpus.py at github.com/balagh-lm/balagh.