Publication Country Classifier
Given a scholarly publication (a title, optionally followed by an abstract) in English, French, Spanish, German or Portuguese, this model returns the countries the text mentions. It covers the 247 OpenAlex countries: ISO 3166-1 alpha-2 codes, plus XK for Kosovo.
It is meant to fill gaps in the geographic indexing of bibliographic databases.
- Base model:
jhu-clsp/mmBERT-base, a multilingual ModernBERT-style encoder, with a 247-way multi-label sigmoid head. - Training data:
Geraldine/pub-country, about 146k OpenAlex works. Their labels come from multilingual rules and were then checked by an LLM in two steps. - Code and full methodology (in French): https://github.com/gegedenice/country-classifier
En français : ce modèle détecte les pays mentionnés dans le titre et le résumé d'une publication scientifique, en cinq langues. Il sert à l'indexation géographique assistée. La méthodologie complète est sur GitHub.
What it predicts
A country is predicted when the text names it explicitly: its name, a variant or a demonym ("France", "USA", "Côte d'Ivoire", "Kenyan", "brasileiro"), in its geographic or political sense. Historical and colonial forms count, and so do dependent territories.
The following are not mentions: homonyms (Anguilla anguilla, Salvador de Bahia, New Jersey, Georgia in the US), person names, journal or society names, and language names. The model does not judge whether the paper is about the country: a passing mention is predicted like the main study area.
Decision rule: a country is predicted when its probability is ≥ 0.5. The same threshold applies to every country (thresholds.json).
Usage
import json, torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer, AutoModelForSequenceClassification
repo = "Geraldine/pub-country-classifier"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
codes = json.load(open(hf_hub_download(repo, "codes.json")))
thr = json.load(open(hf_hub_download(repo, "thresholds.json"))) # 0.5 for every country
text = "La pauvreté rurale au Sénégal et au Mali" # title, then "\n" + abstract if any
with torch.no_grad():
p = torch.sigmoid(model(**tok(text, truncation=True, max_length=512, return_tensors="pt")).logits)[0]
print([(c, round(float(x), 3)) for c, x in zip(codes, p) if x >= thr[c]]) # [('ML', 0.99), ('SN', 0.99)]
Evaluation
Results on the test split (7,954 works) against the dataset's final labels, with a single 0.5 threshold:
| subset | precision | recall | F1 (micro) | F1 (macro) | precision of the top country | false-positive rate on no-country rows | n |
|---|---|---|---|---|---|---|---|
| all | 0.977 | 0.943 | 0.960 | 0.955 | 0.992 | 4.7 % | 7,954 |
| English | 0.985 | 0.960 | 0.973 | 0.965 | 0.996 | 4.2 % | 4,899 |
| French | 0.964 | 0.909 | 0.936 | 0.903 | 0.991 | 6.3 % | 1,070 |
| Spanish | 0.963 | 0.919 | 0.940 | 0.902 | 0.975 | 6.1 % | 933 |
| German | 0.965 | 0.913 | 0.938 | 0.887 | 0.988 | 3.2 % | 618 |
| Portuguese | 0.949 | 0.902 | 0.925 | 0.864 | 0.980 | 5.6 % | 434 |
| title only | 0.978 | 0.955 | 0.967 | 0.962 | 0.988 | 3.1 % | 534 |
- The test labels come from rules and an LLM check, not from human annotators, so they contain a few percent of noise. Some of the "errors" counted above are actually correct predictions that the reference misses.
- 228 of the 246 countries present in the test split have an F1 ≥ 0.9. The weakest are rare territories with only 0 to 20 test examples (
MP,PM,JE,WF,MF,IO,KN,GE), so their scores are not reliable. Details are inper_label_metrics.json.
Examples
| input | prediction |
|---|---|
| Ebola outbreak response in the Democratic Republic of the Congo | CD |
| Soil survey of New Jersey | – |
| Published in the American Journal of Physics: a new laser cooling method | – |
| Circadian rhythm in the eel Anguilla anguilla | – |
| Urban violence in Salvador, Bahia | – |
| MEDICAL LEGISLATION FOR THE ISLE OF MAN | IM |
| A agricultura familiar no semiárido brasileiro | BR |
| Migrationspolitik in Deutschland und Österreich | DE, AT |
| Napoleon's exile on Saint Helena | SH |
| Enseignement du français langue étrangère à l'université | – |
| Urban heat islands in Nairobi | KE (0.87: the country is not named; a generalisation beyond the definition) |
| Grammar of Romanian dialects | RO (0.91: a language name, a known weakness) |
Training
| data | Geraldine/pub-country, train split: 145,781 rows, 24 % with no country |
| epochs | 2 (about 9,100 steps) |
| batch | 16 × 2 gradient accumulation = 32 |
| learning rate | 3e-5, linear decay, warmup on 5 % of the steps |
| weight decay / gradient clipping | 0.01 / 1.0 |
| max length | 512 tokens |
| precision | bf16 |
| loss | BCEWithLogitsLoss (multi-label) |
| hardware | 1 × NVIDIA A10G (Hugging Face Jobs), 1 h 24 |
| seed | 42 |
Limitations
- Weak supervision. The labels come from rules checked by an LLM; no human annotation was done.
- Explicit mentions only. A country referred to only through a city or region is not reliably detected.
- Mention is not topic. Passing mentions are detected like the main study area.
- Known weak spots: language names ("Romanian dialects") and some borderline homonyms ("Bermuda grass").
- Languages. The model was trained on five languages. Other languages may work through mmBERT, with no guarantee.
- Input length. Inputs are truncated to 512 tokens.
Intended use: assisted geographic indexing (suggestions checked by a person or filtered by score) and exploration of bibliographic corpora. Out of scope: inferring authors' nationality or affiliation, and any decision about people.
- Downloads last month
- -
Model tree for Geraldine/pub-country-classifier
Base model
jhu-clsp/mmBERT-base