EuroVoc Multilabel Classifier 🇪🇺 (multilingual)

EuroVoc is the EU's large multidisciplinary, multilingual (24 languages) hierarchical thesaurus of more than 7,000 concepts covering the activities of the EU institutions. Given the volume of legal documents produced daily, high-quality automated or semi-automated subject classification is valuable in this domain.

This version supersedes the earlier English-only RoBERTa classifier: it is built on the multilingual jhu-clsp/mmBERT-base encoder and reads documents in all 24 official EU languages natively, predicting 7,062 EuroVoc concept labels (multi-label).

Architecture

ModernBertModel (mmBERT) encoder, then a masked mean-pool over token states, then a single linear head (classifier.pt) giving one logit per concept (sigmoid, BCE). No trust_remote_code is needed: the encoder is a standard ModernBertModel and the head loads from classifier.pt.

What it is for

A tagging assistant: it proposes tags, a person confirms them. At the broad subject level roughly 3 of 4 suggestions are correct and it finds about 7 of 10 of the correct ones. It is not accurate enough to tag documents unattended at the most specific level, where it must pick one exact concept out of 7,062.

How good it is

Held-out validation set, 800 batches. EuroVoc is a 3-level hierarchy, and accuracy depends heavily on which level you ask for.

level	picks from	F1 (overall)	F1 (per label)	threshold
Domain	21 broad subjects	0.733	0.683	0.70
Micro-thesaurus	127 topics	0.621	0.473	0.80
Concept	7,062 exact tags	0.429	0.045	0.90

F1 (overall) weights every prediction equally; F1 (per label) averages each label's own score, so it is dominated by the thousands of rare concepts and sits much lower. Use the per-level thresholds above, not 0.5: they are the tuned operating points.

Choosing an operating point

use	level	threshold	suggestions correct	correct tags found
default (suggest-and-confirm)	Domain	0.75	74%	72%
precision-critical	Domain	0.90	80%	63%
mid-level suggest	Micro-thesaurus	0.80	61%	63%

Raising the threshold trades finding fewer tags for being right more often.

Languages

Handles all 24 scripts natively, with no unreadable characters, and quality is close to uniform: best Swedish 0.507 and Spanish 0.500, English 0.488, lowest Maltese 0.466 and Greek 0.468 (the share of a document's correct tags landing in the model's top 5). An earlier version of this tagger used an English-only tokenizer and could not read Greek or Cyrillic at all (65 to 68% of their characters were unknown); moving to mmBERT's multilingual tokenizer fixed that, and it is where much of the overall gain came from. Croatian and Irish have under 40 validation documents each, so treat their numbers as noise.

What makes it work

The one idea behind this model: each of the 7,062 concept classifiers is seeded from the meaning of its own EuroVoc name rather than a random start, so rarely-seen concepts inherit language understanding they could never learn from their handful of training documents. Measured against an earlier version of the model without this step (same architecture, same data, same evaluation), it lifts concept-level F1 by 11% (0.385 to 0.429) and the rare-concept score by 20% (0.038 to 0.045), with gains at every level.

Known limits

• Picks a sibling or parent concept instead of the exact one. The model reliably reads the subject but can choose a neighbouring vocabulary term at the most specific level. On a wood-industry document it predicted "wood product" and "sustainable forest management" where the annotator had written "wood industry" and "sustainable development": correct understanding, wrong vocabulary choice. This is the main reason concept-level scores trail the coarser levels, and the main reason to operate at Domain or Micro-thesaurus level for unattended use.
• Rare concepts are still under-predicted. Most of the 7,062 tags appear in very few documents; the name-based initialisation reduced this (per-label F1 +20%) but did not erase it. The concept-level tail remains the weakest area.
• Long documents are cut at ~2,048 tokens (about 68% fit whole; the median fits).

How it was trained

Fine-tuned from jhu-clsp/mmBERT-base on the EuroVoc CELLAR corpus (~327,000 unique documents, 24 official languages). Trained in two stages on 3x NVIDIA RTX A5000, bf16 mixed precision, effective batch 216: a main run with the classifier seeded from EuroVoc name embeddings (learning rate 5e-5, ~9.9 passes over the data), then a short low-rate anneal (7.5e-6, ~2.2 passes) that produced the shipped checkpoint. Total training time roughly 35.5 hours (a 29h main run plus a 6h anneal), about 107 GPU-hours.

Usage

The repo ships a predict.py, or inline:

import torch, pickle, json
from huggingface_hub import hf_hub_download as dl
from transformers import AutoTokenizer, AutoModel
R = "EuropeanParliament/EuroVoc"           # add revision="<branch>" while staged
tok = AutoTokenizer.from_pretrained(R); enc = AutoModel.from_pretrained(R).eval()
head = torch.load(dl(R, "classifier.pt"), weights_only=True)
mlb = pickle.load(open(dl(R, "mlb.pickle"), "rb")); names = json.load(open(dl(R, "names.json")))
b = tok("your EU document text", truncation=True, max_length=2048, return_tensors="pt")
with torch.no_grad():
    h = enc(**b).last_hidden_state; m = b['attention_mask'].unsqueeze(-1).float()
    p = ((h*m).sum(1)/m.sum(1).clamp(min=1e-9) @ head['weight'].T + head['bias']).sigmoid()[0]
for s, i in zip(*p.topk(10)):
    if s > 0.9:   # concept threshold; MT 0.80, DO 0.70
        print(names.get(str(mlb.classes_[i]), mlb.classes_[i]), float(s))

Authors

Original EuroVoc classifier (English RoBERTa): Sébastien Campion, Andreas Papagiannis (European Parliament).

Multilingual mmBERT version: Harun Kalkanci, Andreas Papagiannis (European Parliament).