Rishabh157/wikiann-multilingual-ner-mdeberta

WikiANN Multilingual NER (mDeBERTa-v3-base)

This model is a robust Multilingual Named Entity Recognition (NER) model fine-tuned on the WikiANN (PAN-X) benchmark across 7 major languages.

Built on top of microsoft/mdeberta-v3-base, it provides highly accurate extraction for the core classic NER entities: Person (PER), Organization (ORG), and Location (LOC) (7 BIO output logits).

Developed by Rishabh Kumar.

---

1. Model Description & Architecture

• Base Model: microsoft/mdeberta-v3-base (278M parameters, 12 layers, 768 hidden size, 250k multilingual vocabulary)
• Fine-Tuning Method: LoRA (r=32, alpha=32, target modules: query_proj, key_proj, value_proj, intermediate.dense, output.dense) merged into standalone base weights.
• Precision: FP16 (torch.float16, ~530 MB footprint)
• Task: Multilingual Classic Token Classification / NER
• Languages Supported (7): English (en), German (de), Spanish (es), French (fr), Italian (it), Portuguese (pt), Swedish (sv).

---

2. Dataset & Preprocessing Pipeline

A. Source Data & Splits

• Source: WikiANN (PAN-X) benchmark (unimelb-nlp/wikiann).
• Train Set: 140,000 sentences (20k per language across 7 languages).
• Validation Set: 70,000 sentences (10k per language).
• Test Set: 70,000 sentences (10k per language).

B. Preprocessing & Formatting

1. CoNLL Standardization: Raw PAN-X splits are formatted into clean 2-column CoNLL format (token tag) with sentence boundary separation.
2. Subword Token Alignment: Aligned with mDeBERTa-v3-base SentencePiece subword tokenizer with add_prefix_space=True. Leading subwords receive the ground-truth BIO tag (B-PER, B-LOC, B-ORG), and following subwords are masked with -100.
3. Tag Space: Clean 7-tag output vocabulary (O, B-PER, I-PER, B-LOC, I-LOC, B-ORG, I-ORG).

---

3. Entity Taxonomy (3 Classes / 7 BIO Tags)

Entity Tag	Category	Description
PER	Person	Names of individuals, historical figures, politicians, artists
ORG	Organization	Companies, institutions, government bodies, sports teams
LOC	Location	Countries, cities, geographical regions, mountains, rivers

---

4. Training Results & Metrics

The model was fine-tuned for 5 epochs (6,250 steps) using AdamW (lr=2e-5, linear scheduler, effective batch size 16).

Validation Performance (Best Checkpoint - Epoch 5)

Metric	Score
Validation Entity F1	0.7814 (78.14%)
Validation Precision	0.7570 (75.70%)
Validation Recall	0.8075 (80.75%)
Validation Accuracy	0.9129 (91.29%)
Validation Loss	0.3134

Per-Epoch Progression

Epoch	Step	Precision	Recall	Entity F1	Accuracy	Loss
1	1,250	68.39%	75.61%	71.82%	89.04%	0.3800
2	2,500	73.27%	79.12%	76.08%	90.59%	0.3361
3	3,750	73.99%	79.78%	76.78%	91.01%	0.3234
4	5,000	75.58%	80.66%	78.04%	91.28%	0.3132
5	6,250	75.70%	80.75%	78.14%	91.29%	0.3134

---

5. How to Use

Quick Inference via Hugging Face Pipeline

from transformers import pipeline

ner = pipeline(
    "token-classification",
    model="Rishabh157/wikiann-multilingual-ner-mdeberta",
    aggregation_strategy="simple"
)

text = "Barack Obama visited the United Nations headquarters in New York City."
entities = ner(text)

for entity in entities:
    print(f"{entity['word']:<25} | {entity['entity_group']:<15} | Score: {entity['score']:.4f}")

PyTorch Direct Usage

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

model_name = "Rishabh157/wikiann-multilingual-ner-mdeberta"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

text = "Albert Einstein wurde in Ulm geboren und lebte in Bern."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.argmax(outputs.logits, dim=2)

tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
labels = [model.config.id2label[p.item()] for p in predictions[0]]

for token, label in zip(tokens, labels):
    if label != "O":
        print(f"{token:<20} -> {label}")

---

6. Citation

@misc{kumar2026wikiann,
  author = {Rishabh Kumar},
  title = {WikiANN Multilingual NER with mDeBERTa-v3},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Rishabh157/wikiann-multilingual-ner-mdeberta}}
}