applied-nlp-ner-stage3-distilbert

Stage 3 output of an applied-NLP NER project: distilbert-base-cased fine-tuned for token classification on applied-nlp-ner-stage2-expanded-798 (679 train / 119 test records), predicting the 8-label schema PERSON, ORGANIZATION, LOCATION, TIMEDATE, PRODUCT, WORKOFART, JOB, AMOUNT via a standard BIO tag scheme (17 tags total).

Why this model

distilbert-base-cased (65M params) was chosen as the smallest reasonable encoder for this task: it keeps case information, which is one of the strongest signals for PERSON/ORGANIZATION/LOCATION recognition (an uncased model, or a much smaller model like BERT-mini, would throw that away). DistilBERT halves BERT-base's parameter count and layer count via knowledge distillation while retaining most of its token-level performance, which matters here since training and inference both ran on CPU.

Size / resource footprint

Parameters	65,203,985
Checkpoint size on disk	249.4 MB
Training time (CPU, 8 epochs, 679 records)	1384 s (~23.1 min)
Inference throughput (CPU, batch 16)	19.8 records/sec

No GPU was available in the training environment; all numbers above are CPU-only.

Results on the Stage 2 test split (119 records)

Overall: precision 0.658, recall 0.717, F1 0.686, token accuracy 0.945.

Label	Precision	Recall	F1
PERSON	0.871	0.885	0.878
ORGANIZATION	0.577	0.619	0.597
LOCATION	0.598	0.792	0.682
TIMEDATE	0.651	0.726	0.686
PRODUCT	0.375	0.600	0.462
WORKOFART	0.388	0.491	0.433
JOB	0.714	0.667	0.690
AMOUNT	0.761	0.700	0.729

PERSON is the strongest label (large, easy-signal training class). PRODUCT and WORKOFART are the weakest, which tracks their size in the training set (138 and 295 spans respectively, versus 351-666 for every other label) — see the "Known limitation" note in the Stage 2 dataset card.

QA on out-of-distribution samples

The model was also run on 20 hand-written sentences that appear nowhere in Stage 1 or Stage 2 (see data/stage3/wild_samples.json / data/stage3/wild_predictions.json in the project repo), to see how it behaves outside the register mix of Wikipedia/Wikinews/Gutenberg/bill-text. Ten concrete, reproducible problem patterns came out of that pass; the full list with examples is in the project report (report/REPORT.md, Stage 3 section). In short: multi-token span truncation (entities cut off after the first token), a handful of spurious single-stopword spans ("The", "the" tagged alone), and label confusion between ORGANIZATION/LOCATION/PRODUCT for proper nouns whose surface form is ambiguous out of context (e.g. a sports club sharing its name with a city) were the dominant failure modes -- not outright missed entities.

Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline

tok = AutoTokenizer.from_pretrained("distilbert-base-cased")  # placeholder, use this repo id
model = AutoModelForTokenClassification.from_pretrained("MatinAsgarov/applied-nlp-ner-stage3-distilbert")
tok = AutoTokenizer.from_pretrained("MatinAsgarov/applied-nlp-ner-stage3-distilbert")
nlp = pipeline("ner", model=model, tokenizer=tok, aggregation_strategy="simple")
nlp("Elon Musk announced that Tesla will open a factory in Shanghai.")