Ali-0111/applied-nlp-ner-stage3-distilbert

Applied NLP / NER — Stage 3: First Model Run

Fine-tuned distilbert-base-cased (65.2M params, ~250MB fp32 on disk) on Ali-0111/applied-nlp-ner-stage2-expanded (644 train / 115 test records) for 8-label NER token classification (BIO scheme, 17 tags).

Why this model (decision D1)

NER leans heavily on capitalization as a signal, so a cased model was preferred over uncased. distilbert-base-cased is the smallest well-supported cased token-classification checkpoint with a strong NER track record, over a genuinely-smaller-but-uncased-only alternative (google/electra-small-discriminator, 14M) or an even tinier uncased option (prajjwal1/bert-mini/bert-small, 11-29M) that would have sacrificed the capitalization signal and risked quality on the dataset's thinnest labels (WORKOFART, PRODUCT).

Model stats

• Parameters: 65,203,985
• Size on disk (fp32 safetensors): ~250 MB
• CPU inference (4 threads, batch size 1, RTX-3060-laptop-class dev machine, CUDA_VISIBLE_DEVICES=""):
  • RSS after model load: ~884 MB (includes PyTorch/Transformers runtime overhead, not just weights)
  • RSS during inference: ~1.1 GB
  • Throughput: ~35 samples/sec, ~1,780 tokens/sec (single-threaded batch-of-1; ~29ms/sample latency)
• Training hardware: RTX 3060 Laptop GPU (6GB VRAM), 6 epochs, batch size 16, lr 5e-5, ~23s total training time (644 records)

Training setup (reproducible)

• Base model: distilbert-base-cased
• Max sequence length: 512 (only 1 of 759 records exceeds this token count)
• Epochs: 6, batch size: 16, learning rate: 5e-5, weight decay: 0.01, seed: 42
• No validation split used (per the task brief's own guidance — not enough data/compute for a 3-way split at this scale; train/test is the recommended setup)
• Scripts: stage3_model/scripts/prepare_bio.py (char-span to BIO alignment via tokenizer offset mapping), stage3_model/scripts/train.py (training + evaluation), stage3_model/scripts/measure_cpu_inference.py

Metrics

Train set

Label	Precision	Recall	F1	Support
AMOUNT	0.944	0.960	0.952	977
JOB	0.928	0.957	0.942	536
LOCATION	0.904	0.925	0.915	603
ORGANIZATION	0.933	0.957	0.945	717
PERSON	0.942	0.929	0.936	666
PRODUCT	0.815	0.837	0.826	190
TIMEDATE	0.931	0.931	0.931	825
WORKOFART	0.939	0.946	0.942	277

Overall: P=0.927, R=0.940, F1=0.934, Accuracy=0.986

Test set

Label	Precision	Recall	F1	Support
AMOUNT	0.915	0.950	0.932	260
JOB	0.723	0.754	0.738	114
LOCATION	0.723	0.802	0.760	91
ORGANIZATION	0.661	0.757	0.706	111
PERSON	0.825	0.819	0.822	138
PRODUCT	0.667	0.636	0.651	22
TIMEDATE	0.762	0.752	0.757	145
WORKOFART	0.719	0.767	0.742	60

Overall: P=0.786, R=0.820, F1=0.803, Accuracy=0.933

The train/test gap (F1 0.93 vs 0.80) reflects the small training set (644 records) relative to a 65M-param model over 6 epochs — expected at this scale, and exactly what Stage 4's diagnose/relabel/retrain step is meant to address.