sc00byd00/bert-advanced-focal-llrd-ner

Advanced Custom BERT for Token Classification (NER)

This checkpoint features a fine-tuned bert-base-uncased architecture using custom training dynamics.

Key Engineering Highlights

• Custom Focal Loss Head: Replaced standard Cross-Entropy loss with Focal Loss (gamma=2.0) to mitigate severe background class (O) token imbalance.
• Layer-wise Learning Rate Decay (LLRD): Applied exponential decay (ext{decay rate}=0.85) across transformer layers to preserve lower-level syntax representations while rapidly adapting top classification heads.
• Optimized Gradient Dynamics: Implemented Gradient Accumulation (N=4, effective batch size 32) coupled with max norm gradient clipping (1.0) and FP16 mixed-precision acceleration.

Metrics

Evaluated on CoNLL-2003 validation set using seqeval.